- developers - lists.mariadb.org

Re: [MariaDB commits] [PATCH] MDEV-36934: semi sync makes the master unresponsive when a replica is stopped
by Kristian Nielsen 08 Jul '25

08 Jul '25

Hi Brandon, Kristian Nielsen via commits <commits(a)lists.mariadb.org> writes: > Ensure that a pending semi-sync transaction can only be signalled on its THD > while it is waiting in commit_trx(); not if the wait in commit_trx() is > skipped for some reason. Can you review this patch for MDEV-36934 so it can get into the upcoming release? You've already seen and agreed to the first version of the patch. I've included your test case for the bug, thanks a lot for writing that! I only modified it slightly, to increase the timeout for potentially slow Buildbot hosts, and to avoid an unnecessary use of DBUG injection. I also added your suggestions to (in addition to my original defensive-coding patch to only signal the THD while it is marked as waiting) also avoid adding a transaction to the list if it will never be ack'ed due to no waiting slaves; and to unlink_thd_as_waiter() if we skip waiting for ack in commit_trx(). - Kristian. > > There was a bug that if no semi-sync slaves were connected and > rpl_semi_sync_master_wait_no_slave was off, the THD pointer could be left in > the list of pending transactions after THD was deleted, and an invalid > THD/condition variable could be signalled, causing hang or other corruption. > > Testcase based on work by Brandon Nesterenko <brandon.nesterenko(a)mariadb.com>. > > Signed-off-by: Kristian Nielsen <knielsen(a)knielsen-hq.org> > --- > .../r/rpl_semi_sync_cond_var_per_thd.result | 63 ++++++++++- > .../rpl/t/rpl_semi_sync_cond_var_per_thd.test | 104 +++++++++++++++++- > sql/semisync_master.cc | 50 +++++++-- > sql/semisync_master.h | 10 +- > 4 files changed, 210 insertions(+), 17 deletions(-) > > diff --git a/mysql-test/suite/rpl/r/rpl_semi_sync_cond_var_per_thd.result b/mysql-test/suite/rpl/r/rpl_semi_sync_cond_var_per_thd.result > index 18ad5d3d2cc..96e7aa43fbf 100644 > --- a/mysql-test/suite/rpl/r/rpl_semi_sync_cond_var_per_thd.result > +++ b/mysql-test/suite/rpl/r/rpl_semi_sync_cond_var_per_thd.result > @@ -7,6 +7,8 @@ call mtr.add_suppression("Could not read packet"); > call mtr.add_suppression("Could not write packet"); > set @save_bgc_count= @@global.binlog_commit_wait_count; > set @save_bgc_usec= @@global.binlog_commit_wait_usec; > +set @old_master_wait_point= @@global.rpl_semi_sync_master_wait_point; > +set @old_master_wait_no_slave= @@global.rpl_semi_sync_master_wait_no_slave; > set @save_debug_dbug= @@global.debug_dbug; > set @@global.binlog_commit_wait_count=3; > set @@global.binlog_commit_wait_usec=10000000; > @@ -46,8 +48,6 @@ drop table t1, t2, t3; > # the binlogging to semi-sync, and starting the wait for ACK; and during > # this pause, semi-sync is manually switched off and on. > connection master; > -set @old_master_wait_point= @@global.rpl_semi_sync_master_wait_point; > -set @old_master_wait_no_slave= @@global.rpl_semi_sync_master_wait_no_slave; > set @@global.rpl_semi_sync_master_wait_point= AFTER_SYNC; > set @@global.rpl_semi_sync_master_wait_no_slave= 1; > set @@global.debug_dbug="+d,semisync_log_skip_trx_wait"; > @@ -100,7 +100,66 @@ commit; > # Cleanup > connection master; > drop table tn; > +set @@global.debug_dbug=@save_debug_dbug; > +# > +# MDEV-36934 > +# The server could indefinitely hang due to a memory leak which tried to > +# pthread signal on a destroyed condition variable. In effect, no > +# connections could commit transactions because there would be a thread > +# stuck on a never-returning call to pthread_cond_signal() while > +# holding Repl_semi_sync_master::LOCK_log. > +connection master; > +set @@global.rpl_semi_sync_master_wait_point= AFTER_COMMIT; > +set @@global.rpl_semi_sync_master_wait_no_slave= 0; > +# Ensure servers are in proper state > +connection master; > +connection slave; > +# Test case initial set-up > +connection master; > +create table t_36934 (a int) engine=innodb; > +include/save_master_gtid.inc > +connection slave; > +include/sync_with_master_gtid.inc > +# Pause the user transaction before inserting into Active_tranx > +connect user_con,localhost,root,,; > +SET debug_sync= 'semisync_at_write_tranx_in_binlog SIGNAL at_write_tranx_in_binlog WAIT_FOR resume_write_tranx_in_binlog'; > +insert into t_36934 values (1); > +connection server_1; > +set debug_sync="now wait_for at_write_tranx_in_binlog"; > +# Disconnect the slave (note that the binlog dump thread won't yet be > +# notified of a binlog update from the last transaction, so the slave > +# should neither receiver nor ACK the transaction). > +connection slave; > +include/stop_slave.inc > +# Waiting for master to realize the slave has disconnected.. > +connection server_1; > +# ..done > +# Resuming transaction (it will exit commit_trx early without waiting) > +set debug_sync="now signal resume_write_tranx_in_binlog"; > +connection user_con; > +disconnect user_con; > +# Force delete the user thread (FLUSH THREADS ensures the thread won't > +# stay in the thread cache) > +connection master; > +FLUSH THREADS; > +# BUG: Re-connect slave. MDEV-36934 reports that the master would hang > +# when the slave would re-connect and try to ACK the last transaction > +# who's thread has been deleted > +connection slave; > +include/start_slave.inc > +# Try to commit another transaction (prior to MDEV-36934 fixes, this > +# would hang indefinitely) > +connection master; > +set debug_sync="RESET"; > +insert into t_36934 values (2); > +connection server_1; > +# Waiting 30s for last query to complete.. > +connection master; > +# ..done > +# Cleanup > +connection master; > set @@global.rpl_semi_sync_master_wait_point= @old_master_wait_point; > set @@global.rpl_semi_sync_master_wait_no_slave= @old_master_wait_no_slave; > set @@global.debug_dbug=@save_debug_dbug; > +drop table t_36934; > include/rpl_end.inc > diff --git a/mysql-test/suite/rpl/t/rpl_semi_sync_cond_var_per_thd.test b/mysql-test/suite/rpl/t/rpl_semi_sync_cond_var_per_thd.test > index 537a4bf7fa8..5d6702f8c42 100644 > --- a/mysql-test/suite/rpl/t/rpl_semi_sync_cond_var_per_thd.test > +++ b/mysql-test/suite/rpl/t/rpl_semi_sync_cond_var_per_thd.test > @@ -34,6 +34,8 @@ call mtr.add_suppression("Could not read packet"); > call mtr.add_suppression("Could not write packet"); > set @save_bgc_count= @@global.binlog_commit_wait_count; > set @save_bgc_usec= @@global.binlog_commit_wait_usec; > +set @old_master_wait_point= @@global.rpl_semi_sync_master_wait_point; > +set @old_master_wait_no_slave= @@global.rpl_semi_sync_master_wait_no_slave; > set @save_debug_dbug= @@global.debug_dbug; > set @@global.binlog_commit_wait_count=3; > set @@global.binlog_commit_wait_usec=10000000; > @@ -98,8 +100,6 @@ drop table t1, t2, t3; > > > --connection master > -set @old_master_wait_point= @@global.rpl_semi_sync_master_wait_point; > -set @old_master_wait_no_slave= @@global.rpl_semi_sync_master_wait_no_slave; > set @@global.rpl_semi_sync_master_wait_point= AFTER_SYNC; > set @@global.rpl_semi_sync_master_wait_no_slave= 1; > --eval set @@global.debug_dbug="+d,semisync_log_skip_trx_wait" > @@ -190,8 +190,108 @@ commit; > --echo # Cleanup > --connection master > drop table tn; > +set @@global.debug_dbug=@save_debug_dbug; > + > + > +--echo # > +--echo # MDEV-36934 > +--echo # The server could indefinitely hang due to a memory leak which tried to > +--echo # pthread signal on a destroyed condition variable. In effect, no > +--echo # connections could commit transactions because there would be a thread > +--echo # stuck on a never-returning call to pthread_cond_signal() while > +--echo # holding Repl_semi_sync_master::LOCK_log. > + > +--connection master > +set @@global.rpl_semi_sync_master_wait_point= AFTER_COMMIT; > +set @@global.rpl_semi_sync_master_wait_no_slave= 0; > + > +--echo # Ensure servers are in proper state > +--connection master > +let $status_var= rpl_semi_sync_master_status; > +let $status_var_value= ON; > +source include/wait_for_status_var.inc; > +--connection slave > +let $status_var= rpl_semi_sync_slave_status; > +let $status_var_value= ON; > +source include/wait_for_status_var.inc; > + > +--echo # Test case initial set-up > +--connection master > +create table t_36934 (a int) engine=innodb; > +--source include/save_master_gtid.inc > +--connection slave > +--source include/sync_with_master_gtid.inc > + > +--echo # Pause the user transaction before inserting into Active_tranx > +--connect(user_con,localhost,root,,) > +SET debug_sync= 'semisync_at_write_tranx_in_binlog SIGNAL at_write_tranx_in_binlog WAIT_FOR resume_write_tranx_in_binlog'; > +--send insert into t_36934 values (1) > + > +--connection server_1 > +set debug_sync="now wait_for at_write_tranx_in_binlog"; > + > +--echo # Disconnect the slave (note that the binlog dump thread won't yet be > +--echo # notified of a binlog update from the last transaction, so the slave > +--echo # should neither receiver nor ACK the transaction). > +--connection slave > +--source include/stop_slave.inc > + > +--echo # Waiting for master to realize the slave has disconnected.. > +--connection server_1 > +let $status_var= rpl_semi_sync_master_clients; > +let $status_var_value= 0; > +source include/wait_for_status_var.inc; > +--echo # ..done > + > +--echo # Resuming transaction (it will exit commit_trx early without waiting) > +set debug_sync="now signal resume_write_tranx_in_binlog"; > + > +--connection user_con > +--reap > +--let $user_con_tid= `SELECT connection_id()` > +--disconnect user_con > +--source include/wait_until_disconnected.inc > + > +--echo # Force delete the user thread (FLUSH THREADS ensures the thread won't > +--echo # stay in the thread cache) > +--connection master > +FLUSH THREADS; > + > +--echo # BUG: Re-connect slave. MDEV-36934 reports that the master would hang > +--echo # when the slave would re-connect and try to ACK the last transaction > +--echo # who's thread has been deleted > +--connection slave > +--source include/start_slave.inc > + > +--echo # Try to commit another transaction (prior to MDEV-36934 fixes, this > +--echo # would hang indefinitely) > +--connection master > +set debug_sync="RESET"; > +--send insert into t_36934 values (2) > + > +--connection server_1 > +--echo # Waiting 30s for last query to complete.. > +--let $wait_timeout= 30 > +--let $wait_condition= SELECT count(*)=0 FROM information_schema.processlist WHERE info LIKE 'insert into t_36934%'; > +--source include/wait_condition.inc > + > +# Variable `success` is set by wait_condition.inc > +if (!$success) > +{ > + --echo # ..error > + --die Query is hung > +} > + > +--connection master > +--reap > +--echo # ..done > + > + > +--echo # Cleanup > +--connection master > set @@global.rpl_semi_sync_master_wait_point= @old_master_wait_point; > set @@global.rpl_semi_sync_master_wait_no_slave= @old_master_wait_no_slave; > set @@global.debug_dbug=@save_debug_dbug; > +drop table t_36934; > > --source include/rpl_end.inc > diff --git a/sql/semisync_master.cc b/sql/semisync_master.cc > index 43afb8f996e..854b6a61eeb 100644 > --- a/sql/semisync_master.cc > +++ b/sql/semisync_master.cc > @@ -68,15 +68,20 @@ static ulonglong timespec_to_usec(const struct timespec *ts) > return (ulonglong) ts->tv_sec * TIME_MILLION + ts->tv_nsec / TIME_THOUSAND; > } > > -int signal_waiting_transaction(THD *waiting_thd, const char *binlog_file, > - my_off_t binlog_pos) > +static int > +signal_waiting_transaction(THD *waiting_thd, bool thd_valid, > + const char *binlog_file, my_off_t binlog_pos) > { > /* > It is possible that the connection thd waiting for an ACK was killed. In > such circumstance, the connection thread will nullify the thd member of its > Active_tranx node. So before we try to signal, ensure the THD exists. > + > + The thd_valid is only set while the THD is waiting in commit_trx(); this > + is defensive coding to not signal an invalid THD if we somewhere > + accidentally did not remove the transaction from the list. > */ > - if (waiting_thd) > + if (waiting_thd && thd_valid) > mysql_cond_signal(&waiting_thd->COND_wakeup_ready); > return 0; > } > @@ -182,6 +187,7 @@ int Active_tranx::insert_tranx_node(THD *thd_to_wait, > ins_node->log_name[FN_REFLEN-1] = 0; /* make sure it ends properly */ > ins_node->log_pos = log_file_pos; > ins_node->thd= thd_to_wait; > + ins_node->thd_valid= false; > > if (!m_trx_front) > { > @@ -263,7 +269,8 @@ void Active_tranx::clear_active_tranx_nodes( > if ((log_file_name != NULL) && > compare(new_front, log_file_name, log_file_pos) > 0) > break; > - pre_delete_hook(new_front->thd, new_front->log_name, new_front->log_pos); > + pre_delete_hook(new_front->thd, new_front->thd_valid, > + new_front->log_name, new_front->log_pos); > new_front = new_front->next; > } > > @@ -355,13 +362,17 @@ void Active_tranx::unlink_thd_as_waiter(const char *log_file_name, > } > > if (entry) > + { > entry->thd= NULL; > + entry->thd_valid= false; > + } > > DBUG_VOID_RETURN; > } > > -bool Active_tranx::is_thd_waiter(THD *thd_to_check, const char *log_file_name, > - my_off_t log_file_pos) > +Tranx_node * > +Active_tranx::is_thd_waiter(THD *thd_to_check, const char *log_file_name, > + my_off_t log_file_pos) > { > DBUG_ENTER("Active_tranx::assert_thd_is_waiter"); > mysql_mutex_assert_owner(m_lock); > @@ -377,7 +388,7 @@ bool Active_tranx::is_thd_waiter(THD *thd_to_check, const char *log_file_name, > entry = entry->hash_next; > } > > - DBUG_RETURN(static_cast<bool>(entry)); > + DBUG_RETURN(entry); > } > > /******************************************************************************* > @@ -863,6 +874,10 @@ int Repl_semi_sync_master::commit_trx(const char *trx_wait_binlog_name, > > if (!rpl_semi_sync_master_clients && !rpl_semi_sync_master_wait_no_slave) > { > + lock(); > + m_active_tranxs->unlink_thd_as_waiter(trx_wait_binlog_name, > + trx_wait_binlog_pos); > + unlock(); > rpl_semi_sync_master_no_transactions++; > DBUG_RETURN(0); > } > @@ -922,6 +937,9 @@ int Repl_semi_sync_master::commit_trx(const char *trx_wait_binlog_name, > } > } > > + Tranx_node *tranx_entry= > + m_active_tranxs->is_thd_waiter(thd, trx_wait_binlog_name, > + trx_wait_binlog_pos); > /* In between the binlogging of this transaction and this wait, it is > * possible that our entry in Active_tranx was removed (i.e. if > * semi-sync was switched off and on). It is also possible that the > @@ -932,8 +950,7 @@ int Repl_semi_sync_master::commit_trx(const char *trx_wait_binlog_name, > * rpl_semi_sync_master_yes/no_tx consistent with it, we check for a > * semi-sync restart _after_ checking the reply state. > */ > - if (unlikely(!m_active_tranxs->is_thd_waiter(thd, trx_wait_binlog_name, > - trx_wait_binlog_pos))) > + if (unlikely(!tranx_entry)) > { > DBUG_EXECUTE_IF( > "semisync_log_skip_trx_wait", > @@ -952,6 +969,16 @@ int Repl_semi_sync_master::commit_trx(const char *trx_wait_binlog_name, > break; > } > > + /* > + Mark that our THD is now valid for signalling to by the ack thread. > + It is important to ensure that we can never leave a no longer valid > + THD in the transaction list and signal it, eg. MDEV-36934. This way, > + we ensure the THD will only be signalled while this function is > + running, even in case of some incorrect error handling or similar > + that might leave a dangling THD in the list. > + */ > + tranx_entry->thd_valid= true; > + > /* Let us update the info about the minimum binlog position of waiting > * threads. > */ > @@ -1284,6 +1311,8 @@ int Repl_semi_sync_master::write_tranx_in_binlog(THD *thd, > > DBUG_ENTER("Repl_semi_sync_master::write_tranx_in_binlog"); > > + DEBUG_SYNC(current_thd, "semisync_at_write_tranx_in_binlog"); > + > lock(); > > /* This is the real check inside the mutex. */ > @@ -1317,7 +1346,8 @@ int Repl_semi_sync_master::write_tranx_in_binlog(THD *thd, > m_commit_file_name_inited = true; > } > > - if (is_on()) > + if (is_on() && > + (rpl_semi_sync_master_clients || rpl_semi_sync_master_wait_no_slave)) > { > DBUG_ASSERT(m_active_tranxs != NULL); > if(m_active_tranxs->insert_tranx_node(thd, log_file_name, log_file_pos)) > diff --git a/sql/semisync_master.h b/sql/semisync_master.h > index a1c57959165..28de3ecf480 100644 > --- a/sql/semisync_master.h > +++ b/sql/semisync_master.h > @@ -30,6 +30,7 @@ extern PSI_cond_key key_COND_binlog_send; > > struct Tranx_node { > char log_name[FN_REFLEN]; > + bool thd_valid; /* thd is valid for signalling */ > my_off_t log_pos; > THD *thd; /* The thread awaiting an ACK */ > struct Tranx_node *next; /* the next node in the sorted list */ > @@ -126,7 +127,9 @@ class Tranx_node_allocator > > trx_node= &(current_block->nodes[++last_node]); > trx_node->log_name[0] = '\0'; > + trx_node->thd_valid= false; > trx_node->log_pos= 0; > + trx_node->thd= nullptr; > trx_node->next= 0; > trx_node->hash_next= 0; > return trx_node; > @@ -298,7 +301,8 @@ class Tranx_node_allocator > its invocation. See the context in which it is called to know. > */ > > -typedef int (*active_tranx_action)(THD *trx_thd, const char *log_file_name, > +typedef int (*active_tranx_action)(THD *trx_thd, bool thd_valid, > + const char *log_file_name, > my_off_t trx_log_file_pos); > > /** > @@ -381,8 +385,8 @@ class Active_tranx > * matches the thread of the respective Tranx_node::thd of the passed in > * log_file_name and log_file_pos. > */ > - bool is_thd_waiter(THD *thd_to_check, const char *log_file_name, > - my_off_t log_file_pos); > + Tranx_node * is_thd_waiter(THD *thd_to_check, const char *log_file_name, > + my_off_t log_file_pos); > > /* Given a position, check to see whether the position is an active > * transaction's ending position by probing the hash table.

2 2

Re: 59aafa4ee46: MDEV-33633 Trigger tries to access already dropped view
by Sergei Golubchik 08 Jul '25

08 Jul '25

Hi, Aleksey, On Jun 27, Aleksey Midenkov wrote: > revision-id: 59aafa4ee46 (mariadb-10.6.22-50-g59aafa4ee46) > parent(s): 1221f49f312 > author: Aleksey Midenkov > committer: Aleksey Midenkov > timestamp: 2025-06-04 23:42:36 +0300 > message: > > MDEV-33633 Trigger tries to access already dropped view > > diff --git a/mysql-test/main/ps_ddl.result b/mysql-test/main/ps_ddl.result > index 35998c05dc9..ceb85835623 100644 > --- a/mysql-test/main/ps_ddl.result > +++ b/mysql-test/main/ps_ddl.result > @@ -333,7 +333,7 @@ a > drop view v1; > create view v1 as select a from t3; > insert into t1 (a) values (6); > -ERROR 42S02: Table 'test.t2' doesn't exist > +ERROR HY000: Table definition has changed, please retry transaction Is it possible to the fallback and reopen here? This is a rather unexpected error, particularly when no transactions are involved. > diff --git a/sql/sql_base.cc b/sql/sql_base.cc > index 1a4fa315c3c..c89e88d0f03 100644 > --- a/sql/sql_base.cc > +++ b/sql/sql_base.cc > @@ -1737,6 +1737,18 @@ bool is_locked_view(THD *thd, TABLE_LIST *t) > DBUG_RETURN(FALSE); > } > > + Reprepare_observer reprepare_observer; > + auto scope_exit= make_scope_exit( > + [thd]() mutable { thd->m_reprepare_observer= NULL; }, > + false); > + if (!thd->m_reprepare_observer) > + { > + reprepare_observer.reset_reprepare_observer(); > + reprepare_observer.error_code= ER_TABLE_DEF_CHANGED; > + thd->m_reprepare_observer= &reprepare_observer; > + scope_exit.engage(); > + } And you didn't use SCOPE_EXIT to use a manual engage? Would be clearer, I'd say, to use SCOPE_EXIT and have an if() inside, like if (m_reprepare_observer == &reprepare_observer). But ok, as you like. > if (!tdc_open_view(thd, t, CHECK_METADATA_VERSION)) > { > DBUG_ASSERT(t->view != 0); > @@ -3059,9 +3071,7 @@ bool tdc_open_view(THD *thd, TABLE_LIST *table_list, uint flags) > > DBUG_ASSERT(share->is_view); > > - err= mysql_make_view(thd, share, table_list, (flags & OPEN_VIEW_NO_PARSE)); > - > - if (!err && (flags & CHECK_METADATA_VERSION)) > + if (flags & CHECK_METADATA_VERSION) > { > /* > Check TABLE_SHARE-version of view only if we have been instructed to do > @@ -3076,6 +3086,8 @@ bool tdc_open_view(THD *thd, TABLE_LIST *table_list, uint flags) > goto ret; > } > > + err= mysql_make_view(thd, share, table_list, (flags & OPEN_VIEW_NO_PARSE)); But this one - I don't understand how it works. mysql_make_view() reads the view frm which contains versions that are checked by check_and_update_table_version(). How can check_and_update_table_version() work before mysql_make_view()? > ret: > tdc_release_share(share); > > diff --git a/sql/table.cc b/sql/table.cc > index 9bbc29d4ecf..6d2a80aeace 100644 > --- a/sql/table.cc > +++ b/sql/table.cc > @@ -5799,6 +5799,12 @@ void TABLE::init(THD *thd, TABLE_LIST *tl) > DBUG_ASSERT(!auto_increment_field_not_null); > auto_increment_field_not_null= FALSE; > > + tl->view= NULL; > + tl->derived= NULL; > + tl->derived_type= VIEW_ALGORITHM_UNDEFINED; > + tl->updatable= true; > + tl->effective_with_check= VIEW_CHECK_NONE; > + I agree it's conceptually correct and for the new table these fields should be reset. But it's rather unexpected that TABLE::init() modifies its arguments. Could this reinitialization be done somewhere else? Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 2

Prepared statements analytics problem
by Dmitrii Kochetov 03 Jul '25

03 Jul '25

Most database monitoring systems, such as Newrelic, Datadog, Dolphie, Ottertune, Releem, Solarwinds, use data from the performance_schema database from the events_statements_summary_by_digest and events_statements_* tables to get query analytics for databases. This information allows to calculate system performance metrics, in particular Latency. Also, the data from these tables allows you to get a complete picture of the database operation - what queries are executed, their execution statistics, to get query templates based on digest_text, and based on SQL_TEXT field from events_statements_* or QUERY_SAMPLE_TEXT from events_statements_summary_by_digest (for MySQL >= 8) to get a sample query for digest and optimize it. The prepared statements analytics are contained in a single table prepared_statements_instances, which is managed by Performance_schema as follows: - Statement preparation If the statement is successfully instrumented, a new row is added to the prepared_statements_instances table. If the statement cannot be instrumented, Performance_schema_prepared_statements_lost status variable is incremented. - Prepared statement execution Execution of a EXECUTE command for an instrumented prepared statement instance updates the corresponding prepared_statements_instances table row. - Prepared statement deallocation Execution of a CLOSE command for an instrumented prepared statement instance removes the corresponding prepared_statements_instances table row. To avoid resource leaks, removal occurs even if the prepared statement instruments described previously are disabled. In this regard, there are the following problems with analytics for prepared statements compared to regular queries: 1) Most applications work with prepared statements using the following method when processing a single query - creating a prepared statement, executing the prepared statement, and then dropping the prepared statement. This leads to the fact that query statistics exist for a very short period of time, during which it may not be possible to fetch them. 2) The second problem of prepared statement analytics in comparison with regular queries is the absence of the query hash (digest) and information about the schema_name in which the prepared statement is executed, which makes it difficult to identify unique queries, since a separate record in the Prepared_Statements_instances table is created for each creation of a prepared statement. 3) Execution of prepared statements is possible in two protocols: Binary and text. Execution of a prepared statement in the binary protocol does not create an entry in the events_statements_* tables, but execution in the text protocol creates an entry in the specified tables, which leads to different behaviors of the same prepared statement but in different protocols. 4) For prepared statements, there is no option in getting an example request without a ? placeholder. The SQL_TEXT field in Prepared_Statements_instances contains The prepared statement text, with ? placeholder markers, and in some cases (when executing in the text protocol) it is possible to find only the execute command of this statement, which further requires mapping ? placeholder markers with parameters from the execute request. But there are limitations that the binary protocol does not store EXECUTE requests for prepared statements, and most of the interactions with the database occur through it, through drivers for different programming languages, and another limitation may be the use of variables in the EXECUTE command, the value of which still needs to be obtained. Proposed solution: 1) Records should not be deleted from the Prepared_Statements_instances table after dropping prepared statements. If the table is full, the counter Performance_schema_prepared_statements_lost and If no row has the statement digest value for the statement that just completed, and the table is full, the statistics for the statement that just completed are added to a special “catch-all” row with DIGEST = NULL, which is created if necessary. 2) Each statement must be unique by the fields schema_name, in which the given statement is executed and its hash, digest, which will be the same for the same prepared statement. 3) In the tables events_statements_* must be recorded information about the fact of PREPARE, EXECUTE and DEALLOCATE execution for prepared statement, where in the DIGEST_TEXT field is stored the call of the execute command, and in SQL_TEXT - the normalized prepared query, in which ? placeholder markers are replaced by concrete values.

2 2

Points of low-level design for MDEV-34705, binlog-in-engine
by Kristian Nielsen 30 Jun '25

30 Jun '25

As the new binlog format (MDEV-34705 Binlog-in-engine) is taking shape, I wanted to write up some more details on a number of the low-level design decisions made so far. Re-implementing the binlog format is a huge undertaking due to way it interacts with many different parts of the server code, as well as user-visible interfaces. I am not sure how many people actually read these detailed write-ups, but I think it is still important to at least try to present the design and give others a chance to comment. So many other large changes to or around the server code are being done without even an attempt to proper design review, often with serious defects as a result. All of the design elements mentioned in this mail can be seen in the code available in the branch "knielsen_binlog_in_engine" on Github: https://github.com/MariaDB/server/commits/knielsen_binlog_in_engine File format =========== It was clear from the start that a page-based file format should be used. Initially, the idea was to use a new type of InnoDB tablespace. But after discussions with Marko, this was revised to use a simplified custom format for the binlog files. This avoids much of the overhead of a full InnoDB tablespace and page format, but at the cost of having to implement new code to take over the functionality of the InnoDB buffer pool and recovery code. The InnoDB redo log is still used, with only minor modifications. The page size is currently 4kB, I'm considering 16kB to reduce per-page runtime overhead. The page size is independent of the InnoDB page size and can easily be made configurable. The first page of the file is reserved as a header page; rest of pages have only a four-byte overhead for a CRC32. The CRC32 replaces the iffy per-event checksum in the old legacy binlog. Using a page-based format means it is possible to random-access read the binlog file (this is not possible in the legacy binlog, as there is no easy way to determine the boundaries between events). The GTID indexes are thus no longer necessary; instead we simply binary-search the binlog file for the starting GTID position. To facilitate this binary search, the current GTID binlog state is written (compressed) every --innodb-binlog-state-interval bytes. Similarly, at restart restart, binary search is used to find the position at which to continue writing the last binlog file (as opposed to forcing a binlog rotation as in the legacy binlog). A similar mechanism is used to recover the current gtid_binlog_state without the need for a master-bin.state file. In the new binlog implementation, binlog files are identified solely by their number (starting from 0). The binlog file name is engine-specific, not user configurable. This considerably simplifies the binlog handling, avoiding the need for the binlog index file. A new --binlog-directory option allows to still place the binlog on a file system of choice. Generally 64-bit numbers are used to identify binlog file numbers, sizes, offsets, etc. to avoid limitations like the legacy binlog's 32-bit file offsets. A simple compression scheme is used to store small numbers in few bytes. See storage/innobase/include/ut0compr_int.h for the implementation, which is written to be efficient on modern superscalar CPUs, and helps avoid 64-bit penalty on data storage sizes. In the current design, the new binlog format is not very InnoDB specific, but it is still implemented inside the storage engine, in principle in an engine-dependent format. The advantage of this is that another engine has the flexibility to implement a different format, perhaps even interleaving the binlog data with other data in existing data files. On the other hand, fixing the binlog file format like it is with InnoDB would allow to move part of the code to the server layer and share it between engines. I believe this decision would be best revisited when/if another engine binlog implementation becomes relevant. The function ibb_write_header_page() shows how the header page is written. The function innodb_binlog_prealloc_thread() runs the background thread that pre-allocates new binlog files and closes fully written files. Binlog recovery =============== Just two InnoDB tablespace IDs (for redo records) are reserved for binlog files, used for odd/even files alternately. To be able to unambiguously match a redo record to a binlog file during recovery, we make sure that only the two most recent binlog files will need recovery. When the writer switches to binlog file N, we first pre-allocate (in a background thread) file N+1, and then we fsync() to disk the file (N-1). The page header contains an InnoDB LSN corresponding to the start of the binlog data in the file. Thus, when binlog recovery processes a redo record, it can select the last or second-to-last existing binlog file that has a matching tablespace id and compare the record's LSN against the LSN in the file header. If the record's LSN is smaller, then we know this record corresponds to a binlog file that was already durably fsync()ed to disk, so we can skip it; otherwise the record is applied to the file (possibly creating the file anew if required). This way, we avoid any need to stall during commits to fsync() any binlog data (except when binlog write load starts exceeding the disk I/O capacity). And InnoDB checkpoints can span arbitrary number of binlog files, so that we also avoid stalling binlog rotation while waiting to complete a checkpoint. As a special case, RESET MASTER (or the initial binlog creation at first server restart) fsync()s both the InnoDB redo log and the newly created file header to disk before starting to write data. This simplifies the binlog recovery as we then will always have at least one file header available with starting LSN to compare against the LSN in redo records. In the case of --innodb-flush-log-at-trx-commit=0|2, binlog recovery will truncate extra binlog data for transactions that were not durably written to the redo log. This allows the code to write binlog data out to disk immediately, without waiting for redo log sync to happen first, unlike for InnoDB table/index data. This works because binlog files are always written strictly append-only; data is never modified in place. In fact, each page is only ever written once to the file system, except for one special case. During an InnoDB checkpoint, binlog data prior to the checkpoint LSN need to be synced durably to disk. If there is not enough binlog traffic during this to fill up the last page, we might need to write a partial last page to disk. When this happens, the page is marked, and the next write to that page will redo log the complete page (as opposed to just the appended data). This avoids the need for any double-write buffer for the binlog writes. The class binlog_recovery implements the binlog recovery algorithm. Out-of-band binlogging ====================== The new binlog implementation removes the limitation in the legacy binlog that each event group (transaction) must be written consecutively to a single binlog file. When the binlog transaction cache is full, instead of spilling it to a temporary file, the cache data is written directly into the binlog as so-called out-of-band data. At commit time, large transactions only need to binlog a small commit record that back-references the previously written partial event data. This avoids a large stall at commit time of huge transactions. It also makes it possible to keep a fixed binlog file size and still allow transactions much larger than that. If a transaction rolls back, any already written out-of-band data is simply left in the binlog file as garbage, to disappear when that file is eventually purged. The individual out-of-band records are structured on disk as a forest of complete binary trees. This is used to allow efficient reading of the data starting from the commit record (used to re-assemble complete event groups to send to slaves from the binlog dump threads); and still allow to write the binlog files strictly append-only, without needing to go back to update already written records (eg. with forward-pointers). The header page of each binlog contains a reference to any earlier binlog file that may contain out-of-band data referenced by commit records in this file. This is used to prevent the purge of old binlog files that may still be needed by active dump threads now reading a newer file. This mechanism can also be used in a future version to implement proper XA replication support following the design described in MDEV-32020. The class innodb_binlog_oob_reader implements the logic that re-assembles the out-of-band pieces of data into a complete event group for the reader. The struct binlog_oob_context is the corresponding object that keeps track of the construction of the forest of complete binary trees in the writer. Page FIFO ========= The binlog implements a page FIFO as a replacement for the InnoDB buffer pool used for other tablespaces. As the binlog is written strictly append-only, the "buffer pool" functionality is much simplified. Pages are only added to the end, and can be written always in sequence from the start, hence "FIFO". The page FIFO is accessed by three main parts of the code: 1. By threads writing to the binlog, allocating new pages and writing data into them in-memory. Binlog writes are serialized by LOCK_commit_ordered. 2. By binlog dump thread readers (eg. connected slaves). Readers must read data from pages in the page FIFO until they have been written to disk (and removed from the FIFO); then they must be read from the file through the file system. 3. By the flush thread which writes out completed pages to the binlog files in the background, and a couple other similar non-performance-critical places, such as tablespace close and InnoDB checkpoint to flush binlog files to disk. A simple latching mechanism is used to allow writer and readers to access pages concurrently, to avoid readers stalling the writers. Atomic variable binlog_cur_end_offset[] is used to make readers never read ahead of the writer without requiring mutex locking during the read; see also the description of binlog_cur_durable_offset[] below under "Crash-safe replication". Writer and readers (1) and (2) are prioritised over background flushing (3); the latter will wait for a page to be no longer latched. A global mutex is used to protect the latching and data structure integrity of the page FIFO. This mutex is only held for a brief time (few instructions), to make the page FIFO scalable. The page FIFO is implemented in class fsp_binlog_page_fifo. Crash-safe replication ====================== One of the goals of the new binlog format is to make the binlog and replication crash-safe also when --innodb-flush-log-at-trx-commit=0|2. To achieve this, it is necessary to make sure that binlog events are not sent to slaves until they have become durable on the master; otherwise, if the master crashes, the slave might have a transaction that no longer exists on the master when it comes back up after crash recovery, and replication will diverge. (This is the default setup; the converse property that a slave is always ahead of the master and any master crash will demote the old master as a slave can be provided in a later version using semi-synchronous replication with an "AFTER-PREPARE" configuration). Binlog events become durable (together with the table-data part of the associated transaction) when the corresponding LSN in the InnoDB redo log has been durably flushed to disk. The binlog code keeps track of each binlog write and the associated redo log LSN in the singleton class pending_lsn_fifo. A binlog write of a commit record inserts an entry into this. Then when that LSN has been durably written, the atomic binlog_cur_durable_offset[] is updated to reflect the point up to which the binlog is now known to be durable. The dump thread readers (connected slaves) will use this to not read too far ahead. When a reader reaches the end of the durable part of the binlog, that reader can then initiate an InnoDB redo log flush to make more of the binlog durable and allow the read to continue. This way, we achieve that transactions can complete without waiting for fsync-per-commit (which can massively increase commits-per-second depending on disk latency); at the same time, waiting readers will trigger redo log sync immediately and provide low latency to slaves. Backup ====== With the new binlog implementation, binlog files are maintained transactionally using the same InnoDB crash recovery mechanism used for its other tablespaces. This allows mariabackup to include the binlog files in the backup, something that was not easily available before if not using something like LVM snapshots. The binlog files are copied at the end of the backup, after releasing backup locks. The backup LSN is used together with the InnoDB/binlog recovery algorithm (during mariabackup --prepare) to make the binlog in the backup consistent with the table data. Apart from the obvious desirability of having a backup facility for the binlogs, this also allows a better way to get a corresponding binlog position to use the backup to provision a new replication slave. After setting up a new slave from a backup of the master, the new slave can simply be started (using GTID) from the @@gtid_binlog_pos restored with the binlog (CHANGE MASTER TO ... MASTER_DEMOTE_TO_SLAVE can be used for this). When setting up from a backup of another slave, the starting GTID position is directly available from the restored mysql.gtid_slave_pos table (as is already the case in existing MariaDB versions). This avoids the need for the special code in InnoDB that transactionally stores the current binlog position in the undo segments, as well as the need to hold backup locks while querying @@gtid_binlog_pos or SHOW MASTER STATUS. Storage engine API ================== The initial implementation of the new binlog is inside InnoDB, but the intention is that another (transactional) storage engine could do its own implementation to achive similar benefits as InnoDB. Some effort has gone into making a clean and well-defined interface between the server/binlog layer and the storage engine to facilitate this. The API is seen as new handlerton methods in struct handlerton, as well as new classes handler_binlog_reader, handler_binlog_event_group_info, and handler_binlog_purge_info. Here is an overview of the new handlerton methods: bool (*binlog_init)(size_t binlog_size, const char *directory); This is used at server startup to initialize the engine's binlog data structures, when --binlog-storage-engine=ENGINE is configured by the user. bool (*binlog_write_direct_ordered)(IO_CACHE *cache, handler_binlog_event_group_info *binlog_info, const rpl_gtid *gtid); bool (*binlog_write_direct)(IO_CACHE *cache, handler_binlog_event_group_info *binlog_info, const rpl_gtid *gtid); These are called by the server to write binlog data into the engine-implemented binlog. They are dual methods similar to commit_ordered() and commit(); the _ordered variant runs serialized (in binlog and GTID order) under LOCK_commit_ordered, while the non-ordered runs without holding locks. In addition, the engine is expected to binlog the transaction itself as part of binlog group commit (during commit_ordered), obtaining the binlog data to write from the server-provided binlog_get_cache() function. The class handler_binlog_event_group_info object is used to hold context data related to the event group while it is being binlogged. void (*binlog_group_commit_ordered)(THD *thd, handler_binlog_event_group_info *binlog_info); This is called at the end of binlog group commit (after releasing LOCK_commit_ordered), with the binlog_info of the last commit in the group. It is used to optimise the handling of the crash-safe replication described above, so that the engine can perform operations that need only be done once per group commit (not once per commit), and which can be done without holding the performance-critical LOCK_commit_ordered mutex. bool (*binlog_oob_data_ordered)(THD *thd, const unsigned char *data, size_t data_len, void **engine_data); bool (*binlog_oob_data)(THD *thd, const unsigned char *data, size_t data_len, void **engine_data); These are called by the server layer to binlog large event groups (transactions) in multiple pieces that can be interleaved with other events in the binlog. This allows large transactions to span multiple binlog files, avoids stalls around commits of large transactions, and allows event groups larger than the maximum size of an InnoDB mini-transaction. void (*binlog_oob_reset)(THD *thd, void **engine_data); void (*binlog_oob_free)(THD *thd, void *engine_data); These are simple functions to coordinate (re-)initialisation and de-allocation of engine-specific data associated with the binlogging of event groups. handler_binlog_reader * (*get_binlog_reader)(bool wait_durable); This is the main function used to implement the reading of the binlog by binlog dump threads (eg. connected slaves). The class handler_binlog_reader implements an interface for the server to read binlog data and for the dump threads to wait for new data to arrive when they reach the current EOF. void (*binlog_status)(uint64_t * out_fileno, uint64_t *out_pos); This is used to implement SHOW MASTER STATUS. The file number and offset is mostly informational, the new binlog format is intended for GTID use. void (*get_filename)(char name[FN_REFLEN], uint64_t file_no); In the new binlog implementation, binlog files are identified only by number (the file name is engine-specific and not user configurable). This function allows the server to provide the engine-specific file name for informational purposes to the user. binlog_file_entry * (*get_binlog_file_list)(MEM_ROOT *mem_root); This is used to implement SHOW BINARY LOGS. bool (*binlog_flush)(); This is used to implement FLUSH BINARY LOGS. bool (*binlog_get_init_state)(rpl_binlog_state_base *out_state); This is used to implement FLUSH BIANRY LOGS DELETE_DOMAIN_ID, to get the binlog state at the start of the current (non-purged) binlog. bool (*reset_binlogs)(); This is used to implement RESET MASTER. int (*binlog_purge)(handler_binlog_purge_info *purge_info); This is used to implement PURGE BINARY LOGS as well as auto-purge. The class handler_binlog_purge_info is used to pass the various purge options to the engine, as well as to inform about active dump threads that would prevent purging past their current read position.

1 0

Re: 0ddb9744cea: MDEV-33957 UPDATE fails on replica replicating blob virtual column in NOBLOB mode when replica logging is off
by Sergei Golubchik 24 Jun '25

24 Jun '25

Hi, Aleksey, On Jun 23, Aleksey Midenkov wrote: > commit 0ddb9744cea > Author: Aleksey Midenkov <midenok(a)mariadb.com> > Date: Sun May 25 23:23:29 2025 +0300 > > MDEV-33957 UPDATE fails on replica replicating blob virtual column in > NOBLOB mode when replica logging is off please, put the whole commit subject in one line. various tools assume the subject is one line only and don't handle line wrapping > > The sequence that causes the issue: > > 1. file->row_logging is false because slave-bin was not open; > 2. TABLE::mark_columns_per_binlog_row_image() didn't mark column for > read because file->row_logginbg is false. This was implemented in > e53ad95b733 (MDEV-6877); > 3. TABLE::update_virtual_fields() didn't update virtual field value > because column is not marked for read; > 4. calc_row_difference() sees o_len as UNIV_SQL_NULL, but new row > value is "1". The virtual column is added to update vector; > 5. row_upd() tries to update secondary index, but row_upd_sec_step() > doesn't see old value in the index. > > The patch does mark_virtual_column_with_deps() in case of rgi_slave in > mark_columns_per_binlog_row_image() so that non-stored virtual columns > are marked for update in slave thread. > > diff --git a/sql/ha_partition.h b/sql/ha_partition.h > index 01d15ca70ac..d926bd03137 100644 > --- a/sql/ha_partition.h > +++ b/sql/ha_partition.h > @@ -576,6 +576,17 @@ class ha_partition final :public handler > { > m_file[part_id]->update_create_info(create_info); > } > + > + void column_bitmaps_signal() override > + { > + for (uint i= bitmap_get_first_set(&m_opened_partitions); > + i < m_tot_parts; > + i= bitmap_get_next_set(&m_opened_partitions, i)) > + { > + m_file[i]->column_bitmaps_signal(); > + } > + } That's a good one. Although it's not really related to the bug in question. Please at least mention this in the commit comment, like "also fixed..." > + > private: > int copy_partitions(ulonglong * const copied, ulonglong * const deleted); > void cleanup_new_partition(uint part_count); > diff --git a/sql/table.cc b/sql/table.cc > index 3a51228d1f3..b48aaadb6e1 100644 > --- a/sql/table.cc > +++ b/sql/table.cc > @@ -8019,6 +8024,12 @@ void TABLE::mark_columns_per_binlog_row_image() > } > file->column_bitmaps_signal(); > } > +#if MYSQL_VERSION_ID<110201 > + else if (thd->rgi_slave) > + { > + file->column_bitmaps_signal(); > + } 1. This doesn't match the commit comment, please update the latter to describe the actual code change. 2. why MYSQL_VERSION_ID<110201 ? 3. you can also move `rpl_write_set= write_set` from the beginning of the function here, inside the else branch. This makes the code clearer: rpl_write_set was changed, so we have to signal it. > +#endif > > DBUG_VOID_RETURN; > } Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 1

Re: [PATCH] MDEV-36866 Oracle outer join syntax (+): query with checking for null of non-null column uses wrong query plan and returns wrong result
by Yuchen Pei 24 Jun '25

24 Jun '25

Hi Oleksandr, > From 660e9eefd7c5c5a2b083573b8185be8b5748b0de Mon Sep 17 00:00:00 2001 > From: Oleksandr Byelkin <sanja(a)mariadb.com> > Date: Thu, 19 Jun 2025 14:40:03 +0200 > Subject: [PATCH] MDEV-36866 Oracle outer join syntax (+): query with checking > for null of non-null column uses wrong query plan and returns wrong result > Recalculate maybe_null for parts witch was fixed before converting s/witch/which/ > oracle joins. > --- > [... 97 lines elided] > diff --git a/sql/item.h b/sql/item.h > index 345d99d43cc..5b4ca045cc9 100644 > --- a/sql/item.h > +++ b/sql/item.h > @@ -2292,6 +2292,7 @@ class Item :public Value_source, > return 0; > } > virtual bool ora_join_processor(void *arg) { return 0; } > + virtual bool recalc_maybe_null_processor(void *arg) { return 0; } Can you add a comment here? Something like "update maybe_nulls after converting oracle (+) outer joins" > virtual bool remove_ora_join_processor(void *arg) > { > with_flags&= ~item_with_t::ORA_JOIN; > @@ -2905,6 +2906,15 @@ class Item_args > } > return false; > } > + bool arg_check_maybe_null() > + { > + for (uint i= 0; i < arg_count; i++) > + { > + if (args[i]->maybe_null()) > + return true; > + } > + return false; > + } I suggest rename this function to some_maybe_null, to mean at least one of the args is maybe_null (some vs. all) > bool transform_args(THD *thd, Item_transformer transformer, uchar *arg); > void propagate_equal_fields(THD *, const Item::Context &, COND_EQUAL *); > bool excl_dep_on_table(table_map tab_map) > @@ -3945,6 +3955,11 @@ class Item_field :public Item_ident, > return 0; > } > bool ora_join_processor(void *arg) override; > + bool recalc_maybe_null_processor(void *arg) override > + { > + set_maybe_null(field->maybe_null()); > + return 0; > + } > bool check_ora_join(Item **reference, bool outer_ref_fixed); > void cleanup() override; > Item_equal *get_item_equal() override { return item_equal; } > @@ -5887,6 +5902,11 @@ class Item_func_or_sum: public Item_result_field, > return true; > return (this->*processor)(arg); > } > + bool recalc_maybe_null_processor(void *arg) override > + { > + set_maybe_null(arg_check_maybe_null()); > + return 0; > + } In the testcase, we come across this method for the item "t2.a is null" (aka "isnull(t2.a)"). And here it sets its maybe_null to true. But isnull/is null returns either 0 or 1, so can never have value NULL. Is it ok to set its maybe_null to true then? How about add an Item_func_isnull::recalc_maybe_null_processor() that simply does set_maybe_null(false)? (rr) p this $18 = (Item_func_isnull * const) 0x7f9e1801a000 (rr) p arg_check_maybe_null() $19 = true (rr) dbp this $20 = 0x5602a0000700 <dbug_item_print_buf> "t2.a is null" > /* > Built-in schema, e.g. mariadb_schema, oracle_schema, maxdb_schema > */ > @@ -6216,6 +6236,11 @@ class Item_ref :public Item_ident > return cleanup_processor(arg); > } > bool ora_join_processor(void *arg) override; > + bool recalc_maybe_null_processor(void *arg) override > + { > + set_maybe_null((*ref)->maybe_null()); > + return 0; > + } > Item *field_transformer_for_having_pushdown(THD *thd, uchar *arg) override > { return (*ref)->field_transformer_for_having_pushdown(thd, arg); } > }; Why only implement the processor for these four subclasses only (Item_field, Item_func_or_sum, Item_ref, Item_in_subselect)? Is it because oracle (+) only affects these types of items? > diff --git a/sql/item_cmpfunc.h b/sql/item_cmpfunc.h > index 86c8807a75a..c65d5253c34 100644 > --- a/sql/item_cmpfunc.h > +++ b/sql/item_cmpfunc.h > @@ -3341,7 +3341,7 @@ class Item_cond :public Item_bool_func > void split_sum_func(THD *thd, Ref_ptr_array ref_pointer_array, > List<Item> &fields, uint flags) override; > friend int setup_conds(THD *thd, TABLE_LIST *tables, TABLE_LIST *leaves, > - COND **conds); > + COND **conds, List<Item> *all_fields); > void copy_andor_arguments(THD *thd, Item_cond *item); > bool walk(Item_processor processor, void *arg, > item_walk_flags flags) override; > diff --git a/sql/item_subselect.h b/sql/item_subselect.h > index cb58c62196c..e2c160b0ae3 100644 > --- a/sql/item_subselect.h > +++ b/sql/item_subselect.h > @@ -800,7 +800,12 @@ class Item_in_subselect :public Item_exists_subselect > } > return FALSE; > } > - > + bool recalc_maybe_null_processor(void *arg) override > + { > + if (left_expr->maybe_null()) > + set_maybe_null(true); > + return 0; > + } Can you add a test that covers the Item_in_subselect method here? > friend class Item_ref_null_helper; > friend class Item_is_not_null_test; > friend class Item_in_optimizer; > diff --git a/sql/sql_base.cc b/sql/sql_base.cc > index e4c914f8e7c..7f4cff4e253 100644 > --- a/sql/sql_base.cc > +++ b/sql/sql_base.cc > @@ -8913,10 +8913,11 @@ bool setup_on_expr(THD *thd, TABLE_LIST *table, bool is_update) > [... 53 lines elided] > #endif /* SQL_BASE_INCLUDED */ > diff --git a/sql/sql_oracle_outer_join.cc b/sql/sql_oracle_outer_join.cc > index 524dd3391db..84988d0147a 100644 > --- a/sql/sql_oracle_outer_join.cc > +++ b/sql/sql_oracle_outer_join.cc > @@ -744,7 +744,8 @@ static bool init_tables_array(TABLE_LIST *tables, > bool setup_oracle_join(THD *thd, Item **conds, > TABLE_LIST *tables, > SQL_I_List<TABLE_LIST> &select_table_list, > - List<TABLE_LIST> *select_join_list) > + List<TABLE_LIST> *select_join_list, > + List<Item> *all_fields) > { > DBUG_ENTER("setup_oracle_join"); > uint n_tables= select_table_list.elements; > @@ -964,6 +965,9 @@ bool setup_oracle_join(THD *thd, Item **conds, > { > DBUG_ASSERT(curr->on_conds.elements > 0); > curr->table->outer_join|=JOIN_TYPE_LEFT; > + // it is done after setting table map so maybe_null also set I assume "setting table map" here refers to a call to setup_table_map(). If so, I suggest that the comment be made more readable, such as: // update maybe_null which was previously set in setup_table_map() > + if (curr->table->table) > + curr->table->table->maybe_null= JOIN_TYPE_LEFT; > if (curr->on_conds.elements == 1) > { > curr->table->on_expr= curr->on_conds.head(); > @@ -1028,6 +1032,34 @@ bool setup_oracle_join(THD *thd, Item **conds, > if (add_conditions_to_where(thd, conds, std::move(return_to_where))) > DBUG_RETURN(TRUE); > + // refresh nulability of already fixed parts (WHERE, SELECT list, moved ON) Thoughts: - s/nulability/nullability/ - all_fields is not just the SELECT list. - conds[0] = WHERE + moved ON So I suggest update the comment to be something more accurate like: // refresh nullability of already fixed parts: WHERE and moved ON, // all_fields including the SELECT list, and original ON > + if (conds[0]) > + { > + conds[0]->update_used_tables(); > + conds[0]->walk(&Item::recalc_maybe_null_processor, 0, 0); > + } > + if (all_fields) > + { > + List_iterator<Item> it(*all_fields); > + Item *item; > + while((item= it++)) > + { > + item->update_used_tables(); > + item->walk(&Item::recalc_maybe_null_processor, 0, 0); > + } > + } > + for (i= 0; i < n_tables; i++) > + { > + // we have to count becaust this lists are included in other lists > + List_iterator<Item> it(*all_fields); > + Item *item; > + for (uint j= 0; j < tab[i].on_conds.elements && (item= it++); j++) > + { > + item->update_used_tables(); > + item->walk(&Item::recalc_maybe_null_processor, 0, 0); > + } > + } > + > DBUG_RETURN(FALSE); > } Are the three calls to Item::update_used_tables() needed? I commented them out and nothing bad happened to the test. > diff --git a/sql/sql_select.cc b/sql/sql_select.cc > index d3bfcb8475c..9473746c150 100644 > --- a/sql/sql_select.cc > +++ b/sql/sql_select.cc > @@ -954,7 +954,7 @@ setup_without_group(THD *thd, Ref_ptr_array ref_pointer_array, > DBUG_ENTER("setup_without_group"); > thd->lex->allow_sum_func.clear_bit(select->nest_level); > - res= setup_conds(thd, tables, leaves, conds); > + res= setup_conds(thd, tables, leaves, conds, &all_fields); > /* it's not wrong to have non-aggregated columns in a WHERE */ > select->set_non_agg_field_used(saved_non_agg_field_used); > diff --git a/sql/table.cc b/sql/table.cc > index 8717c740c4f..1ee7184bdb8 100644 > --- a/sql/table.cc > +++ b/sql/table.cc > @@ -10569,7 +10569,7 @@ bool TR_table::query(ulonglong trx_id) > Item *field= newx Item_field(thd, &slex.context, (*this)[FLD_TRX_ID]); > Item *value= newx Item_int(thd, trx_id); > COND *conds= newx Item_func_eq(thd, field, value); > - if (unlikely((error= setup_conds(thd, this, dummy, &conds)))) > + if (unlikely((error= setup_conds(thd, this, dummy, &conds, NULL)))) > return false; > select= make_select(table, 0, 0, conds, NULL, 0, &error); > if (unlikely(error || !select)) > @@ -10608,7 +10608,7 @@ bool TR_table::query(MYSQL_TIME &commit_time, bool backwards) > conds= newx Item_func_ge(thd, field, value); > else > conds= newx Item_func_le(thd, field, value); > - if (unlikely((error= setup_conds(thd, this, dummy, &conds)))) > + if (unlikely((error= setup_conds(thd, this, dummy, &conds, NULL)))) > return false; > // FIXME: (performance) force index 'commit_timestamp' > select= make_select(table, 0, 0, conds, NULL, 0, &error); > -- I'm not sure what the TR_table is about - from git blame it looks like it has something to do with versioning. Is the oracle (+) join relevant here? If not (e.g. these tables are not accessible by users) then I suppose it's ok to pass NULL here. > 2.39.2 Best, Yuchen

1 1

How to verify a clean MariaDB shutdown in logs before a major version upgrade?
by Otto Kekäläinen 22 Jun '25

22 Jun '25

Hi! Congrats on the MariaDB 11.8 release! So far in Debian and Ubuntu no regressions have been reported and looks everything is working well. The most frequent thing I see people complain about is not new to 11.8, but I think introduced in 10.4 with InnoDB refusing to start if crash recovery is needed for a database of an earlier major version. This is explained in e.g. https://web.archive.org/web/20240912062010/https://mariadb.com/kb/en/innodb… To help DBAs ensure they have a clean shutdown before doing a major version upgrade, I was wondering what are the best ways to confirm a clean shutdown by checking the MariaDB logs or files in /var/log/mysql? Specifically, what log entries or patterns in the MariaDB logs indicate a clean shutdown? Additionally, are there particular files or indicators in /var/log/mysql that reliably show the clean shutdown status? For example, if all buffers were flushed, should I expect that some files have size zero?

3 2

Re: [PATCH] MDEV-36883 Oracle outer join syntax (+): operator (+) is not processed in condition like "(t2.b(+) , t1.b) in (select ...)"
by Yuchen Pei 19 Jun '25

19 Jun '25

> From 6311e48c8b60b091c9030612e73ae2d42ec3da9b Mon Sep 17 00:00:00 2001 > From: Oleksandr Byelkin <sanja(a)mariadb.com> > Date: Mon, 16 Jun 2025 19:05:04 +0200 > Subject: [PATCH] MDEV-36883 Oracle outer join syntax (+): operator (+) is not > processed in condition like "(t2.b(+) , t1.b) in (select ...)" > Fixed transfer of ora join flag from left expression to whole query > (fix for all flags and versions?) > Prohibit (+) in row operations. What is a row operation? I can infer some rules from this patch and some related code, where the following is not allowed: 1. An Item_row has a (+) somewhere 2. The LHS of an IN function or subquery is a tuple of at least two items and has a (+) somewhere 3. The RHS of an IN function has a (+) somewhere Note that Items 1 and 2 above are introduced in this patch, and Item 3 was already there. It is hard to tell from these rules what a row operation is. For example, "t1.c1 IN (t2.c2(+), t2.c3(+))" is banned because of Item 3 above, but Item 1 causes "(t2.a, t2.b) = (t1.c(+), t1.b(+))" to be banned. Does that mean that the latter is a row operation but not the former? As another example, "t2.a(+) IN (t1.c, t1.b)" is allowed so it can't be a row operation, but Item 1 bans "(t2.a(+), t2.b(+)) = (t1.c, t1.b)". If there is no simple definition of row operations, can you list the actual Rules 1 and 2 in the commit message? Items 1-3 above is also not the complete set of rules of all disallowed cases. And the division between what is allowed and what is not allowed is getting quite complex (another example: "t1.c1(+) IN (t2.c2, t2.c2)" is allowed but "t1.c1 IN (t2.c2(+), t2.c2(+))" is not). I suppose the complete set of rules will be in the docs when the feature is released. But it would be nice to document them in the code as well, especially that the beginning of sql_oracle_outer_join.cc already mentions some forbidden cases such as cycles, so it would be natural for the rest of the rules to go there. > --- > .../compat/oracle/r/ora_outer_join.result | 4 +-- > .../compat/oracle/r/ora_outer_join_err.result | 22 ++++++++++++++ > .../compat/oracle/t/ora_outer_join_err.test | 29 +++++++++++++++++++ > sql/item_cmpfunc.cc | 6 ++++ > sql/item_row.h | 11 +++++++ > sql/item_subselect.cc | 4 ++- > sql/item_subselect.h | 11 +++++++ > sql/share/errmsg-utf8.txt | 2 +- > 8 files changed, 85 insertions(+), 4 deletions(-) > diff --git a/mysql-test/suite/compat/oracle/r/ora_outer_join.result b/mysql-test/suite/compat/oracle/r/ora_outer_join.result > index ac0a09d5c43..1b99e1b3014 100644 > --- a/mysql-test/suite/compat/oracle/r/ora_outer_join.result > +++ b/mysql-test/suite/compat/oracle/r/ora_outer_join.result > @@ -294,9 +294,9 @@ Warning 4232 Oracle outer join operator (+) ignored in '"test"."tj1"."a" = 1' > # ORA-01719 > # > SELECT * FROM tj1, tj2 WHERE tj1.a IN (tj2.c(+), tj2.d(+)); > -ERROR HY000: Invalid usage of (+) operator: used in OR or IN > +ERROR HY000: Invalid usage of (+) operator: used in OR, IN or ROW operation > SELECT * FROM tj1, tj2 WHERE tj1.a NOT IN (tj2.c(+), tj2.d(+)); > -ERROR HY000: Invalid usage of (+) operator: used in OR or IN > +ERROR HY000: Invalid usage of (+) operator: used in OR, IN or ROW operation > # > # Outer join in 'IN' condition with a single expression > # This is also allowed in oracle since the expression is > diff --git a/mysql-test/suite/compat/oracle/r/ora_outer_join_err.result b/mysql-test/suite/compat/oracle/r/ora_outer_join_err.result > index 59ea247ccb3..f54ee14bbf4 100644 > --- a/mysql-test/suite/compat/oracle/r/ora_outer_join_err.result > +++ b/mysql-test/suite/compat/oracle/r/ora_outer_join_err.result > @@ -83,3 +83,25 @@ select u2.a, (select t1.a, t2.a from t1,t2 where u1.a(+) = t2.a) > from t1 as u1, t2 as u2 where u1.a(+) = u2.a; > ERROR HY000: Invalid usage of (+) operator with outer reference a > DROP TABLE t1,t2; > +# > +# MDEV-36883: Oracle outer join syntax (+): operator (+) is not > +# processed in condition like "(t2.b(+) , t1.b) in (select ...)" > +# > +create table t1 ( c int, b char(1)); > +insert into t1 values (1,'b'); > +create table t2 ( a int , b char(1)); > +insert into t2 values (1,'a'); > +create table t3 (c1 char(1), c2 char(2)); > +insert into t3 values ('c','d'); > +insert into t3 values ('c','d'); > +SELECT t2.b > +FROM t1, t2 WHERE t1.c = t2.a(+) AND (t2.b(+), t1.b) IN (SELECT * from t3); > +ERROR HY000: Invalid usage of (+) operator: used in OR, IN or ROW operation Question: I made a little modification to this query: SELECT t2.b FROM t1, t2 WHERE t1.c = t2.a(+) AND (t2.b(+), t2.b) IN (SELECT c1, c1 from t3); Instead of ER_INVALID_USE_OF_ORA_JOIN_WRONG_FUNC, I got ER_INVALID_USE_OF_ORA_JOIN_CYCLE. Does it make sense to count this as a cycle? > +SELECT t2.b > +FROM t1, t2 WHERE t1.c = t2.a(+) AND (t2.b(+), t1.b) IN (('a','a'),('b','b')); > +ERROR HY000: Invalid usage of (+) operator: used in OR, IN or ROW operation > +SELECT t2.b > +FROM t1, t2 WHERE t1.c = t2.a(+) AND (t2.b(+), t1.b) = (SELECT * from t3 > +ORDER BY a LIMIT 1); > +ERROR HY000: Invalid usage of (+) operator: used in OR, IN or ROW operation > +drop tables t1,t2,t3; > diff --git a/mysql-test/suite/compat/oracle/t/ora_outer_join_err.test b/mysql-test/suite/compat/oracle/t/ora_outer_join_err.test > index 22bb8514b79..94faa293323 100644 > --- a/mysql-test/suite/compat/oracle/t/ora_outer_join_err.test > +++ b/mysql-test/suite/compat/oracle/t/ora_outer_join_err.test > @@ -107,3 +107,32 @@ select u2.a, (select t1.a, t2.a from t1,t2 where u1.a(+) = t2.a) > from t1 as u1, t2 as u2 where u1.a(+) = u2.a; > DROP TABLE t1,t2; > + > +--echo # > +--echo # MDEV-36883: Oracle outer join syntax (+): operator (+) is not > +--echo # processed in condition like "(t2.b(+) , t1.b) in (select ...)" > +--echo # > +create table t1 ( c int, b char(1)); > +insert into t1 values (1,'b'); > + > +create table t2 ( a int , b char(1)); > +insert into t2 values (1,'a'); > + > +create table t3 (c1 char(1), c2 char(2)); > +insert into t3 values ('c','d'); > +insert into t3 values ('c','d'); > + > +--error ER_INVALID_USE_OF_ORA_JOIN_WRONG_FUNC > +SELECT t2.b > +FROM t1, t2 WHERE t1.c = t2.a(+) AND (t2.b(+), t1.b) IN (SELECT * from t3); > + > +--error ER_INVALID_USE_OF_ORA_JOIN_WRONG_FUNC > +SELECT t2.b > +FROM t1, t2 WHERE t1.c = t2.a(+) AND (t2.b(+), t1.b) IN (('a','a'),('b','b')); > + > +--error ER_INVALID_USE_OF_ORA_JOIN_WRONG_FUNC > +SELECT t2.b > +FROM t1, t2 WHERE t1.c = t2.a(+) AND (t2.b(+), t1.b) = (SELECT * from t3 > +ORDER BY a LIMIT 1); > + > +drop tables t1,t2,t3; > diff --git a/sql/item_cmpfunc.cc b/sql/item_cmpfunc.cc > index ca03b13f724..c94e185aa33 100644 > --- a/sql/item_cmpfunc.cc > +++ b/sql/item_cmpfunc.cc > @@ -5037,6 +5037,12 @@ bool Item_func_in::ora_join_processor(void *arg) > { > if (with_ora_join()) > { > + if (cols() > 1 && args[0]->with_ora_join()) rr is telling me that cols() always returns 1 here (see below), so this if condition will always be false (a run of compat/oracle.ora_outer_join_err also does go into this if branch). Do you mean args[0]->cols()? If so, then testcases such as "(t2.b(+), t1.b) IN (('a','a'),('b','b'))" already works because the Item_row method is already returning error. But I suppose there's no harm doing another check here. (rr) list Item_func_in::cols 2523 current value and pointer passed via parameter otherwise. 2524 */ 2525 virtual Item **this_item_addr(THD *thd, Item **addr_arg) { return addr_arg; } 2526 2527 // Row emulation 2528 virtual uint cols() const { return 1; } 2529 virtual Item* element_index(uint i) { return this; } 2530 virtual Item** addr(uint i) { return 0; } 2531 virtual bool check_cols(uint c); 2532 bool check_type_traditional_scalar(const LEX_CSTRING &opname) const; > + { > + // used in ROW operaton > + my_error(ER_INVALID_USE_OF_ORA_JOIN_WRONG_FUNC, MYF(0)); > + return TRUE; > + } > uint n= argument_count(); > DBUG_ASSERT(n >= 2); > // first argument (0) is right part of IN where oracle joins are allowed > diff --git a/sql/item_row.h b/sql/item_row.h > index 2c884c0f858..b234a0499c5 100644 > --- a/sql/item_row.h > +++ b/sql/item_row.h > @@ -152,6 +152,17 @@ class Item_row: public Item_fixed_hybrid, > Item *do_get_copy(THD *thd) const override > { return get_item_copy<Item_row>(thd, this); } > Item *do_build_clone(THD *thd) const override; > + > + bool ora_join_processor(void *arg) override > + { > + if (with_ora_join()) > + { > + // Oracle join operator is used inside rows. > + my_error(ER_INVALID_USE_OF_ORA_JOIN_WRONG_FUNC, MYF(0)); > + return(TRUE); > + } > + return (FALSE); > + } > }; Just a note: this part does not apply to "t1.c1 IN (t2.c2(+), t2.c3(+))" because no Item_row is involved here even though the rhs looks exactly the same as an Item_row. The Item_func_in has three args. This is banned is because anything on the RHS of Item_func_in with a (+) is banned. > #endif /* ITEM_ROW_INCLUDED */ > diff --git a/sql/item_subselect.cc b/sql/item_subselect.cc > index 57a1f01294c..e14f1a01763 100644 > --- a/sql/item_subselect.cc > +++ b/sql/item_subselect.cc > @@ -3614,11 +3614,13 @@ bool Item_in_subselect::fix_fields(THD *thd_arg, Item **ref) > } > } > - if (left_expr && left_expr->fix_fields_if_needed(thd_arg, &left_expr)) > + if (!left_expr || left_expr->fix_fields_if_needed(thd_arg, &left_expr)) > goto err; What is this change for? The test in the patch compat/oracle.ora_outer_join_err do not reach inside the if branch. Neither does the other test compat/oracle.ora_outer_join > else > if (Item_subselect::fix_fields(thd_arg, ref)) > goto err; > + if (left_expr->with_ora_join()) > + copy_flags(left_expr, item_with_t::ORA_JOIN); What are the circumstances when this is needed? All statements in compat/oracle.ora_outer_join_err that reach the copy_flags call here have left_expr already having the ORA_JOIN flag. The other test compat/oracle.ora_outer_join does not reach here. > base_flags|= item_base_t::FIXED; > thd->where= save_where; > DBUG_RETURN(FALSE); > diff --git a/sql/item_subselect.h b/sql/item_subselect.h > index f75fafc4812..cb58c62196c 100644 > --- a/sql/item_subselect.h > +++ b/sql/item_subselect.h > @@ -790,6 +790,17 @@ class Item_in_subselect :public Item_exists_subselect > Subq_materialization_tracker *get_materialization_tracker() const > { return materialization_tracker; } > + bool ora_join_processor(void *arg) override > + { > + if (left_expr->with_ora_join() && left_expr->cols() > 1) > + { > + // used in ROW operaton > + my_error(ER_INVALID_USE_OF_ORA_JOIN_WRONG_FUNC, MYF(0)); > + return TRUE; > + } > + return FALSE; > + } > + This if branch is not entered in the test, likely because the Item_row method already returns error before the walk calls the method on the Item_in_subselect. Is it possible to add a test that covers this? > [... 19 lines elided] Best, Yuchen

1 0

Re: b69bd351c1f: MDEV-35617: DROP USER should leave no active session for that user
by Sergei Golubchik 12 Jun '25

12 Jun '25

Hi, Dmitri, On Jun 06, Dmitri Shulga wrote: > commit b69bd351c1f > Author: Dmitri Shulga <dmitry.shulga(a)mariadb.com> > Date: Wed May 28 00:05:05 2025 +0700 > > MDEV-35617: DROP USER should leave no active session for that user > > On handling the DROP USER statement it is counted a number of sessions > established on behalf every user being dropped. In case the DROP USER > statement is executed in sql_mode = oracle the error > ER_CANNOT_USER > if there are active connections for any of the users listed at > the DROP USER statement. For sql_mode != oracle the warning > ER_ACTIVE_CONNECTIONS_FOR_USER_TO_DROP > if there are active connections. This is difficult to read. Better: DROP USER looks for sessions by the do-be-dropped user and if found: * fails with ER_CANNOT_USER in Oracle mode * continues with ER_ACTIVE_CONNECTIONS_FOR_USER_TO_DROP warning otherwise New connections for that user are rejected with ER_CONNECT_WHILE_DROP_USER_IN_PROGRESS > > Every user being dropped is marked with flag that disallow establishing > a new connections on behalf this user. In case the user tries to establish > a new session while the DROP USER is executed for this account, > connection establishing is failed with the error > ER_CONNECT_WHILE_DROP_USER_IN_PROGRESS > > diff --git a/mysql-test/main/create_drop_user.result b/mysql-test/main/create_drop_user.result > index 67717f3e4e0..4f46481dfc7 100644 > --- a/mysql-test/main/create_drop_user.result > +++ b/mysql-test/main/create_drop_user.result > @@ -41,3 +41,57 @@ Warnings: > Note 1974 Can't drop user 'u1'@'%'; it doesn't exist > DROP USER u2; > ERROR HY000: Operation DROP USER failed for 'u2'@'%' > +# > +# MDEV-35617: DROP USER should leave no active session for that user > +# > +CREATE USER u1; > +CREATE USER u2; > +CREATE USER u3; > +GRANT ALL on test.* to u1; > +GRANT ALL on test.* to u2; > +GRANT ALL on test.* to u3; > +# Establish two connections on behalf the users u1, u3 > +# A connection on behalf the user u2 isn't etablished intentionally "established" > +connect con1, localhost, u1, , test; > +connect con3, localhost, u3, , test; > +# Drop the users u1, u2, u3. Since the users u1 and u3 have active > +# connections to the server, the warning about it be output "will be" > +connection default; > +DROP USER u1, u2, u3; > +Warnings: > +Note 4226 Dropped users ['u1'@'%','u3'@'%'] have active connections. Use KILL CONNECTION if they should not be used anymore. I don't think we have any other error message that puts a list in square brackets. So let's not do it here either. See e.g. ER_CANNOT_USER > +# None of the users u1, u2, u3 should be presence in the system "be present" > +SELECT user, host FROM mysql.user WHERE user IN ('u1', 'u2', 'u3'); > +User Host > +disconnect con1; > +disconnect con3; > +# Check behaviour of the DROP USER statement in > +# oracle compatibility mode > +SET @save_sql_mode = @@sql_mode; > +SET sql_mode="oracle"; > +CREATE USER u1; > +CREATE USER u2; > +CREATE USER u3; > +GRANT ALL on test.* to u1; > +GRANT ALL on test.* to u2; > +GRANT ALL on test.* to u3; > +# Established two connections on behalf the users u1, u3; > +# A connection on behalf the user u2 isn't etablished intentionally "established" > +connect con1, localhost, u1, , test; > +connect con3, localhost, u3, , test; > +connection default; > +# In oracle compatibility mode, DROP USER fails in case > +# there are connections on behalf the users being dropped. > +DROP USER u1, u2, u3; > +ERROR HY000: Operation DROP USER failed for 'u1'@'%','u3'@'%' See? No square brackets :) > +# It is expected to see all three users in output of the query I don't think it's a good idea. DROP USER drops everything it can and reports an error for everything it couldn't drop. It's not all-or-nothing behavior. Same for all other statements that can fail with ER_CANNOT_USER. Same for DROP TABLES. All-or-nothing here is rather unexpected and inconsisent with pretty much every other multi-object command in the server. > +SELECT user, host FROM mysql.user WHERE user IN ('u1', 'u2', 'u3'); > +User Host > +u1 % > +u2 % > +u3 % > +SET sql_mode= @save_sql_mode; > +disconnect con1; > +disconnect con3; > +# Clean up > +DROP USER u1, u2, u3; > diff --git a/sql/sql_acl.cc b/sql/sql_acl.cc > index 26905f0d415..b519eec15c8 100644 > --- a/sql/sql_acl.cc > +++ b/sql/sql_acl.cc > @@ -210,6 +210,12 @@ class ACL_USER_PARAM > > class ACL_USER :public ACL_USER_BASE, public ACL_USER_PARAM > { > + /* > + Ephemeral state (meaning it is not stored anywhere in the Data Dictionary) > + to disable establishing sessions in case the user is being dropped. > + */ > + bool dont_accept_conn= false; should be in ACL_USER_PARAM, I suppose, for consistency. password_errors is there, and it's ephemeral too. > + > public: > ACL_USER() = default; > ACL_USER(THD *, const LEX_USER &, const Account_options &, const privilege_t); > @@ -243,6 +249,7 @@ class ACL_USER :public ACL_USER_BASE, public ACL_USER_PARAM > dst->host.hostname= safe_strdup_root(root, host.hostname); > dst->default_rolename= safe_lexcstrdup_root(root, default_rolename); > bzero(&dst->role_grants, sizeof(role_grants)); > + dst->dont_accept_conn= dont_accept_conn; not needed, there's `*dst= *this` ~20 lines above. > return dst; > } > > @@ -11337,6 +11354,85 @@ bool mysql_create_user(THD *thd, List <LEX_USER> &list, bool handle_as_role) > DBUG_RETURN(result); > } > > + > +struct count_user_threads_callback_arg > +{ > + explicit count_user_threads_callback_arg(LEX_USER *user_arg) > + : user(user_arg), counter(0) > + {} > + > + LEX_USER *user; > + uint counter; > +}; > + > + > +/** > + Callback function invoked for every active THD to count a number of > + sessions established by specified user > + > + @param[in] thd Thread context > + @param arg Account info for that a number of active connections > + should be countered > +*/ > + > +static my_bool count_threads_callback(THD *thd, > + count_user_threads_callback_arg *arg) > +{ > + if (thd->security_ctx->user) > + { > + /* > + Check that hostname (if given) and user name matches. > + > + host.str[0] == '%' means that host name was not given. See sql_yacc.yy > + */ > + if (((arg->user->host.str[0] == '%' && !arg->user->host.str[1]) || > + !strcmp(thd->security_ctx->host_or_ip, arg->user->host.str)) && > + !strcmp(thd->security_ctx->user, arg->user->user.str)) eh, no. Don't invent new username matching rules. In particular, incorrect ones :) This session was already matched, so do if (!strcmp(arg->user->host.str, thd->main_security_ctx.priv_host) && !strcmp(arg->user->user.str, thd->main_security_ctx.priv_user)) And let's add a test, like create user u@localhost; create user u@'%'; connect u,localhost,u; connection default; drop user u@'%'; and there should be neither warning nor an error because of the existing connection by u@localhost. > + arg->counter++; > + } > + return false; > +} > + > + > +/** > + Get a number of connections currently established on behalf the user > + > + @param[in] thd Thread context > + @param[in] user User credential to count up a number of connections > + > + @return a number of connections established by the user at the moment of > + the function call > +*/ > + > +static int count_sessions_for_user(LEX_USER *user) > +{ > + count_user_threads_callback_arg arg(user); > + bool ret __attribute__((unused)); > + > + ret= server_threads.iterate(count_threads_callback, &arg); iterate() takes a lock. You want to release it as soon as possible. And you don't really use the value of a counter, so why bother? Abort the search and return on the first match. > + DBUG_ASSERT(ret == false); > + > + return arg.counter; > +} > + > + > +/** > + Find the specified user and mark it as not accepting incoming sessions > + > + @param user_name the user for that accept of incoming connections > + should be disabled > +*/ > + > +static void disable_connections_for_user(LEX_USER *user) > +{ > + ACL_USER *found_user= find_user_or_anon(user->host.str, user->user.str, > + nullptr); No, find_user_exact(). And let's add this test case: create user u1@'%'; --error ER_CANNOT_USER drop user u1@localhost; connect u1,localhost,u1; select user(), current_user(); > + > + if (found_user != nullptr) > + found_user->disable_new_connections(); > +} > + > + > /* > Drop a list of users and all their privileges. > > @@ -11395,6 +11496,39 @@ bool mysql_drop_user(THD *thd, List <LEX_USER> &list, bool handle_as_role) > continue; > } > > + if (count_sessions_for_user(user_name) != 0) > + append_user(thd, &connected_users, user_name); > + > + correct_users_list.push_back(user_name); > + /* > + Prevent new connections to be established on behalf the user > + being dropped. > + */ > + disable_connections_for_user(user_name); > + } > + > + if (!connected_users.is_empty() && > + (thd->variables.sql_mode & MODE_ORACLE)) > + { > + /* > + Throw error in case there are connections for the user being dropped AND > + sql_mode = oracle > + */ > + mysql_mutex_unlock(&acl_cache->lock); > + mysql_rwlock_unlock(&LOCK_grant); > + > + my_error(ER_CANNOT_USER, MYF(0), > + (handle_as_role) ? "DROP ROLE" : "DROP USER", There can be no DROP ROLE here, everything you're doing above shouldn't even be done for DROP ROLE, check handle_as_role to avoid unnecessary locks. > + connected_users.c_ptr_safe()); > + > + DBUG_RETURN(true); ok, another test case to add: create user u@localhost; set sql_mode=oracle; connect u,localhost,u; connection default; --error ER_CANNOT_USER drop user u@'%'; connect u,localhost,u; select user(), current_user(); > + } > + > + user_list.init(correct_users_list); you don't need correct_users_list, because there shouldn't be inconsistent all-or-nothing behavior > + while ((user_name= user_list++)) > + { > + int rc; > + > if ((rc= handle_grant_data(thd, tables, 1, user_name, NULL)) > 0) > { > // The user or role was successfully deleted > @@ -15127,6 +15267,18 @@ bool acl_authenticate(THD *thd, uint com_change_user_pkt_len) > } > #endif > > + /* > + Attempt to establish a new connection on behalf the user that is > + currently being dropped from a concurrent session. > + Terminate the connection. > + */ > + if (acl_user->dont_accept_new_connections()) > + { > + my_error(ER_CONNECT_WHILE_DROP_USER_IN_PROGRESS, MYF(0), > + acl_user->get_username()); I'd just say user not found. That is, no need to a special error, but find_mpvio_user() can simply skip entries that are being dropped. Also, how can a connection see a user being dropped, if mysql_drop_user() takes acl_cache->lock and doesn't release it until everything is dropped? > + DBUG_RETURN(1); > + } > + > if (set_privs_on_login(thd, acl_user)) > DBUG_RETURN(1); > } Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 1

Re: MDEV-36851 COALESCE() returns nullable column while IFNULL() does not — unexpected behavior in VIEW definition
by Alexander Barkov 05 Jun '25

05 Jun '25

Hello Raghu, This is a review for your patch for MDEV-36851 > · > MDEV-36581: Fix nullability logic of COALESCE The MDEV number in the comment is incorrect. Should be MDEV-36851 Please next time make sure to copy the MDEV number together with its title from Jira, to avoid typos. > COALESCE currently uses a generic logic- "Result of a function > is nullable if any of the arguments is nullable", which is wrong. > > This patch fixes the logic behind the nullability of COALESCE > to make the result NULL, only if all of the arguments are NULL. > > Tests included in `mysql-test/main/func_hybrid_type.test` The general idea of the patch is correct. I have small suggestions for the implementation. <cut> > --- a/sql/item_cmpfunc.h > +++ b/sql/item_cmpfunc.h > @@ -1148,6 +1148,14 @@ class Item_func_interval :public Item_long_func > > class Item_func_coalesce :public Item_func_case_expression > { > +protected: > + bool all_args_maybe_null= true; > + inline void set_nullability_with(const Item *item) override > + { > + if (all_args_maybe_null && !(bool) (item->base_flags & item_base_t::MAYBE_NULL)) > + all_args_maybe_null= false; > + base_flags|= item->base_flags & item_base_t::MAYBE_NULL > + } I don't think we need a new member and a new virtual method. This is too complex to fix this problem. Please (instead of trying to build-in the new code inside fix_fields()) add a loop into fix_length_and_dec() searching for a non-NULL argument and removing the MAYBE_NULL flag if such argument is found. I also suggest additional tests: without view and without derived tables. Just using regular tables. Thanks. > public: > Item_func_coalesce(THD *thd, Item *a, Item *b): > Item_func_case_expression(thd, a, b) {} > @@ -1162,6 +1170,12 @@ class Item_func_coalesce :public Item_func_case_expression > bool native_op(THD *thd, Native *to) override; > bool fix_length_and_dec(THD *thd) override > { > + /* > + Result of COALESCE is nullable iff all arguments > + are NULL, otherwise NOT NULL. > + */ > + if (!all_args_maybe_null) > + base_flags &= ~item_base_t::MAYBE_NULL; > if (aggregate_for_result(func_name_cstring(), args, arg_count, true)) > return TRUE; > fix_attributes(args, arg_count); <cut>

1 0

11.62 to 11.72 Database Upgrade Failure
by jashley＠deadacated.com 03 Jun '25

03 Jun '25

I updated and upgraded the database from 1.32 to 1.62 successfully. 11.62 to 11.72 database upgrade is failing. Is this a known issue? Any suggestions for fixing? Thank you, Jay

2 1

Re: 39f1b4c93e7: MDEV-36053 CURSOR declarations in PACKAGE BODY
by Sergei Golubchik 30 May '25

30 May '25

Hi, Alexander, Phew, it was a big patch. Took me 20 days. Please, see comments below. On May 09, Alexander Barkov wrote: > commit 39f1b4c93e7 > Author: Alexander Barkov <bar(a)mariadb.com> > Date: Mon Apr 14 12:00:57 2025 +0400 > > MDEV-36053 CURSOR declarations in PACKAGE BODY > > This change adds CURSOR declarations inside PACKAGE BODY. > > PL/SQL mode: > > SET sql_mode=ORACLE; > CREATE PACKAGE BODY pkg AS > CURSOR mc0 IS SELECT c0, c1 FROM t1; > PROCEDURE p1 AS > rec mc0%ROWTYPE; > BEGIN > OPEN mc0; > FETCH mc0 INTO rec; > CLOSE mc0 > END; > END; > / > > SQL/PSM mode: > > SET sql_mode=DEFAULT; > CREATE PACKAGE BODY pkg > mc0 CURSOR FOR SELECT c0, c1 FROM t1; > PROCEDURE p1() > BEGIN > DECLARE rec ROW TYPE OF mc0; > OPEN mc0; > FETCH mc0 INTO rec; > CLOSE mc0 > END; > END; > / > PACKAGE BODY cursors like local cursors > (declared inside a FUNCTION or a PROCEDURE) support: > - OPEN/FETCH/CLOSE > - FOR rec IN cur - an explicit cursor loop > - Using a cursor row as an anchored variable data type: > * DECLARE var cur%ROWTYPE; -- sql_mode=ORACLE > * DECLARE var ROW TYPE OF cur; -- sql_mode=DEFAULT > > The patch details: > > - Changing various class members and function/method parameters which > store a CURSOR run-time address from "uint" to sp_rcontext_addr. > A few classes now derive from sp_rcontext_addr instead of having > a "uint m_cursor;" member. > > This change uses the same idea with what we did for SP variables, > when we implemented PACKAGE BODY variables a few years ago. > > - Fixing the grammar in sql_yacc.yy to allow CURSOR declarations > inside PACKAGE BODY. > > - Moving the C++ code handing the "CLOSE_SYM ident" grammar > and creating an sp_instr_cclose instance from sql_yacc.yy > into a new method LEX::sp_close(). > This is to have the grammar file smaller. > Also, this code has significantly changed anyway. > > - Adding a new class sp_pcontext_top. It's used for the top level parse > context (of an sp_head). Note, its children contexts still use the old > class sp_pcontext. > > sp_pcontext_top context additionally to sp_pcontext has: > > const sp_head *m_sp; -- The pointer to the sp_head owning this context > Dynamic_array<sp_pcursor> m_member_cursors; -- PACKAGE BODY wide cursors > > m_sp->m_parent->get_parse_context() is used to find the sp_pcontext > belonging to the parent PACKAGE BODY from a sp_pcontext_top instance > belonging to a PROCEDURE/FUNCTION sp_pcontext_top. > > - Adding a new member in sp_rcontext: > Dynamic_array<sp_cursor*> m_member_cursors; > It's used to store run-time data of PACKAGE BODY wide cursors. > > - Adding a new class sp_instr_copen2. It's used to open PACKAGE BODY cursors. > Unlike the usual cursors, PACKAGE BODY cursors: > * do not use the cursor stack (sp_rcontext::m_cstack) > * do not need a preceeding sp_instr_cpush > * do not need a following sp_instr_cpop > > All cursor information such as "sp_lex_cursor" resides inside > sp_instr_copen2 itself (rather than inside sp_instr_cpush which is > used to store "sp_lex_cursor" in case of sp_instr_copen). > > Note, the other cursor related instructions: > sp_instr_cfetch > sp_instr_cclose > sp_instr_cursor_copy_struct > do not need sp_instr_xxx2 counter-parts. > Thy just use sp_rcontext_addr to address cursors. > > - Adding Sp_rcontext_handler_member > It's used to handle PACKAGE BODY members: > cursors and variables declared in the PACKAGE BODY, > when they are accessed from its executable initialization section: > > CREATE PACKAGE BODY pkg AS > CURSOR mc0 IS SELECT c0, c1 FROM t1; -- A member (PACKAGE BODY cursor) > mv0 mc0%ROWTYPE; -- A member (PACKAGE BODY variable) > PROCEDURE p1 AS > BEGIN > -- Accessing members from here use sp_rcontext_handler_package_body > -- (members of the parent PACKAGE BODY) > OPEN mc0; > FETCH mc0 INTO mv0; > CLOSE mc0; > END; > BEGIN > -- NEW: > -- Accessing members from here use sp_rcontext_handler_member > -- (PACKAGE BODY own members) > OPEN mc0; > FETCH mc0 INTO mv0; > CLOSE mc0; > END; > / > > Member variables and cursor are now marked with the "MEMBER." prefix > in the "SHOW PACKAGE BODY code" output. > Some old MTR tests have been re-recorded accordingly. > > - Adding new virtual methods into Sp_rcontext_handler: > > virtual const sp_variable *get_pvariable(const sp_pcontext *pctx, > uint offset) const; > > virtual const sp_pcursor *get_pcursor(const sp_pcontext *pctx, > uint offset) const; > > They're used from sp_instr::print() virtual implementations. > They internally calculate a proper sp_pcontext using as a parameter > the sp_pcontext pointed by sp_instr::m_ctx. > > For example, Sp_handler_package_body::get_pvariable()/get_pcursor() > accesses to this sp_pcontext: > m_ctx->top_context()->m_sp->m_parent->get_parse_context(), > i.e. the parse context of the PACKAGE BODY which is the parent for > the current package PROCEDURE of FUNCTION an sp_instr belongs to. > > - Adding a new method LEX::find_cursor(). It searches for a cursor in > this order: > * Local cursors in the nearst upper BEGIN/END block. > * A member cursor of the current PACKAGE BODY > (used from the PACKAGE BODY initialization section) > * A member cursor of the parrent PACKAGE BODY > (used from a package PROCEDURE or a package FUNCTION) better write here (and in the function comment below), like 1. Local cursors... 2. A member cursor of the current PACKAGE BODY OR 2. A member cursor of the parrent PACKAGE BODY to emphasize that second and third are alternatives and it never search both in "current PACKAGE BODY" and then in the "parrent PACKAGE BODY" in this order. > > Adding a new method LEX::find_cursor_with_error(). > In case when a cursor is not found, it automatically > raises the ER_SP_CURSOR_MISMATCH SQL condition into > the diagnostics area. > > - Adding a new method sp_head::add_instr_copenX(). > It creates sp_instr_copen for local cursors, > or sp_instr_copen2 for non-local cursors. > > - Adding a new abstract class sp_lex_cursor_instr. > It's used a common parent class for a few sp_instr_xxx classes, > including the new sp_instr_copen2. > This change is needed to avoid code duplication. > > - Adding a new protected method sp_instr::print_cmd_and_var(), to print > an instruction using this format: "command name@offset". > It's used from a few implementations of sp_instr_xxx::print(), > including sp_instr_copen2::print(). > This change is also needed to avoid code duplication. > > - Moving the definition of "class Sp_rcontext_handler" from item.h > into a new file sp_rcontext_handler.h > This is to maitain header dependencies easier, as well as to move > declarations not directly related to "class Item" outside of item.h > > - Adding a new method sp_pcontext::frame_for_members(), to distinguish > easier between local cursors/variables and PACKAGE BODY cursors/variables. > > - Fixing "struct Lex_for_loop_st" to addionally store > a const pointer to Sp_rcontext_handler, to distinguish between: > * FOR rec IN local_cursor > * FOR rec IN package_body_cursor > > diff --git a/mysql-test/include/sp/sp-cursor-package-body-init.inc b/mysql-test/include/sp/sp-cursor-package-body-init.inc > --- /dev/null > +++ b/mysql-test/include/sp/sp-cursor-package-body-init.inc > @@ -0,0 +1,4 @@ > +--disable_query_log > +--let $oracle=`SELECT @@sql_mode LIKE '%ORACLE%'` > +--let $code=`SELECT COALESCE(@code, 0)` well, sure, but wouldn't it be simpler to do let $code=1; in sp-cursor-package-body-code.test ? > +--enable_query_log > diff --git a/mysql-test/include/sp/sp-cursor-package-body-06.inc b/mysql-test/include/sp/sp-cursor-package-body-06.inc > --- /dev/null > +++ b/mysql-test/include/sp/sp-cursor-package-body-06.inc > @@ -0,0 +1,87 @@ > +--echo # > +--echo # PACKAGE BODY initialization: > +--echo # OPEN package_body_cursor > +--echo # FUNCTION: > +--echo # FETCH package_body_cursor INTO package_body_variable > +--echo # PROCEDURE: > +--echo # CLOSE package_body_cursor > +--echo # > + > +--source sp-cursor-package-body-init.inc > + > +if ($oracle == 0) > +{ > + > +DELIMITER /; > +CREATE PACKAGE pkg > + FUNCTION f1() RETURNS INT; > + PROCEDURE p1close(); > +END; > +/ > +CREATE PACKAGE BODY pkg > + DECLARE cur CURSOR FOR SELECT 1 AS c FROM DUAL; > + DECLARE vc0 INT; > + FUNCTION f1() RETURNS INT > + BEGIN > + FETCH cur INTO vc0; > + RETURN vc0; > + END; > + PROCEDURE p1close() > + BEGIN > + CLOSE cur; > + END; > + -- initialization > + OPEN cur; > +END; cool. that's what I was asking for :) > +/ > +DELIMITER ;/ > + > +} > + > + > +if ($oracle > 0) > +{ > + > +DELIMITER /; > +CREATE PACKAGE pkg AS > + FUNCTION f1 RETURN INT; > + PROCEDURE p1close; > +END; > +/ > +CREATE PACKAGE BODY pkg AS > + CURSOR cur IS SELECT 1 AS c FROM DUAL; > + vc0 INT; > + FUNCTION f1 RETURN INT AS > + BEGIN > + FETCH cur INTO vc0; > + RETURN vc0; > + END; > + PROCEDURE p1close AS > + BEGIN > + CLOSE cur; > + END; > +BEGIN > + OPEN cur; > +END; > +/ > +DELIMITER ;/ > + > +} > + > + > +--disable_view_protocol > +--disable_ps2_protocol > +SELECT pkg.f1() FROM DUAL; > +--enable_ps2_protocol > +CALL pkg.p1close(); > +--error ER_SP_CURSOR_NOT_OPEN > +SELECT pkg.f1() FROM DUAL; > +--enable_view_protocol > + > +if ($code > 0) > +{ > +SHOW PACKAGE BODY CODE pkg; > +SHOW FUNCTION CODE pkg.f1; > +SHOW PROCEDURE CODE pkg.p1close; > +} > +DROP PACKAGE pkg; > diff --git a/sql/sp_rcontext_handler.h b/sql/sp_rcontext_handler.h > --- /dev/null > +++ b/sql/sp_rcontext_handler.h > @@ -0,0 +1,141 @@ > +#ifndef SP_RCONTEXT_HANDLER_INCLUDED > +#define SP_RCONTEXT_HANDLER_INCLUDED > + > +/* Copyright (c) 2009, 2025, MariaDB Corporation. > + > + This program is free software; you can redistribute it and/or modify > + it under the terms of the GNU General Public License as published by > + the Free Software Foundation; version 2 of the License. > + > + This program is distributed in the hope that it will be useful, > + but WITHOUT ANY WARRANTY; without even the implied warranty of > + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > + GNU General Public License for more details. > + > + You should have received a copy of the GNU General Public License > + along with this program; if not, write to the Free Software > + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1335 USA */ > + > + > +class sp_rcontext; > +class sp_pcontext; > +class sp_pcursor; > +class sp_cursor; > + > +/** > + A helper class to handle the run time context of various components of SP: > + Variables: > + - local SP variables and SP parameters > + - PACKAGE BODY routine variables > + - (there will be more kinds in the future) > + Cursors: > + - static local cursors > + - static PACKAGE BODY cursors of the parent PACKAGE BODY > + - static PACKAGE BODY own cursors (when used in the executable secion) > +*/ > + > +class Sp_rcontext_handler > +{ > +public: > + virtual ~Sp_rcontext_handler() = default; > + > + > + /** > + A prefix used for SP variable names in queries: > + - EXPLAIN EXTENDED > + - SHOW PROCEDURE CODE > + Local variables and SP parameters have empty prefixes. > + Package body variables are marked with a special prefix. > + This improves readability of the output of these queries, > + especially when a local variable or a parameter has the same > + name with a package body variable. > + */ > + virtual const LEX_CSTRING *get_name_prefix() const= 0; > + > + // Find a parse time SP variable > + virtual const sp_variable *get_pvariable(const sp_pcontext *pctx, > + uint offset) const= 0; > + > + // Find a parse time SP cursor > + virtual const sp_pcursor *get_pcursor(const sp_pcontext *pctx, > + uint offset) const= 0; > + > + /** > + At execution time THD->spcont points to the run-time context (sp_rcontext) > + of the currently executed routine. > + Local variables store their data in the sp_rcontext pointed by thd->spcont. > + Package body variables store data in separate sp_rcontext that belongs > + to the package. > + This method provides access to the proper sp_rcontext structure, > + depending on the SP variable kind. > + */ > + virtual sp_rcontext *get_rcontext(sp_rcontext *ctx) const= 0; > + > + // Find a run time SP cursor > + virtual sp_cursor *get_cursor(THD *thd, uint offset) const= 0; > +}; > + > + > +/* > + A handler to access local variables and cursors. > +*/ > +class Sp_rcontext_handler_local final: public Sp_rcontext_handler > +{ > +public: > + const LEX_CSTRING *get_name_prefix() const override; > + const sp_variable *get_pvariable(const sp_pcontext *pctx, > + uint offset) const override; > + const sp_pcursor *get_pcursor(const sp_pcontext *pctx, > + uint offset) const override; > + sp_rcontext *get_rcontext(sp_rcontext *ctx) const override; > + sp_cursor *get_cursor(THD *thd, uint offset) const override; > +}; > + > + > +/* > + A handler to access parent members, e.g.: > + PACKAGE BODY variables and cursors when used in package routines. > +*/ > +class Sp_rcontext_handler_package_body final :public Sp_rcontext_handler > +{ > +public: > + const LEX_CSTRING *get_name_prefix() const override; > + const sp_variable *get_pvariable(const sp_pcontext *pctx, > + uint offset) const override; > + const sp_pcursor *get_pcursor(const sp_pcontext *pctx, > + uint offset) const override; > + sp_rcontext *get_rcontext(sp_rcontext *ctx) const override; > + sp_cursor *get_cursor(THD *thd, uint offset) const override; > +}; > + > + > +/* > + A handler to access its own members, e.g.: > + PACKAGE BODY variables and cursors when used in > + the initialization section of PACKAGE BODY. > +*/ > +class Sp_rcontext_handler_member final :public Sp_rcontext_handler > +{ > +public: > + const LEX_CSTRING *get_name_prefix() const override; > + const sp_variable *get_pvariable(const sp_pcontext *pctx, > + uint offset) const override; > + const sp_pcursor *get_pcursor(const sp_pcontext *pctx, > + uint offset) const override; > + sp_rcontext *get_rcontext(sp_rcontext *ctx) const override; > + sp_cursor *get_cursor(THD *thd, uint offset) const override; > +}; > + > + > +extern MYSQL_PLUGIN_IMPORT > + Sp_rcontext_handler_local sp_rcontext_handler_local; why is it MYSQL_PLUGIN_IMPORT? I know that it was and you've just copied it, but still? Plugins aren't supposed to use it, are they? Same below. > + > + > +extern MYSQL_PLUGIN_IMPORT > + Sp_rcontext_handler_package_body sp_rcontext_handler_package_body; > + > + > +extern MYSQL_PLUGIN_IMPORT > + Sp_rcontext_handler_member sp_rcontext_handler_member; > + > +#endif // SP_RCONTEXT_HANDLER_INCLUDED > diff --git a/sql/sp_head.h b/sql/sp_head.h > --- a/sql/sp_head.h > +++ b/sql/sp_head.h > @@ -222,6 +223,7 @@ class sp_head :private Query_arena, > LEX_CSTRING m_defstr; > AUTHID m_definer; > > + Query_arena *query_arena() { return this; } this is practically equivalent to inheriting from Query_arena publicly. Why not to do it instead? > const st_sp_chistics &chistics() const { return m_chistics; } > const LEX_CSTRING &comment() const { return m_chistics.comment; } > void set_suid(enum_sp_suid_behaviour suid) { m_chistics.suid= suid; } > diff --git a/sql/sp_pcontext.h b/sql/sp_pcontext.h > --- a/sql/sp_pcontext.h > +++ b/sql/sp_pcontext.h > @@ -319,6 +320,10 @@ class sp_pcursor: public LEX_CSTRING > class sp_pcontext *param_context() const { return m_param_context; } > class sp_lex_cursor *lex() const { return m_lex; } > bool check_param_count_with_error(uint param_count) const; > + bool eq_name(const LEX_CSTRING &name) const Why name isn't Lex_ident_column? > + { > + return !system_charset_info->strnncoll(name.str, name.length, str, length); > + } > }; > > > @@ -826,4 +861,105 @@ class sp_pcontext : public Sql_alloc > }; // class sp_pcontext : public Sql_alloc > > would be good to have a comment here. and why was it not needed for package-wide variables? > +class sp_pcontext_top: public sp_pcontext > +{ > +public: > + sp_pcontext_top(const sp_head *sp); > + > + const sp_head *sp() const > + { > + return m_sp; > + } > + > + bool add_member_cursor(const LEX_CSTRING *name, sp_pcontext *param_ctx, > + class sp_lex_cursor *lex) > + { > + uint dummy_offset; > + if (find_member_cursor(name, &dummy_offset)) > + { > + my_error(ER_SP_DUP_CURS, MYF(0), name->str); > + return true; > + } > + return m_member_cursors.append(sp_pcursor(name, param_ctx, lex)); > + } > + > + const sp_pcursor *find_member_cursor(const LEX_CSTRING *name, > + uint *poff) const > + { > + for (uint i= 0; i < m_member_cursors.elements(); i++) > + { > + if (m_member_cursors.at(i).eq_name(*name)) > + { > + *poff= i; > + return &m_member_cursors.at(i); > + } > + } > + return nullptr; > + } > + > + const sp_pcursor *find_member_cursor(uint offset) const > + { > + return &m_member_cursors.at(offset); > + } > + > + uint member_cursor_count() const > + { > + return (uint) m_member_cursors.elements(); > + } > + > + /* > + Return a possible parse context frame for members. > + > + Members are: > + - CREATE PACKAGE wide variables and cursors (coming soon - MDEV-13139) > + - CREATE PACKAGE BODY wide variables and cursors > + > + Example: "Example (using Oracle syntax):" > + > + CREATE PACKAGE BODY pkg AS -- sp_package > + a0 INT; -- A member > + CURSOR c0; -- A member > + > + PROCEDURE p1() AS -- A method sp_head (package routine) > + a1 INT; -- A local variable - not an sp_package member > + CURSOR c1; -- A local cursor - not an sp_package member > + BEGIN > + OPEN c0; -- Open a cursor member of the parent sp_package > + CLOSE c0; -- Close a cursor member of the parent sp_package > + END; > + > + BEGIN -- Executable initialization section (constructor) > + DECLARE > + a2 INT; -- a local variable - not a member > + CURSOR c2; -- a local cursor - not a member > + BEGIN > + OPEN c0; -- Open its own cursor member (of sp_package) > + CLOSE c0; -- Close its own cursor member (of sp_package) > + END; > + END; > + > + Members are visible in all subroutines and are treated in a different way > + comparing to local variables/cursors. For example, member cursors > + do not need sp_instr_cpush and sp_instr_cpop, unlike local cursors. > + > + The top level parse context frame of an sp_head stores formal parameters: > + - PROCEDURE and FUNCTION can have formal parameters. > + - PACKAGE and PACKAGE BODY cannot have formal parameters, > + but still the top level context frame exists (it's just empty). > + Members reside in child(0) of the top level parse context. > + > + The owning sp_head will check if this frame is really for members, > + depending on the sp_head type (a package vs a routine). > + */ > + sp_pcontext *frame_for_members_candidate() const > + { > + return child_context(0); > + } > + > +private: > + const sp_head *m_sp; > + Dynamic_array<sp_pcursor> m_member_cursors; > +}; > + > + > #endif /* _SP_PCONTEXT_H_ */ > diff --git a/sql/sp_head.cc b/sql/sp_head.cc > --- a/sql/sp_head.cc > +++ b/sql/sp_head.cc > @@ -3818,7 +3821,26 @@ sp_head::set_local_variable_row_field_by_name(THD *thd, sp_pcontext *spcont, > } > > > +bool sp_head::add_instr_copenX(THD *thd, sp_pcontext *spcont, > + const sp_rcontext_addr &addr) > +{ > + sp_instr *i; > + if (addr.rcontext_handler() == &sp_rcontext_handler_local) > + i= new (thd->mem_root) sp_instr_copen(instructions(), spcont, > + addr.offset()); > + else > + { > + const sp_pcursor *pcursor= addr.rcontext_handler()-> > + get_pcursor(spcont, addr.offset()); or spcont->get_pcursor(addr) that's why you've added it, after all :) > + i= new (thd->mem_root) sp_instr_copen2(instructions(), spcont, > + addr, pcursor->lex()); > + } > + return i == NULL || add_instr(i); > +} > + > + > -bool sp_head::add_open_cursor(THD *thd, sp_pcontext *spcont, uint offset, > +bool sp_head::add_open_cursor(THD *thd, sp_pcontext *spcont, > + const sp_rcontext_addr &addr, > sp_pcontext *param_spcont, > List<sp_assignment_lex> *parameters) > { > diff --git a/sql/sp_instr.cc b/sql/sp_instr.cc > --- a/sql/sp_instr.cc > +++ b/sql/sp_instr.cc > @@ -1912,22 +1925,10 @@ sp_instr_cclose::execute(THD *thd, uint *nextp) > void > sp_instr_cclose::print(String *str) > { > - const LEX_CSTRING *cursor_name= m_ctx->find_cursor(m_cursor); > - > + const LEX_CSTRING cmd= {STRING_WITH_LEN("cclose ")}; > + const sp_pcursor *cursor= m_ctx->get_pcursor(*this); > /* cclose name@offset */ > - size_t rsrv= SP_INSTR_UINT_MAXLEN+8; > - > - if (cursor_name) > - rsrv+= cursor_name->length; > - if (str->reserve(rsrv)) > - return; > - str->qs_append(STRING_WITH_LEN("cclose ")); > - if (cursor_name) > - { > - str->qs_append(cursor_name->str, cursor_name->length); > - str->qs_append('@'); > - } > - str->qs_append(m_cursor); > + print_cmd_and_var(cmd, cursor ? *cursor : empty_clex_str, *this, str); may be print_cmd_and_var() should take LEX_CSTRING* as a second argument, not LEX_CSTRING&, then you wouldn't need to repeat cursor ? *cursor : empty_clex_str every time > } > > > @@ -2068,10 +2060,64 @@ sp_instr_cursor_copy_struct::exec_core(THD *thd, uint *nextp) > { > sp_cursor tmp(thd, true); > // Open the cursor without copying data > + // TODO: Check with DmitryS if hiding ROOT_FLAG_READ_ONLY is OK. and? > + DBUG_ASSERT(thd->lex == m_lex_keeper.lex()); > + auto backup_flags= thd->lex->query_arena()->mem_root->flags; > + thd->lex->query_arena()->mem_root->flags&= ~ROOT_FLAG_READ_ONLY; > + > + /* > + Open the cursor in its sp_rcontext, because > + Items inside the cursor statement have rcontext handler > + relative to the cursor rcontext. In this example package: > + > + CREATE PACKAGE BODY pkg AS > + CURSOR c0(p0 INT, p1 VARCHAR(10)) IS SELECT p0, p1 FROM DUAL; > + PROCEDURE p1 AS > + BEGIN > + FOR r0 IN c0(1,'c1') > + LOOP > + SELECT r0.c0, r0.c1; > + END LOOP; > + END; > + END; > + > + Item_splocal instances for the cursor parameters p0 and p1 > + (in the cursor SELECT statement) use sp_rcontext_handler_local. > + But the cursor is opened from the PROCEDURE p1, which > + has its own sp_rcontext. thd->spcont is pointing to the > + PROCEDURE p1 sp_rcontext at the time of the current > + sp_instr::exec_core(). > + > + Using the above package as an example again, let's do the following: > + - backup thd->spcont (pointing to the PROCEDURE p1 sp_rcontext) > + - reset thd->spcont to the sp_rcontext of the PACKAGE BODY pkg > + - open the cursor in its sp_rcontext > + - restore thd->spcont to the sp_rcontext of the PROCEDURE p1 > + */ > + sp_rcontext *rcontext_backup= thd->spcont; > + thd->spcont= m_rcontext_handler->get_rcontext(thd->spcont); > + ret= tmp.open(thd); > + thd->spcont= rcontext_backup; wouldn't it be logical, if the cursor would know its context instead of relying on thd->spcont? then you wouldn't need this hackish swapping. > + thd->lex->query_arena()->mem_root->flags= backup_flags; > + > - if (!(ret= tmp.open(thd))) > + if (!ret) > { > Row_definition_list defs; > /* > + If m_rcontext_handler is sp_rcontext_handler_member then the cursor > + structure is being exported from a package wide cursor to a > + variable, e.g.: > + CREATE PACKAGE BODY pkg AS > + CURSOR c0 IS SELECT 1 AS c0, 'c1' AS c1 FROM DUAL; > + mr0 c0%ROWTYPE; > + ... > + END; may be you should use primarily non-ORACLE-mode syntax in code comments? > + In this case the cursor structure must have the same life cycle with > + the sp_package (pointed by thd->spcont->m_sp), thus let's use > + thd->spcont->m_sp->query_arena() as the arena. > + > + Otherwise, the structure is being exported for a local variable > + whose life cycle is the current sp_head::execute(). > Create row elements on the caller arena. > It's the same arena that was used during sp_rcontext::create(). > This puts cursor%ROWTYPE elements on the same mem_root > @@ -2106,14 +2155,95 @@ sp_instr_cursor_copy_struct::execute(THD *thd, uint *nextp) > void > sp_instr_cursor_copy_struct::print(String *str) > { > - sp_variable *var= m_ctx->find_variable(m_var); > - const LEX_CSTRING *name= m_ctx->find_cursor(m_cursor); > - str->append(STRING_WITH_LEN("cursor_copy_struct ")); > - str->append(name); > + const LEX_CSTRING cmd= {STRING_WITH_LEN("cursor_copy_struct ")}; > + const sp_pcursor *cursor= m_ctx->get_pcursor(*this); > + const sp_variable *var= m_ctx->get_pvariable(m_var); > + // cursor_copy_struct cur@0 var@0 > + print_cmd_and_var(cmd, cursor ? *cursor : empty_clex_str, *this, str); > str->append(' '); > + str->append(*m_var.rcontext_handler()->get_name_prefix()); > str->append(&var->name); > str->append('@'); > - str->append_ulonglong(m_var); > + str->append_ulonglong(m_var.offset()); > +} > + > + > +/* > + sp_instr_copen2 class functions. > +*/ > + > +PSI_statement_info sp_instr_copen2::psi_info= > +{ 0, "copen2", 0}; > + > + > +int > +sp_instr_copen2::execute(THD *thd, uint *nextp) > +{ > + DBUG_ENTER("sp_instr_copen2::execute"); > + m_lex_keeper.disable_query_cache(); > + int res= m_lex_keeper.cursor_reset_lex_and_exec_core(thd, nextp, false, this); > + *nextp= m_ip + 1; > + DBUG_RETURN(res); > +} > + > + > +int sp_instr_copen2::exec_core(THD *thd, uint *nextp) > +{ > + DBUG_ENTER("sp_instr_copen2::exec_core"); > + sp_cursor *cursor= m_rcontext_handler->get_cursor(thd, m_offset); > + /* > + CREATE PACKAGE BODY pkg > + CURSOR package_body_wide_cursor; > + PROCEDURE p1() > + BEGIN > + OPEN package_body_wide_cursor; -- #2 > + CLOSE package_body_wide_cursor; > + END; > + BEGIN > + OPEN package_body_wide_cursor; -- #1 > + CLOSE package_body_wide_cursor; > + END; > + > + "OPEN package_body_wide_cursor" can be called from two different places: > + #1. From the package initialization secion. In this case > + sp_head::mem_main_root::flags does not have the ROOT_FLAG_READ_ONLY bit > + yet - it will be set in the end of the current sp_head::execute(). > + #2. From a package routine. In this case sp_head::execute() representing > + the initialization section of the package was called earlier and > + its mem_main_root::flags already has the ROOT_FLAG_READ_ONLY flag. Why is it set? > + Unset it temporarily. > + */ > + /* > + TODO: > + this assert crashes due to a problem in the metadata modification code what problem? > + */ > + //DBUG_ASSERT(thd->spcont->m_sp->get_package() || > + // (thd->lex->query_arena()->mem_root->flags & ROOT_FLAG_READ_ONLY)); > + DBUG_ASSERT(thd->lex == m_lex_keeper.lex()); > + auto backup_flags= thd->lex->query_arena()->mem_root->flags; > + thd->lex->query_arena()->mem_root->flags&= ~ROOT_FLAG_READ_ONLY; > + > + /* > + Open the cursor in its sp_rcontext. > + See sp_instr_cursor_copy_struct::exec_core() for details. > + */ > + sp_rcontext *ctx_backup= thd->spcont; > + thd->spcont= m_rcontext_handler->get_rcontext(thd->spcont); > + int rc= cursor->open(thd); > + thd->spcont= ctx_backup; > + > + thd->lex->query_arena()->mem_root->flags= backup_flags; > + DBUG_RETURN(rc); > +} > + > + > +void > +sp_instr_copen2::print(String *str) > +{ > + const LEX_CSTRING cmd= {STRING_WITH_LEN("copen2 ")}; > + const sp_pcursor *cursor= m_ctx->get_pcursor(*this); > + /* copen2 name@offset */ > + print_cmd_and_var(cmd, cursor ? *cursor : empty_clex_str, *this, str); > } > > > diff --git a/sql/sp_rcontext.cc b/sql/sp_rcontext.cc > --- a/sql/sp_rcontext.cc > +++ b/sql/sp_rcontext.cc > @@ -34,17 +34,48 @@ > > Sp_rcontext_handler_local sp_rcontext_handler_local; > Sp_rcontext_handler_package_body sp_rcontext_handler_package_body; > +Sp_rcontext_handler_member sp_rcontext_handler_member; > > sp_rcontext *Sp_rcontext_handler_local::get_rcontext(sp_rcontext *ctx) const > { > return ctx; > } > > +const sp_variable * > +Sp_rcontext_handler_local::get_pvariable(const sp_pcontext *pctx, uint i) const > +{ > + return pctx->find_variable(i); > +} > + > +const sp_pcursor * > +Sp_rcontext_handler_local::get_pcursor(const sp_pcontext *pctx, uint i) const > +{ > + return pctx->find_cursor(i); > +} > + > + > sp_rcontext *Sp_rcontext_handler_package_body::get_rcontext(sp_rcontext *ctx) const > { > return ctx->m_sp->m_parent->m_rcontext; > } > > +const sp_variable * > +Sp_rcontext_handler_package_body::get_pvariable(const sp_pcontext *pctx, uint i) > + const > +{ > + DBUG_ASSERT(0); > + return nullptr; What do you mean? One cannot define a variable in a package body? > +} > + > +const sp_pcursor * > +Sp_rcontext_handler_package_body::get_pcursor(const sp_pcontext *pctx, uint i) > + const > +{ > + return pctx->top_context()->sp()->m_parent-> > + get_parse_context()->find_member_cursor(i); > +} > + > + > const LEX_CSTRING *Sp_rcontext_handler_local::get_name_prefix() const > { > return &empty_clex_str; > @@ -57,6 +88,48 @@ const LEX_CSTRING *Sp_rcontext_handler_package_body::get_name_prefix() const > return &sp_package_body_variable_prefix_clex_str; > } > > +sp_cursor *Sp_rcontext_handler_local::get_cursor(THD *thd, uint offset) const > +{ > + return thd->spcont->get_cursor(offset); this is starting to look fragile, because you're now swapping thd->spcont back and forth. One more argument, why it wasn't a good idea and better to let cursor know its context instead. > +} > + > +sp_cursor *Sp_rcontext_handler_package_body::get_cursor(THD *thd, > + uint offset) const > +{ > + return get_rcontext(thd->spcont)->get_member_cursor(offset); > +} > + > + > +sp_rcontext *Sp_rcontext_handler_member::get_rcontext(sp_rcontext *ctx) const > +{ > + return ctx; > +} > + > +const LEX_CSTRING *Sp_rcontext_handler_member::get_name_prefix() const > +{ > + static const LEX_CSTRING sp_package_body_variable_prefix_clex_str= > + {STRING_WITH_LEN("MEMBER.")}; > + return &sp_package_body_variable_prefix_clex_str; > +} > + > +sp_cursor *Sp_rcontext_handler_member::get_cursor(THD *thd, > + uint offset) const > +{ > + return thd->spcont->get_member_cursor(offset); > +} > + > +const sp_variable * > +Sp_rcontext_handler_member::get_pvariable(const sp_pcontext *pctx, uint i) const > +{ > + return pctx->find_variable(i); > +} > + > +const sp_pcursor * > +Sp_rcontext_handler_member::get_pcursor(const sp_pcontext *pctx, uint i) const > +{ > + return pctx->top_context()->find_member_cursor(i); > +} > + > > /////////////////////////////////////////////////////////////////////////// > // sp_rcontext implementation. > diff --git a/sql/structs.h b/sql/structs.h > --- a/sql/structs.h > +++ b/sql/structs.h > @@ -862,26 +862,47 @@ class Lex_for_loop_bounds_intrange: public Lex_for_loop_bounds_st > }; > > > +/* > + This structure is used as a Bison token type. > + It must stay as small as possible, hence it uses a "union" declaration: > + - m_target_bound is used only in "FOR idx IN 0..9" style loops > + - m_cursor_rcontext_handler is used only in "FOR rec IN cursor" style loops > + They are never needed at the same time. > + See "%union size check" in sql_yacc.yy for details on the token size limit. > +*/ > struct Lex_for_loop_st > { > public: > + enum class Type : uchar > + { > + INT_RANGE= 0, // FOR i IN 1..10 > + CURSOR_IMPLICIT= 1, // FOR r IN (SELECT 1 FROM DUAL) > + CURSOR_EXPLICIT= 2 // FOR r IN cursor1 > + }; > class sp_variable *m_index; // The first iteration value (or cursor) > - class sp_variable *m_target_bound; // The last iteration value > + union > + { > + class sp_variable *m_target_bound; // The last iteration value > + const class Sp_rcontext_handler *m_cursor_rcontext_handler; > + }; > int m_cursor_offset; > int8 m_direction; > - bool m_implicit_cursor; > + Type m_type; you can also use bit fields to reduce sizeof(Lex_for_loop_st) even further. Like m_type:2, m_direction:1, m_cursor_offset:29 > void init() > { > m_index= 0; > m_target_bound= 0; > m_cursor_offset= 0; > m_direction= 0; > - m_implicit_cursor= false; > + m_type= Type::INT_RANGE; > + } > + bool is_for_loop_cursor() const > + { > + return m_type == Type::CURSOR_IMPLICIT || m_type == Type::CURSOR_EXPLICIT; > } > - bool is_for_loop_cursor() const { return m_target_bound == NULL; } > bool is_for_loop_explicit_cursor() const > { > - return is_for_loop_cursor() && !m_implicit_cursor; > + return m_type == Type::CURSOR_EXPLICIT; > } > }; > > diff --git a/sql/sql_lex.cc b/sql/sql_lex.cc > --- a/sql/sql_lex.cc > +++ b/sql/sql_lex.cc > @@ -6537,16 +6537,20 @@ LEX::find_variable(const LEX_CSTRING *name, > const Sp_rcontext_handler **rh) const > { > sp_variable *spv; > - if (spcont && (spv= spcont->find_variable(name, false))) > + const sp_pcontext *frame_pctx; > + if (spcont && (spv= spcont->find_variable(name, &frame_pctx, false))) > { > *ctx= spcont; > - *rh= &sp_rcontext_handler_local; > + if (frame_pctx == sphead->frame_for_members()) > + *rh= &sp_rcontext_handler_member; > + else > + *rh= &sp_rcontext_handler_local; > return spv; > } > sp_package *pkg= sphead ? sphead->m_parent : NULL; > - if (pkg && (spv= pkg->find_package_variable(name))) > + if (pkg && (spv= pkg->find_member_variable(name))) > { > - *ctx= pkg->get_parse_context()->child_context(0); > + *ctx= pkg->frame_for_members(); > *rh= &sp_rcontext_handler_package_body; I don't quite understand. Before your change the first if() was handling local variables, and the second - package-wide, as far as I understand. afte your change the first if() apparently also can find package-wide variables. What does the second if() do now? > return spv; > } > @@ -7182,20 +7248,26 @@ bool LEX::sp_for_loop_cursor_declarations(THD *thd, > thd->parse_error(); > DBUG_RETURN(true); > } > - if (unlikely(!(pcursor= spcont->find_cursor_with_error(&name, &coffs, > - false)) || > + > + sp_rcontext_addr cursor_addr(nullptr, 0); > + > + if (unlikely(!(pcursor= find_cursor_with_error(&name, &cursor_addr)) || > pcursor->check_param_count_with_error(param_count))) > DBUG_RETURN(true); > > if (!(loop->m_index= sp_add_for_loop_cursor_variable(thd, index, > - pcursor, coffs, > + pcursor, > + cursor_addr, > bounds.m_index, > item_func_sp))) > DBUG_RETURN(true); > loop->m_target_bound= NULL; > loop->m_direction= bounds.m_direction; > - loop->m_cursor_offset= coffs; > - loop->m_implicit_cursor= bounds.m_implicit_cursor; > + loop->m_cursor_rcontext_handler= cursor_addr.rcontext_handler(); > + loop->m_cursor_offset= cursor_addr.offset(); why do you store m_cursor_rcontext_handler and m_cursor_offset separately, wouldn't it be more natural to store, like loop->m_cursor_addr= cursor_addr; ? > + loop->m_type= bounds.m_implicit_cursor ? > + Lex_for_loop_st::Type::CURSOR_IMPLICIT : > + Lex_for_loop_st::Type::CURSOR_EXPLICIT; > DBUG_RETURN(false); > } > > @@ -8128,21 +8221,21 @@ Item *LEX::make_item_colon_ident_ident(THD *thd, > Item *LEX::make_item_plsql_cursor_attr(THD *thd, const LEX_CSTRING *name, > plsql_cursor_attr_t attr) > { > - uint offset; > - if (unlikely(!spcont || !spcont->find_cursor(name, &offset, false))) > + Cursor_ref ref(name, sp_rcontext_addr(nullptr, 0)); > + if (unlikely(!spcont || !find_cursor(name, &ref))) wait, why find_cursor() gets the name as an argument if name is already set in Cursor_ref? > { > my_error(ER_SP_CURSOR_MISMATCH, MYF(0), name->str); > return NULL; > } > switch (attr) { > case PLSQL_CURSOR_ATTR_ISOPEN: > - return new (thd->mem_root) Item_func_cursor_isopen(thd, name, offset); > + return new (thd->mem_root) Item_func_cursor_isopen(thd, ref); > case PLSQL_CURSOR_ATTR_FOUND: > - return new (thd->mem_root) Item_func_cursor_found(thd, name, offset); > + return new (thd->mem_root) Item_func_cursor_found(thd, ref); > case PLSQL_CURSOR_ATTR_NOTFOUND: > - return new (thd->mem_root) Item_func_cursor_notfound(thd, name, offset); > + return new (thd->mem_root) Item_func_cursor_notfound(thd, ref); > case PLSQL_CURSOR_ATTR_ROWCOUNT: > - return new (thd->mem_root) Item_func_cursor_rowcount(thd, name, offset); > + return new (thd->mem_root) Item_func_cursor_rowcount(thd, ref); > } > DBUG_ASSERT(0); > return NULL; Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

1 0

RFC on https://jira.mariadb.org/browse/MDEV-36832
by arcadiy＠gmail.com 20 May '25

20 May '25

Hello, I believe there may be an bug wrt x gap lock taking by an UPDATE statement: https://jira.mariadb.org/browse/MDEV-36832 The bug contains the detailed case analysis. Could anyone please take a look and see if this is something that has a plausible fix? Thank you!

1 0

pthread_yield() vs std::this_thread::yield()
by Sergey Vojtovich 17 May '25

17 May '25

Monty and I had an argument recently regarding pthread_yield() and std::this_thread::yield() usage. I do want to make this discussion public, so that we can pick either approach, get it documented and make sure everybody is aware of reasoning and decision. Probably even add a bb check that wouldn't allow unwanted function calls to pass. Lets try to keep this discussion focused on std::this_thread::yield() and avoid discussing other C++11 features. Since I'm biased towards std::this_thread::yield() I'd let Monty elaborate his concerns and explain why he prefers pthread_yield(). But generally it comes to compatibility and portability. My position is: pthread_yield() should be removed in favour of std::this_thread::yield(). Benefits: 1. Must be available to all current MariaDB versions. C++11 is a must since 2018. 2. Compatibility and portability, which we don't have to care about. 3. I have no performance concerns after analysing how it is implemented by clang and gcc. 4. Goes inline with "MDEV-35461 - Remove redundant checks for standard library functions" 5. Standard things looks more community-friendly, e.g. why should MSVC developers care about POSIX threads? Currently pthread_yield() can be defined to: 1. SwitchToThread() - Win32 (Wlad, 2009) MSVC seems to support std::this_thread::yield() since 2015: https://learn.microsoft.com/en-us/cpp/standard-library/thread-functions?vie… 2. thr_yield() - Solaris (Serg, 2009) Oracle Developer Studio 12.6 supports std::this_thread::yield() since 2017: https://docs.oracle.com/cd/E77782_01/html/E77789/bkabe.html#OSSCPgnyio 3. pthread_yield_np() - OS X (Serg, 2009) XCode is based on clang, which supports std::this_thread::yield() since 2013. sched_yield() - default (Serg, 2009) pthread_yield() - deprecated since 2021 (Serg, 2009) Possible concerns: 1. Compatibility with recent platforms. C++11 is widely available, thread models seem to be well supported. InnoDB uses std::this_thread_yield() since 2021. 2. Compatibility with older (ancient?) platforms. We're way off into C++11 since 2018. Making recent MariaDB compile on such platforms is probably going to be a heavy effort. If we do really want to have these platforms supported, they must be present in buildbot. 3. Performance impact of std::this_thread::yield(). See below. In many cases it is just a wrapper around functions like sched_yield. clang: libcxx/include/__thread/this_thread.h:inline _LIBCPP_HIDE_FROM_ABI void yield() _NOEXCEPT { __libcpp_thread_yield(); } libcxx/src/support/win32/thread_win32.cpp:void __libcpp_thread_yield() { SwitchToThread(); } libcxx/include/__thread/support/pthread.h:inline _LIBCPP_HIDE_FROM_ABI void __libcpp_thread_yield() { sched_yield(); } gcc additionally supports VxWorks: yield() noexcept { #if defined _GLIBCXX_HAS_GTHREADS && defined _GLIBCXX_USE_SCHED_YIELD __gthread_yield(); #endif } int __gthread_yield (void) { return taskDelay (0); } __GTHREAD_INLINE int __gthread_yield (void) { return __gthrw_(sched_yield) (); } Regards, Sergey

5 7

Re: MDEV-34705 benchmarking
by Kristian Nielsen 16 May '25

16 May '25

Steve Shaw <steve.shaw(a)hammerdb.com> writes: > One follow-up question, with the new build if I set log_bin=1 (or > skip-log-bin=0) I get the following: > > 2025-05-16 15:17:43 0 [ERROR] Binlog name can not be set with > --log-bin when --binlog-storage-engine is used. Use --binlog-directory > to specify a separate directory for binlogs Sorry, as I wrote originally, you have to use in the config file: log_bin (without any argument). I also missed the detail that you put the "=1" in there, since it looks so natural. (The reason is historical. In the existing server, --log-bin takes the basename to use for the binlog files, it's not an on/off switch. So log_bin=1 means to call the binlog files "1.000001", "1.000002", ... In the new binlog I removed the ability to specify the basename, hence the error rather than silently ignoring the user's specified name). > How do I check its working as intended rather than not having enabled > the binlog. When the new binlog is enabled, the data directory (or --binlog-directory if specified) will contain binlog files named like this: binlog-000000.ibb binlog-000001.ibb ... - Kristian.

1 0

Re: MDEV-34705 benchmarking
by Kristian Nielsen 16 May '25

16 May '25

Steve Shaw <steve.shaw(a)mariadb.com> writes: > pipeline as other testing is currently underway so will probably be early > next week. Ok, sounds great! > To confirm we will test the mariadb-11.4.6 branch as a baseline and then I would suggest to use the commit e176066a9e02cf2bdedf9e79fb150029dde430f6 (which is the commit that knielsen_binlog_in_engine was last rebased onto) as the base, just to avoid the possibility that the 136 extra commits in 11.4.6 could influence anything. > the knielsen_binlog_in_engine branch for comparison. > we will set these parameters for mariadb-11.4.6 > > log_bin=1 > innodb_flush_log_at_trx_commit=1 > sync_binlog=1 > > and then these for knielsen_binlog_in_engine > > log_bin=1 > innodb_flush_log_at_trx_commit=1 > sync_binlog=1 > binlog_storage_engine=innodb Yes, this is probably the most interesting starting comparison. > Where you would like the results posting to? To the MDEV or reply to this > email thread? Well, I think the developers@ mailing list is a good place for discussions like this, but either works for me. And as I said, be sure to let me know in case of any issues, I'll try to help. - Kristian.

1 0

MDEV-34705 benchmarking
by Kristian Nielsen 16 May '25

16 May '25

Hi Steve, Brandon Nesterenko told me you would be interested in benchmarking the MDEV-34705 binlog-in-engine feature. The issue https://jira.mariadb.org/browse/MDEV-34705 has a design writeup. The feature is a new implementation of the binlog that is integrated with the InnoDB storage engine, and can therefore be more efficient. The code is in the branch: knielsen_binlog_in_engine Performance-wise, this feature has two major goals: 1. Reduce the number of fsync() required in durable configuration (--innodb-flush-log-at-trx-commit=1 --sync-binlog=1) from two to one per (group) commit, giving a large speedup. 2. Make the non-durable configuration (--innodb-flush-log-at-trx-commit=0|2) crash-safe for the binlog also, by avoiding the need for two-phase commit between InnoDB and binlog. Thus the primary goal of benchmarking is to verify the speedup in (1), and to verify that crash-safety in (2) is obtained without performance regression. Also interesting is to see how much faster new (2) is than old (1), since this comparison applies when crash-safety is required but durability is not. The speedup is expected to be higher at lower concurrency (as fsync cost is amortised over fewer commits); it is also interesting to test for scalabitily regressions at higher concurrency. Read-only workloads should be mostly unaffected. To enable the new binlog in the branch, use the following configuration: binlog_storage_engine=innodb log_bin # Optional binlog location, defaults to datadir. binlog_directory=/data/m/my/binlogs/ I'd be very interested in any results of testing the new feature, and happy to hear any comments/suggestions/questions. If you start testing the code, I will advise that this is work-in-progress; so a good idea to communicate any findings early. There are still pieces missing (and surely bugs), and I'm happy to quickly fix anything that blocks testing. In my own preliminary benchmarks, I have successfully run sysbench (using Mark Callaghan's scripts), where I see typically 50%-150% speedup in (1), and 10%-20% in (2). Results depend heavily on transaction load, concurrency, and benchmark machine disk performance. - Kristian.

1 0

Windows line endings in MariaDB test result files?
by Otto Kekäläinen 14 May '25

14 May '25

Hi, While importing 11.4.6 into Debian I ran into issues with git and quilt failing to convert text files properly between Unix and Windows line endings. Thus asking are Windows CRLF necessary in the new mysql-test/main/mysql-interactive.result test? commit e5147c8140450f28505a227159d2feba37dd1c39 Author: Sergei Golubchik <serg(a)mariadb.org> Date: Tue Jan 30 16:39:28 2024 +0100 regression introduced by MDEV-14448 diff --git a/mysql-test/main/mysql-interactive.result b/mysql-test/main/mysql-interactive.result new file mode 100644 index 00000000000..a18c018b932 --- /dev/null +++ b/mysql-test/main/mysql-interactive.result @@ -0,0 +1,24 @@ +# +# regression introduced by MDEV-14448 +# +delimiter $^M +select 1;^M +$^M +Welcome to the MariaDB monitor. Commands end with ; or \g.^M +Your MariaDB connection id is X^M +Server version: Y^M +Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.^M +^M +Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.^M +^M +MariaDB [(none)]> delimiter $^M +MariaDB [(none)]> select 1;^M + -> $^M ++---+^M +| 1 |^M ++---+^M +| 1 |^M ++---+^M +1 row in set^M +^M +MariaDB [(none)]> All the other files with Windows line endings seem to be have them intentionally: ± find . -type f -exec file {} \; | grep -i "CRLF" ./storage/connect/mysql-test/connect/std_data/emp.txt: ASCII text, with CRLF line terminators ./storage/connect/mysql-test/connect/std_data/boyswin.txt: ASCII text, with CRLF line terminators ./storage/connect/mysql-test/connect/std_data/expenses.txt: ASCII text, with CRLF line terminators ./storage/connect/mysql-test/connect/r/xml.result: HTML document, UTF-8 Unicode text, with CRLF, LF line terminators ./mysql-test/std_data/loadxml2.dat: XML 1.0 document, ASCII text, with very long lines, with CRLF, LF line terminators ./mysql-test/main/loadxml.result: ASCII text, with CRLF, LF line terminators ./mysql-test/main/func_regexp_pcre.result: UTF-8 Unicode text, with very long lines, with CRLF, CR, LF line terminators ./mysql-test/main/mysql-interactive.result: ASCII text, with CRLF, LF line terminators ./mysql-test/main/mysql_binary_mode.result: ASCII text, with CRLF, LF line terminators ./extra/wolfssl/wolfssl/certs/test/smime-test-canon.p7s: MIME entity, ASCII text, with CRLF, CR line terminators ./extra/wolfssl/wolfssl/wolfssl.rc: C source, Little-endian UTF-16 Unicode text, with CRLF line terminators

2 4

MariaDB near 50% in WordPress usage stats
by Otto Kekäläinen 13 May '25

13 May '25

Hi all, Glad to see that MariaDB has now surpassed MySQL in WordPress usage stats at https://wordpress.org/about/stats/ Version Usage MySQL 8.0 20.77% MariaDB 10.6 19.88% MariaDB 10.11 18.60% MySQL 5.7 17.78% MySQL 5.5 7.03% MariaDB 10.5 6.77% MariaDB 10.3 2.74% MySQL 5.6 1.99% MariaDB 11.4 1.27% MariaDB 10.4 1.07% MariaDB 5.5 0.46% MariaDB 10.1 0.43% MariaDB 10.2 0.32% MySQL 5.1 0.17% MySQL 8.4 0.14% I haven't verified this myself, but I heard that MySQL 8.4 deprecated SQL_CALC_FOUND_ROWS which is heavily used in the WordPress code base, and thus the MySQL 8.4 adoption is nearly zero. Good reminder to have collaboration and joint testing with some of the largest open source projects to avoid accidentally having a new MariaDB version being incompatible with them..

2 2

Commit ff4209fa0d1c66952bc9f00587f525314464e7be not merged on main?
by Otto Kekäläinen 08 May '25

08 May '25

Hi! I was updating the patch https://salsa.debian.org/mariadb-team/mariadb-server/-/blob/debian/latest/d… in Debian when importing MariaDB 11.4.6, and I noticed that the upstream commit that changes the same lines in https://github.com/MariaDB/server/commit/ff4209fa0d1c66952bc9f00587f5253144… only landed on MariaDB '10.11' and '11.4' branch, but never on 'main' or '11.8'. Could it be that 'main' has had zero merges from older branches since April 3? Or was there maybe some mistake and one merge was overlooked and it wasn't properly applied on all newer maintained branches?

2 2

CI failures visible on GitHub (affecting contributors)
by Otto Kekäläinen 29 Apr '25

29 Apr '25

Hi, I see that on the 11.8 branch (https://github.com/MariaDB/server/commits/11.8/) which is supposed to be fully green and passing CI as it is soon to be released as a new LTS version, actually isn't consistently green. Out of the latest 10 CI runs the jobs passing rates are: 12/!2 11/12 12/!2 12/!2 8/8 10/12 8/12 7/8 8/8 11/12 Questions: - What is the current plan to get CI to be consistently green? - How is it possible that a commit is pushed on the 11.8 branch even though CI is not passing? Isn't branch protection preventing it? - If there is a way to get commits on 11.8 branch with failures bypassing branch protection, are all developers committed to follow-up on their commits and fix them before they do any other work? Or what is the plan for how to ensure post-push test failures are fixed? - The jobs buildbot/amd64-debian-11-msan-clang-16 and continuous-integration/appveyor/branch seem to be the most common failing ones. Can they be removed now to make the CI consistently green? And apply full branch protection so that they, or any other jobs, can't be added back to unless they are actually green? - Why is the CI occasionally tunning 8 jobs instead of 12? I also see plenty of red at https://buildbot.mariadb.org/#/grid?branch=11.8. Hopefully we get as much green as possible for the start of the 11.8 maintenance so that any potential regressions in future stable maintenance releases of 11.8 series are easier to catch. Most of the issues seem to be test issues and not real MariaDB bugs. I have also done a bunch of testing on Debian QA systems for MariaDB 11.8 and I have only found a couple small issues, so overall 11.8 is looking good! - Otto

3 3

MariaDB 11.8 and Debian status - information to fellow Linux distro packagers/maintainers
by Otto Kekäläinen 27 Apr '25

27 Apr '25

Hi! Do we have any Linux distro packagers and maintainers on this mailing list? If you are working on importing MariaDB 11.8 into a Linux distro, you may want to take a peek at what was done in Debian, as it may save you some work in debugging build or test issues: * https://salsa.debian.org/mariadb-team/mariadb-server/-/commits/debian/latest * https://salsa.debian.org/mariadb-team/galera-4/-/commits/debian/latest You might also benefit from looking at work-in-progress and open Merge Requests at : * https://salsa.debian.org/mariadb-team/mariadb-server/-/merge_requests In particular the MR https://salsa.debian.org/mariadb-team/mariadb-server/-/merge_requests/115 about a NEWS item to Debian users is likely to have useful text/summary that others might want to re-use: * https://salsa.debian.org/otto/mariadb-server/-/blob/dev/NEWS-11.8.2/debian/… Overall Debian package statuses are best viewed at: * https://tracker.debian.org/pkg/mariadb * https://tracker.debian.org/pkg/galera-4 ..and respective in Ubuntu: * https://launchpad.net/ubuntu/+source/mariadb * https://launchpad.net/ubuntu/+source/galera-4 I've fixed a lot of bugs in past weeks. The Debian tracker MariaDB bugs are down from 60 to 44, and in Ubuntu down to 3. For Galera the numbers are now down to 1 and 0. Looking forward to collaborate with maintainers of MariaDB and Galera packages in other Linux distributions! - Otto

1 0

Remaining InnoDB parts for MDEV-34705 binlog-in-engine?
by Kristian Nielsen 25 Apr '25

25 Apr '25

Hi Marko, I have been making some good progress on the new binlog implementation, I implemented all of what we discussed regarding the InnoDB side I believe, and also got a lot done on the integration with the server side code. I wanted to ask you what outstanding issues on the InnoDB side remain from your point of view, if any? The page format now is implemented in the new design, as described in a separate mail: https://lists.mariadb.org/hyperkitty/list/developers@lists.mariadb.org/thre… All the InnoDB-specific page overhead is gone, only a 4-byte CRC32 remains at the end of each page. The first page is reserved as a header page. The new binlog code no longer uses the InnoDB buffer pool, nor create any InnoDB tablespace structures (except for the two reserved tablespace IDs for the redo records). The code has its own buffer pool replacement (called the "page fifo") and own background flushing. The binlog hooks into the InnoDB crash recovery code following your suggestions, and just the single "memcpy" record is needed. Recovery is implemented (and tested somewhat), and has been carefully designed to recover in all cases (barring any bugs of course) while requiring only the two reserved tablespace IDs. I have two outstanding InnoDB-related issues I know about: 1. In order for a replication setup to be crash-safe, I must ensure that the slave does not receive an event until it has become durable on the master. Otherwise, when the master crashes and comes back up, an event might be applied on the slave that no (longer) exist on the master. For this, I need something in the InnoDB redo log code that will callback into the binlog after a specific LSN has been durably written to disk. This callback will then notify the dump threads (slave connections) somehow that the corresponding part of the binlog can now be safely read. 2. Integration with mariabackup. With the new binlog, mariabackup can backup the binlog together with the rest of the InnoDB data in a consistent way, something that I think is not possible currently? This will also remove the need for mariabackup to extract the binlog position with SHOW MASTER STATUS or for InnoDB to store the binlog position in the undo segments. When the binlog is backed up, the correct GTID position can be obtained after restoring a master backup simply by checking the binlog (SELECT @@gtid_binlog_pos). (For a slave backup, the position is similarly available from the backup of the mysql.gtid_slave_pos table, as before). I am not yet sure of the details of mariabackup support. On the high level, it will involve copying the binlog-NNNNNN.ibb into the backup along with the other InnoDB tablespace files; and applying redo records to those binlog files as part of --prepare. One important detail is to handle that we have only two reserved tablespace IDs for binlog files. The current recovery code requires that only the two most recent binlog files need recovery. We could try to ensure that the same applies duing mariabackup --prepare. An idea is to use the fact that binlog files are strictly append-only. Around the point where mariabackup decides the LSN where it stops copying the redo log that becomes the backup snapshot, we can determine the last binlog file that exists. We will check this just before and just after determining the LSN, and repeat if a new binlog file was created just at that point (which would otherwise leave doubt if the file was created before or after that LSN). Then after the snapshot LSN is determined, we can just copy all the binlog files up to and including (but not after) the snapshot point. This way mariabackup --prepare should be able to run the same binlog recovery code that runs in crash recovery, only needing to consider the two most recent of the copied binlog files. I can imagine other ways to do mariabackup of binlog files, but this is one idea that could be feasible. What do you think? Anything other outstanding issues with the new binlog implementation regarding InnoDB that you are aware of? - Kristian.

1 1

Re: d20e4e10353: MDEV-36536 Add option to not collect statistics for long char/varchars
by Sergei Golubchik 23 Apr '25

23 Apr '25

Hi, Michael, On Apr 22, Michael Widenius wrote: > revision-id: d20e4e10353 (mariadb-10.6.21-79-gd20e4e10353) > parent(s): 4f0a0dbd104 > author: Michael Widenius > committer: Michael Widenius > timestamp: 2025-04-22 10:53:08 +0300 > message: > > MDEV-36536 Add option to not collect statistics for long char/varchars > > This is needed to make it easy for users to automatically ignore long > char and varchars when using ANALYZE TABLE PERSISTENT. These fields > can cause problems as they will consume 'CHARACTERS * > MAX_CHARACTER_LENGTH * 2 * number_of_rows' space on disk during > analyze, which can easily be much bigger than the analyzed table. > > This commit adds a new user variable, analyze_max_length, default > value 4G. Any field that is bigger than this in bytes, will be ignored > by ANALYZE TABLE PERSISTENT unless it is specified in FOR COLUMNS(). > > While doing this patch, I noticed that we do not skip GEOMETRY columns > from ANALYZE TABLE, like we do with BLOB. This should be fixed when > merging to the 'main' branch. At the same time we should add a > resonable default value for analyze_max_length, probably 1024, like we > have for max_sort_length. > > diff --git a/mysql-test/main/mysqld--help.test b/mysql-test/main/mysqld--help.test > index 971983fd66c..44449cd3ad5 100644 > --- a/mysql-test/main/mysqld--help.test > +++ b/mysql-test/main/mysqld--help.test > @@ -20,14 +20,14 @@ exec $MYSQLD_BOOTSTRAP_CMD --symbolic-links=0 --log-bin=foo --lower-case-table-n > > perl; > # Variables which we don't want to display in the result file since > - # their paths may vary: > + # their paths may vary or they may use a default of 4294967295 : why? what's wrong with 4294967295? > @skipvars=qw/basedir open-files-limit general-log-file log plugin-dir plugin-maturity > log-slow-queries pid-file slow-query-log-file log-basename > datadir slave-load-tmpdir tmpdir socket thread-pool-size > large-files-support lower-case-file-system system-time-zone > collation-server character-set-server log-tc-size table-cache > table-open-cache table-open-cache-instances max-connections > - server-uid tls-version version.*/; > + server-uid tls-version version.* analyze-max-length/; > > # Plugins which may or may not be there: > @plugins=qw/innodb archive blackhole federated partition s3 > diff --git a/sql/sql_class.h b/sql/sql_class.h > index 69b021cd41d..aac433181a4 100644 > --- a/sql/sql_class.h > +++ b/sql/sql_class.h > @@ -731,6 +731,7 @@ typedef struct system_variables > ha_rows select_limit; > ha_rows max_join_size; > ha_rows expensive_subquery_limit; > + ulong analyze_max_length; don't use ulong, please. uint or ulonglong. > ulong auto_increment_increment, auto_increment_offset; > #ifdef WITH_WSREP > /* > diff --git a/sql/sys_vars.cc b/sql/sys_vars.cc > index 0d4a8e4c6fe..4ae14a58feb 100644 > --- a/sql/sys_vars.cc > +++ b/sql/sys_vars.cc > @@ -445,6 +445,20 @@ static Sys_var_double Sys_analyze_sample_percentage( > CMD_LINE(REQUIRED_ARG), VALID_RANGE(0, 100), > DEFAULT(100)); > > +/* > + The max length have to be UINT_MAX32 to not remove GEOMETRY fields > + from analyze. > +*/ > + > +static Sys_var_ulong Sys_analyze_max_length( > + "analyze_max_length", > + "Fields that require more storage than analyze_max_length and are not " better "Fields which length in bytes more than this are skipped by ANALYZE TABLE PERSISTENT unless explicitly listed in the FOR COLUMNS () clause" "require more storage" is not very clear > + "listed in ANALYZE ... FOR COLUMNS () will automatically be skipped by " > + "ANALYZE TABLE PERSISTENT", > + SESSION_VAR(analyze_max_length), > + CMD_LINE(REQUIRED_ARG), VALID_RANGE(32, UINT_MAX32), > + DEFAULT(UINT_MAX32), BLOCK_SIZE(1)); > + > static Sys_var_ulong Sys_auto_increment_increment( > "auto_increment_increment", > "Auto-increment columns are incremented by this", > Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 1

Design of new binlog file format (MDEV-34705)
by Kristian Nielsen 21 Apr '25

21 Apr '25

As the MDEV-34705 binlog-in-engine project matures, I'm finalising the file format of the new binlog, and will present it here in case of any final comments or request for changes. The file format of course is an important part of the design, as file formats in general are more difficult to change going forward than many other aspects of the code. Following previous discussions on the list with Marko, the file format of the new binlog is now simplified, and mostly independent of InnoDB. The InnoDB buffer pool is no longer used for binlog files, and InnoDB recovery uses separate code to recover binlog pages than other tablespace pages. The binlog implementation reserves two special tablespace IDs in the redo log to denote binlog files, but otherwise binlog files share little code with other InnoDB tablespaces. 1. Binlog file names -------------------- The innodb binlog files have a fixed naming scheme: binlog-000000.ibb binlog-000001.ibb binlog-000002.ibb ... By default they are written to the data directory, but a separate directory can be configured with --binlog-directory. This, there is no longer any need for a binlog index file. New binlog files are pre-allocated to the configured --max-binlog-size in the background. Unused pages contain zeros until written to. Each binlog file is always written strictly append-only, bytes are never overwritten or written out-of-order. 2. Binlog file header --------------------- Each binlog file is page-based. The page size is 4k, fixed in the initial release, but planned to be configurable in a future release. Pages are checksummed (same algorithm as InnoDB full_crc32). The last four bytes of each page contain the CRC32, rest is available for data (no special page header is needed). The first page of the file is reserved as a file header. It contains the following data (little-endian format): Offset Length Data 0 4 "Magic" file identifier (0x010dfefe) 4 4 Log2 of page size (ie. 12 for 4k pages) 8 4 Major version number (0) 12 4 Minor version number (0) 16 8 The file number from the binlog name, consistency check 24 8 File length, in number of pages 32 8 LSN corresponding to the start of this binlog file 40 8 The "GTID state interval" N. Used to speed up GTID lookup. 48 8 File number containing out-of-band data referenced from this file. 56 8 File number containing XA data referenced from this file (for future expansion to support XA). 508 4 Extra crc32 checksum, to support down to 512 bytes page size. The idea with version numbers is that new minor versions are readable by older code, while new major versions are not and older code must refuse to read the file. The LSN in the header is used to do recovery correctly. Any redo record with smaller LSN will be ignored during recovery and not applied to this file. This way only two tablespace IDs are needed for recovery of binlog files. The GTID state interval (--innodb-binlog-state-interval) is a replacement for the GTID indexes of the legacy binlog. Every N pages in a binlog file, the current binlog GTID state is written. This allows to quickly find a GTID position for the slave to start at, by doing binary search on the GTID state records and then scanning ahead over the last few (<=N) pages to find the starting position. The first GTID state in the file stores the full state, the remaining pages only store the difference to the first state. The out-of-band (OOB) file number reference is used to manage when large transactions are written into the binlog files interleaved with other transaction data. If a large transaction starts binlogging in binlog-000005.ibb say, and then completes and writes the commit record to file binlog-000008.ibb, then the file header for file_no=8 will contain oob-reference=5, to mark that file_no=5 is needed by dump threads(slave connections) reading from file_no=8. An extra crc32 checksum is stored at offset 512 in the header page. This is to be later able to support 512-byte (or larger) configurable page size. This makes it possible to read the first 512 bytes of the file, verify the integrity, and read the actual page size at offset 4. The initial release is planned to have fixed 4k page size. 3. Binlog data page format -------------------------- Overall, the binlog contains a logical sequence of "records". Conceptually, the binlog is one single stream of such records; one record can span two files, and file boundary is not meaningful for the binlog content. Each binlog record is written as a single InnoDB mini-transaction (mtr). This means binlog records will be recovered atomically during InnoDB crash recovery, a partial record will not be found at the end of a binlog file. It also means the size of one record is limited by the maximum size of an mtr. Each record is split into one or more "chunks". A chunk always fits in one page. Each binlog page (apart from the first header page) contains one or more chunks. Then follows 0-3 filler bytes 0xff to pad any remaining space, and last the 4 bytes of CRC32 checksum. Here is the byte layout of a page: Offset Length Data 0 len+3 <type> <len1> <len2> <data 0> <data 1> ... <data len-1> len+3 A+3+B+3 <type> <len1> <len2> <data 0> <data 1> ... <data len-1> ... 4089 1 <filler (optional)> 4090 1 <filler (optional)> 4091 1 <filler (optional)> 4092 4 <CRC0> <CRC1> <CRC2> <CRC3> Thus, each chunk has a simple format with one byte denoting the chunk type, two bytes (little-endian) denoting the length, and then the data. Chunk types: 0 Empty (not yet used data, has no length field) 1 Commit record 2 GTID state record, ie. for --innodb-binlog-state-interval 3 Out-of-band binlog data for large transactions 4 Dummy record (used to fill up the last page when FLUSH BINARY LOGS) 0xff Filler byte, to pad page when there is no room for a whole chunk (no length field). Here for example is a commit record, type=1: 00001000: 41c8 0000 918e 0168 a201 0000 A......h.... 00001010: 0026 0000 0000 0000 0008 0001 0000 0000 .&.............. 00001020: 0000 0000 0000 0029 0000 0000 0000 918e .......)........ 00001030: 0168 0201 0000 00a1 0000 0000 0000 0000 .h.............. 00001040: 0007 0000 0000 0000 0004 0000 2300 0000 ............#... 00001050: 0000 0101 0000 2054 0000 0000 0603 7374 ...... T......st 00001060: 6404 0800 0800 0800 818c 0000 0000 0000 d............... 00001070: 0074 6573 7400 4352 4541 5445 2054 4142 .test.CREATE TAB 00001080: 4c45 2074 3120 2861 2049 4e54 204e 4f54 LE t1 (a INT NOT 00001090: 204e 554c 4c2c 2062 2049 4e54 204e 4f54 NULL, b INT NOT 000010a0: 204e 554c 4c2c 2063 2054 4558 542c 2050 NULL, c TEXT, P 000010b0: 5249 4d41 5259 204b 4559 2861 2c20 6229 RIMARY KEY(a, b) 000010c0: 2920 454e 4749 4e45 3d49 6e6e 6f44 42 ) ENGINE=InnoDB The 0x41 is the type, and the 0xc8 0x00 is the length=0x00c8. Then follows the raw event data, in the same format used in the existing binlog (sql/log_event.h). The type byte has two additional flag bits (hence 0x41 and not 0x01): Bit 6 (0x40) This is the last chunk of a record Bit 7 (0x80) This is a continuation chunk (not first) of a record Thus a commit record can consist of a single chunk: type=0x41 , or two chunks: type=0x01 type=0xc1 , or more than two chunks: type=0x01 type=0x81 type=0x81 ... type=0xc1 Chunks are thus used to split a record across pages (a page will not contain more than one chunk for a single record). 4. Chunk data contents ---------------------- The commit record, type=1, has the following header data, consisting of 5 64-bit numbers: num_oob Number of out-of-band records in this transaction oob_first_file_no The file_no of the first oob record oob_first_offset The offset into oob_first_file_no oob_last_file_no The file_no of the last oob record oob_last_offset The offset into oob_last_file_no For small event groups with no out-of-band data, only num_oob=0 is stored. The numbers are stored in a compressed format that saves space for small numbers, see storage/innobase/include/ut0compr_int.h After follows the raw binlog event data. For a small transaction with no out-of-band records, this is the whole event group (GTID, Query/rows, XID/Commit). For large transactions it is just the GTID event. See above for example commit record. The GTID state record, type=2, is just a list of GTIDs, similar to the existing Gtid_list_log_event: number_of_gtids domain_id_0, server_id_0, seq_no_0 domain_id_1, server_id_1, seq_no_1 ... domain_id_N-1, server_id_N-1, seq_no_N-1 Like for commit records, the numbers are stored compressed to save space for small numbers. Here is an example GTID state record containing one GTID 0-1-575 (see ut0compr_int.h for how to interpret the compressed numbers 0x8 0x0 0x8 0x11f9 as 1, 0, 1, 575): 00001000: 4205 0008 0008 f911 B....... 00001010: The OOB data record, type=3, stores one node in a binary tree of event group data. The out-of-band data is structured as binary trees in a way that can be written strictly append-only, and read efficiently by dump threads/connected slaves. The OOB node is defined by 5 numbers: index The identification of the node (nodes are numbered 0, 1, 2...) left_file_no The file_no of the left child of the node left_offset The offset into left_file_no of the left child. Zero denotes a leaf node. right_file_no The file_no of the right child of the node right_offset The offset into the right_file_no. After this follows the raw event data. 5. Conclusion ------------- So this is how the new binlog format is planned to look, barring any last-minute design change requirements or design review comments. I tried to include a good amount of details, feel free to ask for any of the specific details that I did omit, if interested. Hope this helps, - Kristian.

1 0

Re: [JIRA] (MDEV-36629) Change SLAVE to REPLICA in Server error messages
by Kristian Nielsen 18 Apr '25

18 Apr '25

Taking this discussion out of the MDEV-36629 specifics, as this is a general issue. It concerns a problem with how documentation is now being incorrectly updated: "Alex Hanshaw (Jira)" <jira(a)mariadb.org> writes: > Alex Hanshaw commented on MDEV-36629: > Understood. For context Vikas had told the Doc team "I understand the > compatibility issue - so we need to make both options work and > document only the new terminology." When the Doc team asked about the Ouch. This will need to be reverted ASAP, before more damage is done. The documentation is for documenting how the server works. All aspects of how it works. Not some distorted version, whether good- or ill-intended. We must certainly _not_ deliberately remove documentation of existing syntax and functionality! For example I see that documentation of the syntax START SLAVE was removed. This is just plain wrong. START SLAVE is working syntax in replication, and will remain so for the foreseeable future. It needs to be documented for users to be able to understand how it works. Please take this seriously, and ensure that such changes are stopped immediately, and all incorrect changes reverted. - Kristian. >> Change SLAVE to REPLICA in Server error messages >> ------------------------------------------------ >> >> Key: MDEV-36629 >> URL: https://jira.mariadb.org/browse/MDEV-36629 >> Project: MariaDB Server >> Issue Type: Task >> Components: Replication >> Reporter: Shwetha CM >> Assignee: Sergei Golubchik >> Priority: Major >> Fix For: N/A

2 1

Re: 6d4479d0e32: MDEV-36565 Assertion `src != ((void *)0)' failed in my_casedn_8bit
by Sergei Golubchik 16 Apr '25

16 Apr '25

Hi, Alexander, I don't know, this looks like a strange place to fix this. I'd say that either Item_func_xml_extractvalue shouldn't return {0, 0} or, if we'll treat it as a valid String, caseup() and casedn() should be able to handle it. But in a similarly-looking bug MDEV-35945 Assertion `src != ((void *)0)' failed in my_caseup_8bit you fixed it to use { "", 0 } for an empty string. It's commit c69fb1a6273. Then I'd suggest to do the same here. Let's avoid {0, 0} ? On Apr 16, Alexander Barkov wrote: > revision-id: 6d4479d0e32 (mariadb-11.4.5-25-g6d4479d0e32) > parent(s): d3c9a2ee212 > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2025-04-15 19:30:44 +0400 > message: > > MDEV-36565 Assertion `src != ((void *)0)' failed in my_casedn_8bit > > diff --git a/sql/item_strfunc.cc b/sql/item_strfunc.cc > index 2e3c33059d3..5df25c7f237 100644 > --- a/sql/item_strfunc.cc > +++ b/sql/item_strfunc.cc > @@ -1998,7 +1998,16 @@ String *Item_str_conv::val_str(String *str) > goto err; > > null_value= false; > - len= converter(collation.collation, (char*) res->ptr(), res->length(), > + /* > + Avoid passing nullptr as the second parameter to converter(): > + - val_str() of some Items (e.g. Item_func_xml_extractvalue) can return > + Strings with {Ptr=0,str_length=0} meaning an empty string. > + - But my_charset_handler_st virtual function caseup() and casedn() > + (which "converter" point to) do not expect {src=nullptr,srclen=0} input. > + */ > + DBUG_ASSERT(res->ptr() != nullptr || res->length() == 0); > + len= !res->ptr() ? 0/*Avoid passing nullptr*/ : > + converter(collation.collation, (char*) res->ptr(), res->length(), > (char*) str->ptr(), alloced_length); > DBUG_ASSERT(len <= alloced_length); > str->set_charset(collation.collation); > Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

1 0

Re: MDEV-36290: ALTER TABLE with multi-master can cause data loss (PR #3967)
by Kristian Nielsen 16 Apr '25

16 Apr '25

Brandon Nesterenko <notifications(a)github.com> writes: > MDEV-36290: ALTER TABLE with multi-master can cause data loss > https://github.com/MariaDB/server/pull/3967 Hi Brandon, Here is my initial review of the MDEV-36290 draft patch. Overall, it is clear that this is not something that's appropriate for 10.6. It implements a new way to apply row events, based on column names (if available) rather than column number. It is something for next dev version, 12.x. > commit de27bfbafc22601d53ddc0567401fcb26d28a91c > Author: Brandon Nesterenko <brandon.nesterenko(a)mariadb.com> > Date: Thu Apr 3 12:37:09 2025 -0600 > > MDEV-36290: ALTER TABLE with multi-master can cause data loss Please do not use the term "data loss" in the description (I know it is not you who introduced the term ;-). The issue here is conflicting queries run in parallel against multiple masters in a multi-master setup. In general, the support of multi-master in asynchronous replication requires the user to ensure that no conflicting queries are run, or to ensure that such conflicts will not cause undesirable behaviour. The behaviour addressed here is just one of many ways that conflicting ALTER TABLE could cause different data on slave and master. Using the term "data loss" can cause misunderstanding about what the problem is and what the change does. Of course, every case that is handled more correctly is in principle better. In this case, we introduce the facility to configure (--binlog_row_metadata=FULL) replication so that it is possible to replicate between tables on master and slave with different order of the columns. Not sure how useful such an option is in practice, seems to be very few users that will really need this, and such need has to be weighed against the increase in complexity in the code and potential for bugs. With that said, the approach in the patch generally looks reasonable. A few initial detail comments below, as you mentioned this is still a draft patch. As you also mentioned to me earlier, the logic in the patch is not trivial. I did not yet try to understand in detail all parts of the logic and the different corner cases. > + unsigned char *get_optional_metadata_str() > + { > + return m_optional_metadata; > + } > + > + unsigned int get_optional_metadata_len() > + { > + return m_optional_metadata_len; > + } > + Generally I do not like such do-nothing accessor functions. They just add abstraction and reduce grepability without doing anything useful. That's a style preference of course. > + /* > + Maps column index from master to slave. This is determined using the field > + names (provied by optional metadata when the master is configured with > + binlog_row_metadata=FULL). > + */ > + std::map<uint, uint> master_to_slave_index_map; > + > + /* > + When using field names to map from master->slave columns, this keeps track > + of column indices on the master which aren't present on the slave. > + > + It is used to skip columns when checking type-conversions and unpacking > + row data. > + */ > + std::set<uint> master_unmatched_cols; Is using std::map/std::set going to be expensive here, this is operations that will happen on every row-based event (when --binlog-row-metadata=FULL)? The std:: functions can imply a lot of memory allocations that are not very visible, and we usually try to avoid them in performance-relevant server code. master_to_slave_index_map is just a lookup table of numbers 0..#columns-1, right? So it doesn't need to be an associative array, a normal array is sufficient, and doesn't need to be re-allocated each event. And master_unmatched_cols is again just a set of numbers 0..#columns-1, so a bitmap could be used. - Kristian.

1 0

MariaDB 11.8.1 fully compatible with Y2038
by Otto Kekäläinen 09 Apr '25

09 Apr '25

Hi, I just wanted to report that I tested building MariaDB in 2025 and running MTR in 2037 and 2038 using libfaketime[1]. I can confirm no test started failing in 2038, thus MariaDB had no Y2038 issues in my testing. [1] https://salsa.debian.org/mariadb-team/mariadb-server/-/merge_requests/74

1 0

MariaDB Memory Map
by o.halaiev＠gmail.com 08 Apr '25

08 Apr '25

Hi MariaDB team I'm Oleksii Halaev, studying Python and DataScience I'm currently working on a GSoC 2025 proposal and would like to contribute to MariaDB with an idea I'm developing - it's called ‘MariaDB Memory Map’ The idea is to create an interactive visualisation tool that shows how data changes in the MariaDB database over time - how it is added, modified, deleted and how it ‘travels’ between tables Think of it as a sort of ‘life map’ of your data I believe this can be a useful tool for developers and DBAs to gain a deeper understanding of their databases, make debugging easier, and even track down data anomalies Here is my suggestion: https://drive.google.com/file/d/1vOMF0MZRTueEKkavWdwvEIP6_t2UgJLA/view?usp=… I'd really appreciate any feedback, advice or thoughts - and of course, if there's anyone out there who might be interested in mentoring this project, I'd be happy to chat and learn more Thanks for your time! I look forward to hearing from you GitHub: https://github.com/Alekcentijules Email: o.halaiev(a)gmail.com

2 1

Re: 0ed7370d133: MDEV-24726 Assertion `0' failed in Field_varstring_compressed::key_cmp
by Sergei Golubchik 04 Apr '25

04 Apr '25

Hi, Aleksey, Despite the email subject, this is a combined diff of three commits, git diff 071c20fd23b2 2fecd944d6dc On Apr 04, Aleksey Midenkov wrote: > diff --git a/sql/field.cc b/sql/field.cc > index 8b96c473ab6..d6be66cf399 100644 > --- a/sql/field.cc > +++ b/sql/field.cc > @@ -2516,7 +2521,7 @@ Field *Field::new_key_field(MEM_ROOT *root, TABLE *new_table, > uchar *new_null_ptr, uint new_null_bit) > { > Field *tmp; > - if ((tmp= make_new_field(root, new_table, table == new_table))) > + if ((tmp= make_new_field({root, new_table, table == new_table}))) it's a creative trick, I admit, but let's not do it. looks very weird, we don't have it anywhere else. it's not needed either, because you don't need to pass down a new argument, see below > { > tmp->ptr= new_ptr; > tmp->null_ptr= new_null_ptr; > @@ -8278,12 +8281,28 @@ int Field_varstring::cmp_binary(const uchar *a_ptr, const uchar *b_ptr, > } > > > -Field *Field_varstring::make_new_field(MEM_ROOT *root, TABLE *new_table, > - bool keep_type) > +Field *Field_varstring::make_new_field(make_new_field_args args) > { > - Field_varstring *res= (Field_varstring*) Field::make_new_field(root, > - new_table, > - keep_type); > + Field_varstring *res= (Field_varstring*) Field::make_new_field(args); > + if (res) > + res->length_bytes= length_bytes; > + return res; > +} > + > + > +Field *Field_varstring_compressed::make_new_field(make_new_field_args args) > +{ > + Field_varstring *res; > + if (args.tmp_field) tmp field is a field of a tmp table. Unfortunately, INTERNAL_TMP_TABLE value of new_table->s->tmp_table is very overloaded and no longer means internal temporary table anymore. I think we'll need to fix it someday. But a smaller fix for today could be, like @@ -1113,6 +1113,11 @@ struct TABLE_SHARE return (tmp_table == SYSTEM_TMP_TABLE) ? 0 : table_map_id; } + bool is_optimizer_tmp_table() + { + return tmp_table == INTERNAL_TMP_TABLE && !db.length && table_name.length; + } + using the fact that optimizer tmp tables are all named "(temporary)" or something like that and have no db. Using this one can do a fix, like @@ -8281,11 +8279,23 @@ int Field_varstring::cmp_binary(const uchar *a_ptr, con> Field *Field_varstring::make_new_field(MEM_ROOT *root, TABLE *new_table, bool keep_type) { - Field_varstring *res= (Field_varstring*) Field::make_new_field(root, + Field_varstring *res; + if (new_table->s->optimizer_tmp_table()) + { + res= new (root) Field_varstring(ptr, field_length, length_bytes, null_ptr, + null_bit, Field::NONE, &field_name, + new_table->s, charset()); + if (res) + res->init_for_make_new_field(new_table, orig_table); + } + else + { + res= (Field_varstring*) Field::make_new_field(root, new_table, keep_type); if (res) res->length_bytes= length_bytes; + } return res; } > + { > + /* Compressed fields can't have keys (see Field_varstring_compressed::key_cmp()). */ > + res= new (args.root) Field_varstring(*this); > + res->init_new_field(args); > + /* See Column_definition::create_length_to_internal_length_string() */ > + res->field_length--; > + } > + else > + res= (Field_varstring*) Field::make_new_field(args); > if (res) > res->length_bytes= length_bytes; > return res; Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

1 0

MDEV-36107: Proposal for Enhancements in MariaDB Operator Handling
by Harshit Vijay Kumar 02 Apr '25

02 Apr '25

Dear MariaDB Developer Community, I hope you are doing well. I am *Harshit Vijay Kumar*, a final-year undergraduate student in Computer Science. I have been researching *MDEV-36107 <https://jira.mariadb.org/browse/MDEV-36107>* for *GSoC 2025* and, during my exploration of operator operations in MariaDB, I came across some behaviors that caught my attention. To further investigate, I compared these behaviors with MySQL to understand their differences. *Observations: * *(1) Handling Large Number Multiplications Using Exponents* - Currently, multiplying two large integers results in an *overflow*, returning 0 instead of using an exponent-based representation. SELECT 18014398509481984 * 18014398509481984; *MariaDB Output:* 0 *SQL Output:* 3.2451855365842673e+32 - Interestingly, if one of the operands is a *floating-point number*, the result is computed without overflow: SELECT 18014398509481984 * 18014398509481984399405765456789087654390954.0; *Output:* 324518553658426733978210646499548233026655740664701555572736.0 - This inconsistency between integer-only and mixed-type operations is confusing. A standardized approach, possibly leveraging exponent notation, could improve clarity. *(2) Rounding Floating-Point Outputs* - MariaDB currently returns full precision in floating-point calculations, which can lead to unnecessary precision in practical use cases. SELECT 1.0376728938476536278903984765 + 1.039876453672890938476536278902345678908765434567890; *MariaDB Output:* 2.07754934752054456636693475540234567891 *SQL Output:* 2.077549347520544 - A rounding mechanism would allow for concise representation and reduce storage overhead, especially for users unaware of excessive precision in their datasets. *(3) Eliminating Unnecessary Zeros in Floating-Point Representation* SELECT 1.0000 * 1.0000000000; *MariaDB Output:* 1.00000000000000 - While useful in some cases, the inclusion of excessive trailing zeros unnecessarily increases storage usage. A more compact representation should be considered, where only *significant digits* are retained. *(4) Implementation of == Equality Operator* - Currently, MariaDB allows the *assignment operator (=)* for comparisons. However, since = is primarily used for assignments, this dual-purpose behavior can be confusing. - Proposal: Introduce a dedicated *== equality operator* for clear and intuitive comparisons across *INT, FLOAT, BOOL, and CHAR* types. - Expected behavior: - If values match → return 1 - If values don’t match → return 0 Testing & Next Steps To implement these improvements, I plan to add *test cases* in mysql-test covering all edge cases for these operators. I will follow the approach used in the following implementations: - func_equal.result <https://github.com/MariaDB/server/blob/main/mysql-test/main/func_equal.resu…> - func_equal.test <https://github.com/MariaDB/server/blob/main/mysql-test/main/func_equal.test> I would really appreciate your insights on these ideas. If any of these behaviors have intentional design choices, I would love to understand the rationale behind them. Looking forward to your feedback! Best Regards, Harshit Vijay Kumar

1 0

Re: 0a0c5f349c6: MDEV-36425 Extend read_only to also block share locks and super user
by Sergei Golubchik 02 Apr '25

02 Apr '25

Hi, Monty, On Apr 01, Michael Widenius wrote: > revision-id: 0a0c5f349c6 (mariadb-11.8.1-42-g0a0c5f349c6) > parent(s): 21321025e5e > author: Michael Widenius > committer: Michael Widenius > timestamp: 2025-03-31 20:09:03 +0300 > message: > > MDEV-36425 Extend read_only to also block share locks and super user > > The main purpose of this allow one to use the --read-only > option to ensure that no one can issue a query that can > block replication. > > The --read-only option can now take 4 different values: > 0 No read only (as before). > 1 Blocks changes for users without the 'READ ONLY ADMIN' > privilege (as before). > 2 Blocks in addition LOCK TABLES and SELECT IN SHARE MODE > for not 'READ ONLY ADMIN' users. > 3 Blocks in addition 'READ_ONLY_ADMIN' users for all the > previous statements. > diff --git a/mysql-test/main/read_only.result b/mysql-test/main/read_only.result > index 65cc12ffce9..89390ad65d5 100644 > --- a/mysql-test/main/read_only.result > +++ b/mysql-test/main/read_only.result > @@ -69,11 +69,14 @@ set global read_only=1; > connection con1; > select @@global.read_only; > @@global.read_only > -0 > +OFF > unlock tables ; > +Timeout in wait_condition.inc for SELECT @@global.read_only= 1 > +Id User Host db Command Time State Info Progress > +5 test localhost test Query 0 starting show full processlist 0.000 Forgot to fix? It should wait for @@global.read_only='READ_ONLY' now > select @@global.read_only; > @@global.read_only > -1 > +READ_ONLY > connection default; > reap; > connection default; > diff --git a/sql/lock.cc b/sql/lock.cc > index 1296e5455d5..8f9f97f88b7 100644 > --- a/sql/lock.cc > +++ b/sql/lock.cc > @@ -179,19 +198,48 @@ lock_tables_check(THD *thd, TABLE **tables, uint count, uint flags) > > /* > Prevent modifications to base tables if READ_ONLY is activated. > - In any case, read only does not apply to temporary tables. > + In any case, read only does not apply to temporary tables or slave > + threads. > */ > - if (!(flags & MYSQL_LOCK_IGNORE_GLOBAL_READ_ONLY) && !t->s->tmp_table) > + if (unlikely(opt_readonly) && > + !(flags & MYSQL_LOCK_IGNORE_GLOBAL_READ_ONLY) && !t->s->tmp_table && > + !thd->slave_thread) > { > - if (t->reginfo.lock_type >= TL_FIRST_WRITE && > - !ignore_read_only && opt_readonly && !thd->slave_thread) > + switch (opt_readonly) // Impossible you have "Impossible" comment on a wrong line > { > - my_error(ER_OPTION_PREVENTS_STATEMENT, MYF(0), "--read-only"); > - DBUG_RETURN(1); > + case READONLY_OFF: > + DBUG_ASSERT(0); > + break; > + case READONLY_ON: // Compatibility > + if (!(thd->security_ctx->master_access & PRIV_IGNORE_READ_ONLY) && > + t->reginfo.lock_type >= TL_FIRST_WRITE) > + { > + mariadb_error_read_only(); > + DBUG_RETURN(1); I'd make mariadb_error_read_only() to return 1. Then you can write DBUG_RETURN(mariadb_error_read_only()); here and also in all other places, check it out, it's always followed by return 1. > + } > + break; > + case READONLY_NO_LOCK: > + if (!(thd->security_ctx->master_access & PRIV_IGNORE_READ_ONLY) && > + (thd->lex->sql_command == SQLCOM_LOCK_TABLES || > + t->reginfo.lock_type >= TL_FIRST_WRITE || > + t->reginfo.lock_type == TL_READ_WITH_SHARED_LOCKS)) may be t->reginfo.lock_type >= TL_READ_WITH_SHARED_LOCKS ? > + { > + mariadb_error_read_only(); > + DBUG_RETURN(1); > + } > + break; > + case READONLY_NO_LOCK_NO_ADMIN: > + if (thd->lex->sql_command == SQLCOM_LOCK_TABLES || > + t->reginfo.lock_type >= TL_FIRST_WRITE || > + t->reginfo.lock_type == TL_READ_WITH_SHARED_LOCKS) and here > + { > + mariadb_error_read_only(); > + DBUG_RETURN(1); > + } > + break; > } > } > } > - > /* > Locking of system tables is restricted: > locking a mix of system and non-system tables in the same lock > diff --git a/sql/sql_class.h b/sql/sql_class.h > index d65cec37154..34ee8391009 100644 > --- a/sql/sql_class.h > +++ b/sql/sql_class.h > @@ -3448,9 +3452,12 @@ class THD: public THD_count, /* this must be first */ > */ > inline bool is_read_only_ctx() rename to, for example, check_read_only_with_errror() > { > - return opt_readonly && > - !(security_ctx->master_access & PRIV_IGNORE_READ_ONLY) && > - !slave_thread; > + if (likely(!opt_readonly) || slave_thread || > + ((security_ctx->master_access & PRIV_IGNORE_READ_ONLY) && > + opt_readonly != READONLY_NO_LOCK_NO_ADMIN)) > + return false; > + mariadb_error_read_only(); > + return true; > } > > private: > diff --git a/sql/sys_vars.cc b/sql/sys_vars.cc > index 0247afd91a6..545b4674da1 100644 > --- a/sql/sys_vars.cc > +++ b/sql/sys_vars.cc > @@ -3253,15 +3253,24 @@ static bool fix_read_only(sys_var *self, THD *thd, enum_var_type type) > transition (especially when transitioning from false to true) and > synchronizes both booleans in the end. > */ > -static Sys_var_on_access_global<Sys_var_mybool, > + > +const char *read_only_mode_names[]= > +{"OFF", "READ_ONLY", "READ_ONLY_NO_LOCK", "READ_ONLY_NO_LOCK_NO_ADMIN"}; > + > +static Sys_var_on_access_global<Sys_var_enum, > PRIV_SET_SYSTEM_GLOBAL_VAR_READ_ONLY> > Sys_readonly( > "read_only", > - "Make all non-temporary tables read-only, with the exception for " > - "replication (slave) threads and users with the 'READ ONLY ADMIN' " > - "privilege", > - GLOBAL_VAR(read_only), CMD_LINE(OPT_ARG), DEFAULT(FALSE), > - NO_MUTEX_GUARD, NOT_IN_BINLOG, > + "Do not allow changes to non-temporary tables. Options are:" > + "OFF changes allowed. " > + "READ_ONLY blocks changes for users without the 'READ ONLY ADMIN' " "Disallow changes for users without the READ ONLY ADMIN privilege" > + "privilege. " > + "READ_ONLY_NO_LOCK blocks in addition LOCK TABLES and " > + "SELECT IN SHARE MODE for not ADMIN users. " "Additionally disallows LOCK TABLES and SELECT IN SHARE MODE" > + "READ_ONLY_NO_LOCK_NO_ADMIN blocks in addition 'ADMIN' users. " "Disallows also for users with READ_ONLY ADMIN privilege" > + "Replication (slave) threads are not affected by this option", > + GLOBAL_VAR(read_only), CMD_LINE(OPT_ARG), > + read_only_mode_names, DEFAULT(0), NO_MUTEX_GUARD, NOT_IN_BINLOG, > ON_CHECK(check_read_only), ON_UPDATE(fix_read_only)); > > // Small lower limit to be able to test MRR Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 1

Inquiry About GSoC 2025 and Proposal Submission
by Harsh Gupta 31 Mar '25

31 Mar '25

*Dear MariaDB Developers,* I hope you are doing well. My name is *Harsh Gupta*, and I am a *second-year B.Tech Computer Science and Design student at Madhav Institute of Technology and Science, Gwalior*. I am passionate about databases and open-source development, and I am eager to contribute to MariaDB through *Google Summer of Code 2025*. I have gone through the project ideas listed for GSoC and found the MDEV-28395 <https://jira.mariadb.org/browse/MDEV-28395> LOAD DATA plugins particularly interesting. I have experience with *C++ and database systems*, and I have been exploring MariaDB’s codebase to get familiar with its architecture. I have prepared a draft proposal outlining my approach to this project, including my implementation plan and timeline. I would appreciate your feedback to refine it further before submitting it officially. Could you please review my proposal and provide any suggestions for improvement? Also, I would love to connect with potential mentors to discuss the project's scope and feasibility. Looking forward to your guidance and feedback! Best regards, *Harsh Gupta* 📧 harshgupta9329(a)gmail.com

1 0

Re: b7dbe187b1a: MDEV-34680 Asynchronous and Buffered Logging for Audit Plugin.
by Sergei Golubchik 31 Mar '25

31 Mar '25

Hi, Alexey, See few comments below. But the main question, could you have used FILE or IO_CACHE instead of a custom implementation? On Mar 08, Alexey Botchkov wrote: > revision-id: b7dbe187b1a (mariadb-11.8.1-8-gb7dbe187b1a) > parent(s): 6f1161aa34c > author: Alexey Botchkov > committer: Alexey Botchkov > timestamp: 2025-03-02 16:27:23 +0400 > message: > > MDEV-34680 Asynchronous and Buffered Logging for Audit Plugin. > > Buffering added to the file logger. > Two buffers are created so writing to the file doesn't > block other threads from logging. > Operations isolated with two mutexes. > diff --git a/include/mysql/service_logger.h b/include/mysql/service_logger.h > index 35c2eb1e3a9..adce7bc9038 100644 > --- a/include/mysql/service_logger.h > +++ b/include/mysql/service_logger.h > @@ -64,41 +64,44 @@ extern struct logger_service_st { > void (*logger_init_mutexes)(); > LOGGER_HANDLE* (*open)(const char *path, > unsigned long long size_limit, > - unsigned int rotations); > + unsigned int rotations, size_t buffer_size); incompatible API change. MUST change the major service version. > int (*close)(LOGGER_HANDLE *log); > int (*vprintf)(LOGGER_HANDLE *log, const char *fmt, va_list argptr) > ATTRIBUTE_FORMAT_FPTR(printf, 2, 0); > int (*printf)(LOGGER_HANDLE *log, const char *fmt, ...) > ATTRIBUTE_FORMAT_FPTR(printf, 2, 3); > - int (*write)(LOGGER_HANDLE *log, const char *buffer, size_t size); > + int (*write)(LOGGER_HANDLE *log, const void *data, size_t size); > int (*rotate)(LOGGER_HANDLE *log); > + int (*sync)(LOGGER_HANDLE *log); > } *logger_service; > > #ifdef MYSQL_DYNAMIC_PLUGIN > > #define logger_init_mutexes logger_service->logger_init_mutexes > -#define logger_open(path, size_limit, rotations) \ > - (logger_service->open(path, size_limit, rotations)) > +#define logger_open(path, size_limit, rotations, buffer_size) \ > + (logger_service->open(path, size_limit, rotations, buffer_size)) > #define logger_close(log) (logger_service->close(log)) > #define logger_rotate(log) (logger_service->rotate(log)) > +#define logger_rsync(log) (logger_service->sync(log)) > #define logger_vprintf(log, fmt, argptr) (logger_service->\ > vprintf(log, fmt, argptr)) > #define logger_printf (*logger_service->printf) > -#define logger_write(log, buffer, size) \ > - (logger_service->write(log, buffer, size)) > +#define logger_write(log, data, size) \ > + (logger_service->write(log, data, size)) > #else > > void logger_init_mutexes(); > LOGGER_HANDLE *logger_open(const char *path, > unsigned long long size_limit, > - unsigned int rotations); > + unsigned int rotations, size_t buffer_size); > int logger_close(LOGGER_HANDLE *log); > int logger_vprintf(LOGGER_HANDLE *log, const char *fmt, va_list argptr) > ATTRIBUTE_FORMAT(printf, 2, 0); > int logger_printf(LOGGER_HANDLE *log, const char *fmt, ...) > ATTRIBUTE_FORMAT(printf, 2, 3); > - int logger_write(LOGGER_HANDLE *log, const char *buffer, size_t size); > + int logger_write(LOGGER_HANDLE *log, const void *data, size_t size); > int logger_rotate(LOGGER_HANDLE *log); > + int logger_sync(LOGGER_HANDLE *log); > #endif > > > diff --git a/mysys/file_logger.c b/mysys/file_logger.c > index a753c049f68..5464e43eddf 100644 > --- a/mysys/file_logger.c > +++ b/mysys/file_logger.c > @@ -145,85 +230,108 @@ static int do_rotate(LOGGER_HANDLE *log) > namebuf[log->path_len]= 0; > result= my_rename(namebuf, logname(log, log->path, 1), MYF(0)); > log->file= my_open(namebuf, LOG_FLAGS, MYF(0)); > + log->in_file= 0; > exit: > errno= my_errno; > return log->file < 0 || result; > } > > > -/* > - Return 1 if we should rotate the log > -*/ > - > -my_bool logger_time_to_rotate(LOGGER_HANDLE *log) > -{ > - my_off_t filesize; > - if (log->rotations > 0 && > - (filesize= my_tell(log->file, MYF(0))) != (my_off_t) -1 && > - ((ulonglong) filesize >= log->size_limit)) > - return 1; > - return 0; > -} > - > - > int logger_vprintf(LOGGER_HANDLE *log, const char* fmt, va_list ap) > { > - int result; > char cvtbuf[1024]; > size_t n_bytes; > > - flogger_mutex_lock(&log->lock); > - if (logger_time_to_rotate(log) && do_rotate(log)) > - { > - result= -1; > - errno= my_errno; > - goto exit; /* Log rotation needed but failed */ > - } > - > n_bytes= my_vsnprintf(cvtbuf, sizeof(cvtbuf), fmt, ap); > if (n_bytes >= sizeof(cvtbuf)) > n_bytes= sizeof(cvtbuf) - 1; > > - result= (int)my_write(log->file, (uchar *) cvtbuf, n_bytes, MYF(0)); > - > -exit: > - flogger_mutex_unlock(&log->lock); > - return result; > + return logger_write(log, cvtbuf, n_bytes); > } > > > -static int logger_write_r(LOGGER_HANDLE *log, my_bool allow_rotations, > - const char *buffer, size_t size) > +int logger_write(LOGGER_HANDLE *log, const void *data, size_t size) > { > int result; > + size_t in_buffer; > + > + mysql_mutex_lock(&log->lock); > > - flogger_mutex_lock(&log->lock); > - if (allow_rotations && logger_time_to_rotate(log) && do_rotate(log)) > + if (log->buffer_left >= size) > + { > + memcpy(log->buffer_pos, data, size); > + log->buffer_pos+= size; > + log->buffer_left-= size; > + mysql_mutex_unlock(&log->lock); > + return size; > + } > + > + mysql_mutex_lock(&log->lock_buffer); > + > + in_buffer= log->buffer_pos - log->buffer; > + > + { /* swap buffers. */ > + uchar *tmp_buf= log->buffer; > + log->buffer= log->idle_buffer; > + log->idle_buffer= tmp_buf; > + } > + > + set_buffer_pos_and_left(log); > + > + /* Now other threads can write to the new buffer. */ > + mysql_mutex_unlock(&log->lock); > + > + if ((in_buffer && > + my_write(log->file, log->idle_buffer, in_buffer, MYF(0)) != in_buffer)|| > + my_write(log->file, (uchar *) data, size, MYF(0)) != size) > { > result= -1; > errno= my_errno; > - goto exit; /* Log rotation needed but failed */ > + goto exit_buf; > } > + else > + result= 0; > > - result= (int)my_write(log->file, (uchar *) buffer, size, MYF(0)); > + log->in_file+= in_buffer + size; > + if (log->rotations > 0 && log->in_file >= log->size_limit) > + { > + if (do_rotate(log)) > + { > + result= -1; > + errno= my_errno; > + goto exit_buf; /* Log rotation needed but failed */ > + } > + } > > -exit: > - flogger_mutex_unlock(&log->lock); > +exit_buf: > + mysql_mutex_unlock(&log->lock_buffer); > return result; > } > > > -int logger_write(LOGGER_HANDLE *log, const char *buffer, size_t size) > -{ > - return logger_write_r(log, TRUE, buffer, size); > -} > - > int logger_rotate(LOGGER_HANDLE *log) > { > int result; > - flogger_mutex_lock(&log->lock); > - result= do_rotate(log); > - flogger_mutex_unlock(&log->lock); > + mysql_mutex_lock(&log->lock); > + mysql_mutex_lock(&log->lock_buffer); > + result= flush_buffer(log) || do_rotate(log); I'd though you can switch buffers here too and flush/rotate only under the lock_buffer mutex > + mysql_mutex_unlock(&log->lock_buffer); > + mysql_mutex_unlock(&log->lock); > + return result; > +} > + > + > +int logger_sync(LOGGER_HANDLE *log) Why did you introduce an explicit sync API? > +{ > + int result; > + mysql_mutex_lock(&log->lock); > + mysql_mutex_lock(&log->lock_buffer); > + if ((result= flush_buffer(log))) > + { > + result= fsync(log->file); > + } > + mysql_mutex_unlock(&log->lock_buffer); > + mysql_mutex_unlock(&log->lock); > return result; > } > > diff --git a/plugin/server_audit/server_audit.c b/plugin/server_audit/server_audit.c > index 5fc0e3373f3..e51df7ab6b5 100644 > --- a/plugin/server_audit/server_audit.c > +++ b/plugin/server_audit/server_audit.c > @@ -1393,26 +1401,15 @@ static void change_connection(struct connection_info *cn, > static int write_log(const char *message, size_t len, int take_lock) > { > int result= 0; > + > if (take_lock) > - { > - /* Start by taking a read lock */ > mysql_prlock_rdlock(&lock_operations); > - } > > if (output_type == OUTPUT_FILE) > { > if (logfile) > { > - my_bool allow_rotate= !take_lock; /* Allow rotate if caller write lock */ > - if (take_lock && logger_time_to_rotate(logfile)) > - { > - /* We have to rotate the log, change above read lock to write lock */ > - mysql_prlock_unlock(&lock_operations); > - mysql_prlock_wrlock(&lock_operations); > - allow_rotate= 1; > - } > - if (!(is_active= (logger_write_r(logfile, allow_rotate, message, len) == > - (int) len))) > + if (!(is_active= (logger_write(logfile, message, len) == (int) len))) nice, rotation is indeed the logger's headache, not the caller's > { > ++log_write_failures; > result= 1; > @@ -2899,6 +2906,36 @@ static void update_file_rotations(MYSQL_THD thd __attribute__((unused)), > } > > > +static void update_file_buffer_size(MYSQL_THD thd __attribute__((unused)), > + struct st_mysql_sys_var *var __attribute__((unused)), > + void *var_ptr __attribute__((unused)), const void *save) > +{ > + file_buffer_size= *(unsigned int *) save; > + > + error_header(); > + fprintf(stderr, "Log file buffer size was changed to '%d'.\n", file_buffer_size); > + > + if (!logging || output_type != OUTPUT_FILE) > + return; > + > + ADD_ATOMIC(internal_stop_logging, 1); > + if (!maria_55_started || !debug_server_started) > + mysql_prlock_wrlock(&lock_operations); > + > + stop_logging(); > + if (start_logging()) this could be an API call in the logger. Like "reinit(new params)" and it'll close, change params, open - all under a log mutex, which doesn't need to be destroyed and reinitialized. > + { > + error_header(); > + fprintf(stderr, "Logging was disabled..\n"); > + CLIENT_ERROR(1, "Logging was disabled.", MYF(ME_WARNING)); > + } > + > + if (!maria_55_started || !debug_server_started) > + mysql_prlock_unlock(&lock_operations); > + ADD_ATOMIC(internal_stop_logging, -1); > +} > + > + > static void update_file_rotate_size(MYSQL_THD thd __attribute__((unused)), > struct st_mysql_sys_var *var __attribute__((unused)), > void *var_ptr __attribute__((unused)), const void *save) Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 1

Introduction & Interest in GSoC 2025 with MariaDB
by H.NAVEENKUMAR 30 Mar '25

30 Mar '25

*Hello MariaDB Developers,* I hope you're all doing well. My name is *Naveen Kumar*, and I am currently a student with a strong interest in open-source development, databases, and backend engineering. I have experience working with *Python, JavaScript, MySQL, MongoDB, and full-stack development*. I recently registered for *Google Summer of Code (GSoC) 2025* with MariaDB and am eager to contribute to the project. I am still exploring the available project ideas and would appreciate guidance on how to choose the best fit based on my skills. It would be great if mentors or experienced contributors could suggest beginner-friendly issues or key areas to focus on while getting familiar with the MariaDB codebase. Looking forward to learning, collaborating, and making meaningful contributions to the community! Best regards, *Naveen Kumar*

2 1

MariaDB 11.8.1 in Debian
by Otto Kekäläinen 23 Mar '25

23 Mar '25

Hi, I just wanted to let you know that MariaDB 11.8.1 just landed in Debian testing, which will be the upcoming Debian 13 "Trixie" release. I will continue with various quality assurance tasks so that eventually when 11.8.2 comes out it will be as polished as possible from Debian/Ubuntu perspective. Hopefully .2 does not have any big changes, but just bug fixes.

2 1

MariaDB 11.8(.1) to Debian
by Otto Kekäläinen 15 Mar '25

15 Mar '25

Hi, Just FYI, I am done with the February maintenance releases to Debian and Ubuntu, and this weekend started working on importing MairaDB 11.8(.1) to Debian. Draft is visible at https://salsa.debian.org/mariadb-team/mariadb-server/-/merge_requests/107

1 2

Using GitHub directly from inside Emacs
by Otto Kekäläinen 15 Mar '25

15 Mar '25

Hi! I don't use Emacs myself, but I learned last weekend at FOSDEM that some Emacs users love using the Emacs extension Magit [1] with the Forge [2] feature to list and review GitHub Pull Requests directly from Emacs without opening a browser. Just curious to know if any Emacs users here (Monty?) have tried Magit for interacting with https://github.com/MariaDB/server/pulls? There are currently 240 open PRs and 157 have zero reviews [3] from people with GitHub permissions to give reviews. To get more contributors, we need to have more people participating in reviewing/approving incoming contributions. Also some parts of the contributions process are not very optimal, and the process is likely to become smoother only if enough of current core developers at least occasionally "eat their own cooking" and experience (at least parts) of the contribution process. - Otto [1] https://magit.vc/ [2] https://magit.vc/manual/forge.html [3] https://github.com/MariaDB/server/pulls?q=is%3Apr+is%3Aopen+review%3Anone

2 3

Review of MDEV-35915 Implement Global temporary tables, part 2
by Michael Widenius 12 Mar '25

12 Mar '25

Review of Global temporary tables, part 2 commit a9478ca44b44118cf70de493569d64afbb5cd925 Author: Nikita Malyavin <nikitamalyavin(a)gmail.com> Date: Fri Nov 22 02:15:56 2024 +0100 Fix deallocation of Sql_cmd_dml::result Made as a part of MDEV-34309 Sql_cmd_dml::result is using mem_root to control memory allocation, but Sql_alloc allows to mark the memory as freed via destructor. Normal runs have DTRACE disabled, so accesses in MYSQL_DML_DONE are turned into no-op, and that is the bug is not shown. Enabling DTRACE brings a crash yet in a bootstrap. The fix is to carefully set Sql_cmd_dml::result to nullptr whenever it is freed, or is marked as freed. Please include the title of MDEV-354309 in the commit description @@ -34253,6 +34253,7 @@ bool Sql_cmd_dml::prepare(THD *thd) MYSQL_DML_START(thd); + result= nullptr; // It could've been set since the last run. lex->context_analysis_only|= CONTEXT_ANALYSIS_ONLY_DERIVED; if (open_tables_for_query(thd, lex->query_tables, &table_count, 0, How +++ b/sql/sql_delete.cc @@ -1882,6 +1882,7 @@ bool Sql_cmd_delete::execute_inner(THD *thd) { select_result *sel_result= NULL; delete result; + result= nullptr; /* This is DELETE ... RETURNING. It will return output to the client */ if (thd->lex->analyze_stmt) { @@ -1929,10 +1930,7 @@ bool Sql_cmd_delete::execute_inner(THD *thd) } if (result) - { res= false; - delete result; - } Why remove the delete. If the delete may free wrong things, then the newly added result is also likely to free wrong things. Anyway, the code is clearly wrong here. One cannot set res to 0 if res is 1. This means we are ignoring a possible error from the upper level (probably OOM) and we are probably going to assert later in the code that checks if we can send ok. Try to remove the seteting of res= false and run mtr. Same logics goes for all changes in Sql_cmd_update::execute_inner(THD *thd diff --git a/sql/sql_update.cc b/sql/sql_update.cc if (unlikely(res)) + if (unlikely(res) && result) result->abort_result_set(); else { The above is wrong. Should be: + if (unlikely(res)) { if (result) result->abort_result_set(); } else { As otherwise you may call send_explain() when you shouldn't ------------------ Suggestion: - Move delete of result to unprepare. This will ensure that result is always 0 on Sql_cmd_dml::prepare() and excute_innert and you can replace delete result with 'DBUG_ASSERT(result == 0)' in most of your changes. - Remove setting res= 0 if we have a result and check with mtr. Conclusion: - As this is not cricial and you are going on vacation, I will take over this issue --------- commit 6a000835f5f58aba221ab89c120adae4be16abf3 Author: Nikita Malyavin <nikitamalyavin(a)gmail.com> Date: Thu Feb 20 16:12:37 2025 +0100 fix show create diff --git a/sql/sql_show.cc b/sql/sql_show.cc index da0081fd1f8..201aa8e3110 100644 --- a/sql/sql_show.cc +++ b/sql/sql_show.cc @@ -1413,8 +1413,13 @@ mysqld_show_create(THD *thd, TABLE_LIST *table_list) if (table_list->table && table_list->table->s->tmp_table != NO_TMP_TABLE) { TMP_TABLE_SHARE *share= (TMP_TABLE_SHARE*)table_list->table->s; - if (share->from_share && table_list->table->file->records() == 0) - thd->drop_temporary_table(table_list->table, NULL, true); + if (share->from_share) + { + if (table_list->table->file->info(HA_STATUS_VARIABLE) + || (table_list->table->file->records() == 0 + && thd->drop_temporary_table(table_list->table, NULL, true))) + error= TRUE; + } Please move || and && to previous lines so we have the test lined up nicely in the editor. This is how most of the MariaDB code is written. commit 55ff0fb18bbe4539db27ea73cc2b795e53521772 Author: Nikita Malyavin <nikitamalyavin(a)gmail.com> Date: Thu Feb 20 16:12:49 2025 +0100 add rpl assertions if (thd->temporary_tables - && thd->temporary_tables->global_temporary_tables_count) // TODO this may be buggy on slave + && thd->temporary_tables->global_temporary_tables_count) { move && up to previous line. @@ -6051,6 +6053,7 @@ my_bool open_global_temporary_table(THD *thd, TABLE_SHARE *source, if (thd->mdl_context.clone_ticket(&share->mdl_request)) return TRUE; table->reginfo.lock_type= TL_IGNORE; + share->table_creation_was_logged= 1; } The above means that we assume that slave have the same global temporary table, which is probably not the case. Better to set share->table_creation_was_logged=0 for the local temporary tables we are creating which means that any update of the table through the GTT will use row based replication. Please also add a function comment before my_bool open_global_temporary_table(THD *thd, TABLE_SHARE *source, TABLE_LIST *out_table, MDL_ticket *mdl_ticket) commit a64bbaf483ad714493db401f68049ce85a886c96 Author: Nikita Malyavin <nikitamalyavin(a)gmail.com> Date: Thu Feb 27 01:07:45 2025 +0100 CREATE: use frm header to store GTT bits diff --git a/sql/temporary_tables.cc b/sql/temporary_tables.cc index 45bca833d17..ea46c4035ba 100644 --- a/sql/temporary_tables.cc +++ b/sql/temporary_tables.cc @@ -1324,10 +1324,10 @@ int THD::commit_global_tmp_tables() All_share_tables_list::Iterator tables_it(share->all_tmp_tables); TABLE *table= tables_it++; DBUG_ASSERT(table); - if (!share->on_commit_delete) + if (!share->on_commit_delete()) table->file->info(HA_STATUS_VARIABLE); // update records() stat - if (share->on_commit_delete || table->file->records() == 0) + if (share->on_commit_delete() || table->file->records() == 0) error= drop_tmp_table_share(share, NULL, true); } I still think it would be better to not drop the temporary file based on file->records(). The reason is that some temporary table will be slow to drop and if there are users that is using these for many queries things will be slower than they need to be. - Not critical, can be discussed and changed later. commit 21f9ac1e56d6c81a431e5bdbc9b6b89c8b565ca2 Author: Nikita Malyavin <nikitamalyavin(a)gmail.com> Date: Thu Feb 27 01:12:34 2025 +0100 minor (typo, style) fixes -struct Closefrm_context +static my_bool open_gtt_on_error(TABLE *table) { - TABLE *table; - ~Closefrm_context() { closefrm(table); } -}; + closefrm(table); + return TRUE; +} Change return to void as this should never fail. Yes, it will require a few changes in callers, but it better to do return TRUE; in the caller than have the reviewer have to look at what another function is returning. +++ b/sql/temporary_tables.cc @@ -329,7 +329,9 @@ bool THD::internal_open_temporary_table(TABLE_LIST *tl, TABLE **table, { if (tmp_share->from_share && sql_command_flags() & (CF_ADMIN_COMMAND |CF_SCHEMA_CHANGE)) - DBUG_RETURN(false); + DBUG_RETURN(false); /* We want to use real global temporary table + when ALTER/DROP is executed. + */ Move the comment before the DBUG_RETURN() commit 9e1f5413f619bf4f88eef0f0a50fb7df4b8e7d44 Author: Nikita Malyavin <nikitamalyavin(a)gmail.com> Date: Sat Mar 1 01:27:04 2025 +0100 fix ALTER TABLE long wait @@ -11676,7 +11677,7 @@ do_continue:; */ if (alter_info->requested_lock != Alter_info::ALTER_TABLE_LOCK_EXCLUSIVE && thd->mdl_context.upgrade_shared_lock(mdl_ticket, MDL_SHARED_NO_WRITE, - lock_wait_timeout)) + thd->variables.lock_wait_timeout)) goto err_new_table_cleanup; Add a comment why we should not use lock_wait_timeout here. This is needed as we are using it a few lines above and it is important for coders to know why the value is not reused here. commit f7629d745fe729ea445688f5b710a97108da7c76 Author: Nikita Malyavin <nikitamalyavin(a)gmail.com> Date: Sat Mar 1 02:09:20 2025 +0100 Don't drop local data tables for ON COMMIT PRESERVE when records()==0 connect con1,localhost,root,,; +insert t values(1); +connection default; +alter table t add j int; +ERROR HY000: Lock wait timeout exceeded; try restarting transaction +set statement lock_wait_timeout=0 for drop table t; +ERROR HY000: Lock wait timeout exceeded; try restarting transaction We have to add better error messages for known cases. "Cannot alter global temporary tables as we have open instances of it" "Cannot alter global temporary tables as there are other users for it" Can be done after vacation. diff --git a/mysql-test/main/global_temporary_table.test b/mysql-test/main/global_temporary_table.test index 0edc6eee493..17195eaf3ea 100644 --- a/mysql-test/main/global_temporary_table.test +++ b/mysql-test/main/global_temporary_table.test +--} insert into t1 values(1, 'one'); + select * from t1; No reason for having an extra empty line here. This commit is using: bool THD::use_real_global_temporary_share() const { return (sql_command_flags() & (CF_ALTER_TABLE | CF_SCHEMA_CHANGE | CF_STATUS_COMMAND) && lex->sql_command != SQLCOM_CREATE_TABLE) || lex->sql_command == SQLCOM_CREATE_VIEW; } Please add a comment and move |, && and || to the end of lines. Please also fix the indentation. You can also add a new flag CF_USE_LOCAL_GTT table and mark those commands for which using GTT will open a local GTT copy. In sql_show.cc:get_all_tables(THD *thd, TABLE_LIST *tables, COND *cond) there is this code: if (thd->use_real_global_temporary_share()) continue; This looks wrong. We are ignoring all temporary tables if the above case is true. To ignore GTT ables we have to verify that the found share is a GTT table. Shouldn't we also test for tmp_share->from_share ? Why are we testing for use_real_global_temporary_share() in this code? This is called only be DML. Shouldn't that be safe for GTT tables? For what commands are thd->use_real_global_temporary_share() not true here? commit 718afd78dae4a37ed009ab349dfe9bca68fbc3d7 Author: Nikita Malyavin <nikitamalyavin(a)gmail.com> Date: Tue Mar 4 18:30:07 2025 +0100 Extract THD::has_open_global_temporary_tables() No comments commit cc35b9cb54e74a2f0fbbf4a2400599c7fddb750d Author: Nikita Malyavin <nikitamalyavin(a)gmail.com> Date: Tue Mar 4 21:01:07 2025 +0100 Add CF_ALTER_TABLE No comments commit 381243f4e75f61a31305c656f9b6be20cec042d2 Author: Nikita Malyavin <nikitamalyavin(a)gmail.com> Date: Wed Mar 5 00:18:42 2025 +0100 add global_temporary_tables plugin, move relevant code to temporary_tables.cc + MariaDB_PLUGIN_MATURITY_STABLE Should probably not be stable ;) Please change to BETA. We just have to ensure that this does not cause problems in Jenkin. If it does, you can change it to GAMMA or STABLE (depending on what Jenkins supports) commit a7fa6f835998bc33f1528420df5fe3f1b8a5dbfd Author: Nikita Malyavin <nikitamalyavin(a)gmail.com> Date: Wed Mar 5 00:50:50 2025 +0100 refactor drop_tmp_table_share + while ((tab= it++)) { + if (tab != table && tab->query_id != 0) + { + /* Found a table instance in use. This table cannot be dropped. */ + my_error(ER_CANT_REOPEN_TABLE, MYF(0), table->alias.c_ptr()); + result= true; + goto end; + } Suggested comment change: /* Found a table instance in use. This table cannot be dropped. */ -> /* Found a table instance in use (query_id != 0). This table cannot be dropped. */ ------- commit bb9ff1b2ec45b437bd39b6dacef8885fadeb818b Author: Nikita Malyavin <nikitamalyavin(a)gmail.com> Date: Wed Mar 5 02:51:23 2025 +0100 refactor THD::internal_open_temporary_table No comments commit 72914bfd92ae7390f8e7924944c92a1dc9591e1a Author: Nikita Malyavin <nikitamalyavin(a)gmail.com> Date: Wed Mar 5 22:06:02 2025 +0100 enforce limitations + if (create_info->table_options & HA_OPTION_GLOBAL_TEMPORARY_TABLE) + { + const char *error= NULL; + for (const Key &fk: create_info->alter_info->key_list) + if (fk.type == Key::FOREIGN_KEY) + { + error= "FOREIGN KEY"; + break; + } You can replace the above loop with: if (thd->lex->alter_info_flags & ALTER_ADD_FOREIGN_KEY) Note also that in your old code, later errors could overwrite earlier ones. Changing this to a normal if, like the others, will solve that. commit a2831f8e8efcfecd548b8ced6e19c3fc4d6966f9 Author: Nikita Malyavin <nikitamalyavin(a)gmail.com> Date: Thu Mar 6 20:55:00 2025 +0100 information_schema support mysql-test/main/global_temporary_table.result +GLOBAL TEMPORARY +show table status where temporary='Y'; +Name Engine Version Row_format Rows Avg_row_length Data_length Max_data_length Index_length Data_free Auto_increment Create_time Update_time Check_time Collation Checksum Create_options Comment Max_index_length Temporary Please add a test that show the output from information_schema.tables for all fields that are stable for mtr. By the way, why does the above show table status not show any data while +select TABLE_TYPE from information_schema.tables where table_name = 't1'; +TABLE_TYPE +TEMPORARY +GLOBAL TEMPORARY Show two rows just above show table status; At least 'TEMPORARY' should have 'Temporary' set to 'Y'' commit a580c6bc85791829c850d7eedc03d62a5ee99ffd Author: Nikita Malyavin <nikitamalyavin(a)gmail.com> Date: Thu Mar 6 23:43:38 2025 +0100 VIEW support Views already open the correct tables automatically, so it just works. Almost ;) + || lex->sql_command == SQLCOM_CREATE_VIEW ; .... commit d4c00419ebb39a07f87eabf0eee3660a768cbbed Author: Nikita Malyavin <nikitamalyavin(a)gmail.com> Date: Fri Mar 7 02:16:23 2025 +0100 AS SELECT support + if (table->s->tmp_table && + table->s->db_create_options & + (HA_OPTION_GLOBAL_TEMPORARY_TABLE|HA_OPTION_ON_COMMIT_DELETE_ROWS)) + table= NULL; // It will be deleted on commit + This will not work because of the following code: if (WSREP(thd) && table->file->ht->db_type == DB_TYPE_INNODB) You either have to add a goto or fix all references to 'table'. Note this code later in the same function. if (m_plock) { MYSQL_LOCK *lock= *m_plock; *m_plock= NULL; m_plock= NULL; if (create_info->pos_in_locked_tables) { /* If we are under lock tables, we have created a table that was originally locked. We should add back the lock to ensure that all tables in the thd->open_list are locked! */ table->mdl_ticket= create_info->mdl_ticket; This should be safe as long as on cannot use LOCK TABLES on gtt tables. Please add a check for that! Another thing to check is with GALERA that local gtt tables are not replicated. commit 45e2f14bcbdd61c1ea5712f5b79b07a096aa46d2 Author: Nikita Malyavin <nikitamalyavin(a)gmail.com> Date: Mon Mar 10 20:16:16 2025 +0100 CREATE TABLE ... LIKE support sql_table.cc: local_create_info.table_options&= ~HA_OPTIONS_TO_RESET_CREATE_LIKE; Looking at the code, I belive the above is always 0 here. Please replace with DBUG_ASSERT(local_create_info.table_options == 0); commit 1c44e4dabe22555a3b517db7f67c1e4d8b03fc70 Author: Nikita Malyavin <nikitamalyavin(a)gmail.com> Date: Mon Mar 10 21:49:11 2025 +0100 RENAME support +++ b/sql/sql_parse.cc @@ -6351,6 +6351,13 @@ check_rename_table(THD *thd, TABLE_LIST *first_table, check_grant(thd, INSERT_ACL | CREATE_ACL, &new_list, FALSE, 1, FALSE))) return 1; + + if (table->table && + table->table->s->db_create_options & HA_OPTION_GLOBAL_TEMPORARY_TABLE) + { + my_error(ER_LOCK_WAIT_TIMEOUT, MYF(0)); + return 1; + } We need a better error here. Something like: "Cannot rename a global temporary tables as this connection has open instances of it" commit b0f02644c22522c325bfbddd847030d61237b4e8 Author: Nikita Malyavin <nikitamalyavin(a)gmail.com> Date: Tue Mar 11 00:48:11 2025 +0100 FLUSH support, forbid LOCK FLUSH TABLE on Global temporary table is basically no-op, since it never has open instances. - It is ok to ignore flush table on GTT tables. There is no point in doing that. Same goes for temporary tables. +/** + Check if any global temporary tables are still opened in this session. +*/ +static int flush_check_for_global_temporary_tables(THD *thd, TABLE_LIST *tl) +{ + if (thd->rgi_slave || !thd->has_open_global_temporary_tables()) + return 0; + + for (;tl; tl= tl->next_global) + { + TABLE_SHARE *share= thd->find_tmp_table_share(tl); + if (share && share->db_create_options & HA_OPTION_GLOBAL_TEMPORARY_TABLE) + { + my_error(ER_LOCK_WAIT_TIMEOUT, MYF(0)); + return 1; + } + } + return 0; +} I think you can remove the above. +flush table t; +lock table t write; +ERROR 42S02: Unknown table 't' I think it is wrong that we get an err for the lock. We should fix it so that we open the table local GTT table and do not take a lock. (In other words, it should work as normal lock tables for tmp tables). @@ -5970,6 +5970,12 @@ my_bool open_global_temporary_table(THD *thd, TABLE_SHARE *source, { DBUG_ASSERT(!thd->rgi_slave); // slave won't use global temporary tables + if (thd->lex->sql_command == SQLCOM_LOCK_TABLES) + { + my_error(ER_BAD_TABLE_ERROR, MYF(0), source->table_name.str); + return TRUE; + } + See above. We should remove the above and instead open the table normally. Author: Nikita Malyavin <nikitamalyavin(a)gmail.com> Date: Tue Mar 11 02:08:10 2025 +0100 Replication support Please do not use 'show binlog events' as this will cause result conflicts with any changes in the binlog format and server changes. You have now things like Server ver: 11.8.0-MariaDB-debug-log, Binlog ver: 4 in the test output use instead source include/show_binlog_events.inc; One can also does not need to use ../.. to access main tests. Here is a diff for the test: +++ b/mysql-test/suite/binlog/t/global_temporary_table.test @@ -4,7 +4,7 @@ set @old_binlog_annotate_row_events= @@global.binlog_annotate_row_events; set global binlog_annotate_row_events= OFF; set session binlog_annotate_row_events= OFF; -source ../../main/global_temporary_table.test; +source main/global_temporary_table.test; +source include/show_binlog_events.inc; -show binlog events; set global binlog_annotate_row_events= @old_binlog_annotate_row_events; ----- +bool is_user_tmp_table(TMP_TABLE_SHARE *share) +{ + if (share->db_create_options & HA_OPTION_GLOBAL_TEMPORARY_TABLE) + return false; + return share->tmp_table == TRANSACTIONAL_TMP_TABLE || + share->tmp_table == NON_TRANSACTIONAL_TMP_TABLE; +} The normal way to test for temporary table is: share->tmp_table != NO_TMP_TABLE; see is_temporary_table() This should be safe also for the above function. Add () around the return functions, to get editors to indent it correctly by default. I think the above function needs some comments. Please explain how things are: if (share->db_create_options & HA_OPTION_GLOBAL_TEMPORARY_TABLE) is true, this is a GTT definition table and tmp_table would be NO_TMP_TABLE. If this is a local GTT tables, then (share->db_create_options & HA_OPTION_GLOBAL_TEMPORARY_TABLE) is false and share->from_share is true. It would be good if the above function would have good enough comments to explain all the above. We need to describe exactly how things are in some please in the source. Could you add the description how to detect GTT tables and local GTT tables and how they differ from normal temporary tables at the start of temporary_tables.cc Especially how share->db_create_options and share->tmp_table are set. If you add this function first in tempoary_table.cc, it may explain how things are: bool is_user_tmp_table(TMP_TABLE_SHARE *share) { if (share->db_create_options & HA_OPTION_GLOBAL_TEMPORARY_TABLE) { /* This is either a global temporary table (GTT) definition table or local GTT table created from the GTT definition. If (share->tmp_table != NO_TMP_TABLE) then this is local GTT table. In this case share->from_share is also set and points to the global GTT definition. */ return false; } /* Return true of the table is non GTT temporary table */ return (share->tmp_table != NO_TMP_TABLE) }

1 0

Re: [PATCH] Fix redundant ER_PRIOR_COMMIT_FAILED in parallel replication
by Kristian Nielsen 11 Mar '25

11 Mar '25

Hi Monty, thanks for review! Michael Widenius <michael.widenius(a)gmail.com> writes: > Review of patch: > > wait_for_prior_commit() can be called multiple times per event group, > only do my_error() the first time the call fails. > > > --- a/sql/sql_class.cc > +++ b/sql/sql_class.cc > @@ -8365,6 +8365,18 @@ wait_for_commit::wait_for_prior_commit2(THD > *thd, bool allow_kill) > } > > > +void > +wait_for_commit::prior_commit_error(THD *thd) > +{ > + /* > + Only raise a "prior commit failed" error if we didn't already raise > + an error. > + */ > + if (!thd->get_stmt_da()->is_set()) > + my_error(ER_PRIOR_COMMIT_FAILED, MYF(0)); > +} > + > + > /* > Wakeup anyone waiting for us to have committed. > > diff --git a/sql/sql_class.h b/sql/sql_class.h > index 4cee31b296e..69b021cd41d 100644 > --- a/sql/sql_class.h > +++ b/sql/sql_class.h > @@ -2398,8 +2398,8 @@ struct wait_for_commit > return wait_for_prior_commit2(thd, allow_kill); > else > { > - if (wakeup_error) > - my_error(ER_PRIOR_COMMIT_FAILED, MYF(0)); > + if (unlikely(wakeup_error)) > + prior_commit_error(thd); > return wakeup_error; > } > } > The only small objection I have to this patch is that if we get first > one error and then a wait_for_prior_commit error, then we will not get > the second error. > I would suggest that instead of checking for if > (!thd->get_stmt_da()->is_set()) instead add a new flag > 'got_prior_commit_error' to check if we have already > got a prior commit error for this thread. > Adding bool get_prior_commit_error into THD after bool > wakeup_blocked; has no storage cost. It is actually deliberate to skip the ER_PRIOR_COMMIT_FAILED if we already gave a different error. The reason is that the ER_PRIOR_COMMIT_FAILED is not a real error - it is an internal error thrown only to make sure that a following transaction does not succeed after a prior transaction failed, which would result in inconsistent commit order. Therefore, if a worker thread already got a real error, for example duplicate key or row not found, then it seems preferable to keep that error rather than override it with a default, internal-only error. (Arguably it might make sense to not write ER_PRIOR_COMMIT_FAILED errors to the error log at all, as they do not seem very useful; they are not an error that the user should do anything about, on the contrary they indicate that the worker correctly handled an earlier error. But that would be a separate patch, probably). > It up to you to decide which solution to choose. Ok, then I will keep this, but I will add a comment explaining why the original error is preferred over ER_PRIOR_COMMIT_FAILED. > This should be pushed to 10.6 to avoid the 63*4 warnings we can get > with parallel=4. Ok, will do then. > I will try to spend some time today to try to simulate the case where > we get new calls to wait_for_prior_commit after signaling a previous > failure from it. I hope you find a way to reproduce, I did not manage to do so when I tried. It would be good to find the root cause of those repeated ER_PRIOR_COMMIT_FAILED. - Kristian.

1 0

Re: 0cc1c382dfc: MDEV-34712 Add support to sha2 and pbkdf2 key derivation in file_key_management
by Sergei Golubchik 10 Mar '25

10 Mar '25

Hi, Yuchen, On Mar 10, Yuchen Pei wrote: > revision-id: 0cc1c382dfc (mariadb-11.8.1-9-g0cc1c382dfc) > parent(s): 3a81664cb80 > author: Yuchen Pei > committer: Yuchen Pei > timestamp: 2025-03-05 13:33:53 +1100 > message: > > MDEV-34712 Add support to sha2 and pbkdf2 key derivation in file_key_management > > Add two new variables: > > file_key_management_digest={sha1|sha224|sha256|sha384|sha512} > > This specify the digest function to use in key derivation of the key > used for decryption of the keyfile > > file_key_management_use_pbkdf2=N > > This specify whether pbkdf2 is used in the key derivation, and if > so (N != 0), how many thousands iterations. > diff --git a/include/mysql/service_my_crypt.h b/include/mysql/service_my_crypt.h > index 1a4ebcfd067..3a58d0828fc 100644 > --- a/include/mysql/service_my_crypt.h > +++ b/include/mysql/service_my_crypt.h > @@ -70,6 +73,10 @@ extern struct my_crypt_service_st { > unsigned int (*my_aes_get_size)(enum my_aes_mode mode, unsigned int source_length); > unsigned int (*my_aes_ctx_size)(enum my_aes_mode mode); > int (*my_random_bytes)(unsigned char* buf, int num); > + void (*my_bytes_to_key)(const unsigned char *salt, const unsigned char *input, > + unsigned int input_len, unsigned char *key, > + unsigned char *iv, enum my_digest digest, > + unsigned int use_pbkdf2); backward compatible API change - you need to increment the service version > } *my_crypt_service; > > #ifdef MYSQL_DYNAMIC_PLUGIN > diff --git a/mysql-test/suite/encryption/t/filekeys_encfile.test b/mysql-test/suite/encryption/t/filekeys_encfile.test > index a0611a38ec4..89e13834a79 100644 > --- a/mysql-test/suite/encryption/t/filekeys_encfile.test > +++ b/mysql-test/suite/encryption/t/filekeys_encfile.test > @@ -1 +1,34 @@ > +# # The encrypted key files were generated using the following: > + > +# let $plain_keyfile=$MYSQL_TMP_DIR/keys.txt; > +# let $enc_keyfile_prefix=$MTR_SUITE_DIR/t/filekeys-data; > + > +# write_file $plain_keyfile; > +# # > +# # this is a comment > +# # > +# 1;770A8A65DA156D24EE2A093277530142 > +# 2;4D92199549E0F2EF009B4160F3582E5528A11A45017F3EF8 > +# # another comment > +# 33;B374A26A71490437AA024E4FADD5B497FDFF1A8EA6FF12F6FB65AF2720B59CCF > +# 4;18420B5CBA31CCDFFE9716E91EB61374D05914F3ADE23E03 --> ignored > + > +# 5;966050D7777350B6FD5CCB3E5F648DA45C63BEFB6DEDDFA13443F156B7D35C84 > +# 6;B5EA210C8C09EF20DB95EC584714A89F # and yet another > + > +# EOF > + > +# exec openssl enc -aes-256-cbc -md sha1 -pass pass:secret -in $plain_keyfile -out $enc_keyfile_prefix-sha1.enc; > +# exec openssl enc -aes-256-cbc -md sha224 -pass pass:secret -in $plain_keyfile -out $enc_keyfile_prefix-sha224.enc; > +# exec openssl enc -aes-256-cbc -md sha256 -pass pass:secret -in $plain_keyfile -out $enc_keyfile_prefix-sha256.enc; > +# exec openssl enc -aes-256-cbc -md sha384 -pass pass:secret -in $plain_keyfile -out $enc_keyfile_prefix-sha384.enc; > +# exec openssl enc -aes-256-cbc -md sha512 -pass pass:secret -in $plain_keyfile -out $enc_keyfile_prefix-sha512.enc; > +# exec openssl enc -aes-256-cbc -md sha1 -pbkdf2 -pass pass:secret -in $plain_keyfile -out $enc_keyfile_prefix-sha1-pbkdf2.enc; better say -iter 1000 instead of -pbkdf2 (to match plugin parameters) And change some tests to use a non-default value, please > +# exec openssl enc -aes-256-cbc -md sha224 -pbkdf2 -pass pass:secret -in $plain_keyfile -out $enc_keyfile_prefix-sha224-pbkdf2.enc; > +# exec openssl enc -aes-256-cbc -md sha256 -pbkdf2 -pass pass:secret -in $plain_keyfile -out $enc_keyfile_prefix-sha256-pbkdf2.enc; > +# exec openssl enc -aes-256-cbc -md sha384 -pbkdf2 -pass pass:secret -in $plain_keyfile -out $enc_keyfile_prefix-sha384-pbkdf2.enc; > +# exec openssl enc -aes-256-cbc -md sha512 -pbkdf2 -pass pass:secret -in $plain_keyfile -out $enc_keyfile_prefix-sha512-pbkdf2.enc; > + > +# echo Done; > + > source filekeys_goodtest.inc; > diff --git a/mysys_ssl/my_crypt.cc b/mysys_ssl/my_crypt.cc > index 346eff51b1b..ef74121d530 100644 > --- a/mysys_ssl/my_crypt.cc > +++ b/mysys_ssl/my_crypt.cc > @@ -382,4 +384,39 @@ int my_random_bytes(uchar *buf, int num) > return MY_AES_OK; > } > > +static inline const EVP_MD *get_digest(enum my_digest digest) > +{ > + switch (digest) > + { > + case MY_DIGEST_SHA1: return EVP_sha1(); > + case MY_DIGEST_SHA224: return EVP_sha224(); > + case MY_DIGEST_SHA256: return EVP_sha256(); > + case MY_DIGEST_SHA384: return EVP_sha384(); > + case MY_DIGEST_SHA512: return EVP_sha512(); > + default: > + { > + assert(0); > + return NULL; > + } > + } > +} > + > +void my_bytes_to_key(const unsigned char *salt, const unsigned char *input, > + uint input_len, unsigned char *key, unsigned char *iv, > + enum my_digest digest, uint use_pbkdf2) > +{ > + if (use_pbkdf2 == 0) > + EVP_BytesToKey(EVP_aes_256_cbc(), get_digest(digest), salt, > + input, input_len, 1, key, iv); > + else > + { > + uchar keyiv[KEY_LENGTH + IV_LENGTH]; > + PKCS5_PBKDF2_HMAC((const char*) input, input_len, salt, 8, > + use_pbkdf2 * 1000, get_digest(digest), I suggest not to do *1000. It doesn't help with anything and introduces a limitation that we don't need. Perhaps somebody has a policy of using 2^N iterations in PBKDF2? Also, `openssl enc -iter` doesn't do *1000 > + KEY_LENGTH + IV_LENGTH, keyiv); > + memcpy(key, keyiv, KEY_LENGTH); > + memcpy(iv, keyiv + KEY_LENGTH, IV_LENGTH); > + } > +} > + > } > diff --git a/plugin/file_key_management/file_key_management_plugin.cc b/plugin/file_key_management/file_key_management_plugin.cc > index 8702785a1fd..f37cfe05960 100644 > --- a/plugin/file_key_management/file_key_management_plugin.cc > +++ b/plugin/file_key_management/file_key_management_plugin.cc > @@ -54,10 +68,22 @@ static MYSQL_SYSVAR_ENUM(encryption_algorithm, encryption_algorithm, > "Encryption algorithm to use" recommendation, > NULL, NULL, 0, &encryption_algorithm_typelib); > > +static MYSQL_SYSVAR_ENUM(digest, digest, > + PLUGIN_VAR_RQCMDARG | PLUGIN_VAR_READONLY, > + "Digest to use", "Digest algorithm used for decrypting the key file. The value of the openssl enc -md option" > + NULL, NULL, 0, &digest_typelib); > + > +static MYSQL_SYSVAR_UINT(use_pbkdf2, use_pbkdf2, > + PLUGIN_VAR_RQCMDARG | PLUGIN_VAR_READONLY, > + "Use pbkdf2", "Number of PBKDF2 iteration used for decrypting the key file. Use 0 to disable PBKDF2, otherwise it's a value of the openssl enc -iter option" > + NULL, NULL, 0, 0, UINT_MAX / 1000, 0); > + > static struct st_mysql_sys_var* settings[] = { > MYSQL_SYSVAR(filename), > MYSQL_SYSVAR(filekey), > MYSQL_SYSVAR(encryption_algorithm), > + MYSQL_SYSVAR(digest), > + MYSQL_SYSVAR(use_pbkdf2), > NULL > }; Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

1 0

Has anyone implemented InnoDB redo log-based physical replication?
by Otto Kekäläinen 04 Mar '25

04 Mar '25

Hi, Has anyone attempted or seen any existing implementations "physical" replication between two MariaDB instances, where a secondary read-only instance would follow the primary by directly reading and applying page changes from the primary's redo logs? The secondary instance would start (and restart if necessary) using a full copy of the data from the primary instance, but while running keep itself current by reading redo logs from the primary instance, and apply pages based on the increasing LSN number. I call this "physical" replication to distinguish it from normal logical replication based on e.g. binlogs, and this would obviously only work for InnoDB. Thanks, Otto

2 2

Review request for MariaDB 2025 Q1 uploads to Debian and Ubuntu
by Otto Kekäläinen 24 Feb '25

24 Feb '25

Hi! Thanks Vicentiu for reviewing https://salsa.debian.org/mariadb-team/mariadb-server/-/merge_requests/102 The next uploads in line are ready for review at: https://salsa.debian.org/mariadb-team/mariadb-server/-/merge_requests/98 Prepare MariaDB Server 1:10.11.11-0+deb12u1 minor maintenance release for Debian stable (Bookworm) https://salsa.debian.org/mariadb-team/mariadb-server/-/merge_requests/106 Prepare upload to MariaDB 10.11.9 to Ubuntu 24.04 "Noble" https://salsa.debian.org/mariadb-team/mariadb-server/-/merge_requests/96 Prepare upload to MariaDB 10.6.19 to Ubuntu 22.04 "Jammy" https://salsa.debian.org/mariadb-team/mariadb-10.5/-/merge_requests/20 New upstream version MariaDB 10.5.28 for Debian 11 "Bullseye" Once I am done with these, I will start testing MariaDB 11.8 with the goal of uploading it into Debian in the coming weeks.

1 0

Re: [PATCH] MDEV-36099 Ensure that creation and usage of temporary tables in replication is predictable
by Kristian Nielsen 23 Feb '25

23 Feb '25

Hi Monty, > From: Monty <monty(a)mariadb.org> > > The purpose of this commit is to ensure that creation and changes of > temporary tables are properly and predicable logged to the binary > log. It also fixes some bugs where ROW logging was used in MIXED mode, > when STATEMENT would be a better (and expected) choice. Here is my review of the patch for MDEV-36099 predictable binlog format for temporary tables: Overall this seems like solid work, thanks! As Elena also remarked, this does significant changes to a very tricky and bug-ridden area of replication, that of temporary tables, this is scary stuff. But the patch does look good and relatively simple. A main change with this patch is that now all use of temporary tables will by default be binlogged in row mode. I think this is a reasonable change. There are still remaining problems with temporary tables in statement-based mode; and it also helps with parallel replication, which cannot run in parallel statement-logged temporary tables. And you provided --create-temporary-table-binlog-formats as a way to enable backwards-compatible behaviour. I think it is important to mention this prominently in the release notes. There will invariably be some users that manipulate a large number of rows with temporary tables, eg. BEGIN; CREATE TEMPORARY TABLE tmp AS SELECT a, b, c FROM large_table1 JOIN large_table2 ON ... INSERT INTO other_table SELECT b, c FROM tmp WHERE a <100; DROP TEMPORARY TABLE tmp; COMMIT; This will change from binlogging a few hundred bytes of Query events to potentially many megabytes of row events, which could cause problems in some cases. I think the change is still justified from the benefits it gives and the relative few cases where this will cause serious problems, but it is still important to try to document it as well as possible. I read the code part of the patch through in detail, but I did not spot anything in it that looks wrong. There may be things missing from the patch that are hard to spot in a review, I think that's unavoidable in a change like this. But I did try to think of possible issues without finding anything, and the patch does look quite simple compared to what it is doing. So looks good I think. Just a few questions on details I did not understand, and some minor suggestions for changes. Questions: > diff --git a/sql/sql_parse.cc b/sql/sql_parse.cc > index c2f842aea56..3d1df5ac42f 100644 > --- a/sql/sql_parse.cc > +++ b/sql/sql_parse.cc > @@ -531,6 +531,7 @@ void init_update_queries(void) > CF_SCHEMA_CHANGE; > sql_command_flags[SQLCOM_CREATE_SEQUENCE]= (CF_CHANGES_DATA | > CF_REEXECUTION_FRAGILE | > + CF_FORCE_ORIGINAL_BINLOG_FORMAT | > CF_AUTO_COMMIT_TRANS | > CF_SCHEMA_CHANGE); I did not get the reason for this change. I assume some bug fix found during test case writing? > --- a/sql/sql_class.h > +++ b/sql/sql_class.h > @@ -817,6 +817,7 @@ typedef struct system_variables > /* Flags for slow log filtering */ > ulong log_slow_rate_limit; > ulong binlog_format; ///< binlog format for this thd (see enum_binlog_format) > + ulong create_temporary_table_binlog_formats; Is there any way to make this take less space than 8 bytes (on 64-bit)? It goes into every THD, and only two bits are used. > + > + Temporary tables will be logged only on CREATE in STMT format > + or on INSERT if all changes to the table is in the binlog. > */ > if ((WSREP_EMULATE_BINLOG(thd) || mysql_bin_log.is_open()) && > - (likely(!error) || thd->transaction->stmt.modified_non_trans_table || > - thd->log_current_statement())) > + (table->s->using_binlog() || > + ((in_create_table && > + (!table->s->tmp_table || thd->binlog_create_tmp_table())))) && > + (likely(!error) || > + (!in_create_table && > + (thd->transaction->stmt.modified_non_trans_table || > + thd->log_current_statement())))) This conditional is now very hard to understand. I think I managed to understand all parts of it, except the part "thd->log_current_statement()", but that is unchanged with your patch. This if () would now benefit from being split into multiple parts, each of which explained in a comment. > diff --git a/mysql-test/main/statistics.result b/mysql-test/main/statistics.result > index bb6469c46da..ebd9cb7f36e 100644 > --- a/mysql-test/main/statistics.result > +++ b/mysql-test/main/statistics.result > @@ -1278,7 +1278,7 @@ Table Op Msg_type Msg_text > mysql.column_stats analyze error Invalid argument > ANALYZE TABLE mysql.column_stats; > Table Op Msg_type Msg_text > -mysql.column_stats analyze status OK > +mysql.column_stats analyze status Table is already up to date > SELECT * FROM mysql.table_stats; > db_name table_name cardinality > SELECT * FROM mysql.column_stats; I did not understand why this .result change happens after your patch? Minor things: > diff --git a/mysql-test/suite/rpl/t/create_or_replace.inc b/mysql-test/suite/rpl/t/create_or_replace.inc > index e8fa95cb94a..b4e91d7b412 100644 > --- a/mysql-test/suite/rpl/t/create_or_replace.inc > +++ b/mysql-test/suite/rpl/t/create_or_replace.inc > @@ -4,7 +4,24 @@ > --let $rpl_topology=1->2 > --source include/rpl_init.inc > > -# Create help tables > +select @@binlog_format, @@create_temporary_table_binlog_formats; > + > +# Copy create_temporary_table_binlog_formats from master to slave > +# This is done to get same results as with older versions of MariaDB. > +# The slave will work even if this is not done. However in mixed > +# format on slave temporary tables would not be logged to in the > +# slaves binary log > +let $format=`select @@create_temporary_table_binlog_formats`; > +connection server_2; > +--eval set @@global.create_temporary_table_binlog_formats='$format' > +# Ensure that the slave threads uses the new values. > +stop slave; > +start slave; Here, use --source include/stop_slave.inc and --source include/start_slave.inc. For consistent with other test cases. > - Rename of not binlogged temporary tables where binlogged to binary log > which caused replication to stop. s/where/were/ > + /* > + Mark if query was logged as statement. Used to check it tmp table changes > + are logged. > + */ > + bool query_binlogged_as_stmt; s/check it table/check if table/ + /* + This is set for all normal tables and for temporary tables that has + the CREATE binary logged. + + 0 table table is not in binary log (not logged temporary table) + 1 table create was logged (normal table or logged temp table) + 2 table create was logged but not all changes are in the binary log. + ROW LOGGING will be used for the table. s/that has the CREATE/that have the CREATE/ s/table table is not/table is not/ > @@ -794,6 +821,20 @@ void THD::mark_tmp_table_as_free_for_reuse(TABLE *table) > > DBUG_ASSERT(table->s->tmp_table); > > + /* > + Ensure that table changes where either binary logged or the table > + is marked as not up to date. > + */ s/where/were/ Thanks, Monty! Reviewed-by: Kristian Nielsen <knielsen(a)knielsen-hq.org> - Kristian.

2 1

MDEV-33671: MTR Safe Process Hardcodes RLIMIT_NOFILE
by mohanadkhaled87＠gmail.com 10 Feb '25

10 Feb '25

Hello I am Mohanad and this is my first issue here I started working on this issue a while ago but only began testing it recently. As per the comment in this issue (https://jira.mariadb.org/browse/MDEV-33671?focusedCommentId=281856&page=com…) "You can run ulimit from Perl, between the fork and exec." I implemented this by using the BSD::Resource library, as it appears to be part of the core Perl libraries and does not introduce an external dependency and as a sanity check I added ```perl my $has_bsd_resource = eval { require BSD::Resource; BSD::Resource->import(qw(setrlimit getrlimit RLIMIT_NOFILE)); 1; }; ``` Would this be an okay to use ? Also, I have manually verified the implementation by printing the limit values. However, is it possible to add test files or automate testing for this issue in MTR ?

1 0

Current status on MDEV-34705 binlog-in-engine
by Kristian Nielsen 09 Feb '25

09 Feb '25

Hi Brandon (and other interested), It is great that you want to look closer at the MDEV-34705, binlog-in-engine project. I wanted to write up a high-level summary of the current status to help you get an overview. For reference, this is a project to implement a new binlog format that solves some of the complexity and performance problems with the old format that is not integrated with the storage engine transaction log (eg. InnoDB binlog). The code is being developed on the branch knielsen_binlog_in_engine on Github, https://github.com/MariaDB/server/commits/knielsen_binlog_in_engine There has been some previous design discussions on the mailing list, eg.: https://lists.mariadb.org/hyperkitty/list/developers@lists.mariadb.org/thre… https://lists.mariadb.org/hyperkitty/list/developers@lists.mariadb.org/thre… - Kristian. Current status: - Feature is off by default, enabled by --binlog-storage-engine=innodb. - Binlog files are now identified solely with their file_no, starting from 0 (no more master-bin.index). Binlog file names are hard-coded by the engine (binlog-NNNNNN.ibb), directory can be selected with --binlog-directory. - Binlog files are pre-allocated to --max-binlog-size, thus their size is fixed. Event groups and even events can span multiple binlog files. - New binlog format is page-based. On server start, binary search is used to find the current position (no need for scanning a whole binlog file). GTID start position is likewise found by binary search, no need for separate GTID index. - Binlog files are written strictly append-only, data is never updated after initial write. The InnoDB redo log is used to ensure that files are recovered into a consistent state with the engine data after a crash. - Large event groups (larger than --binlog-cache-size) are split into pieces that are binlogged "out-of-band", meaning they are written in separate records during transaction execution, before commit. The final commit record merely has a reference to the out-of-band data. Out-of-band data for rolled-back transactions remain as "garbage" until binlog purge. - The binlog dump thread re-assembles out-of-band data into consecutive event groups for the slaves. Thus, slaves see event data in the same format as before, no slave code changes needed. Out-of-band data is structured as complete binary trees for efficient reading. - Basic slave replication is working, not much is tested though. Currently GTID mode only, might make sense to not support filename/position mode for simplicity. - RESET MASTER, FLUSH BINARY LOGS, PURGE BINARY LOGS are implemented. - Initial benchmarking show good speedup (x2) in durable configuration, potentially huge (3 to sevelal hundred times faster) speedup in crash-safe configuration. A main challenge with this project is that the binlog internals are visible in so many places to users, and to other parts of the code. Here is an (incomplete) list of major issues still to implement: - The current implementation is using the normal InnoDB buffer pool and checkpoint mechanism. See the mailing list discussions for Marko's ideas for implementing a simpler integration with the InnoDB redo log. - When --sync-binlog=1, implement that binlog events are not sent to slaves until durably written to redo log. When --sync-binlog=0, implement that events are not sent until redo log is (non-durably) written (--innodb-flush-log-at-trx-commit=2). - mysqlbinlog ability to read new binlog format. The code is there, mostly a matter of finding a good way to share code between the server/InnoDB and mysqlbinlog.cc. - Disable all parts of the legacy binlog when --binlog-storage-engine is set (in current code some parts still remain active). - SHOW BINLOG EVENTS. - Binlog checksum and encryption. Will checksum and encrypt the individual pages, not use the legacy per-event checksums or encryption. - Storage engine API. The functionality is implemented, but the design of the API and interfaces might need review and cleanup, ie. maybe parts to be implemented as services (include/service_versions.h etc.) - Lots of smaller ToDo:s, some marked in comments in the code. - There is a --suite=binlog_in_engine, but a lot more testing is neeed. Some parts may be initially not supported in the first release, but implemented later: - Semi-synchronous replication. Should be straight-forward to implement later. Might make sense to use GTID to send back-and-forth instead of filename/offset. AFTER_SYNC no longer makes sense, but could consider implementing AFTER_PREPARE. - External XA, to be implemented following MDEV-32020. The out-of-band mechanism will be used so that XA COMMIT can reference the binlog data and replicate without the need for any transactions to be in XA PREPARED state on the slave or in backups.

1 0

Next step on MDEV-34705, implement binlog in InnoDB
by Kristian Nielsen 30 Jan '25

30 Jan '25

Hi Marko, Time flies, somehow it's already more than a year since our first discussions on implementing the binlog in InnoDB and avoiding the extra fsync() and complexity of the two-phase commit between InnoDB and binlog. But things have progressed, and I have now reached the point where most of the basic groundwork is implemented. Event groups are binlogged to InnoDB tablespaces. Binlog dump thread can read the binlog and send to slave, and replication is working. Large event groups are split into pieces, bounding the amount of data that needs to be atomically written in mini-transactions and at commit time. There are still many details left, but mostly in the server-layer replication code which should be manageable, just will take some time to get completed. I think now is a good time for you to take a first real look at the InnoDB part of the changes, I would really value your input. The main part of the InnoDB code is in two files: 1. handler/handler0binlog.cc for the high-level part that deals mostly with the new binlog file format and interfacing to the server layer. 2. fsp/fsp0binlog.cc for the low-level part that most tightly interacts with the InnoDB mini-transactions and buffer pool. The most interesting part for you to look at is fsp/fsp0binlog.cc (~1k lines), though I'm happy to hear comments on any part of the patch, of course. The code is pushed to GitHub in the branch knielsen_binlog_in_engine: https://github.com/MariaDB/server/commits/knielsen_binlog_in_engine and I've also attached the complete patch. This is my first major patch for InnoDB, so there will undoubtedly be a number of style changes required. But the overall structure of the code should now be close to what I imagine would be the final result, with some pending ToDo steps marked in comments in the code, and detailed in the below list, some of which we discussed a bit already. I hope you will take a look at the patch and let me know of any questions or other things you need from me. Maybe we can also find a chance to discuss further if you will come to FOSDEM start of February, or I could visit sometimes in Finland. - Kristian. ---- Some known outstanding issues in the InnoDB part: - We previously discussed removing some of the page header overhead for binlog tablespaces. Currently the code just leaves alone the first FIL_PAGE_DATA bytes (38) and the last FIL_PAGE_DATA_END (8 IIRC). - We discussed previously to write the current LSN at the start of the tablespace, and use this in recovery to handle that we have only two tablespace IDs that are reused. So we need code in recovery that checks the LSN at the start of the tablespace, and skips redo records with LSN smaller than this. - We want to avoid the double-write buffer for binlog pages, at least for the first page write (most pages will only be written as full pages). You mentioned an idea to completely avoid the double-write buffer and instead do some specific code for recovery in the uncommon case where a partial binlog page is written to disk due to low commit activity. - The flushing of binlog pages to disk currently happens in a dedicated thread in the background. I'd welcome ideas on how to do this differently. It is good to flush binlog pages quickly and re-use their buffer pool entries for something better. Also writing the pages to disk quickly (not necessarily fsync()'ing) makes the data readable by mysqlbinlog. - Checksum and encryption should use the standard InnoDB mechanism. I assume checksum is already handled in the code through using the buffer pool and mini-transactions to read/write pages. Not sure about encryption. I need to implement that the code handles checksum and decryption when reading the pages manually from the file (not through buffer pool).

2 10

2025 Q1 release - near final
by Daniel Black 29 Jan '25

29 Jan '25

The issue where this is tracked is: https://jira.mariadb.org/browse/MDEV-35789 Current blockers: https://jira.mariadb.org/issues/?filter=31019&jql=project%20in%20(mdev)%20A… The "fix version" in the above url list all are 11.4+ so all LTS versions should be stable. Branches bb-10.5-release (LTS) bb-10.6-release (LTS) bb-10.11-release (LTS) bb-11.4-release (LTS) bb-11.7-release (stable for first time) bb-11.8-release (gamma - no LTS announcement made) If there's regression particularly impacting please ensure the issue has a blocker priority, with a good business case reason why it should block, and make as blocks MDEV-35789. In addition to compiling the source yourselves: Containers: quay.ion/mariadb-foundation/mariadb-devel:{release} list: https://quay.io/repository/mariadb-foundation/mariadb-devel?tab=tags note: * pushes to *-release branch also gains the same tag as the main 10.X, 11.X tags. Check commit at the beginning of the server startup to determine actual commit. Repos for testing (packages may not last more than a few weeks at urls): https://ci.mariadb.org/ contains for deb and rpm distros a repo file in the form: bb-{version}-release-latest-{arch}-{distro}-{version}-{deb|rpm}-autobake.{sources|repo}

1 0

Parameterized queries
by E.S. Rosenberg 27 Jan '25

27 Jan '25

Hi, Recently I found out about parameterized queries in MySQL, it seems to have been supported at least since 8.0.x but it is not listed as an incompatibility between mariadb and mysql yet the package spec for COM_QUERY on mariadb does not mention any support for parameterized queries. So is this an incompatibility, some out of date docs or am I just looking at the wrong com_query page? Refs: https://dev.mysql.com/doc/dev/mysql-server/8.0.40/page_protocol_com_query.h… https://mariadb.com/kb/en/com_query/ Thanks! Eli

4 9

Re: 6a4b85ce1cc: MDEV-20281 "[ERROR] Failed to write to mysql.slow_log:" without error reason
by Sergei Golubchik 25 Jan '25

25 Jan '25

Hi, Oleksandr, Ok to push after fixing spagetty (either by fixing a typo or by changing the code pattern as suggested below). On Jan 25, Oleksandr Byelkin wrote: > revision-id: 6a4b85ce1cc (mariadb-10.5.27-202-g6a4b85ce1cc) > parent(s): d261fa5c703 > author: Oleksandr Byelkin > committer: Oleksandr Byelkin > timestamp: 2025-01-24 17:24:17 +0100 > message: > > MDEV-20281 "[ERROR] Failed to write to mysql.slow_log:" without error reason > > Add "backup" (in case of absence issued by error) reasons for failed logging. > > @@ -998,6 +1047,9 @@ bool Log_to_csv_event_handler:: > close_log_table(thd, &open_tables_backup); > thd->time_zone_used= save_time_zone_used; > DBUG_RETURN(result); > +store_err: // spagetty, but reduce size of the code > + cause= "Can't write data (possible incorrect log table structure)"; > + goto err; > } typo. it's "spaghetti" but may be better to avoid it altogether by rewriting as cause= "..."; if (field[0]->store(...)) goto err; if (field[1]->store(...)) goto err; if (field[2]->store(...)) goto err; ... cause= NULL; Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

1 0

Re: 0b403e03939: MDEV-35661 Assertion `!cache_list ||...' fails in online_alter_close_connection
by Sergei Golubchik 22 Jan '25

22 Jan '25

Hi, Nikita, On Jan 19, Nikita Malyavin wrote: > revision-id: 0b403e03939 (mariadb-11.7.1-16-g0b403e03939) > parent(s): 40e502eca4c > author: Nikita Malyavin > committer: Nikita Malyavin > timestamp: 2025-01-03 00:54:52 +0100 > message: > > MDEV-35661 Assertion `!cache_list ||...' fails in > online_alter_close_connection > > After MDEV-35182 (88adcbf3) fix, online_alter still wasn't fully ready > for `XA PREPARE` to be skipped. > > In fact, according to XA standard, even xa_commit() call may happen > without preceding xa_prepare(), so this stage is completely optional. > > This commit removes any relying on xa_prepare() call. The only thing > that will be done there is an equivalent of "statement-level commit", > which basically just flushes a pending rows event to an online alter > cache. > > This is done by always storing an online alter cache in the XA_data > object whenever XA is active, rather than in THD::ha_data. Why? I thought it'd be simpler to keep the cache in the THD and take it off it in commit or rollback. That is, almost as now, but commit/rollback won't rely on prepare being called before. Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 1

Re: 33e36944232: MDEV-30111 InnoDB: Failing assertion: update->n_fields == 0 in row_ins_sec_index_entry_by_modify
by Sergei Golubchik 22 Jan '25

22 Jan '25

Hi, Alexander, On Jan 22, Alexander Barkov wrote: > revision-id: 33e36944232 (mariadb-10.6.20-19-g33e36944232) > parent(s): 25be7da2024 > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2024-11-14 10:19:24 +0400 > message: > > MDEV-30111 InnoDB: Failing assertion: update->n_fields == 0 in row_ins_sec_index_entry_by_modify > > Also fixes: > > MDEV-32190 Index corruption with unique key and nopad collation (without DESC or HASH keys) > > The code in strings/strcoll.inl when comparing an empty string > to a string like 0x0001050001 did not take into account > that the leftmost weight in the latter can be zero, while > there are some more weights can follow the zero weight. > > Rewriting the code to treat the shorter string as smaller than > a longer string. > > --- a/strings/strcoll.inl > +++ b/strings/strcoll.inl > @@ -192,10 +192,10 @@ MY_FUNCTION_NAME(strnncoll)(CHARSET_INFO *cs __attribute__((unused)), > >0 >0 Two weights were scanned, check weight difference. > */ > if (!a_wlen) > - return b_wlen ? -b_weight : 0; > + return b_wlen ? -1 : 0; > > if (!b_wlen) > - return b_is_prefix ? 0 : a_weight; > + return b_is_prefix ? 0 : +1; > > if ((res= (a_weight - b_weight))) > return res; Sorry, I don't understand it. The function returns >0, 0, or <0. It shouldn't matter whether it returns -b_weight or -1, a_weight or +1. Except if the weight can be <= 0 itself. But in that case, if ((res= (a_weight - b_weight))) return res; is rather suspicious too. Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

1 0

On the question on producing bintars going forward
by Kristian Nielsen 22 Jan '25

22 Jan '25

Hi Vicentiu, I read through your proposal for how to handle MariaDB bintars. I know a lot of the old history on this (like from working in the MySQL build team back in 2005), I also know first-hand of the tough spot between what people think is "easily possible" and the practicalities of packaging for real linux systems. So I hope my input can be useful in some way: 1. To me, providing bintars is not important. Nor is providing any binaries for that matter. MariaDB is open source, the most important thing is it is easy to compile. Others can provide binaries as needed, and in fact distros are already doing so very well. So I personally am fine with not releasing bintars going forward. I know binaries are considered important by others. 2. I agree a clear definition is lacking of exactly what binaries should be/provide. My opinion for bintars is that they are a way to use MariaDB easily similar to compiling the source, but without the need for having installed a compiler toolchain. So bintars should be the result of a mostly default compile of the source, on a stable/oldstable linux distro like Debian or some redhat with minimal optional dependencies installed. Most work should go into making default compile do the right thing, as opposed to endless tweaking in build scripts or Buildbot for how bintars are compiled. 3. In your proposal, I think there is too much focus (out of good intentions, I'm sure) about solving all possible problems. No install method is perfect for all use cases. Bintars provide very simple install or even easy-of-use without installing, but they have other limitations. Let us not try to make bintars "perfect". Rather make them simple to produce, and simple to use in the limited scenarious where they can work well. 4. Let us firmly give up on building static binaries. Static binaries do not work well, and have not done so in many years. Here is a reference for a problem with linking libc statically: https://sourceware.org/glibc/wiki/FAQ#Even_statically_linked_programs_need_… And mixing static and dynamic linking creates some very serious problems, see MDEV-32251 and MDEV-25633. We should just build a default `cmake` binary with dynamic linking against the required library dependencies, do the build on a minimal linux where most optional dependencies are not installed. This will then work well on user's systems that have compatible libraries. On systems that have not, there are other options for running MariaDB. 5. I think we should _definitely_ not maintain an infrastructure for building our own static versions of various system libraries / 3rd-party libraries, see also (4). 6. I agree with concerns about too much effort needed to maintain bintars for many years. Let us be pragmatic, make things simple, and only do what can be done with small amount of effort. Build the bintars on stable linux systems for compatibility, and use default installs as much as possible for simplicity. When those systems become too old to easily continue building bintars, just stop building them. Existing users can easily transition to compiling themselves if they really need, thanks to (2); or use another of the different options available for running MariaDB. And finally Vicentiu, let me commend you for taking responsibility of the task of producing bintars and working so constructively on getting to a working result. It is a task the difficulty of which is easily underappreciated. So these are my opinions and suggestions, hope this helps. - Kristian.

2 1

Re: d47b1ebc06d: MDEV-35326: Memory Leak in init_io_cache_ext upon SHUTDOWN
by Sergei Golubchik 10 Jan '25

10 Jan '25

Hi, Oleksandr, On Jan 09, Oleksandr Byelkin wrote: > revision-id: d47b1ebc06d (mariadb-10.5.27-143-gd47b1ebc06d) > parent(s): 3bbbeae7924 > author: Oleksandr Byelkin > committer: Oleksandr Byelkin > timestamp: 2025-01-08 14:14:10 +0100 > message: > > MDEV-35326: Memory Leak in init_io_cache_ext upon SHUTDOWN > > The problems were that: > 1) resources was freed "asimetric" normal execution in send_eof, > in case of error in destructor. > 2) destructor was not called in case of SP for result objects. > (so if the last SP execution ended with error resorces was not > freeded on reinit before execution (cleanup() called before next > execution) and destructor also was not called due to lack of > delete call for the object) > > All result method revised and freeing resources made "symetric". > > Destructor of result object called for SP. > > Added skipped invalidation in case of error in insert. > > Removed misleading naming of reset(thd) (could be mixed with > with reset()) > > diff --git a/sql/sp_head.cc b/sql/sp_head.cc > --- a/sql/sp_head.cc > +++ b/sql/sp_head.cc > @@ -3665,6 +3665,18 @@ int sp_lex_keeper::cursor_reset_lex_and_exec_core(THD *thd, uint *nextp, > return res; > } > > +sp_lex_keeper::~sp_lex_keeper() > +{ > + if (m_lex_resp) > + { > + /* Prevent endless recursion. */ > + m_lex->sphead= NULL; > + if (m_lex->result) you don't need this if(), the standard explicitly says that delete of a NULL pointer is valid. > + delete m_lex->result; > + lex_end(m_lex); > + delete m_lex; > + } > +} > > /* > sp_instr class functions > diff --git a/sql/sql_class.cc b/sql/sql_class.cc > --- a/sql/sql_class.cc > +++ b/sql/sql_class.cc > @@ -3192,10 +3193,22 @@ bool select_send::send_eof() > Handling writing to file > ************************************************************************/ > > +bool select_to_file::free_recources() > +{ > + if (file >= 0) > + { > + (void) end_io_cache(&cache); > + bool error= mysql_file_close(file, MYF(MY_WME)); > + file= -1; > + return error; > + } > + return FALSE; > +} > + > bool select_to_file::send_eof() > { > int error= MY_TEST(end_io_cache(&cache)); > - if (unlikely(mysql_file_close(file, MYF(MY_WME))) || > + if (unlikely(free_recources()) || free_recources() incluses end_io_cache(), you don't need to call it separately above. > unlikely(thd->is_error())) > error= true; > > @@ -5934,7 +5935,15 @@ class select_to_file :public select_result_interceptor { > { path[0]=0; } > ~select_to_file(); > bool send_eof() override; > + void abort_result_set() override; > void cleanup() override; > + /* > + It is separated from cleanup() becaouse: 1. typo 2. if you rename cleanup() to reset_for_next_ps_execution() then you won't need that comment, because, obviously, "freeing resources" is distinct from "resetting for next ps execution" but it's not obvious that "freeing resources" is distinct from "cleanup" > + 1. cleanup can not be called (conventional execution) > + 2. cleanup not only free recources but reset counters (some sends > + counter updates in destructor) > + */ > + bool free_recources(); > }; > > > diff --git a/sql/sql_delete.cc b/sql/sql_delete.cc > --- a/sql/sql_delete.cc > +++ b/sql/sql_delete.cc > @@ -1422,6 +1422,13 @@ void multi_delete::abort_result_set() > { > DBUG_ENTER("multi_delete::abort_result_set"); > > + /**************************************************************************** > + > + NOTE: if you change here be aware that almost the same code is in > + multi_delete::send_eof(). > + > + ***************************************************************************/ Let's talk about it separately. > + > /* the error was handled or nothing deleted and no side effects return */ > if (error_handled || > (!thd->transaction->stmt.modified_non_trans_table && !deleted)) > diff --git a/sql/sql_update.cc b/sql/sql_update.cc > --- a/sql/sql_update.cc > +++ b/sql/sql_update.cc > @@ -2743,6 +2743,13 @@ void multi_update::abort_result_set() > (!thd->transaction->stmt.modified_non_trans_table && !updated))) > return; > > + /**************************************************************************** > + > + NOTE: if you change here be aware that almost the same code is in > + multi_delete::send_eof(). multi_update::send_eof() > + > + ***************************************************************************/ > + > /* Something already updated so we have to invalidate cache */ > if (updated) > query_cache_invalidate3(thd, update_tables, 1); Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

1 0

Re: 331430a6a73: MDEV-33658 2/2 Cannot add a foreign key on a table with a matching long UNIQUE
by Sergei Golubchik 09 Jan '25

09 Jan '25

Hi, Nikita, On Jan 01, Nikita Malyavin wrote: > diff --git a/sql/sql_class.cc b/sql/sql_class.cc > index de688b4cb31..2ffceb3932b 100644 > --- a/sql/sql_class.cc > +++ b/sql/sql_class.cc > @@ -217,6 +218,16 @@ Foreign_key::Foreign_key(const Foreign_key &rhs, MEM_ROOT *mem_root) > > bool is_foreign_key_prefix(Key *a, Key *b) > { > + ha_key_alg a_alg= a->key_create_info.algorithm; > + ha_key_alg b_alg= b->key_create_info.algorithm; > + > + // The real algorithm will be BTREE is none was given by user. Technically, this is not correct, the algorithm will be whatever the engine wants. For MEMORY engine it'll be HASH. But only InnoDB can do foreign keys now, and for InnoDB th3 default key algorithm is BTREE. Let's reflect it in the comment, like, for example (also fixed a typo "is" -> "if") // The real algorithm in InnoDB will be BTREE if none was given by user > + a_alg= a_alg == HA_KEY_ALG_UNDEF ? HA_KEY_ALG_BTREE : a_alg; > + b_alg= b_alg == HA_KEY_ALG_UNDEF ? HA_KEY_ALG_BTREE : b_alg; > + > + if (a_alg != b_alg) > + return false; > + > /* Ensure that 'a' is the generated key */ > if (a->generated) > { > diff --git a/sql/sql_table.cc b/sql/sql_table.cc > index 3f035defbd3..d7aeb5d7cbf 100644 > --- a/sql/sql_table.cc > +++ b/sql/sql_table.cc > @@ -3556,6 +3556,303 @@ key_add_part_check_null(const handler *file, KEY *key_info, > } > > > +static > +my_bool key_check_without_overlaps(THD *thd, HA_CREATE_INFO *create_info, > + Alter_info *alter_info, > + Key &key) > +{ > + DBUG_ENTER("key_check_without_overlaps"); > + if (!key.without_overlaps) > + DBUG_RETURN(FALSE); > + // append_system_key_parts is already called, so we should check all the > + // columns except the last two. > + const auto &period_start= create_info->period_info.period.start; > + const auto &period_end= create_info->period_info.period.end; > + List_iterator<Key_part_spec> part_it_forwarded(key.columns); > + List_iterator<Key_part_spec> part_it(key.columns); > + part_it_forwarded++; > + part_it_forwarded++; > + while (part_it_forwarded++) > + { > + Key_part_spec *key_part= part_it++; > + > + if (period_start.streq(key_part->field_name) > + || period_end.streq(key_part->field_name)) > + { > + my_error(ER_KEY_CONTAINS_PERIOD_FIELDS, MYF(0), key.name.str, > + key_part->field_name.str); > + DBUG_RETURN(TRUE); > + } > + } It was simpler to have this check inside append_system_key_parts(), why did you move it? > + > + if (key.key_create_info.algorithm == HA_KEY_ALG_HASH || > + key.key_create_info.algorithm == HA_KEY_ALG_LONG_HASH) > + > + { > + my_error(ER_KEY_CANT_HAVE_WITHOUT_OVERLAPS, MYF(0), key.name.str); > + DBUG_RETURN(TRUE); > + } <...> > + KEY_CREATE_INFO *key_cinfo= &key.key_create_info; > + > + if (is_hash_field_needed) > + { > + if (key_cinfo->algorithm == HA_KEY_ALG_UNDEF) > + key_cinfo->algorithm= HA_KEY_ALG_LONG_HASH; Perhaps you shouldn't set key algorithm in add_hash_field() then? or, rather replace the assignment with an assert? > + > + if (key_cinfo->algorithm != HA_KEY_ALG_HASH && > + key_cinfo->algorithm != HA_KEY_ALG_LONG_HASH) > + { > + my_error(ER_TOO_LONG_KEY, MYF(0), max_key_length); > + DBUG_RETURN(TRUE); > + } > + } > + } > + } > + DBUG_RETURN(FALSE); > +} > + > + > /* > Preparation for table creation Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 2

MDEV-29744 fix required when merging to 11.4
by Kristian Nielsen 09 Jan '25

09 Jan '25

Hi Sanja, others who might need to merge 10.6->10.11->11.4: The patch for MDEV-29744 in 10.6 needs a fix when merged to 11.4, otherwise test binlog.max_binlog_total_size (at least) will fail. I have attached the required fix. I could not push this fix to 11.4 myself, as branch protection is broken and rejected my push on a false positive. - Kristian.

1 0

Re: 87188cd3309: MDEV-29744: Fix incorrect locking order of LOCK_log/LOCK_commit_ordered and LOCK_global_system_variables
by Sergei Golubchik 08 Jan '25

08 Jan '25

Hi, Kristian, Mostly ok, but I'd prefer not to access max_binlog_size without any protection, so suggest to do something like ulong saved= max_binlog_size; mysql_mutex_unlock(&LOCK_global_system_variables); mysql_bin_log.set_max_size(saved); mysql_mutex_lock(&LOCK_global_system_variables); On Jan 08, Kristian Nielsen wrote: > revision-id: 87188cd3309 (mariadb-10.6.20-47-g87188cd3309) > parent(s): 7372ecc3965 > author: Kristian Nielsen > committer: Kristian Nielsen > timestamp: 2024-12-05 09:51:17 +0100 > message: > > MDEV-29744: Fix incorrect locking order of LOCK_log/LOCK_commit_ordered and LOCK_global_system_variables > > --- a/sql/sys_vars.cc > +++ b/sql/sys_vars.cc > @@ -1660,7 +1660,9 @@ Sys_max_binlog_stmt_cache_size( > > static bool fix_max_binlog_size(sys_var *self, THD *thd, enum_var_type type) > { > + mysql_mutex_unlock(&LOCK_global_system_variables); > mysql_bin_log.set_max_size(max_binlog_size); > + mysql_mutex_lock(&LOCK_global_system_variables); > return false; > } > static Sys_var_on_access_global<Sys_var_ulong, Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 1

Why is innodb-log-file-mmap is available only on amd64/arm64/ppc64el but not other Linux architectures?
by Otto Kekäläinen 08 Jan '25

08 Jan '25

Hi, The documentation at https://mariadb.com/kb/en/innodb-system-variables/#innodb_log_file_mmap says this is supposed to be enabled by default on Linux, regardless of the architecture. However, in my testing it is not available on all architectures. Can somebody help me read the code path of innodb-log-file-mmap to figure out why the visibility/availability of this option it is currently architecture-dependent? ## Context In Debian I maintain an extra test called "configuration-trace" which basically dumps the output of these four commands into a file: /usr/bin/mariadb --print-defaults /usr/bin/mariadb --verbose --help /usr/sbin/mariadbd --print-defaults /usr/sbin/mariadbd --verbose --help Then, using the static file containing the text output of these files, it compares to the output of these commands on a running system to detect if the MariaDB defaults or features randomly changed (for example due to changes in dependencies or platform). This "configuration-trace" is also a handy way to detect what changed upstream in minor MariaDB releases, which aren't supposed to have much changes. The update of the static text output files for MariaDB 11.4.4 were done in https://salsa.debian.org/mariadb-team/mariadb-server/-/commit/e2851fcfca9e8… and they show among others a new variable innodb-log-file-mmap ## Problem The issues I am running into is that this new innodb-log-file-mmap only seems to exist on amd64 and arm64? I discovered this in the Debian CI system (autpkgtests) at https://ci.debian.net/packages/m/mariadb/ which is passing for amd64/arm64/ppc64el, but failing on other platforms: https://ci.debian.net/packages/m/mariadb/unstable/armel/55869348/ https://ci.debian.net/packages/m/mariadb/unstable/armhf/55869349/ https://ci.debian.net/packages/m/mariadb/unstable/i386/55869350/ https://ci.debian.net/packages/m/mariadb/unstable/loong64/55869351/ https://ci.debian.net/packages/m/mariadb/unstable/s390x/55869354/ (The output also shows that mariadbd --verbose --help prints out wsrep-provider twice, once wsrep-provider=ON and once wsrep-provider=none, but that is a different issue.) On systems where it isn't available, shouldn't the option still show up (but as permanently disabled)? It seems odd to have system variables appear or disappear depending on the Linux architecture.

2 8

Re: b16319bdd0f: MDEV-35469 Heap tables are calling mallocs to often
by Sergei Golubchik 30 Dec '24

30 Dec '24

Hi, Monty, > commit b16319bdd0f > Author: Michael Widenius <monty(a)mariadb.org> > AuthorDate: Thu Nov 21 12:28:57 2024 +0200 > Commit: Michael Widenius <monty(a)mariadb.org> > CommitDate: Fri Dec 27 13:53:37 2024 +0200 > > diff --git a/storage/heap/hp_block.c b/storage/heap/hp_block.c > index 324efc8b4af..44bca363746 100644 > --- a/storage/heap/hp_block.c > +++ b/storage/heap/hp_block.c > @@ -75,9 +75,11 @@ int hp_get_new_block(HP_SHARE *info, HP_BLOCK *block, size_t *alloc_length) > Next time we allocate data for X rows. > When level 1 is full, we allocate data for HP_PTRS_IN_NOD at level 2 and 1 > + X rows at level 0. > */ > *alloc_length= (sizeof(HP_PTRS) * ((i == block->levels) ? i : i - 1) + > (ulonglong)block->records_in_block * block->recbuffer); > + /* Alloc in blocks of powers of 2 */ > + *alloc_length= MY_MAX(*alloc_length, block->alloc_size); What will happen to block->alloc_size - *alloc_length bytes? they'll be unused? > if (!(root=(HP_PTRS*) my_malloc(hp_key_memory_HP_PTRS, *alloc_length, > MYF(MY_WME | > (info->internal ? > diff --git a/storage/heap/hp_create.c b/storage/heap/hp_create.c > index c07a1e968c4..29812f08107 100644 > --- a/storage/heap/hp_create.c > +++ b/storage/heap/hp_create.c > @@ -15,11 +15,23 @@ > Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1335 USA */ > > #include "heapdef.h" > +#include <my_bit.h> > > static int keys_compare(void *heap_rb, const void *key1, const void *key2); > static void init_block(HP_BLOCK *block,uint reclength,ulong min_records, > ulong max_records); > > + > +/* > + In how many parts are we going to do allocations of memory and indexes > + If we assigne 1M to the heap table memory, we will allocate roughly > + (1M/16) bytes per allocaiton > +*/ > +int heap_allocation_parts= 16; static. technically, it could be `static const`, But I presume the compiler can figure out the `const` part if the variable is static. And e.g. use >> 4. Otherwise it has to assume it can be modified in some other file. > + > +/* min block allocation */ > +ulong heap_min_allocation_block= 16384; same. static. > + > /* Create a heap table */ > > int heap_create(const char *name, HP_CREATE_INFO *create_info, > @@ -170,7 +182,8 @@ int heap_create(const char *name, HP_CREATE_INFO *create_info, > share->keydef= (HP_KEYDEF*) (share + 1); > share->key_stat_version= 1; > keyseg= (HA_KEYSEG*) (share->keydef + keys); > - init_block(&share->block, visible_offset + 1, min_records, max_records); > + init_block(&share->block, hp_memory_needed_per_row(visible_offset + 1), visible_offset is already MY_MAX(reclength, sizeof (char*)). perhaps you don't need visible_offset and should use hp_memory_needed_per_row(reclength) here? > + min_records, max_records); > /* Fix keys */ > memcpy(share->keydef, keydef, (size_t) (sizeof(keydef[0]) * keys)); > for (i= 0, keyinfo= share->keydef; i < keys; i++, keyinfo++) > @@ -266,44 +279,90 @@ static int keys_compare(void *heap_rb_, const void *key1_, > heap_rb->search_flag, not_used); > } > > + > +/* > + Calculate length needed for storing one row > +*/ > + > +size_t hp_memory_needed_per_row(size_t reclength) > +{ > + /* Data needed for storing record + pointer to records */ > + reclength= MY_MAX(reclength, sizeof(char*) + 1); > + reclength= MY_ALIGN(reclength, sizeof(char*)); > + return reclength; > +} > + > +/* > + Calculate the number of rows that fits into a given memory size > +*/ > + > +ha_rows hp_rows_in_memory(size_t reclength, size_t index_size, > + size_t memory_limit) > +{ > + /* We need one extra byte for the delete flag */ > + reclength= hp_memory_needed_per_row(reclength+1); you have +1 for the delete flag in hp_memory_needed_per_row() > + if ((memory_limit - sizeof(HP_PTRS)) < (index_size + reclength)) > + return 0; /* Wrong arguments */ Why do you need this if()? If its condition is true the expression below will return 0 naturally, you don't need a separate check for that. > + return (ha_rows) ((memory_limit - sizeof(HP_PTRS)) / > + (index_size + reclength)); > +} > + > + > static void init_block(HP_BLOCK *block, uint reclength, ulong min_records, > ulong max_records) > { > diff --git a/storage/heap/hp_write.c b/storage/heap/hp_write.c > index 5469784c8c1..cb079eac757 100644 > --- a/storage/heap/hp_write.c > +++ b/storage/heap/hp_write.c > @@ -145,6 +145,8 @@ static uchar *next_free_record_pos(HP_SHARE *info) > DBUG_PRINT("exit",("Used old position: %p", pos)); > DBUG_RETURN(pos); > } > + if (!(block_pos=(info->records % info->block.records_in_block))) > + { why? > if ((info->records > info->max_records && info->max_records) || > (info->data_length + info->index_length >= info->max_table_size)) > { Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 2

Re: 448e82f9f0d: MDEV-35720 Add query_time to statistics
by Sergei Golubchik 30 Dec '24

30 Dec '24

Hi, Monty, Only a couple of comments about the test case: > commit 448e82f9f0d > Author: Michael Widenius <monty(a)mariadb.org> > Date: Fri Dec 27 19:26:55 2024 +0200 > > diff --git a/mysql-test/main/log_state.test b/mysql-test/main/log_state.test > --- a/mysql-test/main/log_state.test > +++ b/mysql-test/main/log_state.test > @@ -321,6 +321,48 @@ SET GLOBAL general_log_file = @old_general_log_file; > > --enable_ps_protocol > > +--echo # > +--echo # MDEV-35720 Add query_time to statistics > +--echo # > + > +connect (con2,localhost,root,,); > +let $s1=`show status like "query_time"`; > +select sleep(3) into @a; > +let $s2=`show status like "query_time"`; > +let $s3=`show global status like "query_time"`; > +--disable_query_log > +--eval SET @s1= SUBSTRING_INDEX('$s1', '\t', -1); > +--eval SET @s2= SUBSTRING_INDEX('$s2', '\t', -1); > +--eval SET @s3= SUBSTRING_INDEX('$s3', '\t', -1); Eh, that's not how one is supposed to do it. I mean, it works, but the easier approach is let $s1=query_get_value(show status like "query_time", Value); or purely in SQL set @s1=(select variable_value from information_schema.global_status where variable_name='query_time'); > +select @s1 >= 0.00 and @s1 <= 2.00 as "should be true"; > +--eval select @s2 >= 2.00 and @s2 < 10.00 as "should be true"; > +--eval select @s3 >= 3.00 as "should be true"; 1. you don't need eval here 2. I expect this test, like almost all other timing tests will be horribly unstable in buildbot and will be eventually disabled. No, I don't have a recipe for a stable timing test. > +--enable_query_log > + > +# select @s1,@s2,@s3; > +disconnect con2; > +connection default; > + > +delimiter |; > +create procedure p1() > +begin > +select sleep(1) into @a; > +select sleep(2) into @a; > +end| > +delimiter ;| > + > +connect (con2,localhost,root,,); > +call p1(); > +let $s1=`show status like "query_time"`; > +--disable_query_log > +--eval SET @s1= SUBSTRING_INDEX('$s1', '\t', -1); > +--eval select @s1 > 2.00 and @s1 < 10.00 as "should be true"; > +--enable_query_log > + > +# select @s1; > +disconnect con2; > +connection default; > + > # > # Cleanup > # Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 1

Re: dfc1632da73: Change default MEM_ROOT allocations to decrease calls to malloc
by Sergei Golubchik 27 Dec '24

27 Dec '24

Hi, Monty, Please, cherry-pick it into 10.11 and push as bb-10.11-merge. There will be *a lot* of conflicts in memroot code because of your "Added support to MEM_ROOT for write protected memory" change, it'd be best if you resolve your conflicts yourself. On Dec 22, Michael Widenius wrote: > revision-id: dfc1632da73 (mariadb-10.6.20-60-gdfc1632da73) > parent(s): 3297680579f > author: Michael Widenius > committer: Michael Widenius > timestamp: 2024-12-17 10:45:25 +0200 > message: > > Change default MEM_ROOT allocations to decrease calls to malloc ... > --- a/mysys/my_alloc.c > +++ b/mysys/my_alloc.c > @@ -173,7 +180,7 @@ void *alloc_root(MEM_ROOT *mem_root, size_t length) > DBUG_ENTER("alloc_root"); > - DBUG_PRINT("enter",("root: %p", mem_root)); > + DBUG_PRINT("enter",("root: %p length: %ld", mem_root, (long) length)); FYI, my_vsnprintf() supports standard "%zu" to print size_t values. Not important here, of course. > DBUG_ASSERT(alloc_root_inited(mem_root)); Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 1

Re: 2ba3bbb2463: Updated misc_session_status.test to not fail if select does not fail
by Sergei Golubchik 27 Dec '24

27 Dec '24

Hi, Monty, On Dec 22, Michael Widenius wrote: > revision-id: 2ba3bbb2463 (mariadb-10.6.20-61-g2ba3bbb2463) > parent(s): dfc1632da73 > author: Michael Widenius > committer: Michael Widenius > timestamp: 2024-12-17 10:45:25 +0200 > message: > > Updated misc_session_status.test to not fail if select does not fail > > The test with memory restrictions randomly works or fails depending on > server configurations. > As the test was there to just check if the server crashes when run with > small memory configurations, I disabled testing if the query would fail > or not. The test still has its original purpose. This is very questionable. The test would still serve its original purpose indeed if it'll hit OOM at least somewhere. But with your change it's not clear that it'll ever do. The query might as well succeed everywhere and we'll never know. Or it might start succeeding in the future. That is, technically you are right, the test still does its job (presuming you've verified that it hits OOM at least on some builders). But your change basically guarantees that it'll become meaningless in the future. Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 1

Re: 6c5c1d0bf00: MDEV-35469 Heap tables are calling mallocs to often
by Sergei Golubchik 23 Dec '24

23 Dec '24

Hi, Monty, On Dec 21, Michael Widenius wrote: > revision-id: 6c5c1d0bf00 (mariadb-10.6.20-57-g6c5c1d0bf00) > parent(s): 671f80c7388 > author: Michael Widenius > committer: Michael Widenius > timestamp: 2024-12-17 10:45:25 +0200 > message: > > MDEV-35469 Heap tables are calling mallocs to often > > Heap tables are allocated blocks to store rows according to > my_default_record_cache (mapped to the server global variable > read_buffer_size). > This causes performance issues when the record length is big > (> 1000 bytes) and the my_default_record_cache is small. > > Changed to instead split the default heap allocation to 1/16 of the > allowed space and not use my_default_record_cache anymore when creating > the heap. The allocation is also aligned to be just under a power of 2. > > For some test that I have been running, which was using record length=633, > the speed of the query doubled thanks to this change. > > Other things: > - Fixed calculation of max_records passed to hp_create() to take > into account padding between records. > > diff --git a/storage/heap/hp_create.c b/storage/heap/hp_create.c > index c07a1e968c4..fcfcf607822 100644 > --- a/storage/heap/hp_create.c > +++ b/storage/heap/hp_create.c ... > + of the block. > + */ > + records_in_block= > + (my_round_up_to_next_power((uint32) > + (records_in_block * recbuffer + > + sizeof(HP_PTRS) + MALLOC_OVERHEAD)/2) / > + recbuffer); Strictly speaking, it should be (my_round_up_to_next_power(records_in_block * recbuffer + extra) - extra ) / recbuffer And why /2 ? Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 1

Re: 07364d701b3: Added status variable "Max_memory_used" to SHOW STATUS
by Sergei Golubchik 22 Dec '24

22 Dec '24

Hi, Michael, On Dec 22, Michael Widenius wrote: > revision-id: 07364d701b3 (mariadb-10.6.20-64-g07364d701b3) > parent(s): 13fed908291 > author: Michael Widenius > committer: Michael Widenius > timestamp: 2024-12-17 10:45:26 +0200 > message: > > Added status variable "Max_memory_used" to SHOW STATUS > > This shows the maximum memory allocations used by the current connection. > The value for @@global.max_memory_used is 0 as we are not collecting this > value as it would cause a notable performance issue registering this for > all threads for every memory allocation Showing 0 is very confusing. I suggest not to print this variable in SHOW GLOBAL STATUS at all (probably possible, by returning a zero-length SHOW_ARRAY). Or show it as NULL. Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

1 0

Re: 13fed908291: Added more information to errors reported by report_reply_packet()
by Sergei Golubchik 22 Dec '24

22 Dec '24

Hi, Monty, On Dec 22, Michael Widenius wrote: > revision-id: 13fed908291 (mariadb-10.6.20-63-g13fed908291) > parent(s): ca5ac5ac490 > author: Michael Widenius > committer: Michael Widenius > timestamp: 2024-12-17 10:45:26 +0200 > message: > > Added more information to errors reported by report_reply_packet() > > In particular ""Read semi-sync reply magic number error" now prints out > what was wrong with the packet. > > @@ -622,7 +626,8 @@ int Repl_semi_sync_master::report_reply_packet(uint32 server_id, > log_file_len = packet_len - REPLY_BINLOG_NAME_OFFSET; > if (unlikely(log_file_len >= FN_REFLEN)) > { > - sql_print_error("Read semi-sync reply binlog file length too large"); > + sql_print_error("Read semi-sync reply binlog file length too large: %llu", > + (longlong) log_file_pos); You used unsigned "%llu" but cast to a signed longlong. Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

1 0

Can find sponsor: Would anyone be interested in patching mha to be MariaDB gtid compatible?
by highclass99 22 Dec '24

22 Dec '24

Hello, Would anyone be interested in patching mha to be MariaDB gtid compatible? I think I would be able to find a sponsor to pay a bit for it. reference links) https://github.com/yoshinorim/mha4mysql-manager https://github.com/yoshinorim/mha4mysql-node https://github.com/charlesdirk/MariaDB-MHA -> could not get this to work: if anyone got it to work please let me know how to get it to work. Thank you. Have a nice day.

3 3

Re: 3297680579f: Remove end newlines from CHECK TABLE output.
by Sergei Golubchik 21 Dec '24

21 Dec '24

Hi, Michael, On Dec 21, Michael Widenius wrote: > revision-id: 3297680579f (mariadb-10.6.20-59-g3297680579f) > parent(s): 4038ec2cc2c > author: Michael Widenius > committer: Michael Widenius > timestamp: 2024-12-17 10:45:25 +0200 > message: > > Remove end newlines from CHECK TABLE output. > > InnoDB has in some cases (not always) an newline at the end of error > messages. This messes up the CHECK TABLE output. > Fixed CHECK TABLE to remove the end newlines in its result set. I'd rather fix InnoDB to not do that, instead of working around InnoDB bug that garbles the CHECK TABLE output Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

1 0

Re: e0afb5f08b5: cleanup: further replace thd->alloc() with operator new
by Sergei Golubchik 10 Dec '24

10 Dec '24

Hi, Nikita, Despite the commit has in the title, comments below apply to four commits: $ git log --oneline 7d5165894282^..0f6fd23e1604 0f6fd23e160 cleanup: add new(thd) to Sql_alloc and Item e0afb5f08b5 cleanup: further replace thd->alloc() with operator new 8daea2e14b2 cleanup: add operator new(size_t, const THD*) for single-object allocations 7d516589428 cleanup: add operator new[](size_t, const THD*) On Dec 06, Nikita Malyavin wrote: > diff --git a/sql/sql_class.h b/sql/sql_class.h > index aff02df69ec..299c8298606 100644 > --- a/sql/sql_class.h > +++ b/sql/sql_class.h > @@ -1279,11 +1279,5 @@ class Query_arena > { return state == STMT_CONVENTIONAL_EXECUTION; } > > template <typename T=char> > - inline T* alloc(size_t size) const > - { > - return (T*)alloc_root(mem_root, sizeof(T)*size); > - } > - > - template <typename T=char> > inline T* calloc(size_t size) const Hmm, you didn't remove thd->calloc<>() I understand why, but that kind of makes the API inconsistent, perhaps it would've been cleaner to keep thd->alloc<>() and only implement new (thd) and new (arena) for Sql_alloc? May be not only for Sql_alloc. There are 1. pod types. they can be alloc'ed an calloc'ed, I think thd->alloc/thd->calloc is fine for them 2. non-pod types, can be allocated on the heap and on a mem_root. your new (thd)/delete (thd) operators are best for them. 3. non-pod types that can only be allocated on a mem_root. Sql_alloc can be used for them, if it'll disallow non-placement new. I feel like we don't need to make a distinction between 2 and 3, in that case Sql_alloc can simply go away. > { > diff --git a/plugin/feedback/feedback.cc b/plugin/feedback/feedback.cc > index bc8a98c11ae..dcc2e4417e5 100644 > --- a/plugin/feedback/feedback.cc > +++ b/plugin/feedback/feedback.cc > @@ -103,22 +103,22 @@ static COND* make_cond(THD *thd, TABLE_LIST *tables, LEX_STRING *filter) > nrc->resolve_in_table_list_only(tables); > nrc->select_lex= tables->select_lex; > > - res= new (thd) Item_cond_or(thd); > + res= new (thd) Item_cond_or(thd); > if (!res) > return OOM; > > for (; filter->str; filter++) > { > - Item_field *fld= new (thd) Item_field(thd, nrc, db, table, > + Item_field *fld= new (thd) Item_field(thd, nrc, db, table, > field); > - Item_string *pattern= new (thd) Item_string(thd, filter->str, > + Item_string *pattern= new (thd) Item_string(thd, filter->str, > (uint) filter->length, cs); you might want to go through the diff and fix the indentation > - Item_string *escape= new (thd) Item_string(thd, "\\", 1, cs); > + Item_string *escape= new (thd) Item_string(thd, "\\", 1, cs); > > if (!fld || !pattern || !escape) > return OOM; > > - Item_func_like *like= new (thd) Item_func_like(thd, fld, pattern, > + Item_func_like *like= new (thd) Item_func_like(thd, fld, pattern, > escape, 0); > > if (!like) > diff --git a/sql/item.h b/sql/item.h > index 631fd1b19f8..b0cf03c1435 100644 > --- a/sql/item.h > +++ b/sql/item.h > @@ -857,8 +857,9 @@ class Item :public Value_source, > static void *operator new(size_t size); > > public: > - static void *operator new(size_t size, MEM_ROOT *mem_root) throw () > + static void *operator new(size_t size, MEM_ROOT *mem_root) noexcept > { return alloc_root(mem_root, size); } > + static void *operator new(size_t size, const THD *thd) noexcept; > static void operator delete(void *ptr,size_t size) { TRASH_FREE(ptr, size); } > static void operator delete(void *ptr, MEM_ROOT *mem_root) {} Better inherit it from Sql_alloc > diff --git a/sql/item_cmpfunc.cc b/sql/item_cmpfunc.cc > index d86fdf5c5d6..e0adc5c3bb4 100644 > --- a/sql/item_cmpfunc.cc > +++ b/sql/item_cmpfunc.cc > @@ -2007,7 +2007,7 @@ bool Item_func_interval::fix_length_and_dec(THD *thd) > > if (not_null_consts) > { > - intervals= thd->alloc<interval_range>(rows - 1); > + intervals= new(current_thd) interval_range[rows - 1]; current_thd ? > if (!intervals) > return true; > > diff --git a/sql/item_subselect.cc b/sql/item_subselect.cc > index 008384a6590..21364286260 100644 > --- a/sql/item_subselect.cc > +++ b/sql/item_subselect.cc > @@ -6086,8 +6086,8 @@ bool Ordered_key::init(int col_idx) > > // TIMOUR: check for mem allocation err, revert to scan > > - key_columns= thd->alloc<Item_field*>(1); > - compare_pred= thd->alloc<Item_func_lt*>(1); > + key_columns= new(thd) Item_field*();; > + compare_pred= new(thd) Item_func_lt*();; ;; > > key_columns[0]= new (thd) Item_field(thd, tbl->field[col_idx]); > /* Create the predicate (tmp_column[i] < outer_ref[i]). */ > diff --git a/sql/opt_subselect.h b/sql/opt_subselect.h > index 8b4dd1f0092..37902c6eebe 100644 > --- a/sql/opt_subselect.h > +++ b/sql/opt_subselect.h > @@ -357,7 +357,7 @@ uint get_number_of_tables_at_top_level(JOIN *join); > the tuple. > */ > > -class SJ_TMP_TABLE : public Sql_alloc > +class SJ_TMP_TABLE why? > { > public: > /* > diff --git a/sql/sql_alloc.h b/sql/sql_alloc.h > index f5d2d4e8b1a..e643a9f817d 100644 > --- a/sql/sql_alloc.h > +++ b/sql/sql_alloc.h > @@ -23,17 +23,21 @@ > class Sql_alloc > { > public: > - static void *operator new(size_t size) throw () > + static void *operator new(size_t size) noexcept > { > return thd_alloc(_current_thd(), size); > } can this be disallowed completely? > - static void *operator new[](size_t size) throw () > + static void *operator new[](size_t size) noexcept > { > return thd_alloc(_current_thd(), size); > } > - static void *operator new[](size_t size, MEM_ROOT *mem_root) throw () > + > + static void *operator new[](size_t size, const THD *thd) noexcept; > + static void *operator new(size_t size, const THD *thd) noexcept; > + > + static void *operator new[](size_t size, MEM_ROOT *mem_root) noexcept > { return alloc_root(mem_root, size); } > - static void *operator new(size_t size, MEM_ROOT *mem_root) throw() > + static void *operator new(size_t size, MEM_ROOT *mem_root) noexcept > { return alloc_root(mem_root, size); } > static void operator delete(void *ptr, size_t size) { TRASH_FREE(ptr, size); } > static void operator delete(void *, MEM_ROOT *){} > diff --git a/sql/sql_class.cc b/sql/sql_class.cc > index 7847389dabc..37869da63ae 100644 > --- a/sql/sql_class.cc > +++ b/sql/sql_class.cc > @@ -8831,3 +8831,46 @@ LEX_CSTRING make_string(THD *thd, const char *start_ptr, > size_t length= end_ptr - start_ptr; > return {strmake_root(thd->mem_root, start_ptr, length), length}; > } > +void* operator new(size_t size, const Query_arena *thd) noexcept > +{ > + return alloc_root(thd->mem_root, size); > +} > + > +void operator delete[](void *ptr, const Query_arena *thd) noexcept > +{} > + > +void operator delete(void *ptr, const Query_arena *thd) noexcept > +{} if you put them into sql_class.h - will they be inlined? Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 3

MDEV-34705: Storing binlog in InnoDB
by Kristian Nielsen 06 Dec '24

06 Dec '24

I am in the early stages of a project to hopefully implement a new binlog format that is managed transactionally by InnoDB. This is a complex change, and I want to solicit comments early, so I encourage feedback. The TLDR; of this: - The current binlog format stores events separately from InnoDB. This requires a complex and _very_ constly 2-phase commit protocol between the binlog and InnoDB for every commit. - The binlog format is rather naive, and not well suited as a high-performance transactional log. It exposes internal implementation detail to the slaves, and its flat-file nature makes it inflexible to read (for example finding the last event sequential start from the start). Replacing it with something new has the potential to reap many benefits. - This project proposes to extend the storage engine API to allow an engine to implement binlog storage. It will be implemented in InnoDB so that binlog events are stored in InnoDB tablespaces, and the standard InnoDB transactional and recovery mechanisms are used to provide crash safety and optionally durability. - This mechanism will allow full crash-recovery and consistency without any fsync per commit (--innodb-flush-log-at-trx-commit=0), providing huge improvement in throughput for applications that do not require strict durability requirements. And it will reduce fsync requirement to one per commit for durable config, with greatly improved opportunities for group commit. Below a long and somewhat raw write-up of my current state of design. There was some interest expressed in learning more about this work, an interest that I'm very pleased with ;-). The current state of my prototype implementation is available in the branch knielsen_binlog_in_engine: https://github.com/MariaDB/server/commits/knielsen_binlog_in_engine Architectural considerations. The fundamental problem with the binlog is that it functions as a transactional log, but it is separate from the InnoDB transactional log. This is very unoptimal. To ensure consistency in case of crash, a two-phase commit protocol is needed that requires two fsync per commit, which is very costly in terms of performance. And it causes a lot of complexity in the code as well. We should make it so that there is only a single transactional log ("single source of truth") used, at least for the case where only a single storage engine is used (eg. InnoDB) for all tables. Some different options for doing this can be considered (explained in detail below): 1. Implement a new binlog format that is stored in a new InnoDB tablespace type (or more generally in any storage engine that implements the appropriate new API for binlog). 2. Keep the current binlog format, extend the InnoDB redo log format with write-ahead records for binlog writes. Integrate binlog writes with InnoDB checkpointing and buffer pool flush. 3. Store the binlog events directly inside the InnoDB redo log. Implement log archiving of the InnoDB log to preserve old binlog data as long as desired. 4. (Mentioned only for completeness): Implement a general redo log facility on the server level, and change both InnoDB and binlog to use that facility. I am currently working on option (1), inspired by talks I had with Marko Mäkelä at the MariaDB fest last Autumn. He suggested that a new type of tablespace be added to InnoDB which would hold the binlog, and explained that this would require relative small changes in the InnoDB code. I think this is the best option. The InnoDB log and tablespace implementation is very mature, it is very good to be able to re-use it for the binlog. The integration into InnoDB looks to be very clean, and all the existing infrastructure for log write, recovery, checkpointing, buffer pool can be re-used directly. All the low-level detail of how the binlog is stored on disk can be handled efficiently inside the InnoDB code. This option will require a lot of changes in the binlog and replication code, which is the main cost of choosing this option. I have a _lot_ of experience with the MariaDB replication code, and I think this is feasible. And I think there are another set of huge benefits possible from implementing a new binlog format. The existing binlog format is very naive and neither very flexible, nor well suited as a high-performance transactional log. So changing the binlog format is something that should be done eventually anyway. Option (2) would also be feasible I think, and has the advantage of needing less changes to the replication code. Monty mentioned seeing a patch that sounded something like this. The main disadvantage is that I think this will require complex interaction between low-level InnoDB checkpointing code and low-level binlog writing code, which I fear will eventually lead to complex code that will be hard to maintain and will stiffle further innovation on either side. It also doesn't address the other problems with the current binlog format. To me, letting the storage engine control how it stores data transactionally - including the binlog events - is the "right" solution, and worth persuing over a quicker hack that tries to avoid changing code. Option (3) is also interesting. The idea would be that the binlog becomes identical to the InnoDB redo log, storing the binlog data interleaved with the InnoDB log records. This is conceptually simpler than options (1) and (2), in the sense that we don't have to first write-ahead-log a record for modifying binlog data, and then later write the actual data modification to a separate tablespace/file. But the InnoDB log is cyclic, and old data is overwritten as the log LSN wraps over the end. Thus, this option (3) will require implementing log archiving in InnoDB, which I am not sure how much work will be required. Either the binlog data could be archived by copying out asynchroneously to persistent binlog files, which becomes somewhat similar to option (1) or (2) then perhaps. Or InnoDB could be changed to create a new log file at wrap-over, renaming the old one to archive it, though this will leave the binlog data somewhat "dilluted", mixed up with InnoDB internal redo records. One pragmatic reason I've chosen option (1) over this option (3) for now is that I am more comfortable with doing large changes to replication code than to InnoDB code. Also, option (3) would more drastically change the way users can work with the binlog files, as a large amount of the most recent data would be sitting inside the InnoDB redo log and not in dedicated binlog files, unlike options (1) and (2). Still, option (3) is an interesting idea and has the potential to reduce the write amplification that will occur from (1) and (2). So I'm eager to hear any suggestions around this. Option (4) I mention only for completeness. Somehow, the "clean" way to handle transactional logging would be if the server layer provides a general logging service, and all the components would then use this for logging. InnoDB and the binlog, but also all the other logs or log-like things we have, eg. Aria, DDL log, partitioning, etc. However, I don't think it makes sense to try to discard the whole InnoDB write-ahead logging code and replace it with some new implementation done on the server level. And I'm not even sure it makes sense from a theoretical point of view - somehow, the logging method used is intrinsically a property of the storage engine; it seems dubious if a general server-level log facility could be designed that would be optimal for both InnoDB and RocksDB, for example. High-level design. Introduce a new option --binlog-storage-engine=innodb. Extend the storage engine API with the option for an engine to provide a binlog implementation, supported by InnoDB initially. Initially the option cannot be changed dynamically. When --binlog-storage-engine=innodb, the binlog files will use a different format, stored as a new type of InnoDB tablespace. It should support that the old part of the binlog is the legacy format and the new is the InnoDB format, to facilitate migrations. Maybe also support going back again, though this is less critical initially. A goal is to clean up some of the unnecessary complexity of the legacy binlog format. The old format will be supported for the foreseeable future, so the new format can break backwards compatibility. For example, the name of binlog files will be fixed, binlog-<NNNNNN>.ibb, so each file can be identified solely by its number NNNNNN. The option to decide the directory in which to store the binlog files should be supported though. I think we can require the slaves to use GTID when --binlog-storage-engine=innodb is enabled on the master. This way we can avoid exposing slaves to internal implementation details of the binlog, and the slaves no longer need to update the file relay-log.info at every commit. We should also be able to get rid of the Format_description_log_event and the Rotate_log_event at the start of every binlog file, so that the slave can view the events from the master as one linear stream and not care about how the data is split into separate binlog files on the master. The binlog format will be page based. This will allow pre-allocating the binlog files and efficiently writing them page by page. And it will be possible to access the files without scanning them sequentially from the start; eg. we can find the last event by reading the last page of the file, binary-searching for the last used page if the binlog file is partially written. The GTID indexes can be stored inside the same tablespace as the binlog data (eg. at the end of the tablespace), avoiding the need for a separate index file. GTID index design is still mostly TBD, but I think it can be implemented so that indexes are updated transactionally as part of the InnoDB commit and no separate crash recovery of GTID indexes is needed. With GTID indexes being guaranteed available, we can use them to obtain the GTID state at the start of each binlog file, and avoid the need for the Gtid_list_log_event at the start of the binlog. With a page-based log with extensible format, metadata can be added to the binlog that is only used on the master without having to introduce new replication events that are not relevant to the replication on the slave. This can be used eg. to eliminate the Binlog_checkpoint_log_event, for example. Possibly the binlog checkpoint mechanism can be completely avoided for the --binlog-storage-engine=innodb case, since there is no more 2-phase commit needed in the common case of an InnoDB-only transaction. The existing binlog checksum and encryption will no longer be used, instead the standard InnoDB checksums and encryption will be reused. Implementation plan. The storage engine API must be extended to provide a facilities for writing to the binlog and for reading from the binlog. The design of the API is TBD, should be discussed to try to make it generic and suitable for other engines than InnoDB. When writing to binlog in InnoDB, the central idea is to use the same mini-transaction (mtr) that marks the transaction as committed in InnoDB. This is what makes the binlog data guaranteed consistent with the InnoDB table data without the need for two-phase commit. The initial version I think can use the existing binlog group commit framework; this will simplify the implementation. This will thus keep the LOCK_commit_ordered and the queue_for_group_commit() mechanisms. Later work can then be to see if this can be even further improved in terms of scalability. I want to implement that large event groups can be split into multiple chunks in the binlog that no longer need to be consecutive. The final chunk will be the one that contains the GTID and marks the event group binlogged; this final chunk will then refer back to the other parts. This way, a large transactions can be binlogged without stalling the writing of other (smaller) transactions in parallel, which is a bottleneck in the legacy binlog. And it avoids problems with exceeding the maximum size of an mtr. In the first version, I think the binlog reader in the dump thread will collect and concatenate the different chunks before sending them to the slave, so simplify the initial implementation. But a later change could allow the slave to receive the different chunks interleaved between different event groups. This can even eventually allow the slave to speculatively execute events even before the transaction has committed on the master, to potentially reduce slave lag to less than the time of even a single transaction; this would be a generalisation of the --binlog-alter-two-phase feature. The binlog will be stored in separate tablespace files, each of size --max-binlog-size. The binlog files will be pre-allocated by a background thread. Since event groups can spill over to the next file, each file can be a fixed size, and hopefully also the rotate events will no longer be necessary. A busy server will quickly cycle through tablespace files, and we want to avoid "using up" a new tablespace ID for each file (InnoDB tablespace IDs are limited to 2**32 and are not reused). Two system tablespace IDs will be reserved for the binlog, and new binlog files will alternate between them. This way, the currently written binlog can be active while the previous one is being flushed to disk and the remaining writes checkpointed. Once the previous log has been flushed completely, its tablespace ID can be re-used for the next, pre-allocated binlog file and be ready for becoming active. This way, the switching to the next binlog file should be able to occur seamlessly, without any stalls, as long as page flushing can keep up. The flushing of binlog pages will be prioritised, to avoid stalling binlog writes, to free up buffer pool pages that can be used more efficiently than holding binlog data, and to quickly make the binlog files readable from outside the server (eg. with mysqlbinlog). The binlog dump thread that reads from the binlog and sends the data to slaves will use a binlog reading API implemented in InnoDB. I will prefer to read directly from the binlog files, in order to reduce pressure on the InnoDB buffer pool etc. A slave that connects from an old position back in time may need to read a lot of old data from the binlogs; there is little value in loading this data into the buffer pool, evicting other more useful pages. The reader can lookup in the InnoDB buffer pool with the BUF_GET_IF_IN_POOL flag. This way, the data can be accessed from the buffer pool if it is present. If not present, we can be sure that the data will be in the data file, and can read it from the file directly. If the data is known to be already flushed to disk before the specific binlog position, then the buffer pool lookup can be skipped altogether. The mysqlbinlog program will need to be extended somehow to be able to read from an InnoDB tablespace (or in general other storage engine). I think this means mysqlbinlog needs some kind of engine plugin facility for reading binlog. Note that the -read-from-remote-server option will also be available to read data from a mysqlbinlog that doesn't the new format, or to read the very newest data before it gets written to disk. Final words. This is still an early draft of the feature, as it is being worked on and refined; things are subject to change. And this also means that things are *open* to change, according to any suggestions with a good rationale.

2 7

Review of MDEV-6017 Add support for Indexes on Expressions
by Michael Widenius 03 Dec '24

03 Dec '24

Hi! Here is the review for MDEV-6017 Add support for Indexes on Expressions commit c1e7574c799163f5d8a26ca8e0f91af6c91f7077 Instead of adding bool function virtual bool vcol_subst_analyzer(uchar **) { return false; } Why not add Item_base_t flag: IS_SIMPLE_BOOL_FUNC This takes one bit instead of increasing ALL created item base structures with 8 bytes. As we have 100+ items, adding a virtual function increases code space with 1K or more + the functions. You can also replace vcol_subst_analyzer() with a top level function: bool Item::vcol_subst_analyzer/( { return (type() == Item::FUNC_ITEM && (((Item_func*) this)->bitmap_bit() & (BITMAP_EQ | BITMAP_EQUAL | BITMAP_LT | BITMAP_GT | BITMAP_LE | BITMAP_GE | BITMAP_BETWEEN | BITMAP_IN | BITMAP_ISNULL | BITMAP_ISNOTNULL ) } For this to work, you would have to add the last two bitmaps to enum Bitmap in item_func.h @@ -3272,6 +3292,7 @@ class Item_cond :public Item_bool_func bool excl_dep_on_table(table_map tab_map) override; bool excl_dep_on_grouping_fields(st_select_lex *sel) override; + bool vcol_subst_analyzer(uchar **) override { return true; } I assume you need the above to be able to traverse AND and OR expressions for WHERE. Please add a comment that this is the intent. If you decide to use the bitmaps, add an explanation why AND and OR is in the vcol_subst_analyzer list. ---- +void collect_indexed_vcols_for_table(TABLE *tbl, List<Field> *vcol_fields) +{ Please use table instead of tbl. We use table as a varible much more than tbl: ((/my/maria-10.5/sql)) grep tbl *.* | wc 1143 5371 73297 ((/my/maria-10.5/sql)) grep table *.* | wc 40886 236754 2832202 + for (uint i=0; i < tbl->s->keys; i++) + { + // note: we could also support histograms here + if (!tbl->keys_in_use_for_query.is_set(i)) + continue; I wonder if it would be a good idea to extend bitmap to have an iterator like we have for tables where we only loop over the set bits. From sql_bitmap.h: class Table_map_iterator { ulonglong bmp; public: Table_map_iterator(ulonglong t): bmp(t){} uint next_bit() { if (!bmp) return BITMAP_END; uint bit= my_find_first_bit(bmp); bmp &= ~(1ULL << bit); return bit; } int operator++(int) { return next_bit(); } enum { BITMAP_END= 64 }; }; Would be quite trival to do also for keys. This would help tables with many keys and also would allows us to remove the 'call' to if (!tbl->keys_in_use_for_query.is_set(i)) We could also do an optimized version where we implement bitmap_get_next_set() where we do not have to reset the previous bits for each call. --------- + if (field->vcol_info) + vcol_fields->push_back(field); You have to take care of error handling here. Please change the functions collect_indexed_vcols_for_table(), collect_indexed_vcols_for_join() and other functions that calls push_back() to bool and return 1 in case of error. + item->top_level_compile(ctx->thd, + &Item::vcol_subst_analyzer, &yes, + &Item::vcol_subst_transformer, (uchar*)ctx); You also have to check the return of the above for errors. Another option is to just check thd->fatal_error() as this will be set in case of OOM. .... + { + if (auto nested_join= table->nested_join) + subst_vcols_in_join_list(ctx, &nested_join->join_list); PLEASE do not use auto. I want to know the type while reading and debugging the code! auto can also introduce bugs if a type changes. + @todo: this cannot handle VIEW columns currently. View columns are wrapped + in Item_direct_view_ref objects which do not comsider themselves equal to + Item_field objects they're wrapping. Please add this also to the commit comment. + /* + Do not do the substitution if the datatypes do not match. + Virtual column's datatype can be different from the expression, for + example: + + col3 INT AS (CONCAT(col1, col2)) + + Do not allow substitutions that change the semantics of comparison. + At the moment, we require that datatypes are the same. + This probably could be relaxed. + + For strings: we allow two cases: + - vcol_expr and vcol_field have the same collation. + - vcol_field has the same collation as the comparison collation. + + (Note: MySQL calls resolve_type() after it has done the substitution. + This can potentially update the comparator. The idea is that this + shouldn't be necessary as we do not want to change the comparator. + Changing the comparator will change the semantics of the condition, + our point is that this must not happen) + */ Move to a function comment. It is hard to read read the function code with this long comment in the middle. + if (vcol_expr->type_handler_for_comparison() != + vcol_field->type_handler_for_comparison() || + vcol_expr->maybe_null() != vcol_field->maybe_null()) + fail_cause="type mismatch"; Why cannot we handle this case if vcol_expr->maybe_null is 0 and vcol_field->maybe_null() is set? In this case we know that the vcol_field's we are going to encounter can never be null, which should satisfy our requirements. + Item_field *itf= new (thd->mem_root) Item_field(thd, vcol_field); + if (!itf) + return false; Better to return true in case of errors, like we do in most of the other code. Remove and a blank line instead + //Item *const_expr; ... remove + //const_expr= args[0]; ----- +Item* Item_func_in::vcol_subst_transformer(THD *thd, uchar *arg) +{ + Vcol_subst_context *ctx= (Vcol_subst_context*)arg; + if (!compatible_types_scalar_bisection_possible()) Add comment /* Check that all arguments inside IN() are constants */ Regards, Monty

1 0

Re: 5b6b23563bf: MDEV-34979 generate SBOM from server builds
by Sergei Golubchik 02 Dec '24

02 Dec '24

Hi, Vladislav, Thanks! See comments below On Dec 02, Vladislav Vaintroub wrote: > revision-id: 5b6b23563bf (mariadb-11.7.1-5-g5b6b23563bf) > parent(s): eff9c198e32 > author: Vladislav Vaintroub > committer: Vladislav Vaintroub > timestamp: 2024-11-25 19:04:18 +0100 > message: > > MDEV-34979 generate SBOM from server builds > > This commit adds the capability to generate a Software Bill of Materials > (SBOM) from server builds. > > It introduces a new WITH_SBOM variable, which defaults to ON for package > builds (i.e if BUILD_CONFIG is used) and to OFF otherwise. > > When enabled, the build process will produce an sbom.json document in > CycloneDX format, capturing information about various dependencies, > which is gathered from various sources. > > We use git submodule information and CMake external projects properties > to gather version information for 3rd party code, but also handle > dependencies if external code is part of our repository > (zlib, or Connect storage engine's minizip) > > The SBOM document is stored in the root build directory in sbom.json file, > but is not currently installed. > > diff --git a/CMakeLists.txt b/CMakeLists.txt > --- a/CMakeLists.txt > +++ b/CMakeLists.txt > @@ -581,6 +581,13 @@ ENDIF() > > INCLUDE(build_depends) > > +INCLUDE(CMakeDependentOption) > +CMAKE_DEPENDENT_OPTION(WITH_SBOM "Generate Software Bill of Materials (SBOM)" OFF "BUILD_CONFIG" ON) > +IF(WITH_SBOM) > + INCLUDE(generate_sbom) > + GENERATE_SBOM() > +ENDIF() This doesn't work for me, I mean, cmake -DBUILD_CONFIG=mysql_release . does not generate the file, but cmake -DBUILD_CONFIG=mysql_release . -DWITH_SBOM=ON does. May be it'd be more robust to use OPTION() and set it from cmake/build_configuration/mysql_release.cmake? Like it's done for other options. > + > INCLUDE(CPack) > > IF(WIN32 AND SIGNCODE) > diff --git a/cmake/generate_submodule_info.cmake b/cmake/generate_submodule_info.cmake > --- /dev/null > +++ b/cmake/generate_submodule_info.cmake > @@ -0,0 +1,73 @@ ... > + FOREACH(s ${out_list}) > + IF(${counter} EQUAL 0) > + IF (NOT("${s}" MATCHES "Entering '")) does it depend on the locale? > + MESSAGE(FATAL "Unexpected output ${outvar}") > + ENDIF() > + STRING(LENGTH "${s}" slen) > + MATH(EXPR substr_len "${slen} - 11") ... > diff --git a/cmake/generate_sbom.cmake b/cmake/generate_sbom.cmake > --- /dev/null > +++ b/cmake/generate_sbom.cmake > @@ -0,0 +1,240 @@ > +INCLUDE(generate_submodule_info) > +INCLUDE(ExternalProject) > + > +# Remove all matches from a cmake list > +FUNCTION (REMOVE_ALL_MATCHES list_var_name match_str) > + SET(new_list) > + FOREACH(el ${${list_var_name}}) > + IF(NOT("${el}" MATCHES "${match_str}")) > + LIST(APPEND new_list ${el}) > + ENDIF() > + ENDFOREACH() > + SET(${list_var_name} ${new_list} PARENT_SCOPE) > +ENDFUNCTION() you could've used LIST(FILTER ...) instead. (but not LIST(REMOVE_ITEM ...) which is 3.16+) > + > +# Extract user name and repository name from a github URL. > +FUNCTION (EXTRACT_REPO_NAME_AND_USER repo_url repo_name_var repo_user_var) > + IF(repo_url MATCHES "^git@") > + # normalize to https-style URLs > + STRING(REGEX REPLACE "^git@([^:]+):(.*)$" "https://\\1/\\2" repo_url "${repo_url}") I don't think we have that, is it even tested? :) note, I'm not saying it isn't needed, better to have it than not. > + ENDIF() > + # Extract the repository user > + STRING(REGEX REPLACE "https://([^/]+)/([^/]+)/.*" "\\2" repo_user "${repo_url}") > + > + STRING(REGEX REPLACE ".*/([^/]*)$" "\\1" repo_name "${repo_url}") > + STRING(REGEX REPLACE "\\.git$" "" repo_name "${repo_name}") > + > + SET(${repo_name_var} ${repo_name} PARENT_SCOPE) > + SET(${repo_user_var} ${repo_user} PARENT_SCOPE) > +ENDFUNCTION() > + > +# Add a known 3rd party dependency for SBOM generation > +# Currently used for "vendored" (part of our repository) source code we know about > +# such as zlib, as well ExternalProject_Add() projects > +MACRO(ADD_THIRD_PARTY_DEPENDENCY name url tag rev version description) > + LIST(FIND ALL_THIRD_PARTY ${name} idx) > + IF (idx GREATER -1) > + MESSAGE(FATAL_ERROR "${name} is already in ALL_THIRD_PARTY") > + ENDIF() > + SET(${name}_URL ${url}) > + SET(${name}_TAG ${tag}) > + SET(${name}_REVISION ${rev}) > + SET(${name}_DESCRIPTION "${description}") > + SET(${name}_VERSION "${version}") > + LIST(APPEND ALL_THIRD_PARTY ${name}) > +ENDMACRO() > + > +# Match third party component with supplier > +# CyclonDX documentation says it is > +# "The organization that supplied the component. > +# The supplier may often be the manufacturer, but may also be a distributor or repackager." > +# > +# Perhaps it can always be "MariaDB", but security team recommendation is different > +# more towards "author" > +FUNCTION (sbom_get_supplier repo_name repo_user varname) why a function? just curious > + IF("${repo_name_SUPPLIER}") > + SET(${varname} "${repo_name_SUPPLIER}" PARENT_SCOPE) ... > + # It is possible to provide EXTRA_SBOM_DEPENDENCIES > + # and accompanying per-dependency data, to extend generared sbom > + # document. > + # Example below injects an extra "ncurses" dependency using several > + # command line parameters for CMake. > + # -DEXTRA_SBOM_DEPENDENCIES=ncurses > + # -Dncurses_URL=https://github.com/mirror/ncurses > + # -Dncurses_TAG=v6.4 > + # -Dncurses_VERSION=6.4 > + # -Dncurses_DESCRIPTION="A fake extra dependency" > + SET(ALL_THIRD_PARTY ${ALL_SUBMODULES} ${EXTRA_SBOM_DEPENDENCIES}) okay. Though I personally wouldn't use ALL_SUBMODULES and EXTRA_SBOM_DEPENDENCIES and instead would scan GET_CMAKE_PROPERTY(VARS VARIABLES) for *_TAG or something > + > + IF(TARGET ha_connect OR TARGET connect) > + ADD_THIRD_PARTY_DEPENDENCY(minizip > + "https://github.com/zlib-ng/minizip-ng" > + "" 252588f "" > + "Vendored minizip, inside connect storage engine, storage/connect/zip.c et al") did you verify that our file matches the content of the file from that repo at that commit? also, a though - how can we prevent our minigzip.c from being modified without updaing the dependency? Suggestion, check the commit hash, like git log -n1 --format=%h storage/connect/zip.c > + ENDIF() > + > + IF(TARGET libfmt) > + ExternalProject_GET_PROPERTY(libfmt URL) > + STRING(REGEX MATCH "[0-9]+\\.[0-9]+\\.[0-9]+" libmt_TAG "${URL}") > + ADD_THIRD_PARTY_DEPENDENCY(libfmt > + "https://github.com/fmtlib/fmt" > + "${libmt_TAG}" "${libmt_TAG}" "${libmt_TAG}" > + "header only library, used in server") > + ENDIF() I wonder, wouldn't it be simpler and more robust to overwrite or wrap ExternalProject() to do the above automatically? (using the same name or a different one like MyExternalProject) ... Also, I noticed that the server entry gets "purl": "pkg:github//@5b6b23563bf" perhaps it should have a repo name here? Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

1 0

Re: 534dda1234b: MDEV-35522: MariaDB Audit does not detect all DCLs forms when masking password
by Sergei Golubchik 02 Dec '24

02 Dec '24

Hi, Oleksandr, Ok to push. But do you no longer delete server_audit.log after the test? On Dec 02, Oleksandr Byelkin wrote: > revision-id: 534dda1234b (mariadb-10.5.27-64-g534dda1234b) > parent(s): fdb6db6b47f > author: Oleksandr Byelkin > committer: Oleksandr Byelkin > timestamp: 2024-11-29 11:45:01 +0100 > message: > > MDEV-35522: MariaDB Audit does not detect all DCLs forms when masking password > > 1. skip OR REPLACE > (to make it filter_query_type made recursive) > > 2. skip SET STATEMENT ... FOR before checking > statements with passwords > > diff --git a/mysql-test/suite/plugins/t/server_audit_pwd_mask.test b/mysql-test/suite/plugins/t/server_audit_pwd_mask.test > --- a/mysql-test/suite/plugins/t/server_audit_pwd_mask.test > +++ b/mysql-test/suite/plugins/t/server_audit_pwd_mask.test > --echo # cleaunup > +DROP SERVER s1; > DROP USER u1; > DROP USER u2; > set global server_audit_logging=off; > ---remove_file $SEARCH_FILE > +##--remove_file $SEARCH_FILE why? > --disable_warnings > UNINSTALL PLUGIN ed25519; > UNINSTALL PLUGIN server_audit; > --enable_warnings Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 1

Re: ff3d06f6601: MDEV-22217 Make OS character sets "utf8" and "utf-8" map to MariaDB character set "utf8mb4"
by Sergei Golubchik 16 Nov '24

16 Nov '24

Hi, Alexander, Of course, there can be no comments about the code change here. But the question is, is it something we want to do? 1. Is utf8/utf-8 locale indeed 4-byte utf8? 2. Why is hard-coding to utf8mb4 better than having it dependent on UTF8_IS_UTF8MB3 ? On Nov 16, Alexander Barkov wrote: > revision-id: ff3d06f6601 (mariadb-11.6.1-44-gff3d06f6601) > parent(s): 4b6922a315f > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2024-10-28 14:11:40 +0400 > message: > > MDEV-22217 Make OS character sets "utf8" and "utf-8" map to MariaDB character set "utf8mb4" > > diff --git a/mysql-test/main/mysql_locale_posix.test b/mysql-test/main/mysql_locale_posix.test > index c047b7b1ae9..b37d04baca2 100644 > --- a/mysql-test/main/mysql_locale_posix.test > +++ b/mysql-test/main/mysql_locale_posix.test > @@ -203,3 +203,20 @@ DROP DATABASE `Ã³`; > + > +--exec LC_ALL=en_US.utf8 $MYSQL --default-character-set=auto test -e "SELECT @@character_set_client" > +--exec LC_ALL=en_US.utf8 $MYSQL --default-character-set=auto test -e "SELECT @@character_set_connection" > +--exec LC_ALL=en_US.utf8 $MYSQL --default-character-set=auto test -e "SELECT @@character_set_results" may be do it in one SELECT? or in SHOW VARIABLES LIKE 'character_set%' ? to have it a tiny bit faster? Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

1 0

2024 Q4 release part 2 (11.6.2, 11.7,1)
by Daniel Black 11 Nov '24

11 Nov '24

All, The 11.6.2 and 11.7.1 releases are now built: * 11.6.2 d8dad8c3b54cd09fefce7bc3b9749f427eed9709 (GA) * 11.7.1 74331a48dd3dadfe5041f3440b81a7183e9fd4a8 release notes still work in progress: * https://mariadb.com/kb/en/mariadb-11-6-2-release-notes/ * 11.7.1 still being prepared but will be 11.6.2 like + the features of 11.7.0 release notes + Vector search, + bugs fixes since 11.7.0. Both are rolling releases and once GA, like 11.6.2, will be the last of the release series. containers for testing are available: * quay.io/mariadb-foundation/mariadb-devel:11.6 * quay.io/mariadb-foundation/mariadb-devel:11.7 Build packages for RPM/Deb distros repo/source files: https://ci.mariadb.org/ * 11.6.2 - bb-11.6-release-latest* * 11.7.1 - bb-11.7-release-latest* Sources: * https://ci.mariadb.org/51755/ (11.6.2) * https://ci.mariadb.org/51770/ (11.7.1) Now begins the process of loading packages to mirrors and Docker Official Images releases.

1 0

Contribute to MariaDB compatibility in open source projects?
by Robert Silén 06 Nov '24

06 Nov '24

All, We have been working on community outreach as described in an earlier blog Improving MariaDB support in open source projects <https://mariadb.org/improving-mariadb-support-in-open-source-projects/>. Many projects have made efforts to improve their MariaDB compatibility with or without our help, and also documented compatibility more visibly in READMEs and documentation. However, we at the Foundation don't have the knowhow or resources to help out everybody, so I thought I'd share a list of some issues. Would somebody like to contribute to any of them? Any comment on the issues, or code contributions to them would be much appreciated. Thank you. Do share any thoughts or actions on this. Robert Silén, community advocacy at MariaDB Foundation robert.silen(a)mariadb.org - https://github.com/SOCI/soci/issues/1150: add MariaDB to README - https://github.com/tinode/chat/pull/924: Help set up CI for MariaDB (and MySQL) - https://github.com/inspircd/inspircd-docs/issues/239: tls and srv not available in MariaDB - https://github.com/umami-software/umami/discussions/2943: BIN_TO_UUID() not available in MariaDB (MDEV-15854 <https://jira.mariadb.org/browse/MDEV-15854>) - https://github.com/diesel-rs/diesel/issues/1882: Create a distinct backend for MariaDB - https://github.com/grafana/grafana: Create a dedicated DataStore for MariaDB - https://github.com/drizzle-team/drizzle-orm/pull/1692: WIP: Add support for Maria DB (core API implementation) - https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1079753: Create mariadb specific dbconfig-mariadb - https://core.trac.wordpress.org/ticket/60037: Fix version checking in next release - https://gitlab.com/zoph-project/zoph/-/issues/267: give a thumbs up to their own addition of MariaDB to documentation (preferably recommend MariaDB)

2 1

Re: e8021aaf28e: auth: refactor MYSQL_AUTHENTICATION_INTERFACE_VERSION naming
by Sergei Golubchik 28 Oct '24

28 Oct '24

Hi, Nikita, This looks very confusing, this is not how versions work. What are you trying to achieve by that? On Oct 07, Nikita Malyavin wrote: > revision-id: e8021aaf28e (mariadb-11.6.1-9-ge8021aaf28e) > parent(s): a5b80531fbb > author: Nikita Malyavin > committer: Nikita Malyavin > timestamp: 2024-10-04 00:16:17 +0200 > message: > > auth: refactor MYSQL_AUTHENTICATION_INTERFACE_VERSION naming > > Rename MYSQL_AUTHENTICATION_INTERFACE_VERSION to > MYSQL_AUTH_INTERFACE_VERSION_2_02. > This is the current latest version, and it will be fixed to all auth plugins > for current implementation. > MYSQL_AUTHENTICATION_INTERFACE_VERSION will stay for compatibility with 3-rd > party auth plugins and will be fixed to V. 2.02. > Add MYSQL_AUTHENTICATION_INTERFACE_VERSION_LAST that will be updated as needed. > Add MYSQL_AUTH_INTERFACE_MAJOR_MASK to identify major version. > Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 5

Re: b17308e254a: MDEV-34854 Parsec sends garbage when using an empty password
by Sergei Golubchik 28 Oct '24

28 Oct '24

Hi, Nikita, On Oct 07, Nikita Malyavin wrote: > revision-id: b17308e254a (mariadb-11.6.1-10-gb17308e254a) > parent(s): e8021aaf28e > author: Nikita Malyavin > committer: Nikita Malyavin > timestamp: 2024-10-04 01:33:47 +0200 > message: > > MDEV-34854 Parsec sends garbage when using an empty password > > When an empty password is set, the server doesn't call > st_mysql_auth::hash_password and leaves > MYSQL_SERVER_AUTH_INFO::auth_string empty. > > Fix: > generate hashes for empty passwords as well. This breaks some auth > plugins, so we increment interface version and do it only from Auth V. > 2.03. > > Some empty passwords could be already stored with no though. The user "with no though" ? > will have to call SET PASSWORD once again, anyway the authentication > wouldn't have worked for such password. ok, I presume you mean ed25519 only, because mysql_native_password worked with an empty password and it generates an empty hash for it. > diff --git a/mysql-test/suite/plugins/r/parsec.result b/mysql-test/suite/plugins/r/parsec.result > index 512c066e2d7..b7e3537af29 100644 > --- a/mysql-test/suite/plugins/r/parsec.result > +++ b/mysql-test/suite/plugins/r/parsec.result ... let's add ed25519 test too, for completeness. > diff --git a/sql/sql_acl.cc b/sql/sql_acl.cc > index 2722ea2ea19..ba05a5656c5 100644 > --- a/sql/sql_acl.cc > +++ b/sql/sql_acl.cc > @@ -2402,7 +2402,10 @@ static int set_user_auth(THD *thd, const LEX_CSTRING &user, > res= ER_NOT_VALID_PASSWORD; > goto end; > } > - if (pwtext.length) > + > + // Starting from version 2.03 we also generate hash for empty passwords. > + if ((info->interface_version >= MYSQL_AUTH_INTERFACE_VERSION_2_03 I don't understand this MYSQL_AUTH_INTERFACE_VERSION_2_03 thing. First, again, that's not how a version is supposed to work. Second, this empty-password change isn't a change in the API. You can simply start calling hash_password() for empty passwords and it'll just work. I've tried :) > + && pwtext.str) || pwtext.length) > { > if (info->hash_password) > { Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 3

2024 Q4 release
by Daniel Black 27 Oct '24

27 Oct '24

All, Just letting you know that we're in the progress of getting the releases ready. The issue where this is tracked is: * https://jira.mariadb.org/browse/MDEV-35030 Branches being released are at the tips of these branches: bb-10.5-release (LTS) bb-10.6-release (LTS) bb-10.11-release (LTS) bb-11.2-release bb-11.4-release (LTS) bb-11.6-release bb-11.7-release So far the code is merged up to bb-11.4-release. Later branches will be updated, check history before using. There are two blocking bugs on bb-11.2-release onwards affecting compilation of the columnstore and connect storage engine. In addition to compiling the source yourselves; Containers: quay.ion/mariadb-foundation/mariadb-devel:{release} list: https://quay.io/repository/mariadb-foundation/mariadb-devel?tab=tags note: * 11.4 is from the 11.4 branch rather than bb-11.4-release, * 11.2 didn't build due to blocker bugs). Repos for testing: https://ci.mariadb.org/ contains for deb and rpm distros a repo file in the form: bb-{version}-release-latest-{arch}-{distro}-{version}-{deb|rpm}-autobake.{sources|repo} Please report any serious regressions with a Blocker priority bug, and mark this issue blocking MDEV-35030.

1 0

Re: 692ca003f8c: MDEV-33658 2/2 Cannot add a foreign key on a table with a matching long UNIQUE
by Sergei Golubchik 23 Oct '24

23 Oct '24

Hi, Nikita, On Oct 18, Nikita Malyavin wrote: > revision-id: 692ca003f8c (mariadb-10.5.26-169-g692ca003f8c) > parent(s): ba988513ec7 > author: Nikita Malyavin > committer: Nikita Malyavin > timestamp: 2024-10-13 19:33:14 +0200 > message: > > MDEV-33658 2/2 Cannot add a foreign key on a table with a matching long UNIQUE > > Cannot add a foreign key on a table with a long UNIQUE multi-column index, that > contains a foreign key as a prefix. > > Check that index algorithms match during the "generated" keys duplicate removal. > > diff --git a/mysql-test/main/long_unique_bugs.test b/mysql-test/main/long_unique_bugs.test > index 2f85328bda3..08b49b6c5da 100644 > --- a/mysql-test/main/long_unique_bugs.test > +++ b/mysql-test/main/long_unique_bugs.test > @@ -706,4 +706,19 @@ alter table t1 enable keys; > insert into t1 values (2); > drop table t1; > > +--echo # > +--echo # Cannot add a foreign key on a table with a long UNIQUE multi-column > +--echo # index, that contains a foreign key as a prefix. > +--echo # > +create table a (id int(19) primary key) engine = innodb; > +create table b (id int primary key, > + long_text varchar(1000), > + constraint unique_b unique key (id, long_text) > + ) engine = innodb default charset utf8mb4; > + > +alter table b add constraint b_fk_ida foreign key (id) references a (id); > +show create table b; > +drop table b; > +drop table a; Two problems with this test case. A minor one - it doesn't start with the --echo # MDEV-33658 Cannot add a foreign key on a table with a matching long UNIQUE line. Please add it. A more serious problem - it doesn't fail on a vanilla 10.5. The original set of SQL commands from MDEV-33658 fails with "Foreign key constraint is incorrectly formed", but your test doesn't. You might've oversimplified it. Please, fix the test case to fail on vanilla 10.5 with the above error. > + > --echo # End of 10.5 tests > diff --git a/sql/sql_class.cc b/sql/sql_class.cc > index 99ba9e3587b..3d01b9d5584 100644 > --- a/sql/sql_class.cc > +++ b/sql/sql_class.cc > @@ -203,7 +203,7 @@ Foreign_key::Foreign_key(const Foreign_key &rhs, MEM_ROOT *mem_root) > > /* > Test if a foreign key (= generated key) is a prefix of the given key > - (ignoring key name, key type and order of columns) > + (ignoring key name, key type and order of columns, but minding the algorithm) it doesn't ignore the order of columns and "key type" is basically algorithm, so the above can simply be "(ignoring key name)" > > NOTES: > This is only used to test if an index for a FOREIGN KEY exists > @@ -218,6 +218,17 @@ Foreign_key::Foreign_key(const Foreign_key &rhs, MEM_ROOT *mem_root) > > bool is_foreign_key_prefix(Key *a, Key *b) > { > + ha_key_alg a_alg= a->key_create_info.algorithm; > + ha_key_alg b_alg= b->key_create_info.algorithm; > + > + bool is_a_btree= a_alg == HA_KEY_ALG_BTREE || a_alg == HA_KEY_ALG_UNDEF; > + bool is_b_btree= b_alg == HA_KEY_ALG_BTREE || b_alg == HA_KEY_ALG_UNDEF; > + > + // Either the algorithms are total match, or they both are btree > + // in order to be able to prefix > + if (a_alg != b_alg && !(is_a_btree && is_b_btree)) > + return false; I don't see why algorithms can be not b-tree, if they match. What use case did you have in mind? > + > /* Ensure that 'a' is the generated key */ > if (a->generated) > { > Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 1

Update documentation for non-blocking client library
by Kristian Nielsen 22 Oct '24

22 Oct '24

Hi Georg, Following MDEV-34859, I wrote a small update for the documentation on the new cmake option -DWITH_BOOST_CONTEXT=ON cmake option, patch included below. However, I am not sure how I can include it in the documentation? Feel free to just take it yourself, or let me know a preferred way to submit it. - Kristian. diff --git a/configuration_options.md b/configuration_options.md index 6e8da57..726b4f8 100644 --- a/configuration_options.md +++ b/configuration_options.md @@ -27,6 +27,11 @@ If you want to use a different generator, e.g. for nmake on Windows, you need to |WITH_OPENSSL|ON| Possible values are ON or OFF. Not supported anymore since Connector/C 3.0| |WITH_SSL|SCHANNEL (windows), otherwise OPENSSL|Specifies type of TLS/SSL library. E.g. GNUTLS, OPENSSL or SCHANNEL (Windows only). OFF disables TLS/SSL functionality| +### Non-blocking client library options +| Option | Default | Description | +|-|-|- +|WITH_BOOST_CONTEXT|OFF| Use `boost::context` instead of `ucontext` for the non-blocking client API. Can be used to build the non-blocking API on platforms that do not have `ucontext`. Note that on x86_64 (aka amd64), i386, and aarch64 (aka arm64), a native implementation is always used over `ucontext` or `boost::context`. (Added in 3.3.12)| + ### Client plugins Client plugins can be configured as dynamic plugins (DYNAMIC) or built-in plugins (STATIC) by specifying the plugin name followed by suffix `_PLUGIN_TYPE` as key, and `DYNAMIC` or `STATIC` as value.

2 1

Community request: audit the PARSEC plugin
by Andrew Hutchings 16 Oct '24

16 Oct '24

Hi all, There is a new authentication plugin called PARSEC that is set to become the default authentication plugin in MariaDB soon (MDEV-32618 and MDEV-12320). The current documentation for it can be found here: https://mariadb.com/kb/en/authentication-plugin-parsec/ I personally think that using a single salt and sending it to the client could be a problem. I do not work in the security industry and the documentation has a few errors in it and pieces missing, so I'm unclear how certain things work. I think the ext-salt requirements will likely add complications to load balancing scenarios and replication. This is a request for anyone in the community to please audit this plugin before it becomes the default. Once it is the default, it will be difficult to change it. Kind Regards -- Andrew (LinuxJedi) Hutchings Chief Contributions Officer MariaDB Foundation

2 2

Re: 778c67f7af0: MDEV-18151 Skipped error returning for GRANT/SET PASSWORD
by Sergei Golubchik 09 Oct '24

09 Oct '24

Hi, Oleksandr, On Oct 09, Oleksandr Byelkin wrote: > revision-id: 778c67f7af0 (mariadb-10.5.26-154-g778c67f7af0) > parent(s): e9c999caf48 > author: Oleksandr Byelkin > committer: Oleksandr Byelkin > timestamp: 2024-10-04 09:28:46 +0200 > message: > > MDEV-18151 Skipped error returning for GRANT/SET PASSWORD > > Make error issueing for GRANT and SET PASSWORD the same. typo. "issuing" > Report errors wich were skipped before. > > diff --git a/sql/sql_acl.cc b/sql/sql_acl.cc > --- a/sql/sql_acl.cc > +++ b/sql/sql_acl.cc > @@ -2370,6 +2369,21 @@ static int set_user_auth(THD *thd, const LEX_CSTRING &user, > > res= 0; > end: > + switch(res) > + { > + case ER_OUTOFMEMORY: // should be reported by my_malloc > + case ER_NOT_VALID_PASSWORD: // should be reported by plugin > + case 0: > + break; > + case ER_SET_PASSWORD_AUTH_PLUGIN: > + my_error(res, MYF(0), plugin_name); > + break; > + case ER_PASSWD_LENGTH: > + my_error(ER_PASSWD_LENGTH, MYF(0), SCRAMBLED_PASSWORD_CHAR_LENGTH); this is also reported by the plugin. The caller doesn't know what the correct length is, it's not SCRAMBLED_PASSWORD_CHAR_LENGTH, this value is different for different plugins. > + break; > + default: > + DBUG_ASSERT(0); > + } > if (unlock_plugin) > plugin_unlock(thd, plugin); > return res; > Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

1 0

PRs with one approval pending the second
by Otto Kekäläinen 04 Oct '24

04 Oct '24

Hi *all* developers, Hopefully someone else than Andrew or Sergei replies as a project of the size of MariaDB should have multiple developers who are capable of reviewing PRs, and not overload a few. These PRs already have one approval, but would need a second to be merged. If you feel you are knowledgeable in the domain of the PRs, please help out by reviewing them. Thanks! https://github.com/MariaDB/server/pull/3225: MDEV-29955: add ${PCRE_LIBRARY_PATH} to link directories https://github.com/MariaDB/server/pull/3540: Add MTRs to further test coverage around TLS functionality

1 0

PRs with two approvals but not merged
by Otto Kekäläinen 04 Oct '24

04 Oct '24

Hi, Here are two PRs that have already two approvals, yet they have not been merged for many weeks. Can you please state explicitly on the PR as a comment what the submitter needs to do, or if there is nothing to do, just merge it? * https://github.com/MariaDB/server/pull/3208 * https://github.com/MariaDB/server/pull/2556 Thanks

3 3

More extensive major version upgrade testing
by Otto Kekäläinen 04 Oct '24

04 Oct '24

Hi, If any dev here is interested in the topic of ensuring the major version upgrades (and essentially running mariadb-upgrade) continues to work flawlessly and regressions are less likely to surface unnoticed, please review and consider adding your approvals to https://github.com/MariaDB/server/pull/3503 and share feedback on WIP https://github.com/MariaDB/server/pull/3563. Thanks!

2 1

Re: [PATCH] MDEV-28413 System versioning is incorrect in Spider
by Yuchen Pei 03 Oct '24

03 Oct '24

Hi Aleksey, Thanks for the patch. Please see below for some initial review comments. There are some todos for myself noted below which I will look into tomorrow. > From b4465fc6b51dfc4e645569201e6026ec6405367b Mon Sep 17 00:00:00 2001 > From: Aleksey Midenkov <midenok(a)gmail.com> > Date: Mon, 30 Sep 2024 22:26:51 +0300 > Subject: [PATCH] MDEV-28413 System versioning is incorrect in Spider > Basic functioning of system versioning in Spider: UPDATE, DELETE, > DELETE HISTORY, AS OF, FROM .. TO, BETWEEN .. AND. > More testing should be done for: > LOAD DATA, joins, derived tables, multi-update/delete and what else? > Note that we are still using HA_CAN_DIRECT_UPDATE_AND_DELETE, but not > for periods in DELETE (fwiw there is condition > !table_list->has_period(), but it is missing in UDPATE). > --- > .../mysql-test/spider/r/versioning.result | 142 ++++++++++++++++ > .../mysql-test/spider/t/versioning.test | 151 ++++++++++++++++++ > storage/spider/spd_db_conn.cc | 19 ++- > storage/spider/spd_db_include.h | 1 + > storage/spider/spd_db_mysql.cc | 25 ++- > 5 files changed, 335 insertions(+), 3 deletions(-) > create mode 100644 storage/spider/mysql-test/spider/r/versioning.result > create mode 100644 storage/spider/mysql-test/spider/t/versioning.test > [... 149 lines elided] > diff --git a/storage/spider/mysql-test/spider/t/versioning.test b/storage/spider/mysql-test/spider/t/versioning.test > new file mode 100644 > index 00000000000..6093169bff4 > --- /dev/null > +++ b/storage/spider/mysql-test/spider/t/versioning.test > @@ -0,0 +1,151 @@ > +--source include/have_innodb.inc > +--source suite/versioning/common.inc > + > +install plugin spider soname 'ha_spider'; For init the spider plugin in spider tests we commonly do --disable_query_log --disable_result_log --source ../t/test_init.inc --enable_result_log --enable_query_log > +set spider_same_server_link= on; > + > +--replace_result $MASTER_1_MYPORT MASTER_1_MYPORT > +eval create server s foreign data wrapper mysql options > +(host '127.0.0.1', database 'test', user 'root', port $MASTER_1_MYPORT); Use evalp instead of eval and you can save the --replace_result line > + > + > +create table t1 ( > + x int, y int > +) with system versioning; > + > +create table t2 ( > + x int, y int, > + row_start timestamp(6) as row start invisible, > + row_end timestamp(6) as row end invisible, > + period for system_time (row_start, row_end) > +) with system versioning; > + > +create or replace table t1_sp ( > + x int, y int) > +engine=spider comment='wrapper "mysql", srv "s", table "t1"' > +with system versioning; > + > +create or replace table t2_sp ( > + x int, y int, > + row_start timestamp(6) as row start invisible, > + row_end timestamp(6) as row end invisible, > + period for system_time (row_start, row_end)) > +engine=spider comment='wrapper "mysql", srv "s", table "t2"' > +with system versioning; > + > +--echo # Timestamps are not propagated (subject of MDEV-16546, but not for implicit system fields) > +set timestamp= unix_timestamp('2000-01-01 00:00:00'); I tried executing directly using select spider_direct_sql('set timestamp= unix_timestamp("2000-01-01 00:00:00");', '', 'srv "s"'); But then I get extra rows in subsequent select queries and I think it is caused by the timezones. It could be a separate task to fix this, but we should warn users about it. > +insert into t1_sp values (1, 1), (2, 2); It may be obvious to you, but I find the timestamps at times hard to keep track of, so I suggest adding a comment to each DML statement indicating the time, as well as the select row_start / row_end statements. For example, change the line above to insert into t1_sp values (1, 1), (2, 2); # T1 > +insert into t2_sp values (1, 1), (2, 2); and change this line to insert into t2_sp values (1, 1), (2, 2); # T2 > +--sleep 0.01 > + > +set timestamp= unix_timestamp('2000-01-01 00:11:11'); > +--echo # check dml generating history > +delete from t1_sp where x = 1; add # T3 > +--sleep 0.01 > +delete from t2_sp where x = 1; # T4 > +--sleep 0.01 > + > +select row_end into @t1 from t2 for system_time all where x = 1; And so we can annotate this line with the timestamp above, like so: select row_end into @t1 from t2 for system_time all where x = 1; # @t1 = T4 > + > +update t1_sp set y= y + 1; # T5 > +select row_start into @t2 from t1 where x = 2; # @t2 = T5 > +--sleep 0.01 > +update t2_sp set y= y + 1; # T6 With the annotations I suggested, results from the following block of selects becomes easier to understand. > + > +select * from t1_sp; > +select * from t2_sp; > +select * from t1_sp for system_time as of timestamp @t1; > +select * from t2_sp for system_time as of timestamp @t1; > +select * from t1_sp for system_time as of timestamp @t2; > +select * from t2_sp for system_time as of timestamp @t2; > +select * from t1_sp for system_time from timestamp @t1 to timestamp @t2; > +select * from t2_sp for system_time from timestamp @t1 to timestamp @t2; > +select * from t1_sp for system_time between timestamp @t1 and timestamp @t2; > +select * from t2_sp for system_time between timestamp @t1 and timestamp @t2; > + > +--echo # spider cannot call functions, it fails in spider_create_group_by_handler() > + # at: > + # > + # 1555 if (spider_db_print_item_type(item, null, spider, null, null, 0, > + # 1556 roop_count, true, fields_arg)) > + # 1557 { > + # 1558 dbug_print("info",("spider dbton_id=%d can't create select", roop_count)); > + # 1559 spider_clear_bit(dbton_bitmap, roop_count); > + # 1560 keep_going = false; > + # 1561 break; > + # 1562 } > + # This is fine. The Spider group-by-handler (GBH) is not meant to handle every situation. Either remove this block starting from the --echo, or make it more succinct by replacing it with something like (Note that GBH supports certain functions like trim): # The spider group-by-handler does not support the check_row function > +--echo # but somehow it doesn't fail the query and continues without FOR SYSTEM_TIME ALL clause > +select *, check_row(row_start, row_end) from t1_sp for system_time all order by row_end, y, x; > +select *, check_row(row_start, row_end) from t2_sp for system_time all order by row_end, y, x; I will check why the execution without the group-by-handler results in wrong results. > +--echo # SELECT from base table works as usual: > +select *, check_row(row_start, row_end) from t1 for system_time all order by row_end, y, x; > +select *, check_row(row_start, row_end) from t2 for system_time all order by row_end, y, x; > + > +--echo # here it works nice (append_table_list() has the chance to add FOR SYSTEM_TIME ALL) > + > +# Usage: SHOW_TIMESTAMPS=1 mtr versioning.spider Shouldn't the test name be spider.versioning? > +if ($SHOW_TIMESTAMPS) > +{ > + select @t1, @t2; > +} > +if (!$SHOW_TIMESTAMPS) > +{ > +--replace_regex /\d\d\d\d-\d\d-\d\d \d\d:\d\d:\d\d.\d\d\d\d\d\d/TIMESTAMP/ > +} > +select *, row_start, row_end from t1_sp for system_time all order by row_end, y, x; > + > +if (!$SHOW_TIMESTAMPS) > +{ > +--replace_regex /\d\d\d\d-\d\d-\d\d \d\d:\d\d:\d\d.\d\d\d\d\d\d/TIMESTAMP/ > +} > +select *, row_start, row_end from t2_sp for system_time all order by row_end, y, x; > + > +delete history from t1_sp before system_time @t1; > +delete history from t2_sp before system_time @t1; > + > +if (!$SHOW_TIMESTAMPS) > +{ > +--replace_regex /\d\d\d\d-\d\d-\d\d \d\d:\d\d:\d\d.\d\d\d\d\d\d/TIMESTAMP/ > +} > +select *, row_start, row_end from t1_sp for system_time all order by row_end, y, x; > + > +if (!$SHOW_TIMESTAMPS) > +{ > +--replace_regex /\d\d\d\d-\d\d-\d\d \d\d:\d\d:\d\d.\d\d\d\d\d\d/TIMESTAMP/ > +} > +select *, row_start, row_end from t2_sp for system_time all order by row_end, y, x; > + > +select * from t1 for system_time all; > +select * from t2 for system_time all; > + > +delete history from t1_sp; > +delete history from t2_sp; > + > +if (!$SHOW_TIMESTAMPS) > +{ > +--replace_regex /\d\d\d\d-\d\d-\d\d \d\d:\d\d:\d\d.\d\d\d\d\d\d/TIMESTAMP/ > +} > +select *, row_start, row_end from t1_sp for system_time all order by row_end, y, x; > + > +if (!$SHOW_TIMESTAMPS) > +{ > +--replace_regex /\d\d\d\d-\d\d-\d\d \d\d:\d\d:\d\d.\d\d\d\d\d\d/TIMESTAMP/ > +} > +select *, row_start, row_end from t2_sp for system_time all order by row_end, y, x; > + > +select * from t1 for system_time all; > +select * from t2 for system_time all; > + > +drop tables t1, t2, t1_sp, t2_sp; > +set timestamp= default; > +--source suite/versioning/common_finish.inc > + > +uninstall plugin spider; > +drop table mysql.spider_link_failed_log, mysql.spider_link_mon_servers, mysql.spider_tables, mysql.spider_table_crd, mysql.spider_table_position_for_recovery, mysql.spider_table_sts, mysql.spider_xa, mysql.spider_xa_failed_log, mysql.spider_xa_member; > +drop function spider_direct_sql; > +drop function spider_bg_direct_sql; > +drop function spider_ping_table; > +drop function spider_copy_tables; > +drop function spider_flush_table_mon_cache; For deinit the spider plugin in spider tests we commonly do --disable_query_log --disable_result_log --source ../t/test_deinit.inc --enable_result_log --enable_query_log Please replace the lines from "uninstall plugin spider;" to the end of the test with this block. > diff --git a/storage/spider/spd_db_conn.cc b/storage/spider/spd_db_conn.cc > index 8750e87237e..896fce8adf5 100644 > --- a/storage/spider/spd_db_conn.cc > +++ b/storage/spider/spd_db_conn.cc > @@ -1515,9 +1515,8 @@ int spider_db_append_key_where_internal( > ) { > SPIDER_RESULT_LIST *result_list = &spider->result_list; > SPIDER_SHARE *share = spider->share; > -#ifndef DBUG_OFF > TABLE *table = spider->get_table(); > -#endif Why remove the #ifndef and #endif? > [... 99 lines elided] Best, Yuchen

1 1

[Maria-developers] MariaDB // Temporal tables
by Robert Palm 02 Oct '24

02 Oct '24

2 2

Help wanted: Encryption Plugin
by ssss.aaa2002＠gmail.com 19 Sep '24

19 Sep '24

Hello I wonder if anyone can help me out for this case I'm trying to build an encryption plugin for mariaDB and I built this https://gist.github.com/Mahmoudgalalz/ae1de7a57f8b9f37056efc2018575ce4 I managed to compile it, but I'm running into an issue when I'm trying to use encrypted option MariaDB [test]> CREATE TABLE test_encrypted (id INT PRIMARY KEY, sensitive_data VARCHAR(255)) ENGINE=InnoDB ENCRYPTED=YES; ERROR 2013 (HY000): Lost connection to server during query MariaDB [test]> CREATE TABLE t1 (a VARCHAR(8)) ENGINE=InnoDB ENCRYPTED=YES ENCRYPTION_KEY_ID=1; ERROR 2006 (HY000): Server has gone away No connection. Trying to reconnect... Connection id: 31 Current database: test ERROR 2013 (HY000): Lost connection to server during query so if anyone can help me out to figure out what is wrong and should I correct or consider would be very much appreciated.

2 4

Help wanted: Encryption Plugin
by ssss.aaa2002＠gmail.com 19 Sep '24

19 Sep '24

Hello I wonder if anyone can help me out for this case I'm trying to build an encryption plugin for mariaDB and I built this https://gist.github.com/Mahmoudgalalz/ae1de7a57f8b9f37056efc2018575ce4 I managed to compile it, but I'm running into an issue when I'm trying to use encrypted option MariaDB [test]> CREATE TABLE test_encrypted (id INT PRIMARY KEY, sensitive_data VARCHAR(255)) ENGINE=InnoDB ENCRYPTED=YES; ERROR 2013 (HY000): Lost connection to server during query MariaDB [test]> CREATE TABLE t1 (a VARCHAR(8)) ENGINE=InnoDB ENCRYPTED=YES ENCRYPTION_KEY_ID=1; ERROR 2006 (HY000): Server has gone away No connection. Trying to reconnect... Connection id: 31 Current database: test ERROR 2013 (HY000): Lost connection to server during query so if anyone can help me out to figure out what is wrong and should I correct or consider would be very much appreciated.

1 0

Re: [PATCH] MDEV-32014 Rename binlog cache temporary file to binlog file for large transaction
by Kristian Nielsen 10 Sep '24

10 Sep '24

> From: Libing Song <anders.slb(a)alibaba-inc.com> > > Description > =========== > When a transaction commits, it copies the binlog events from > binlog cache to binlog file. Very large transactions > (eg. gigabytes) can stall other transactions for a long time > because the data is copied while holding LOCK_log, which blocks > other commits from binlogging. Hi Libing Song, Here is my full review of the patch for MDEV-32014 Rename binlog cache temporary file to binlog file for large transaction. First let me say that thanks, reviewing this patch was a pleasant surprise. The binlog code that you are integrating this new feature in is complex to work with, but overall I think you have managed to do it a pretty clean way. That is great work, and I think a lot of effort must have gone into getting it this way. I have a number of comments and requests for changes below, so I wanted to give this overall good impression up-front. Also thanks to Brandon for doing first review. My apologies if I am repeating something already discussed, but Github pull request reviews of non-trivial patches are almost impossible to read, with reviews being split in fragments, patches disappearing after force-pushes, etc. The root cause for the problem being addressed by the patch is a fundamental limitation in the current binlog format. It requires all transactions to be binlogged as a single event group, using consecutive bytes in a single binlog file. Thus, while writing the large transaction to the binlog during commit, all other transaction commits will have to wait before they can commit. Eventually, we should move towards an enhanced binlog format that doesn't have this limitation, so that transactions can be partially binlogged during execution, avoiding this bottlenect. There are a number of other problems caused by this limitation that are not addressed by this patch. Changing the binlog format is a complex task not easily done, so this patch should be seen as a temporary solution to one specific bottleneck, until if/when this underlying limitation can hopefully be removed. First some overall comments: As Brandon also remarked, the name "binlog_free_flush" does not describe the feature very clearly. I would suggest to use the term "rotate" in the name, as the concept of binlog rotation is well known in MariaDB replication, and the patch effectively performs large commits by doing it as a binlog rotation instead of a simple write to the existing binlog file. So I suggest to name the option --binlog-commit-by-rotate-threshold. And I suggest in the code (names and comments) to use the phrase "rotate_by_commit" instead of "free_flush". The patch enables the feature by default, with a threshold of 128MB. We need to consider if that is appropriate, or if the feature should be disabled by default. Reasons for disabling by default: - The feature causes some binlog files to be cut short of the configured --max-binlog-size, which will be a surprise to users. - The feature adds an overhead of 4096 pre-allocated bytes at the start of every trx cache (though the overhead is minimal). - A default of 128MB is somewhat arbitrary. - Releasing this feature disabled by default is somewhat less risky wrt. regressions for users that upgrade. Reasons to enable by default: - Most users will otherwise not know the feature exists, and will thus not benefit from this work. - It will get much less testing "in the wild". So the decision will be a compromise between these two. If I had to make the choice, I would default to this feature being disabled. But I'm open to hear opinions the other way. The patch calls the trx_group_commit_leader() function. That is ackward, as commit-by-rotate does not actually participate in group commit. But the problem is with the existing function trx_group_commit_leader(), which does too much in a single function. So please do a small refactoring of trx_group_commit_leader() and keep only the first part of that function that fetches the group commit queue. Move the second part to a new function trx_group_commit_with_engines() that is called by trx_group_commit_leader() passing the "queue" and "last_in_queue" values. And assert in trx_group_commit_with_engines() that LOCK_log is held on entry and comment that this function will release the lock. Then you can from your Binlog_free_flush::commit() function call into trx_group_commit_with_engines() directly, avoiding the need for adding a true/false flag, and also avoiding that the trx_group_commit_leader() function is sometimes called with LOCK_log held and sometimes not. Same situation with MYSQL_BIN_LOG::write_transaction_to_binlog_events(), this function is also doing too much and makes integrating your code ackward. Please refactor the MYSQL_BIN_LOG::write_transaction_to_binlog_events() function something like this: bool MYSQL_BIN_LOG::write_transaction_to_binlog_events(group_commit_entry *entry) { if (binlog_commit_by_rotate.should_commit_by_rotate(entry) && binlog_commit_by_rotate.commit(entry)) { /* Commit done by renaming the trx/stmt cache. */ } else { if (write_transaction_with_group_commit(entry)) return true; /* Error */ } if (unlikely(entry->error)) { write_transaction_handle_error(entry); return 1; } return 0; } Introducing two new functions: bool write_transaction_with_group_commit(group_commit_entry *entry) This function contains the code from write_transaction_to_binlog_events() up to and including this: if (likely(!entry->error)) return entry->thd->wait_for_prior_commit(); You don't need the wait_for_prior_commit() in the commit-by-rotate case, as you already did this in Binlog_free_flush::commit()). You also don't need to handle the (!opt_optimize_thread_scheduling) case for commit-by-rotate, this is old code that is being deprecated. Just add a condition in should_free_flush() to skip if opt_optimize_thread_scheduling is set. Other new function: void write_transaction_handle_error(group_commit_entry *entry) This function contains the remainder of the write_transaction_to_binlog_events() function starting from "switch (entry->error) ...". This way, the commit-by-rotate case is cleanly separated from the normal group commit case and the code of write_transaction_to_binlog_events() becomes much cleaner. Now follows more detailed comments on specific parts of the patch: > diff --git a/sql/log.cc b/sql/log.cc > index 34f9ad745fc..3dc57b21c05 100644 > --- a/sql/log.cc > +++ b/sql/log.cc > @@ -163,6 +163,111 @@ static SHOW_VAR binlog_status_vars_detail[]= > + void set_reserved_bytes(uint32 header_len) > + { > + // Gtid event length > + header_len+= LOG_EVENT_HEADER_LEN + Gtid_log_event::max_data_length + > + BINLOG_CHECKSUM_LEN; > + header_len= header_len - (header_len % IO_SIZE) + IO_SIZE; This has an off-by-one error, it will round up IO_SIZE to 2*IO_SIZE. Use eg. header_len = (header_len + (IO_SIZE - 1)) & ~(IO_SIZE - 1) You can use static_assert((IO_SIZE & (IO_SIZE - 1)) == 0, "power of two") if you want to catch if anyone would ever change IO_SIZE to be not a power of two. +private: + Binlog_free_flush &operator=(const Binlog_free_flush &); + Binlog_free_flush(const Binlog_free_flush &); Add before this a comment "singleton object, prevent copying by mistake". > + char m_cache_dir[FN_REFLEN]; This member appears to be unused? > + /** The cache_data which will be renamed to binlog. */ > + binlog_cache_data *m_cache_data{nullptr}; Also appears unused. Maybe add a comment that m_entry and m_replaced are protected by LOCK_log? > +static Binlog_free_flush binlog_free_flush; > +ulonglong opt_binlog_free_flush_threshold= 10 * 1024 * 1024; This initializes the value to 10 MB, but the default value in sys_vars.cc is 128MB, so that seems inconsistent. > @@ -4126,8 +4238,7 @@ bool MYSQL_BIN_LOG::open(const char *log_name, > /* Notify the io thread that binlog is rotated to a new file */ > if (is_relay_log) > signal_relay_log_update(); > - else > - update_binlog_end_pos(); > + Why do you remove update_binlog_end_pos() here? Doesn't the binlog dump threads need this update? > @@ -8703,7 +8821,16 @@ MYSQL_BIN_LOG::queue_for_group_commit(group_commit_entry *orig_entry) > bool > MYSQL_BIN_LOG::write_transaction_to_binlog_events(group_commit_entry *entry) > { > - int is_leader= queue_for_group_commit(entry); > + int is_leader; > + > + if (binlog_free_flush.should_free_flush(entry) && > + binlog_free_flush.commit(entry)) > + { > + is_leader= 1; > + goto commit; > + } > + > + is_leader= queue_for_group_commit(entry); As described above, re-factor the write_transaction_to_binlog_events() function to avoid this assignment of is_leader and the goto. > @@ -8863,6 +8992,16 @@ MYSQL_BIN_LOG::trx_group_commit_leader(group_commit_entry *leader) > uint64 commit_id; > DBUG_ENTER("MYSQL_BIN_LOG::trx_group_commit_leader"); > > + /* > + When move a binlog cache to a binlog file, the transaction itself is > + a group. > + */ > + if (unlikely(is_free_flush)) > + { > + last_in_queue= leader; > + queue= leader; > + } > + else As described above, re-factor trx_group_commit_leader() to avoid this check in trx_group_commit_leader(), instead calling directly into the relevant code now in trx_group_commit_with_engines(). > +inline bool Binlog_free_flush::should_free_flush( > + const MYSQL_BIN_LOG::group_commit_entry *entry) const > +{ > + /* Do not free flush if total_bytes smaller than limit size. */ > + if (likely(cache_data->get_byte_position() <= > + opt_binlog_free_flush_threshold)) > + return false; Put this check as the first check in the function. So the common case of a small transaction does not need to run any of the other checks. And as suggested above, put a check: if (unlikely(!opt_optimize_thread_scheduling)) return false; (This is not critical, but just to make things a bit easier; there is no reason for you/us to worry about making commit-by-rotate work with disabled optimize_thread_scheduling, this option isn't used any more. Otherwise you'd need to test that your patch works correctly with optimize_thread_scheduling disabled, which would just be a waste of time.) > +bool Binlog_free_flush::commit(MYSQL_BIN_LOG::group_commit_entry *entry) > +{ > + // It will be released by trx_group_commit_leader > + mysql_mutex_lock(&mysql_bin_log.LOCK_log); After suggested changes, comment should read "It will be released by trx_group_commit_with_engines()." > + mysql_bin_log.trx_group_commit_leader(entry, true /* is_free_flush */); As discussed above, change this to call a new function mysql_bin_log.trx_group_commit_with_engines(), which contains the second part of trx_group_commit_leader() that you need to run. Then there is no longer a need to pass a `true` flag here to skip the first part. Also, before the call, this is the place to set entry->next to nullptr, to convert the stand-alone entry into a (singleton) list (instead of initializing it in MYSQL_BIN_LOG::write_transaction_to_binlog().) > @@ -8301,6 +8418,7 @@ MYSQL_BIN_LOG::write_transaction_to_binlog(THD *thd, > DBUG_RETURN(0); > } > > + entry.next= nullptr; > entry.thd= thd; > entry.cache_mngr= cache_mngr; > entry.error= 0; Do you need to initialize entry->next here, and if so why? At this point, the entry is not a list, it is a stand-alone struct that is not yet linked into any list, the next pointer should be set appropriately when/if the struct is later put into a list. More important than the minor cost of initializing, by redundantly initializing here we are hiding bugs (eg. Valgrind/msan/asan) if some code would incorrectly read the next pointer. > +bool Binlog_free_flush::replace_binlog_file() > +{ > + size_t binlog_size= my_b_tell(&mysql_bin_log.log_file); > + size_t required_size= binlog_size; > + // space for Gtid_log_event > + required_size+= LOG_EVENT_HEADER_LEN + Gtid_log_event::max_data_length + > + BINLOG_CHECKSUM_LEN; Just a remark, here you actually know the real required size of the GTID event, so you don't need to conservatively estimate Gtid_log_event::max_data_length. But I think doing it this way is fine, it is simpler and in most cases we are rounding up to IO_SIZE anyway, so I'm fine with keeping it this way. > + sql_print_information("Could not rename binlog cache to binlog, " > + "require %llu bytes but only %llu bytes reserved.", > + required_size, m_cache_data->file_reserved_bytes()); This log message will perhaps appear a bit out-of-context in the error log. Suggested clarification: "Could not rename binlog cache to binlog (as requested by --binlog-commit-by-rotate-threshold). Required ..." > + // Set the cache file as binlog file. > + mysql_file_close(mysql_bin_log.log_file.file, MYF(MY_WME)); Do not to pass MY_WME here (instead pass MYF(0)). Since you are ignoring the return of the function, so we don't want mysql_file_close to call my_error() leaving an error code set even though the operation succeeds. > +size_t Binlog_free_flush::get_gtid_event_pad_size() > +{ > + size_t begin_pos= my_b_tell(&mysql_bin_log.log_file); > + size_t pad_to_size= > + m_cache_data->file_reserved_bytes() - begin_pos - LOG_EVENT_HEADER_LEN; > + > + if (binlog_checksum_options != BINLOG_CHECKSUM_ALG_OFF) > + pad_to_size-= BINLOG_CHECKSUM_LEN; Adjusting with BINLOG_CHECKSUM_LEN is to account for the 4 bytes of checksum at the end of the GTID_EVENT, right? But why do you subtract LOG_EVENT_HEADER_LEN when calculating pad_to_size? (I mean, it is probably correct, otherwise nothing would work I think. I just did not understand it, maybe you can explain in a short comment. Or maybe Gtid_log_event::write() could be modified to do the adjusting for LOG_EVENT_HEADER_LEN and BINLOG_CHECKSUM_LEN). > + // offset must be saved before replace_binlog_file(), it will update the pos > + my_off_t offset= my_b_tell(&log_file); Suggest to clarify comment to "..., it will update the file position". > @@ -756,18 +759,20 @@ class MYSQL_BIN_LOG: public TC_LOG, private Event_log > new_file() is locking. new_file_without_locking() does not acquire > LOCK_log. > */ > - int new_file_impl(); > + int new_file_impl(bool is_free_flush= false); > void do_checkpoint_request(ulong binlog_id); > - int write_transaction_or_stmt(group_commit_entry *entry, uint64 commit_id); > + int write_transaction_or_stmt(group_commit_entry *entry, uint64 commit_id, > + bool is_free_flush= false); > int queue_for_group_commit(group_commit_entry *entry); > bool write_transaction_to_binlog_events(group_commit_entry *entry); > - void trx_group_commit_leader(group_commit_entry *leader); > + void trx_group_commit_leader(group_commit_entry *leader, > + bool is_free_flush= false); > bool is_xidlist_idle_nolock(); > void update_gtid_index(uint32 offset, rpl_gtid gtid); > > public: > void purge(bool all); > - int new_file_without_locking(); > + int new_file_without_locking(bool is_free_flush= false); > /* > A list of struct xid_count_per_binlog is used to keep track of how many > XIDs are in prepared, but not committed, state in each binlog. And how > @@ -997,7 +1002,8 @@ class MYSQL_BIN_LOG: public TC_LOG, private Event_log > enum cache_type io_cache_type_arg, > ulong max_size, > bool null_created, > - bool need_mutex); > + bool need_mutex, > + bool is_free_flush = false); > bool open_index_file(const char *index_file_name_arg, > const char *log_name, bool need_mutex); > /* Use this to start writing a new log file */ > @@ -1037,7 +1043,7 @@ class MYSQL_BIN_LOG: public TC_LOG, private Event_log > bool is_active(const char* log_file_name); > bool can_purge_log(const char *log_file_name, bool interactive); > int update_log_index(LOG_INFO* linfo, bool need_update_threads); > - int rotate(bool force_rotate, bool* check_purge); > + int rotate(bool force_rotate, bool* check_purge, bool is_free_flush= false); > void checkpoint_and_purge(ulong binlog_id); > int rotate_and_purge(bool force_rotate, DYNAMIC_ARRAY* drop_gtid_domain= NULL); > /** Style preference: please add the these new boolean parameters without default value, and instead change existing callers to pass false explicitly. (And the parameter will be "commit_by_rotate" after name change.) > + > +extern ulonglong opt_binlog_free_flush_threshold; > +static Sys_var_ulonglong Sys_binlog_free_flush_threshold( > + "binlog_free_flush_threshold", > + "Try to rename the binlog cache temporary file of the commiting " > + "transaction to a binlog file when its binlog cache size " > + "is bigger than the value of this variable", > + GLOBAL_VAR(opt_binlog_free_flush_threshold), > + CMD_LINE(REQUIRED_ARG), VALID_RANGE(10 * 1024 * 1024, ULLONG_MAX), > + DEFAULT(128 * 1024 * 1024), BLOCK_SIZE(1)); As described above: - Suggested name: "binlog_commit_by_rotate_threshold". - Suggested text: "For transactions that exceed this size in the binlog, commit by a more efficient method that forces a binlog rotation and re-uses the pre-existing temporary file as the next binlog file." - Let me hear opinions on whether this should default to 128 MB or should default to eg. ULONGLONG_MAX to disable the feature by default. > diff --git a/mysql-test/main/tmp_space_usage.test b/mysql-test/main/tmp_space_usage.test > index af7b295f343..1685dbbc450 100644 > --- a/mysql-test/main/tmp_space_usage.test > +++ b/mysql-test/main/tmp_space_usage.test > @@ -215,7 +215,8 @@ select count(distinct concat(seq,repeat('x',1000))) from seq_1_to_1000; > > set @save_max_tmp_total_space_usage=@@global.max_tmp_total_space_usage; > set @@global.max_tmp_total_space_usage=64*1024*1024; > -set @@max_tmp_session_space_usage=1179648; > +# Binlog cache reserve 4096 bytes at the begin of the temporary file. > +set @@max_tmp_session_space_usage=1179648+65536; > select @@max_tmp_session_space_usage; > set @save_aria_repair_threads=@@aria_repair_threads; > set @@aria_repair_threads=2; > @@ -224,6 +225,7 @@ set @@max_heap_table_size=16777216; > > CREATE TABLE t1 (a CHAR(255),b INT,INDEX (b)); > INSERT INTO t1 SELECT SEQ,SEQ FROM seq_1_to_100000; > +set @@max_tmp_session_space_usage=1179648; > --error 200 > SELECT * FROM t1 UNION SELECT * FROM t1; > DROP TABLE t1; > @@ -266,11 +268,15 @@ SELECT MIN(VARIABLE_VALUE) OVER (), NTILE(1) OVER (), MAX(VARIABLE_NAME) OVER () > > connect(c1, localhost, root,,); > set @@binlog_format=row; > -CREATE OR REPLACE TABLE t1 (a DATETIME) ENGINE=MyISAM; > +CREATE OR REPLACE TABLE t1 (a DATETIME) ENGINE=InnoDB; > +# Use the transaction to keep binlog cache temporary file large enough > +BEGIN; > INSERT INTO t1 SELECT NOW() FROM seq_1_to_6000; > + > SET max_tmp_session_space_usage = 64*1024; > --error 200 > SELECT * FROM information_schema.ALL_PLUGINS LIMIT 2; > +ROLLBACK; > drop table t1; Can you explain why you need these changes to the test case? I was not able to understand that from the patch. With these changes, I approve of this patch / pull-request to merge into the next MariaDB release. - Kristian.

3 3

Re: bd616a3733c: Auth: set thd->scramble in all cases during writing Initial Handshake Packet
by Sergei Golubchik 10 Sep '24

10 Sep '24

Hi, Nikita, On Sep 09, Nikita Malyavin wrote: > revision-id: bd616a3733c (mariadb-11.6.1-13-gbd616a3733c) > parent(s): bbbb429a1eb > author: Nikita Malyavin > committer: Nikita Malyavin > timestamp: 2024-09-07 20:46:19 +0200 > message: > > Auth: set thd->scramble in all cases during writing Initial Handshake Packet > > --- > sql/sql_acl.cc | 19 +++++++++++++------ > 1 file changed, 13 insertions(+), 6 deletions(-) > > diff --git a/sql/sql_acl.cc b/sql/sql_acl.cc > index 93efc5806cc..7845af55413 100644 > --- a/sql/sql_acl.cc > +++ b/sql/sql_acl.cc > @@ -13504,17 +13504,21 @@ static bool send_server_handshake_packet(MPVIO_EXT *mpvio, > mpvio->cached_server_packet.pkt_len= data_len; > } > > - if (data_len < SCRAMBLE_LENGTH) > + if (thd->scramble[SCRAMBLE_LENGTH] != 0) old code was obviously wrong but the new one makes no sense either. How can thd->scramble[SCRAMBLE_LENGTH] be not zero at this point? I'd think you need to execute the code below unconditionally. > { > + DBUG_ASSERT(thd->scramble[SCRAMBLE_LENGTH] == 1); // Sanity > + > if (data_len) > { > /* > the first packet *must* have at least 20 bytes of a scramble. > - if a plugin provided less, we pad it to 20 with zeros > + if a plugin provided less, we pad it to 20 with zeros, > + plus extra zero termination sign is put in thd->scramble. > + If more is provided, we'll use only 20 bytes as a handshake scramble. Not sure it's a good idea. Why not to send the complete scramble? True, it won't fit on the client side into mysql->scramble[]. Two ways to fix it: * Write the first 20 bytes. The server will only store the first 20 bytes too, and for COM_CHANGE_USER they'll use the first 20 bytes only (for the login auth they can use the complete scramble). That's a rather hackish fix, a proper fix would be * store the complete scramble in MYSQL. The first 20 bytes in MYSQL::scramble_buff, the rest in MySQL::extension.scramble_suffix > */ > - memcpy(scramble_buf, data, data_len); > - bzero(scramble_buf + data_len, SCRAMBLE_LENGTH - data_len); > - data= scramble_buf; > + size_t fill_size= MY_MIN(SCRAMBLE_LENGTH, data_len); > + memcpy(thd->scramble, data, fill_size); > + bzero(thd->scramble + fill_size, SCRAMBLE_LENGTH - fill_size + 1); > } Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 3

Re: 982bf06d560: MDEV-12320 configurable default authentication plugin for the server
by Sergei Golubchik 09 Sep '24

09 Sep '24

Hi, Nikita, This looks good. Minor comments below. On Sep 09, Nikita Malyavin wrote: > revision-id: 982bf06d560 (mariadb-11.6.1-14-g982bf06d560) > parent(s): bd616a3733c > author: Nikita Malyavin > committer: Nikita Malyavin > timestamp: 2024-09-07 21:44:13 +0200 > message: > > MDEV-12320 configurable default authentication plugin for the server > > * Add a new cmdline-only variable "default_auth_plugin". > * A default plugin is locked at the server init and unlocked at the deinit > stages. This means that mysql_native_password and old_password_plugin, when > default, are locked/unlocked twice. doesn't matter, compiled-in plugins are only locked in debug builds, otherwise it's a no-op. > > --- a/sql/sys_vars.cc > +++ b/sql/sys_vars.cc > @@ -4538,6 +4538,14 @@ static Sys_var_plugin Sys_enforce_storage_engine( > DEFAULT(&enforced_storage_engine), NO_MUTEX_GUARD, NOT_IN_BINLOG, > ON_CHECK(check_has_super)); > > +extern const char *default_auth_plugin_name; > +extern LEX_CSTRING native_password_plugin_name; Is it ok? The correct type is Lex_ident_plugin. > +static Sys_var_charptr Sys_default_auth_plugin( there's also Sys_var_lexstring, if you prefer that. > + "default_auth_plugin", "Default plugin, that will be tried first when authenticating new connections", reformat the long line, please > + READ_ONLY GLOBAL_VAR(default_auth_plugin_name), CMD_LINE(OPT_ARG), > + DEFAULT(native_password_plugin_name.str), > + NO_MUTEX_GUARD, NOT_IN_BINLOG); > --- a/sql/sql_acl.cc > +++ b/sql/sql_acl.cc > @@ -2543,9 +2542,20 @@ bool acl_init(bool dont_read_acl_tables) > old_password_plugin= my_plugin_lock_by_name(0, > &old_password_plugin_name, MYSQL_AUTHENTICATION_PLUGIN); > > + Lex_cstring_strlen def_plugin_name(default_auth_plugin_name); > + default_auth_plugin= my_plugin_lock_by_name(NULL, &def_plugin_name, > + MYSQL_AUTHENTICATION_PLUGIN); > + > if (!native_password_plugin || !old_password_plugin) > DBUG_RETURN(1); > > + if (!default_auth_plugin) > + { > + sql_print_error("Default plugin %s could not be loaded", > + default_auth_plugin_name); see init_default_storage_engine() in mysqld.cc - it's for --default-storage-engine option. let's use similar wording for consistency: sql_print_error("Unknown/unsupported authentication plugin: %s", default_auth_plugin_name); > + DBUG_RETURN(1); > + } > + > if (dont_read_acl_tables) > { > DBUG_RETURN(0); /* purecov: tested */ Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

1 0

Re: bbbb429a1eb: Auth: release the plugins from responsibility to fill scramble_buff.
by Sergei Golubchik 09 Sep '24

09 Sep '24

Hi, Nikita, Conceptually ok, see one omission below. On Sep 09, Nikita Malyavin wrote: > revision-id: bbbb429a1eb (mariadb-11.6.1-12-gbbbb429a1eb) > parent(s): db5d1cde450 > author: Nikita Malyavin > committer: Nikita Malyavin > timestamp: 2024-09-07 20:44:02 +0200 > message: > > Auth: release the plugins from responsibility to fill scramble_buff. > > Fill it once during the Initial Handshake Packet parsing. This uses the fact > that the server guarantees first 20 bytes to be the scramble. > > Submodule libmariadb de6305915f8..12d78187061: > diff --git a/libmariadb/libmariadb/mariadb_lib.c b/libmariadb/libmariadb/mariadb_lib.c > index 78195d44..07953ea4 100644 > --- a/libmariadb/libmariadb/mariadb_lib.c > +++ b/libmariadb/libmariadb/mariadb_lib.c > @@ -1954,6 +1954,8 @@ restart: > goto error; > } > } > + memmove(mysql->scramble_buff, scramble_data, SCRAMBLE_LENGTH); > + mysql->scramble_buff[SCRAMBLE_LENGTH]= 0; > /* Set character set */ > if (mysql->options.charset_name) > diff --git a/libmariadb/plugins/auth/my_auth.c b/libmariadb/plugins/auth/my_auth.c > index a2fd519d..47d26150 100644 > --- a/libmariadb/plugins/auth/my_auth.c > +++ b/libmariadb/plugins/auth/my_auth.c > @@ -126,10 +126,6 @@ static int native_password_auth_client(MYSQL_PLUGIN_VIO *vio, MYSQL *mysql) > > if (pkt_len != SCRAMBLE_LENGTH + 1) > return CR_SERVER_HANDSHAKE_ERR; > - > - /* save it in MYSQL */ > - memmove(mysql->scramble_buff, pkt, SCRAMBLE_LENGTH); > - mysql->scramble_buff[SCRAMBLE_LENGTH] = 0; This looks ok > } > > if (mysql && mysql->passwd[0]) > diff --git a/sql-common/client.c b/sql-common/client.c > index 6d030ce0a17..28b477c1e9a 100644 > --- a/sql-common/client.c > +++ b/sql-common/client.c > @@ -4178,10 +4178,6 @@ static int native_password_auth_client(MYSQL_PLUGIN_VIO *vio, MYSQL *mysql) > > if (pkt_len != SCRAMBLE_LENGTH + 1) > DBUG_RETURN(CR_SERVER_HANDSHAKE_ERR); > - > - /* save it in MYSQL */ > - memcpy(mysql->scramble, pkt, SCRAMBLE_LENGTH); > - mysql->scramble[SCRAMBLE_LENGTH] = 0; this isn't, you removed memcpy from the plugin, but didn't add it to mysql_real_connect in sql-common/client.c. > } > > if (mysql->passwd[0]) Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 1

Re: b96afca238a: MDEV-16440 merge 5.7 P_S sysvars instrumentation and tables
by Sergei Golubchik 08 Sep '24

08 Sep '24

Hi, Nikita, Despite the email subject, it's a review of everything in bb-10.10-MDEV-16440, *except* 9c57bbfef92. That is 37e1ccbf678..c16b2429f02 without notifiable_work_zone.h I'll send a separate email about notifiable_work_zone.h On Sep 08, Nikita Malyavin wrote: > revision-id: b96afca238a (mariadb-10.6.1-472-gb96afca238a) > parent(s): 37e1ccbf678 > author: Nikita Malyavin > committer: Nikita Malyavin > timestamp: 2022-06-19 14:52:30 +0300 > message: > > MDEV-16440 merge 5.7 P_S sysvars instrumentation and tables > diff --git a/mysql-test/suite/perfschema/t/variables_stress.test b/mysql-test/suite/perfschema/t/variables_stress.test > new file mode 100644 > index 00000000000..be3012bf8af > --- /dev/null > +++ b/mysql-test/suite/perfschema/t/variables_stress.test > @@ -0,0 +1,71 @@ stress tests generally don't belong in mtr and aren't reliable enough for the mtr. Also, you forgot to check in the result file for it. on the other hand, looking at the patch I got the impression that this stress test helped to find a lot of bugs. > +let $i = 64; > +enable_connect_log; > +while ($i) { > + echo i = $i; > + connect(con_$i, localhost, root); > +# set debug_sync = "tp_hanger wait_for banger"; > + send set debug_dbug = "+d,tp_wanger"; > + dec $i; > +} > +connection default; > + > +SELECT THREAD_ID, PROCESSLIST_ID FROM performance_schema.threads; > +set debug_sync = "now signal banger"; > +connect(banger_1, localhost, root); > +let $banger1= `SELECT THREAD_ID FROM performance_schema.threads WHERE PROCESSLIST_ID = CONNECTION_ID()`; > +send select count(*) from performance_schema.variables_by_thread where variable_name="time_zone"; > +connect(banger_2, localhost, root); > +let $banger2= `SELECT THREAD_ID FROM performance_schema.threads WHERE PROCESSLIST_ID = CONNECTION_ID()`; > +send select count(*) from performance_schema.variables_by_thread where variable_name="time_zone"; > +connect(banger_3, localhost, root); > +let $banger3= `SELECT THREAD_ID FROM performance_schema.threads WHERE PROCESSLIST_ID = CONNECTION_ID()`; > +send select count(*) from performance_schema.variables_by_thread where variable_name="time_zone"; > +connect(banger_4, localhost, root); > +let $banger4= `SELECT THREAD_ID FROM performance_schema.threads WHERE PROCESSLIST_ID = CONNECTION_ID()`; > +send select count(*) from performance_schema.variables_by_thread where variable_name="time_zone"; > +connect(banger_5, localhost, root); > +let $banger5= `SELECT THREAD_ID FROM performance_schema.threads WHERE PROCESSLIST_ID = CONNECTION_ID()`; > + > +send select count(*) from performance_schema.variables_by_thread where variable_name="time_zone"; > + > +let $i = 2000; > +let $j=65; > +while ($i) { > + echo i = $i; > + connection banger_1; > + reap; > + send select count(*) from performance_schema.variables_by_thread where variable_name="time_zone"; > + connection banger_2; > + reap; > + send select count(*) from performance_schema.variables_by_thread where variable_name="time_zone"; > + connection banger_3; > + reap; > + send select count(*) from performance_schema.variables_by_thread where variable_name="time_zone"; > + connection banger_4; > + reap; > + send select count(*) from performance_schema.variables_by_thread where variable_name="time_zone"; > + connection banger_5; > + reap; > + send select count(*) from performance_schema.variables_by_thread where variable_name="time_zone"; > + dec $i; > + > + let $threads= `show status where variable_name = 'threads_connected'`; > + if ($threads < 63) { > + connect(con_$i, localhost, root); > + send set debug_dbug = "+d,tp_wanger"; > + inc $j; > + } > +} > +reap; > +select * from performance_schema.variables_by_thread where variable_name="time_zone"; > +echo $banger1; > +echo $banger2; > +echo $banger3; > +echo $banger4; > +echo $banger5; > +# eval select * from performance_schema.variables_by_thread where variable_name like "%id" and thread_id=197; > +connection default; > +# set global debug_dbug = ""; > +set debug_sync = reset; > + > +show status where variable_name = 'threads_connected'; > diff --git a/sql/sql_plugin.h b/sql/sql_plugin.h > index d4df8c6468f..3d02d1c7509 100644 > --- a/sql/sql_plugin.h > +++ b/sql/sql_plugin.h > @@ -186,6 +186,10 @@ extern SHOW_COMP_OPTION plugin_status(const char *name, size_t len, int type); > extern bool check_valid_path(const char *path, size_t length); > extern void plugin_mutex_init(); > > +template <class T> class Dynamic_array; you can simply include sql_array.h, can't you? > + > +void plugin_lock_by_sys_var_array(THD *thd, Dynamic_array<SHOW_VAR> *vars); > + > typedef my_bool (plugin_foreach_func)(THD *thd, > plugin_ref plugin, > void *arg); > diff --git a/sql/sql_parse.cc b/sql/sql_parse.cc > index 0597b086b7b..ec86389d08f 100644 > --- a/sql/sql_parse.cc > +++ b/sql/sql_parse.cc > @@ -1296,6 +1296,9 @@ dispatch_command_return do_command(THD *thd, bool blocking) > goto out; > } > > + if (unlikely(thd->apc_target.have_apc_requests())) > + thd->apc_target.process_apc_requests(); 1. why here? 2. there generally shouldn't be direct checks of apc requests, the code should do check_killed() often enough. So if you really need a check here (see question 1), add if (thd->check_killed())... > + > packet= (char*) net->read_pos; > /* > 'packet_length' contains length of data, as it was stored in packet > diff --git a/vio/viopipe.c b/vio/viopipe.c > index aeaad311b7e..5106c6d6513 100644 > --- a/vio/viopipe.c > +++ b/vio/viopipe.c > @@ -53,8 +53,9 @@ static size_t wait_overlapped_result(Vio *vio, int timeout) > } > else > { > + DWORD last_error= GetLastError(); > /* Error or timeout, cancel the pending I/O operation. */ > - CancelIo(vio->hPipe); > + CancelIoEx(vio->hPipe, &vio->overlapped); why, CancelIoEx resets last error, but CancelIo doesn't? > /* > If the wait timed out, set error code to indicate a > diff --git a/sql/threadpool_win.cc b/sql/threadpool_win.cc > index ed68e31c755..25844d1f3ef 100644 > --- a/sql/threadpool_win.cc > +++ b/sql/threadpool_win.cc > @@ -131,6 +131,11 @@ void TP_pool_win::resume(TP_connection* c) > SubmitThreadpoolWork(((TP_connection_win*)c)->work); > } > > +int TP_pool_win::wake(TP_connection *c) > +{ > + return 0; > +} How is that supposed to work? How do you wake up threadpool threads on Windows? > + > #define CHECK_ALLOC_ERROR(op) \ > do \ > { \ > diff --git a/mysys/my_thr_init.c b/mysys/my_thr_init.c > index cde34e5bdb9..89f190ee989 100644 > --- a/mysys/my_thr_init.c > +++ b/mysys/my_thr_init.c > @@ -432,9 +432,21 @@ const char *my_thread_name(void) > extern void **my_thread_var_dbug() > { > struct st_my_thread_var *tmp; > + int last_error; > if (!my_thread_global_init_done) > return NULL; > +#ifdef WIN32 > + last_error= GetLastError(); > +#else > + last_error= errno; > +#endif > + > tmp= my_thread_var; > +#ifdef WIN32 > + SetLastError(last_error); > +#else > + errno= last_error; > +#endif huh? my_thread_var can change errno? > return tmp && tmp->init ? &tmp->dbug : 0; > } > #endif /* DBUG_OFF */ > diff --git a/storage/perfschema/pfs_variable.h b/storage/perfschema/pfs_variable.h > index e59b02f2af8..f509e97048c 100644 > --- a/storage/perfschema/pfs_variable.h > +++ b/storage/perfschema/pfs_variable.h > @@ -212,7 +212,7 @@ class Find_THD_variable : public Find_THD_Impl > return false; > > /* Hold this lock to keep THD during materialization. */ > - mysql_mutex_lock(&thd->LOCK_thd_data); > + mysql_mutex_lock(&thd->LOCK_thd_kill); see commits 736a54f49c72 and c6c2a2b8d463 > return true; > } > void set_unsafe_thd(THD *unsafe_thd) { m_unsafe_thd= unsafe_thd; } > diff --git a/sql/threadpool_generic.h b/sql/threadpool_generic.h > index b7a35b7cbf0..e16ec4a4966 100644 > --- a/sql/threadpool_generic.h > +++ b/sql/threadpool_generic.h > @@ -88,9 +89,17 @@ struct TP_connection_generic :public TP_connection > ulonglong abs_wait_timeout; > ulonglong enqueue_time; > TP_file_handle fd; > + /** > + Designates whether fd is currently connected to the poll denoted by > + thread_group->pollfd. See also change_group. > + */ > bool bound_to_poll_descriptor; > int waiting; > bool fix_group; > +#ifndef WIN32 > + Notifiable_work_zone work_zone; > + bool leave_work_zone() final; > +#endif Do you need this #ifndef? WIN32 is *never* defined, it's _WIN32. and _WIN32 is never defined *here*, on Windows we use TP_connection_win. > #ifdef _WIN32 > win_aiosocket win_sock{}; > void init_vio(st_vio *vio) override > diff --git a/sql/sql_class.cc b/sql/sql_class.cc > index a0ba1d7e5a3..bfe49cddbf4 100644 > --- a/sql/sql_class.cc > +++ b/sql/sql_class.cc > @@ -1909,13 +1910,13 @@ void THD::awake_no_mutex(killed_state state_to_set) > } > > /* Broadcast a condition to kick the target if it is waiting on it. */ > -void THD::abort_current_cond_wait(bool force) > +void THD::abort_current_cond_wait(bool force, bool mark_abort) a bit illogical, these values aren't orthogonal. you want a 3-state enum, like ABORT_NONE, ABORT_CONNECTIONS_BUT_NOT_SYSTEM_THREADS, ABORT_ALL > { > mysql_mutex_assert_owner(&LOCK_thd_kill); > if (mysys_var) > { > mysql_mutex_lock(&mysys_var->mutex); > - if (!system_thread || force) // Don't abort locks > + if (mark_abort && (!system_thread || force)) // Don't abort locks > mysys_var->abort=1; > > /* > @@ -4373,6 +4374,14 @@ my_bool thd_net_is_killed(THD *thd) > return thd && thd->killed ? 1 : 0; > } > > +/* returns true if any APC was processed */ > +void thd_net_process_apc_requests(THD *thd) even though it's called from net_serv.cc, there's nothing about NET here, so calling it thd_net_process_apc_requests is misleading > +{ > + if (unlikely(thd->apc_target.have_apc_requests())) > + thd->apc_target.process_apc_requests(); you also call it from the threadpool_common and a couple of times you have these two lines verbatim in other places (e.g. sql_parse.cc) why couldn't you call thd->check_killed() instead? > + DEBUG_SYNC(thd, "net_after_apc"); > +} > + > > void thd_increment_bytes_received(void *thd, size_t length) > { > diff --git a/sql/my_apc.h b/sql/my_apc.h > index cc98e36bbe4..ae60bdc2763 100644 > --- a/sql/my_apc.h > +++ b/sql/my_apc.h > @@ -81,7 +83,7 @@ class Apc_target > */ > inline bool have_apc_requests() > { > - return MY_TEST(apc_calls); > + return MY_TEST(apc_calls.load(std::memory_order_acquire)); why? > } > > inline bool is_enabled() { return enabled; } > @@ -95,14 +97,19 @@ class Apc_target > virtual ~Apc_call() {} > }; > > + class Call_request; > /* Make a call in the target thread (see function definition for details) */ > - bool make_apc_call(THD *caller_thd, Apc_call *call, int timeout_sec, bool *timed_out); > + bool make_apc_call(THD *caller_thd, Apc_call *call, > + int timeout_sec, bool *timed_out); > + > + void enqueue_request(Call_request *request_buff, Apc_call *call); > + int wait_for_completion(THD *caller_thd, Call_request *request, > + int timeout_sec); You wrote > In pfs_variable.cc we will have to make an additional step between > enqueue_request and wait_for_completion. These two functions will be called > directly and therefore both should have a public interface and I don't think I quite agree with that. Your "additional step" seems to be abort_current_cond_wait() and notify_apc(). It's not an exotic additional step that only pfs_variable.cc needs, anyone, who makes an apc call needs the target thread to wake up and process it. It's something that should be part of apc code, not done only in pfs_variable.cc And this case you likely won't need enqueue_request and wait_for_completion as a public api > > #ifndef DBUG_OFF > int n_calls_processed; /* Number of calls served by this target */ > #endif > private: > - class Call_request; > > /* > Non-zero value means we're enabled. It's an int, not bool, because one can > diff --git a/sql/threadpool.h b/sql/threadpool.h > index 7737d056b4a..08e85d362e7 100644 > --- a/sql/threadpool.h > +++ b/sql/threadpool.h > @@ -112,6 +119,8 @@ struct TP_connection > > /* Read for the next client command (async) with specified timeout */ > virtual int start_io() = 0; > + IF_WIN(,virtual) bool leave_work_zone(){ return true; } > + IF_WIN(Notifiable_work_zone work_zone;,) // Dummy object. Ugh, this is unreadable. IF_WIN was supposed to be used in expressions. And why do you need virtual methods here, again? > > virtual void wait_begin(int type)= 0; > virtual void wait_end() = 0; > diff --git a/sql/threadpool_generic.cc b/sql/threadpool_generic.cc > index eb08441a4d5..f61dd90606b 100644 > --- a/sql/threadpool_generic.cc > +++ b/sql/threadpool_generic.cc > @@ -114,7 +114,8 @@ struct pool_timer_t > > static pool_timer_t pool_timer; > > -static void queue_put(thread_group_t *thread_group, TP_connection_generic *connection); > +IF_DBUG(,static) you can remove this IF_DBUG, I don't see you use queue_put outside of this file. > +void queue_put(thread_group_t *thread_group, TP_connection_generic *connection); > static void queue_put(thread_group_t *thread_group, native_event *ev, int cnt); > static int wake_thread(thread_group_t *thread_group,bool due_to_stall); > static int wake_or_create_thread(thread_group_t *thread_group, bool due_to_stall=false); > @@ -1485,8 +1520,41 @@ static int change_group(TP_connection_generic *c, > } > > > +#ifndef WIN32 > +bool TP_connection_generic::leave_work_zone() > +{ > + auto leave_state= work_zone.try_leave(); > + while (unlikely(leave_state == Notifiable_work_zone::SIGNAL)) > + { > + if (thd->apc_target.have_apc_requests()) > + thd->apc_target.process_apc_requests(); > + leave_state= work_zone.try_leave(); > + } > + return leave_state == Notifiable_work_zone::OWNER; > +} > +#endif > + > +#ifndef DBUG_OFF > +#include <random> > +static thread_local std::mt19937 mt_random{std::random_device()()}; > +static thread_local std::uniform_int_distribution<int> kill_dist{1, 100000}; > +void queue_put(thread_group_t *thread_group, TP_connection_generic *connection); > +void tp_send(TP_connection_type* c); > +#endif > + > int TP_connection_generic::start_io() > { > + if(DBUG_IF("tp_wanger")) > + { > + bool survive= kill_dist(mt_random) != 555; does that compile? you define kill_dist under #ifndef DBUG_OFF. You can fix this with, like #ifndef DBUG_OFF if (DBUG_IF("tp_wanger")) { static thread_local std::mt19937 ... etc } #endif > + if (survive) > + { > + state= TP_STATE_INTERRUPTED; > + tp_send(this); > + } > + return survive ? 0 : 1; > + } > + > /* > Usually, connection will stay in the same group for the entire > connection's life. However, we do allow group_count to > diff --git a/vio/viosocket.c b/vio/viosocket.c > index 002ff274b74..45c3d80a33d 100644 > --- a/vio/viosocket.c > +++ b/vio/viosocket.c > @@ -909,20 +916,53 @@ static my_bool socket_peek_read(Vio *vio, uint *bytes) > */ > > #ifndef _WIN32 > +#ifdef _GNU_SOURCE > +#define PFD_SIZE 1 > +static inline int vio_poll(struct pollfd pfd[], nfds_t nr, int timeout) > +{ > + struct timespec tm, *tm_arg= NULL; > + sigset_t signals; > + /* Convert the timeout, in milliseconds, to seconds and microseconds. */ > + if (timeout >= 0) > + { > + tm.tv_sec= timeout / 1000; > + tm.tv_nsec= (timeout % 1000) * 1000 * 1000; > + tm_arg= &tm; > + } > + sigemptyset(&signals); > + > + return ppoll(pfd, 1, tm_arg, &signals); > +} > +#else > +#define PFD_SIZE 2 > +#define vio_poll poll > +#endif why do you need that? How is your vio_poll() different from poll() ? > + > int vio_io_wait(Vio *vio, enum enum_vio_io_event event, int timeout) > { > int ret; > short revents __attribute__((unused)) = 0; > - struct pollfd pfd; > + const short revents_read = MY_POLL_SET_IN | MY_POLL_SET_ERR | POLLRDHUP; > + struct pollfd pfd[PFD_SIZE]; > my_socket sd= mysql_socket_getfd(vio->mysql_socket); > + int pfd_size; > MYSQL_SOCKET_WAIT_VARIABLES(locker, state) /* no ';' */ > DBUG_ENTER("vio_io_wait"); > DBUG_PRINT("enter", ("timeout: %d", timeout)); > > - memset(&pfd, 0, sizeof(pfd)); > + memset(pfd, 0, sizeof(pfd)); > > - pfd.fd= sd; > + pfd[0].fd= sd; > > +#ifndef _GNU_SOURCE > + my_socket self_pipe= threadlocal_get_self_pipe(); > + if (self_pipe) > + { > + pfd[1].fd= self_pipe; > + pfd[1].events= MY_POLL_SET_IN; > + } > +#endif > + pfd_size= PFD_SIZE; really? I'd expect it to be essentially pfd_size= self_pipe ? 2 : 1; > /* > Set the poll bitmask describing the type of events. > The error flags are only valid in the revents bitmask. > @@ -970,16 +1023,16 @@ int vio_io_wait(Vio *vio, enum enum_vio_io_event event, int timeout) > } > > #else > - > int vio_io_wait(Vio *vio, enum enum_vio_io_event event, int timeout) > { > int ret; > - struct timeval tm; > - my_socket fd= mysql_socket_getfd(vio->mysql_socket); > - fd_set readfds, writefds, exceptfds; > MYSQL_SOCKET_WAIT_VARIABLES(locker, state) /* no ';' */ > DBUG_ENTER("vio_io_wait"); > > + /* Not read , e.g connect or write - use select() based wait */ > + struct timeval tm; > + my_socket fd= mysql_socket_getfd(vio->mysql_socket); > + fd_set readfds, writefds, exceptfds; it's C, you cannot mix declarations and code > /* Convert the timeout, in milliseconds, to seconds and microseconds. */ > if (timeout >= 0) > { > @@ -1013,14 +1066,17 @@ int vio_io_wait(Vio *vio, enum enum_vio_io_event event, int timeout) > ret= select(0, &readfds, &writefds, &exceptfds, (timeout >= 0) ? &tm : NULL); > > END_SOCKET_WAIT(locker, timeout); > /* Set error code to indicate a timeout error. */ > if (ret == 0) > WSASetLastError(SOCKET_ETIMEDOUT); > > /* Error or timeout? */ > - if (ret <= 0) > + if (ret <= 0){ indentation > + DBUG_PRINT("vio", ("vio_io_wait: error return: %d. Last error: %d", > + ret, socket_errno)); > DBUG_RETURN(ret); > + } > > /* The requested I/O event is ready? */ > switch (event) > diff --git a/sql/sql_connect.cc b/sql/sql_connect.cc > index 9beb3cca8f8..1c8975cfb23 100644 > --- a/sql/sql_connect.cc > +++ b/sql/sql_connect.cc > @@ -23,7 +23,53 @@ > #include "mariadb.h" > #include "mysqld.h" > #include "sql_priv.h" > -#ifndef _WIN32 > +#ifdef _WIN32 > +#ifdef _WIN32_WINNT > +#undef _WIN32_WINNT > +#endif > + > +#define _WIN32_WINNT 0x0500 > +#include "windows.h" // QueueUserAPC2 > +#define WINDOWS_KERNEL32_DLLNAME_W "kernel32" > + > +// https://github.com/dotnet/runtime/pull/55649/files > + > +// These declarations are for a new special user-mode APC feature introduced in Windows. These are not yet available in Windows > +// SDK headers, so some names below are prefixed with "CLONE_" to avoid conflicts in the future. Once the prefixed declarations > +// become available in the Windows SDK headers, the prefixed declarations below can be removed in favor of the SDK ones. > + > +//enum CLONE_QUEUE_USER_APC_FLAGS > +//{ > +// CLONE_QUEUE_USER_APC_FLAGS_NONE = 0x0, > +// CLONE_QUEUE_USER_APC_FLAGS_SPECIAL_USER_APC = 0x1, > +// > +// CLONE_QUEUE_USER_APC_CALLBACK_DATA_CONTEXT = 0x10000 > +//}; > +//typedef BOOL (WINAPI *QueueUserAPC2Proc)(PAPCFUNC ApcRoutine, HANDLE Thread, ULONG_PTR Data, CLONE_QUEUE_USER_APC_FLAGS Flags); > +//void c() > +//{ > +// HMODULE hKernel32 = LoadLibraryExA(WINDOWS_KERNEL32_DLLNAME_W, NULL, LOAD_LIBRARY_SEARCH_SYSTEM32); > +// > +// // See if QueueUserAPC2 exists > +// QueueUserAPC2Proc pfnQueueUserAPC2Proc = (QueueUserAPC2Proc)GetProcAddress(hKernel32, "QueueUserAPC2"); > +// if (pfnQueueUserAPC2Proc == nullptr) > +// { > +// abort(); > +// } > +// > +// // See if QueueUserAPC2 supports the special user-mode APC with a callback that includes the interrupted CONTEXT. A special > +// // user-mode APC can interrupt a thread that is in user mode and not in a non-alertable wait. > +// if (!(*pfnQueueUserAPC2Proc)(EmptyApcCallback, GetCurrentThread(), 0, SpecialUserModeApcWithContextFlags)) > +// { > +// return; > +// } > +// > +// return pfnQueueUserAPC2Proc; > +//} > +// > +//static QueueUserAPC2Proc pfnQueueUserAPC2 = InitializeSpecialUserModeApc(); what's that big commented block for? and win-something defines/includes - still needed? > + > +#else > #include <netdb.h> // getservbyname, servent > #endif > #include "sql_audit.h" > @@ -1353,6 +1399,107 @@ bool thd_is_connection_alive(THD *thd) > return FALSE; > } > > +#if !defined(_WIN32) > +static void self_pipe_write(); > +#if !defined(_GNU_SOURCE) > +class Thread_apc_context; > +static thread_local Thread_apc_context *_THR_APC_CTX= NULL; may be it should be part of THD (rather, Apc_target) or mysys_var? that is, accessible as current_thd->...->self_pipe or my_thread_var->self_pipe the first one (in Apc_target) seems quite logical > +#endif > + > +class Thread_apc_context > +{ > +public: > +#ifndef _GNU_SOURCE > + my_socket self_pipe[2]{}; > +#endif > + > + bool setup_thread_apc() > + { > +#if defined(_GNU_SOURCE) > + struct sigaction act {}; > + act.sa_handler= [](int) -> void {self_pipe_write();}; why? it doesn't do anything, as far as I can see > + act.sa_flags= 0; > + int ret= sigaction(SIG_APC_NOTIFY, &act, NULL); > + DBUG_ASSERT(ret == 0); > + > + sigset_t signals; > + ret|= sigemptyset(&signals); > + DBUG_ASSERT(ret == 0); > + ret|= sigaddset(&signals, SIG_APC_NOTIFY); > + DBUG_ASSERT(ret == 0); > + > + ret|= pthread_sigmask(SIG_BLOCK, &signals, NULL); > + DBUG_ASSERT(ret == 0); > +#else > + // Self-pipe trick. Create a new pipe and store it thread-locally > + // It can be accessed from Vio later. See also vio_io_wait() > + // From the other end, it is accessed through a threadlocal > + // _THR_APC_CTX, from a SIG_APC_NOTIFY signal handler. > + int ret= pipe(self_pipe); > + > + struct sigaction act {}; > + act.sa_handler= [](int) -> void { self_pipe_write(); }; > + act.sa_flags= 0; > + ret|= sigaction(SIG_APC_NOTIFY, &act, NULL); why? you send a signal and convert it to a self-pipe write. why not to write to the pipe in the first place? like Apc_target::wake() which will send a signal or write to a pipe, as appropriate. And TP_pool::wake() will call thd->apc_target->wake(); > + return ret == 0; > +#endif > + return true; > + } > + bool inited; > + Thread_apc_context() > + { > + inited = setup_thread_apc(); > +#ifndef _GNU_SOURCE > + if (inited) > + _THR_APC_CTX= this; > +#endif > + } > +#ifndef _GNU_SOURCE > + ~Thread_apc_context() > + { > + _THR_APC_CTX= NULL; > + closesocket(self_pipe[0]); > + closesocket(self_pipe[1]); > + } > +#endif > +}; > + > +#ifdef _GNU_SOURCE > +static void self_pipe_write() > +{ > + // No self-pipe is actually used. Instead, ppoll is used to wake up on signal. > +} > +#else > + > +my_socket threadlocal_get_self_pipe() > +{ > + return _THR_APC_CTX ? _THR_APC_CTX->self_pipe[0] : 0; > +} > + > +static void self_pipe_write() > +{ > + DBUG_ASSERT(IF_WIN(0, pthread_self() == select_thread) || _THR_APC_CTX); > + if (!_THR_APC_CTX) > + return; // SIGUSR1 is used in thr_alarm, which can wake up main thread. > + char buf[4]{}; > + send(_THR_APC_CTX->self_pipe[1], buf, sizeof buf, 0); > +} > +#endif > +#endif /* !_WIN32 */ > + > +bool thread_per_connection_notify_apc(THD *thd) > +{ > +#ifdef WIN32 > + auto vio= thd->net.vio; > + HANDLE h= > + (vio->type == VIO_TYPE_NAMEDPIPE) ? vio->hPipe : (HANDLE)vio->mysql_socket.fd; > + return CancelIoEx(h, NULL); > +#else > + bool ret = pthread_kill(thd->mysys_var->pthread_self, SIG_APC_NOTIFY) == 0; > + DBUG_ASSERT(ret || errno == EAGAIN); > + return ret; > +#endif > +} 1. I'd prefer APC framework implementation to stay in my_apc.cc. not being spread over three (?) different files. 2. how is it diferent from ::wake() ? > > void do_handle_one_connection(CONNECT *connect, bool put_in_cache) > { > diff --git a/sql/threadpool_common.cc b/sql/threadpool_common.cc > index f72f46b3a6b..8f24bc9e4f7 100644 > --- a/sql/threadpool_common.cc > +++ b/sql/threadpool_common.cc > @@ -320,6 +346,10 @@ static void threadpool_remove_connection(THD *thd) > end_connection(thd); > close_connection(thd, 0); > unlink_thd(thd); > + // The rest of APC requests should be processed after unlinking. > + // This guarantees that no new APC requests can be added. > + // TODO: better notify the requestor with some KILLED state here. > + thd->apc_target.process_apc_requests(); not sure it guarantees much. One can find the thd before the unlink and add a new request after the unlink. or is it impossible? > PSI_CALL_delete_current_thread(); // before THD is destroyed > delete thd; > > diff --git a/sql/my_apc.cc b/sql/my_apc.cc > index 9777deb399a..a89d9c8bc19 100644 > --- a/sql/my_apc.cc > +++ b/sql/my_apc.cc > @@ -47,11 +47,12 @@ void Apc_target::init(mysql_mutex_t *target_mutex) > void Apc_target::enqueue_request(Call_request *qe) > { > mysql_mutex_assert_owner(LOCK_thd_kill_ptr); > - if (apc_calls) > + Call_request *old_apc_calls= apc_calls; eh, what? you modify apc_calls without a lock somewhere? > + if (old_apc_calls) > { > - Call_request *after= apc_calls->prev; > - qe->next= apc_calls; > - apc_calls->prev= qe; > + Call_request *after= old_apc_calls->prev; > + qe->next= old_apc_calls; > + old_apc_calls->prev= qe; > > qe->prev= after; > after->next= qe; > @@ -100,89 +109,118 @@ void init_show_explain_psi_keys(void) > if (PSI_server == NULL) > return; > > - PSI_server->register_cond("sql", show_explain_psi_conds, > - array_elements(show_explain_psi_conds)); > + PSI_server->register_cond("sql", apc_request_psi_conds, > + array_elements(apc_request_psi_conds)); > + PSI_server->register_mutex("sql", apc_request_psi_locks, > + array_elements(apc_request_psi_locks)); > } > #endif > > - > -/* > - Make an APC (Async Procedure Call) to another thread. > - > - @detail > - Make an APC call: schedule it for execution and wait until the target > - thread has executed it. > - > - - The caller is responsible for making sure he's not posting request > - to the thread he's calling this function from. > - > - - The caller must have locked target_mutex. The function will release it. > - > - @retval FALSE - Ok, the call has been made > - @retval TRUE - Call wasnt made (either the target is in disabled state or > - timeout occurred) > -*/ > - > -bool Apc_target::make_apc_call(THD *caller_thd, Apc_call *call, > - int timeout_sec, bool *timed_out) > +/** > + Wait gracefully until the request is completed. > + @retval 0 -- Success > + @retval 1 -- Timeout > + */ > +int Apc_target::wait_for_completion(THD *caller_thd, Call_request *apc_request, > + int timeout_sec) > { > - bool res= TRUE; > - *timed_out= FALSE; > - > - if (enabled) > - { > - /* Create and post the request */ > - Call_request apc_request; > - apc_request.call= call; > - apc_request.processed= FALSE; > - mysql_cond_init(key_show_explain_request_COND, &apc_request.COND_request, > - NULL); > - enqueue_request(&apc_request); > - apc_request.what="enqueued by make_apc_call"; > - > struct timespec abstime; > const int timeout= timeout_sec; > set_timespec(abstime, timeout); > > + DBUG_EXECUTE_IF("apc_timeout", set_timespec_nsec(abstime, 1000000);); > + int res = 1; > int wait_res= 0; > PSI_stage_info old_stage; > - caller_thd->ENTER_COND(&apc_request.COND_request, LOCK_thd_kill_ptr, > + > + mysql_mutex_unlock(LOCK_thd_kill_ptr); I don't understand how this works. After you release LOCK_thd_kill_ptr, the THD might be destroyed. How do you detect that? By the wait timing out? Who will destroy the Call_request? > + > + mysql_mutex_lock(&apc_request->LOCK_request); > + caller_thd->ENTER_COND(&apc_request->COND_request, &apc_request->LOCK_request, > &stage_show_explain, &old_stage); > /* todo: how about processing other errors here? */ > - while (!apc_request.processed && (wait_res != ETIMEDOUT)) > + while (!apc_request->processed && (wait_res != ETIMEDOUT)) > { > - /* We own LOCK_thd_kill_ptr */ > - wait_res= mysql_cond_timedwait(&apc_request.COND_request, > - LOCK_thd_kill_ptr, &abstime); > - // &apc_request.LOCK_request, &abstime); > + wait_res= mysql_cond_timedwait(&apc_request->COND_request, > + &apc_request->LOCK_request, &abstime); > if (caller_thd->killed) > break; > + > + if (caller_thd->apc_target.have_apc_requests()) > + { > + mysql_mutex_unlock(&apc_request->LOCK_request); > + caller_thd->apc_target.process_apc_requests(); > + mysql_mutex_lock(&apc_request->LOCK_request); > + } > } > > - if (!apc_request.processed) > + if (!apc_request->processed) > { > /* > The wait has timed out, or this thread was KILLed. > - Remove the request from the queue (ok to do because we own > - LOCK_thd_kill_ptr) > + We can't delete it from the queue, because LOCK_thd_kill_ptr is already > + released. It can't be reacquired because of the ordering with > + apc_request->LOCK_request. > + However, apc_request->processed is guarded by this lock. > + Set processed= TRUE and transfer the ownership to the processor thread. > + It should free the resources itself. > + apc_request cannot be referred after unlock anymore in this case. > */ > - apc_request.processed= TRUE; > - dequeue_request(&apc_request); > - *timed_out= TRUE; > - res= TRUE; > + apc_request->processed= TRUE; > + res= 1; > } > else > { > /* Request was successfully executed and dequeued by the target thread */ > - res= FALSE; > + res= 0; > } > - /* > - exit_cond() will call mysql_mutex_unlock(LOCK_thd_kill_ptr) for us: > - */ > + > + /* EXIT_COND() will call mysql_mutex_unlock(LOCK_request) for us */ > caller_thd->EXIT_COND(&old_stage); > > - /* Destroy all APC request data */ > - mysql_cond_destroy(&apc_request.COND_request); > + return res; > +} > + > +/** Create and post the request */ > +void Apc_target::enqueue_request(Call_request *request_buff, Apc_call *call) > +{ > + request_buff->call= call; > + request_buff->processed= FALSE; > + enqueue_request(request_buff); > + request_buff->what="enqueued by make_apc_call"; > +} > + > +/** > + Make an APC (Async Procedure Call) to another thread. > + > + @detail > + Make an APC call: schedule it for execution and wait until the target > + thread has executed it. > + > + - The caller is responsible for making sure he's not posting request > + to the thread he's calling this function from. > + > + - The caller must have locked target_mutex. The function will release it. > + > + @retval FALSE - Ok, the call has been made > + @retval TRUE - Call wasnt made (either the target is in disabled state or > + timeout occurred) > +*/ > + > +bool Apc_target::make_apc_call(THD *caller_thd, Apc_call *call, > + int timeout_sec, bool *timed_out) > +{ > + bool res= TRUE; > + *timed_out= FALSE; > + > + if (enabled) > + { > + Call_request *apc_request= new Call_request; > + enqueue_request(apc_request, call); > + res= wait_for_completion(caller_thd, apc_request, timeout_sec); > + *timed_out= res; > + if (!*timed_out) > + delete apc_request; > } > else > { > diff --git a/storage/perfschema/pfs_variable.cc b/storage/perfschema/pfs_variable.cc > index 17b7dfc200c..698ff1eca6c 100644 > --- a/storage/perfschema/pfs_variable.cc > +++ b/storage/perfschema/pfs_variable.cc > @@ -164,7 +167,11 @@ int PFS_system_variable_cache::do_materialize_global(void) > during materialization. > */ > if (!m_external_init) > + { > + mysql_mutex_lock(&LOCK_plugin); > init_show_var_array(OPT_GLOBAL, true); > + mysql_mutex_unlock(&LOCK_plugin); 1. why not move the locking inside init_show_var_array()? 2. init_show_var_array() locks LOCK_system_variables_hash - is that not enough? > + } > > /* Resolve the value for each SHOW_VAR in the array, add to cache. */ > for (SHOW_VAR *show_var= m_show_var_array.front(); > @@ -230,37 +236,26 @@ int PFS_system_variable_cache::do_materialize_all(THD *unsafe_thd) > m_materialized= false; > m_cache.clear(); > > - /* Block plugins from unloading. */ > - mysql_mutex_lock(&LOCK_plugin_delete); also please remove all references of LOCK_plugin_delete > - > /* > Build array of SHOW_VARs from system variable hash. Do this within > LOCK_plugin_delete to ensure that the hash table remains unchanged > while this thread is materialized. > */ > if (!m_external_init) > + { > + mysql_mutex_lock(&LOCK_plugin); > init_show_var_array(OPT_SESSION, false); > + mysql_mutex_unlock(&LOCK_plugin); > + } > > /* Get and lock a validated THD from the thread manager. */ > if ((m_safe_thd= get_THD(unsafe_thd)) != NULL) > { > DEBUG_SYNC(m_current_thd, "materialize_session_variable_array_THD_locked"); > - for (SHOW_VAR *show_var= m_show_var_array.front(); > - show_var->value && (show_var != m_show_var_array.end()); show_var++) > - { > - /* Resolve value, convert to text, add to cache. */ > - System_variable system_var(m_safe_thd, show_var, m_query_scope, false); > - m_cache.push(system_var); > - } > - > - /* Release lock taken in get_THD(). */ > - mysql_mutex_unlock(&m_safe_thd->LOCK_thd_data); > + ret= make_call(&PFS_system_variable_cache::refresh_vars, 1); > > m_materialized= true; > - ret= 0; > } > - > - mysql_mutex_unlock(&LOCK_plugin_delete); > return ret; > } > > @@ -312,6 +307,105 @@ void PFS_system_variable_cache::free_mem_root(void) > } > } > } > +class PFS_system_variable_cache_apc: public Apc_target::Apc_call > +{ > +public: > + typedef PFS_system_variable_cache::Request_func Request; > + PFS_system_variable_cache_apc(PFS_system_variable_cache *pfs, Request func, > + uint param) > + : m_pfs(pfs), m_func(func), m_param(param) {} > +private: > + PFS_system_variable_cache *m_pfs; > + Request m_func; > + uint m_param; > + > + void call_in_target_thread() override > + { > + call(m_pfs, m_func, m_param); > + } > +public: > + static void call(PFS_system_variable_cache *pfs, Request func, uint param) > + { > + THD *safe_thd= pfs->safe_thd(); > + > + DBUG_ASSERT(pfs->query_scope() == OPT_SESSION); > + > + mysql_mutex_lock(&LOCK_global_system_variables); > + if (!safe_thd->variables.dynamic_variables_ptr || > + global_system_variables.dynamic_variables_head > > + safe_thd->variables.dynamic_variables_head) > + { > + mysql_prlock_rdlock(&LOCK_system_variables_hash); > + sync_dynamic_session_variables(safe_thd, false); > + mysql_prlock_unlock(&LOCK_system_variables_hash); > + } > + mysql_mutex_unlock(&LOCK_global_system_variables); > + > + (pfs->*func)(param); > + } > +}; > + > +void PFS_system_variable_cache::refresh_vars(uint all) > +{ > + for (SHOW_VAR *show_var= m_show_var_array.front(); > + show_var->value && (show_var != m_show_var_array.end()); show_var++) > + { > + sys_var *value= (sys_var *)show_var->value; > + > + /* Match the system variable scope to the target scope. */ > + if (all || match_scope(value->scope())) > + { > + /* Resolve value, convert to text, add to cache. */ > + System_variable system_var(m_safe_thd, show_var, m_query_scope, false); > + m_cache.push(system_var); > + } > + } > +} > +void PFS_system_variable_cache::refresh_one_var(uint index) > +{ > + SHOW_VAR *show_var= &m_show_var_array.at(index); > + > + if (show_var && show_var->value && > + (show_var != m_show_var_array.end())) > + { > + sys_var *value= (sys_var *)show_var->value; > + > + /* Match the system variable scope to the target scope. */ > + if (match_scope(value->scope())) > + { > + /* Resolve value, convert to text, add to cache. */ > + System_variable system_var(m_safe_thd, show_var, m_query_scope, false); > + m_cache.push(system_var); > + } > + } > +} > + > + > +#define MAKE_CALL_MAX_RETRIES 3 unused > + > +int PFS_system_variable_cache::make_call(Request_func func, uint param) > +{ > + int ret= 0; > + THD *requestor_thd= m_current_thd; > + if (requestor_thd == m_safe_thd) > + { > + mysql_mutex_unlock(&m_safe_thd->LOCK_thd_kill); > + PFS_system_variable_cache_apc::call(this, func, param); > + } > + else > + { > + PFS_system_variable_cache_apc apc_call(this, func, param); > + auto *request= new Apc_target::Call_request; > + m_safe_thd->apc_target.enqueue_request(request, &apc_call); > + m_safe_thd->abort_current_cond_wait(false, false); > + m_safe_thd->scheduler->notify_apc(m_safe_thd); > + DEBUG_SYNC(requestor_thd, "apc_after_notify"); > + ret= m_safe_thd->apc_target.wait_for_completion(requestor_thd, request, 10); > + if (ret == 0) > + delete request; > + } > + return ret; > +} > > /** > Build a SESSION system variable cache for a pfs_thread. Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

1 0

Re: c46e7c5cb64: MDEV-12252 ROW data type for stored function return values
by Sergei Golubchik 05 Sep '24

05 Sep '24

Hi, Alexander, Looks good, thanks. Great commit comment too. Just a few questions and test suggestions - see below. Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org On Aug 23, Alexander Barkov wrote: > revision-id: c46e7c5cb64 (mariadb-11.4.1-5-gc46e7c5cb64) > parent(s): ae6684d79f4 > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2024-02-20 13:49:54 +0400 > message: > > MDEV-12252 ROW data type for stored function return values > diff --git a/mysql-test/main/sp-anchor-row-type-table.test b/mysql-test/main/sp-anchor-row-type-table.test > index 7123f9160db..2fbcfbffc1b 100644 > --- a/mysql-test/main/sp-anchor-row-type-table.test > +++ b/mysql-test/main/sp-anchor-row-type-table.test > @@ -939,3 +939,70 @@ DROP TABLE t1; > --echo # > --echo # End of 10.4 tests > --echo # > + > +--echo # > +--echo # Start of 11.3 tests > +--echo # > + > +--echo # > +--echo # MDEV-12252 ROW data type for stored function return values > +--echo # > + > +--error ER_PARSE_ERROR > +CREATE FUNCTION f1() RETURNS ROW TYPE OF step1.step2.step3 RETURN ROW(1,2); > + > +CREATE FUNCTION f1() RETURNS ROW TYPE OF t1 RETURN ROW(1,2); you don't resolve t1 at CREATE FUNCTION time. why? is it how TYPE OF works in DECLARE too? > +SHOW CREATE FUNCTION f1; > +--error ER_NO_SUCH_TABLE > +SELECT f1(); > +DROP FUNCTION f1; > + > +CREATE FUNCTION f1() RETURNS ROW TYPE OF test.t1 RETURN ROW(1,2); > +SHOW CREATE FUNCTION f1; > +--error ER_NO_SUCH_TABLE > +SELECT f1(); > +DROP FUNCTION f1; > + > + > +CREATE TABLE t1 (a INT, b VARCHAR(32)); > +CREATE FUNCTION f1() RETURNS ROW TYPE OF test.t1 RETURN (SELECT * FROM t1); > +SHOW CREATE FUNCTION f1; > +DELIMITER $$; > +CREATE PROCEDURE p1() > +BEGIN > + DECLARE r ROW TYPE OF t1 DEFAULT f1(); > + SELECT r.a, r.b; > +END; > +$$ > +DELIMITER ;$$ > + > +# Testing with no rows > +CALL p1(); > +--error ER_OPERAND_COLUMNS > +SELECT f1(); > +SELECT f1()=ROW(1,'b1') AS c; > +SELECT f1()=ROW(NULL,NULL) AS c; > + > +# Testing with one row > +INSERT INTO t1 VALUES (1,'b1'); > +CALL p1(); > +--error ER_OPERAND_COLUMNS > +SELECT f1(); > +SELECT f1()=ROW(1,'b1') AS c; > +SELECT f1()=ROW(1,'') AS c; > +SELECT f1()=ROW(2,'b1') AS c; > +SELECT f1()=ROW(1,NULL) AS c; > +SELECT f1()=ROW(NULL,'b1') AS c; do one more SHOW CREATE FUNCTION here > + > +# Testing with two rows > +INSERT INTO t1 VALUES (2,'b2'); > +--error ER_SUBQUERY_NO_1_ROW > +CALL p1(); > +--error ER_OPERAND_COLUMNS > +SELECT f1(); > +--error ER_SUBQUERY_NO_1_ROW > +SELECT f1()=ROW(1,'b1') AS c; > + > +DROP PROCEDURE p1; > +DROP FUNCTION f1; > +DROP TABLE t1; > diff --git a/mysql-test/main/sp-anchor-type.test b/mysql-test/main/sp-anchor-type.test > index 0e24ef900d8..fed53fd011a 100644 > --- a/mysql-test/main/sp-anchor-type.test > +++ b/mysql-test/main/sp-anchor-type.test > @@ -767,3 +767,78 @@ BEGIN NOT ATOMIC > END; > $$ > DELIMITER ;$$ > + > + > +--echo # > +--echo # Start of 11.3 tests > +--echo # > + > +--echo # > +--echo # MDEV-12252 ROW data type for stored function return values > +--echo # > + > +--error ER_PARSE_ERROR > +CREATE FUNCTION f1() RETURNS TYPE OF a RETURN 1; > + > +CREATE FUNCTION f1() RETURNS TYPE OF t1.a RETURN (SELECT min(a) FROM t1); > +SHOW CREATE FUNCTION f1; > +--error ER_NO_SUCH_TABLE > +SELECT f1(); > +CREATE TABLE t1 (b INT); > +--error ER_BAD_FIELD_ERROR > +SELECT f1(); > +DROP TABLE t1; > +DROP FUNCTION f1; > + > +CREATE DATABASE db1; > +DELIMITER $$; > +CREATE FUNCTION db1.f1() RETURNS TYPE OF db1.t1.a > +BEGIN > + RETURN (SELECT min(a) FROM t1); > +END; > +$$ > +DELIMITER ;$$ > +SHOW CREATE FUNCTION db1.f1; > +--error ER_NO_SUCH_TABLE > +SELECT db1.f1(); > +CREATE TABLE db1.t1 (b TIME); > +--error ER_BAD_FIELD_ERROR > +SELECT db1.f1(); > +DROP TABLE db1.t1; > +CREATE TABLE db1.t1 (a TIME); > +SELECT db1.f1(); > +INSERT INTO db1.t1 VALUES ('10:20:30'); > +SELECT db1.f1(); > +DROP TABLE db1.t1; > +DROP FUNCTION db1.f1; > +DROP DATABASE db1; > + > +CREATE TABLE t1 (a TIME); > +CREATE FUNCTION f1() RETURNS TYPE OF test.t1.a RETURN (SELECT min(a) FROM t1); > +SHOW CREATE FUNCTION f1; > +SELECT f1(); > +DROP FUNCTION f1; > +DROP TABLE t1; > + > +CREATE TABLE t1 (a TIME); > +DELIMITER $$; > +CREATE FUNCTION f1() RETURNS TYPE OF t1.a > +BEGIN > + RETURN (SELECT min(a) FROM t1); > +END; > +$$ > +DELIMITER ;$$ > +SHOW CREATE FUNCTION f1; > +SELECT f1(); > +CREATE TABLE t2 AS SELECT f1(); > +SHOW CREATE TABLE t2; > +SELECT * FROM t2; > +DROP TABLE t2; > +INSERT INTO t1 VALUES ('10:20:30'); > +SELECT f1(); > +CREATE TABLE t2 AS SELECT f1(); > +SHOW CREATE TABLE t2; > +SELECT * FROM t2; > +DROP TABLE t2; please, also try here DROP TABLE t1; CREATE TABLE t1 (a INT); INSERT INTO t1 VALUES (10); CREATE TABLE t2 AS SELECT f1(); the point is to change TYPE OF t1.a after the function has been successfully called, and thus is already in the sp cache. > +DROP FUNCTION f1; > +DROP TABLE t1; > diff --git a/mysql-test/suite/compat/oracle/t/sp-package.test b/mysql-test/suite/compat/oracle/t/sp-package.test > index aefede41c8b..7c231d736c5 100644 > --- a/mysql-test/suite/compat/oracle/t/sp-package.test > +++ b/mysql-test/suite/compat/oracle/t/sp-package.test > @@ -3109,3 +3109,79 @@ DROP TABLE t1; > DROP FUNCTION f1_deterministic; > DROP FUNCTION f2_not_deterministic; > DROP PACKAGE pkg1; > + > +--echo # > +--echo # MDEV-12252 ROW data type for stored function return values > +--echo # > + > +--echo # > +--echo # Testing fixed ROW type with package routines > +--echo # > + > +DELIMITER $$; > +CREATE PACKAGE pkg > +AS > + FUNCTION f1() RETURN ROW(a INT, b VARCHAR(32)); > + PROCEDURE p1(r ROW(a INT, b VARCHAR(32))); > + PROCEDURE p2(); > +END; > +$$ > +CREATE PACKAGE BODY pkg > +AS > + FUNCTION f1() RETURN ROW(a INT, b VARCHAR(32)) AS > + BEGIN > + RETURN ROW(1,'b1'); > + END; > + PROCEDURE p1(r ROW(a INT, b VARCHAR(32))) AS > + BEGIN > + SELECT r.a, r.b; > + END; > + PROCEDURE p2() AS > + BEGIN > + CALL p1(f1()); > + END; > +END; > +$$ > +DELIMITER ;$$ > +CALL pkg.p1(pkg.f1()); > +CALL pkg.p2; > +DROP PACKAGE pkg; > + > + > +--echo # > +--echo # Testing table%ROWTYPE with package routines > +--echo # now you can also add package tests outside of oracle mode > + > +CREATE TABLE t1 (a INT, b VARCHAR(32)); > +INSERT INTO t1 VALUES (1,'b1'); > +DELIMITER /; > +CREATE PACKAGE pkg > +AS > + FUNCTION f1 RETURN t1%ROWTYPE; > + PROCEDURE p1(r t1%ROWTYPE); > + PROCEDURE p2; > +END; > +/ > +CREATE PACKAGE BODY pkg > +AS > + FUNCTION f1 RETURN t1%ROWTYPE AS > + r t1%ROWTYPE; > + BEGIN > + SELECT * INTO r FROM t1; > + RETURN r; > + END; > + PROCEDURE p1(r t1%ROWTYPE) AS > + BEGIN > + SELECT r.a || ' ' || r.b; > + END; > + PROCEDURE p2 AS > + BEGIN > + p1(f1()); > + END; > +END; > +/ > +DELIMITER ;/ > +CALL pkg.p1(pkg.f1()); > +CALL pkg.p2; > +DROP PACKAGE pkg; > +DROP TABLE t1; > diff --git a/sql/field.h b/sql/field.h > index 7f1c243a180..7af069a9735 100644 > --- a/sql/field.h > +++ b/sql/field.h > @@ -1373,6 +1374,20 @@ class Field: public Value_source > in str and restore it with set() if needed > */ > virtual void sql_type(String &str) const =0; > + void sql_type_for_sp_returns(String &str) const This is oddly specific. Strange that we don't need to print the field type with charset/collation anywhere else, but for sp returns we do. > + { > + sql_type(str); > + if (has_charset()) > + { > + str.append(STRING_WITH_LEN(" CHARSET ")); > + str.append(charset()->cs_name); > + if (Charset(charset()).can_have_collate_clause()) > + { > + str.append(STRING_WITH_LEN(" COLLATE ")); > + str.append(charset()->coll_name); > + } > + } > + } > virtual void sql_rpl_type(String *str) const { sql_type(*str); } > virtual uint size_of() const =0; // For new field > inline bool is_null(my_ptrdiff_t row_offset= 0) const > diff --git a/sql/field.cc b/sql/field.cc > index 35be045620d..23b050de221 100644 > --- a/sql/field.cc > +++ b/sql/field.cc > @@ -2768,6 +2768,32 @@ bool Field_row::sp_prepare_and_store_item(THD *thd, Item **value) > } > > > +uint Field_row::cols() const > +{ > + // The table with ROW members must be already instantiated > + DBUG_ASSERT(m_table); > + return m_table->s->fields; > +} > + > + > +void Field_row::sql_type(String &res) const would be more logical to call this Field_row::sql_type_for_sp_returns() > +{ > + res.set_ascii(STRING_WITH_LEN("row(")); > + for (uint i= 0; i < m_table->s->fields; i++) > + { > + if (i > 0) > + res.append(','); > + Field *field= m_table->field[i]; > + res.append(field->field_name); > + res.append(' '); > + StringBuffer<64> col; > + field->sql_type_for_sp_returns(col); > + res.append(col.to_lex_cstring()); > + } > + res.append(')'); > +} > + > + > /**************************************************************************** > Functions for the Field_decimal class > This is an number stored as a pre-space (or pre-zero) string > diff --git a/sql/item.cc b/sql/item.cc > index 54ad511cb6b..0a125752979 100644 > --- a/sql/item.cc > +++ b/sql/item.cc > @@ -3001,10 +3003,31 @@ Item_sp::init_result_field(THD *thd, uint max_length, uint maybe_null, > dummy_table->s->table_name= empty_clex_str; > dummy_table->maybe_null= maybe_null; > > + if (m_sp->m_return_field_def.is_column_type_ref() && > + m_sp->m_return_field_def.column_type_ref()-> > + resolve_type_ref(thd, &m_sp->m_return_field_def)) > + DBUG_RETURN(TRUE); Here you overwrite old m_return_field_def value which was a column_type_ref with its actual type, don't you? What will happen if you change the column's type after the sp was already executed once? > + > if (!(sp_result_field= m_sp->create_result_field(max_length, name, > dummy_table))) > DBUG_RETURN(TRUE); > > + /* > + In case of a ROW return type we need to create Item_field_row > + on top of Field_row, and remember it in sp_result_item_field_row. > + ROW members are later accessed using sp_result_item_field_row->addr(i), > + e.g. when copying the function return value to a local variable. > + For scalar return types no Item_field is needed around sp_result_field, > + as the value is fetched directly from sp_result_field, > + inside Item_func_sp::val_xxx() methods. Sorry, I don't understand that. Why cannot you use Field_row here? > + */ > + if (Field_row *field_row= dynamic_cast<Field_row*>(sp_result_field)) > + { > + if (!(sp_result_item_field_row= > + m_sp->m_return_field_def.make_item_field_row(thd, field_row))) > + DBUG_RETURN(true); > + } > + > if (sp_result_field->pack_length() > sizeof(result_buf)) > { > void *tmp; > diff --git a/sql/sp.cc b/sql/sp.cc > index ddaeeb8cb7a..7eab8304c31 100644 > --- a/sql/sp.cc > +++ b/sql/sp.cc > @@ -1091,18 +1136,19 @@ sp_returns_type(THD *thd, String &result, const sp_head *sp) > bzero((char*) &share, sizeof(share)); > table.in_use= thd; > table.s = &share; > - field= sp->create_result_field(0, 0, &table); > - field->sql_type(result); > + field= create_result_field(0, 0, &table); > > - if (field->has_charset()) > + if (m_return_field_def.is_row()) > { > - result.append(STRING_WITH_LEN(" CHARSET ")); > - result.append(field->charset()->cs_name); > - if (Charset(field->charset()).can_have_collate_clause()) > - { > - result.append(STRING_WITH_LEN(" COLLATE ")); > - result.append(field->charset()->coll_name); > - } > + Field_row *field_row= dynamic_cast<Field_row*>(field); > + if (!field_row->row_create_fields( > + thd, m_return_field_def.row_field_definitions())) > + field->sql_type(result); when is this used? you're creating a TABLE and Field's - only to print types and charsets. This might be tolerable, but only if it's not needed often > + } > + else > + { > + DBUG_ASSERT(m_return_field_def.type_handler()->is_scalar_type()); > + field->sql_type_for_sp_returns(result); > } > > delete field; > diff --git a/sql/sql_yacc.yy b/sql/sql_yacc.yy > index 5977e309ae4..f11e4f89466 100644 > --- a/sql/sql_yacc.yy > +++ b/sql/sql_yacc.yy > @@ -18347,7 +18338,50 @@ sp_parameters: > ; > > sf_returned_type_clause: > - RETURNS_SYM sf_return_type > + RETURNS_SYM > + { > + LEX *lex= Lex; you still do this lex=Lex? :) it became obsolete, like, 15 years ago. If not more. > + lex->init_last_field(&lex->sphead->m_return_field_def, > + &empty_clex_str); > + } > + sf_return_type > + ; > + > +sf_return_type: > + field_type > + { > + if (unlikely(Lex->sf_return_fill_definition($1))) > + MYSQL_YYABORT; > + } > + | ROW_SYM row_type_body > + { > + if (Lex->sf_return_fill_definition_row($2)) > + MYSQL_YYABORT; > + } > + | ROW_SYM TYPE_SYM OF_SYM ident > + { > + if (Lex->sf_return_fill_definition_rowtype_of( > + Qualified_column_ident(&$4))) > + MYSQL_YYABORT; > + } > + | ROW_SYM TYPE_SYM OF_SYM ident '.' ident > + { > + if (Lex->sf_return_fill_definition_rowtype_of( > + Qualified_column_ident(&$4, &$6))) > + MYSQL_YYABORT; > + } > + | TYPE_SYM OF_SYM ident '.' ident > + { > + if (Lex->sf_return_fill_definition_type_of( > + Qualified_column_ident(&$3, &$5))) > + MYSQL_YYABORT; > + } > + | TYPE_SYM OF_SYM ident '.' ident '.' ident > + { > + if (Lex->sf_return_fill_definition_type_of( > + Qualified_column_ident(thd, &$3, &$5, &$7))) > + MYSQL_YYABORT; > + } > ; > > package_implementation_item_declaration:

2 1

Clean up executable test result files:
by Otto Kekäläinen 04 Sep '24

04 Sep '24

Hi! In https://github.com/MariaDB/server/commit/9cb75f333fa2ae816909f95a4e19e6ddac… Alexander made some test result files executable (look at 100644 → 100755 in git diff). I sent twice a PR to fix this: - Oct 2023: https://github.com/MariaDB/server/pull/2779 - Feb 2024: https://github.com/MariaDB/server/pull/3042 Unfortunately, I can see that on the main branch this is executable again: mariadb-server main+ ± find mysql-test -executable -name *.test -or -executable -name *.result mysql-test/main/lowercase_table2.result Could the next person doing merges fix this tiny annoyance once and for all? Thanks :)

1 0

Re: [PATCH] MDEV-32014 Rename binlog cache temporary file to binlog file for large transaction
by Kristian Nielsen 03 Sep '24

03 Sep '24

Andrei Elkin <andrei.elkin(a)mariadb.com> writes: >> So please do a small refactoring of trx_group_commit_leader() and keep >> only ... > > and no need for refactoring really arises. Well, the refactoring is good, even without the MDEV-32014 patch. The function trx_group_commit_leader() is already too long and complicated, and does too much in a single function. And the first part (obtaining the queue) is cleanly separated from the second part (committing each entry in the queue in-order in binlog and participating engines). It's just nice that this improvement of the existing code also makes the MDEV-32014 patch integrate cleaner. > I noted earlier on this part on the PR page and LiBing seems to have > practically proven (by the most recent commit that you must've missed) that > conducting the rename inside the flush stage of > trx_group_commit_leader() is viable: > > https://github.com/MariaDB/server/pull/3402/commits/287511a703f7dff47939633… > > By that measure the flushing is expanded onto possibly multiple "implicitly" > rotating log files Hm, I don't like the change 287511a703f7dff47939633475edd436a542cf6f. In fact, that is one of the things I like the most about Libing Song's patch, that it so cleanly separates the commit-by-rotate logic from the complex group commit logic in queue_for_group_commit() and trx_group_commit_leader(), which is already too complex. The original patch without 287511a703f7dff47939633475edd436a542cf6f seems much cleaner. It is better to do as in the original patch, and treat commit-by-rotate like a normal binlog rotate, not try to participate that transaction in binlog group commit. We cannot share the fsync anyway, as the files are different. And it introduces a lot of complexities to suddenly have binlog rotations in the middle of group commit - what about the complex accounting around xid_count and binlog checkpoints? There will be a lot of corner cases eg. if _two_ commit-by-rotate happens in the same group commit, with lots of potential for subtle bugs that will occur only very rarely in production and be almost impossible to track down. I strongly prefer the original patch I reviewed (with my suggested changes to write_transaction_to_binlog_events() and trx_group_commit_leader()), where the commit-by-rotate logic is cleanly separated from the group commit logic and implemented mostly using the existing binlog rotate logic. - Kristian.

2 2

Re: [PATCH] MDEV-34857: Implement --slave-abort-blocking-timeout
by Michael Widenius 03 Sep '24

03 Sep '24

Review of MDEV-34857: Implement --slave-abort-blocking-timeout If a slave replicating an event has waited for more than @@slave_abort_blocking_timeout for a conflicting metadata lock held by a non-replication thread, the blocking query is killed to allow replication to proceed and not be blocked indefinitely by a user query. <cut> diff --git a/sql/mdl.cc b/sql/mdl.cc index faccd1c9476..9845718e165 100644 --- a/sql/mdl.cc +++ b/sql/mdl.cc <cut> @@ -2397,14 +2398,39 @@ MDL_context::acquire_lock(MDL_request *mdl_request, double lock_wait_timeout) find_deadlock(); - struct timespec abs_timeout, abs_shortwait; + struct timespec abs_timeout, abs_shortwait, abs_abort_blocking_timeout; + bool abort_blocking_enabled= false; + double abort_blocking_timeout= slave_abort_blocking_timeout; Add a comment here. Something like /* This code implements aborting of queries on the slaves that stops replication from continuing. */ + if (abort_blocking_timeout < lock_wait_timeout && + m_owner->get_thd()->rgi_slave) + { + set_timespec_nsec(abs_abort_blocking_timeout, + (ulonglong)(abort_blocking_timeout * 1000000000ULL)); + abort_blocking_enabled= true; + } set_timespec_nsec(abs_timeout, (ulonglong)(lock_wait_timeout * 1000000000ULL)); - set_timespec(abs_shortwait, 1); wait_status= MDL_wait::EMPTY; - while (cmp_timespec(abs_shortwait, abs_timeout) <= 0) + for (;;) { + bool abort_blocking= false; + set_timespec(abs_shortwait, 1); + if (abort_blocking_enabled && + cmp_timespec(abs_shortwait, abs_abort_blocking_timeout) >= 0) + { + /* + If a slave DDL has waited for --slave-abort-select-timeout, then notify + any blocking SELECT once before continuing to wait until the full + timeout. + */ + abs_shortwait= abs_abort_blocking_timeout; + abort_blocking= true; + abort_blocking_enabled= false; + } + if (cmp_timespec(abs_shortwait, abs_timeout) > 0) + break; Is the above right last compare right? Assuming that cmp_timespec(abs_shortwait, abs_abort_blocking_timeout) > 0 then the above will not break and we will not signal other threads to break even if the previous code expectrs us to abort blocking. (I think the cmp_timespec() return test should be the same in both cases) Shouldn't the code be: for (;;) { bool abort_blocking= false; set_timespec(abs_shortwait, 1); if (abort_blocking_enabled && cmp_timespec(abs_shortwait, abs_abort_blocking_timeout) >= 0) { /* If a slave DDL has waited for --slave-abort-select-timeout, then notify any blocking SELECT once before continuing to wait until the full timeout. */ abort_blocking= true; abort_blocking_enabled= false; } else if (cmp_timespec(abs_shortwait, abs_timeout) > 0) break; Or have I missed something? /* abs_timeout is far away. Wait a short while and notify locks. */ wait_status= m_wait.timed_wait(m_owner, &abs_shortwait, FALSE, mdl_request->key.get_wait_state_name()); @@ -2425,9 +2451,8 @@ MDL_context::acquire_lock(MDL_request *mdl_request, double lock_wait_timeout) mysql_prlock_wrlock(&lock->m_rwlock); if (lock->needs_notification(ticket)) - lock->notify_conflicting_locks(this); + lock->notify_conflicting_locks(this, abort_blocking); mysql_prlock_unlock(&lock->m_rwlock); - set_timespec(abs_shortwait, 1); I like the new code in that we are not setting abs_shortwait two times anymore. (Using a while loop in the is case was sub optimal. Ok to push after the above change or if you have a good argument why the current code is correct

1 0

Re: [PATCH] MDEV-32014 Rename binlog cache temporary file to binlog file for large transaction
by andrei.elkin＠pp.inet.fi 02 Sep '24

02 Sep '24

(Re-sending to the list) Howdy Kristian. > > The patch calls the trx_group_commit_leader() function. That is ackward, as > commit-by-rotate does not actually participate in group commit. But the > problem is with the existing function trx_group_commit_leader(), which does > too much in a single function. I noted earlier on this part on the PR page and LiBing seems to have practically proven (by the most recent commit that you must've missed) that conducting the rename inside the flush stage of trx_group_commit_leader() is viable: https://github.com/MariaDB/server/pull/3402/commits/287511a703f7dff47939633… By that measure the flushing is expanded onto possibly multiple "implicitly" rotating log files > > So please do a small refactoring of trx_group_commit_leader() and keep > only ... and no need for refactoring really arises. Could you please assess that approach, cheers, Andrei

1 0

Next LTS version in February 2025?
by Otto Kekäläinen 29 Aug '24

29 Aug '24

Hi! Traditionally there has been a new major MariaDB version in February of the year Debian has a new major version. What is the plan for February 2025? I was reading https://mariadb.org/about/#maintenance-policy and old mariadb.orb blog posts but none was explicit about this.

2 2

Re: fec3696fef6: MDEV-15751 CURRENT_TIMESTAMP should return a TIMESTAMP [WITH TIME ZONE?]
by Sergei Golubchik 29 Aug '24

29 Aug '24

Hi, Alexander, This has waited far too long :( > commit fec3696fef6 > Author: Alexander Barkov <bar(a)mariadb.com> > Date: Tue Feb 14 13:27:46 2023 +0400 > > diff --git a/sql/sql_class.h b/sql/sql_class.h > index f0bf695a1c1..7378225eaa2 100644 > --- a/sql/sql_class.h > +++ b/sql/sql_class.h > @@ -3900,7 +3900,8 @@ class THD: public THD_count, /* this must be first */ > } > const Type_handler *type_handler_for_datetime() const; > bool timestamp_to_TIME(MYSQL_TIME *ltime, my_time_t ts, > - ulong sec_part, date_mode_t fuzzydate); > + ulong sec_part, date_mode_t fuzzydate, > + bool zero_timesteamp_means_zero_datetime); I don't understand it. NOW() cannot return a zero timestamp, how this feature can possibly need changes to zero timestamp treatment? > inline my_time_t query_start() { return start_time; } > inline ulong query_start_sec_part() > { used|= QUERY_START_SEC_PART_USED; return start_time_sec_part; } > diff --git a/sql/item_timefunc.h b/sql/item_timefunc.h > index a5f6d9307c6..1c463c8a332 100644 > --- a/sql/item_timefunc.h > +++ b/sql/item_timefunc.h > @@ -707,6 +707,49 @@ class Item_datetimefunc :public Item_func > }; > > > +class Item_timestampfunc: public Item_func > +{ > +protected: > + Datetime to_datetime(THD *thd) > + { > + return Timestamp_or_zero_datetime_native_null(thd, this).to_datetime(thd); how does it differ from Datetime(this) ? > + } > +public: > + Item_timestampfunc(THD *thd): Item_func(thd) {} > + Item_timestampfunc(THD *thd, Item *a): Item_func(thd, a) {} > + const Type_handler *type_handler() const override > + { return &type_handler_timestamp2; } > + bool get_date(THD *thd, MYSQL_TIME *ltime, date_mode_t fuzzydate) override > + { > + return null_value= to_datetime(thd).copy_to_mysql_time(ltime); > + } > + double val_real() override > + { > + Datetime dt= to_datetime(current_thd); > + null_value= !dt.is_valid_datetime(); > + return dt.to_double(); may be `return null_value ? 0 : dt.to_double()` ? why to call to_double if it isn't a valid datetime? (same everywhere below) > + } > + longlong val_int() override > + { > + Datetime dt= to_datetime(current_thd); > + null_value= !dt.is_valid_datetime(); > + return dt.to_longlong(); > + } > + my_decimal *val_decimal(my_decimal *to) override > + { > + Datetime dt= to_datetime(current_thd); > + null_value= !dt.is_valid_datetime(); > + return dt.to_decimal(to); > + } > + String *val_str(String *to) override > + { > + Datetime dt= to_datetime(current_thd); > + null_value= !dt.is_valid_datetime(); > + return dt.to_string(to, decimals); > + } > +}; > + > + > /* Abstract CURTIME function. Children should define what time zone is used */ > > class Item_func_curtime :public Item_timefunc > @@ -837,20 +880,48 @@ class Item_func_now :public Item_datetimefunc > }; > > > -class Item_func_now_local :public Item_func_now > +/* > + This is a replacement for Item_func_now_local, > + returning TIMESTAMP instead of DATETIME. why did you rename it? why didn't you change Item_func_now_utc? > +*/ > +class Item_func_current_timestamp: public Item_timestampfunc > { > public: > - Item_func_now_local(THD *thd, uint dec): Item_func_now(thd, dec) {} > + Item_func_current_timestamp(THD *thd, uint dec) > + :Item_timestampfunc(thd) > + { decimals= dec; } why not Item_timestampfunc(thd, dec) ? > LEX_CSTRING func_name_cstring() const override > { > static LEX_CSTRING name= {STRING_WITH_LEN("current_timestamp") }; > return name; > } > - int save_in_field(Field *field, bool no_conversions) override; > - void store_now_in_TIME(THD *thd, MYSQL_TIME *now_time) override; > + void print(String *str, enum_query_type query_type) override > + { > + str->append(func_name_cstring()); > + str->append('('); > + if (decimals) > + str->append_ulonglong(decimals); > + str->append(')'); > + } > + bool fix_length_and_dec(THD *thd) override > + { > + if (check_decimal_scale_or_error(TIME_SECOND_PART_DIGITS)) > + return true; > + fix_attributes_datetime(decimals); > + return false; > + } > + bool val_native(THD *thd, Native *to) override; > + bool check_vcol_func_processor(void *arg) override > + { > + /* > + NOW is safe for replication as slaves will run with same time as > + master > + */ what does this replication comment do in check_vcol_func_processor? (I know it was in the old code too, but "old code did it" never stopped your cleanups before :) > + return mark_unsupported_function(func_name(), "()", arg, VCOL_TIME_FUNC); > + } > enum Functype functype() const override { return NOW_FUNC; } > Item *get_copy(THD *thd) override > - { return get_item_copy<Item_func_now_local>(thd, this); } > + { return get_item_copy<Item_func_current_timestamp>(thd, this); } > }; > > > diff --git a/sql/item_timefunc.cc b/sql/item_timefunc.cc > index 52b9ae7a682..3e224e3e6f7 100644 > --- a/sql/item_timefunc.cc > +++ b/sql/item_timefunc.cc > @@ -1642,32 +1633,10 @@ void Item_func_now::print(String *str, enum_query_type query_type) > } > > > -int Item_func_now_local::save_in_field(Field *field, bool no_conversions) > +bool Item_func_current_timestamp::val_native(THD *thd, Native *to) > { > - if (field->type() == MYSQL_TYPE_TIMESTAMP) > - { > - THD *thd= field->get_thd(); > - my_time_t ts= thd->query_start(); > - ulong sec_part= decimals ? thd->query_start_sec_part() : 0; > - sec_part-= my_time_fraction_remainder(sec_part, decimals); > - field->set_notnull(); > - field->store_timestamp(ts, sec_part); > - return 0; > - } > - else > - return Item_datetimefunc::save_in_field(field, no_conversions); > -} > - > - > -/** > - Converts current time in my_time_t to MYSQL_TIME representation for local > - time zone. Defines time zone (local) used for whole NOW function. > -*/ > -void Item_func_now_local::store_now_in_TIME(THD *thd, MYSQL_TIME *now_time) > -{ > - thd->variables.time_zone->gmt_sec_to_TIME(now_time, thd->query_start()); > - set_sec_part(thd->query_start_sec_part(), now_time, this); > - thd->used|= THD::TIME_ZONE_USED; > + Timestamp ts(Timeval(thd->query_start(), thd->query_start_sec_part())); > + return null_value= ts.trunc(decimals).to_native(to, decimals); can this possibly fail? > } > > > @@ -2732,7 +2693,6 @@ String *Item_func_tochar::val_str(String* str) > > bool Item_func_from_unixtime::fix_length_and_dec(THD *thd) > { > - thd->used|= THD::TIME_ZONE_USED; would be nice to have a test case for this. Like direct insert into a timestamp column no longer uses timezone - e.g. timestamp within dst switch time is inserted correctly. and vice versa, when from_unixtime is converted to a string, like `insert ... values(concat(from_unixtime(...)))` then TIME_ZONE_USED is still set and show binlog events (or mysqlbinlog) should show that the timezone was correctly binlogged, as before. > tz= thd->variables.time_zone; > Type_std_attributes::set( > Type_temporal_attributes_not_fixed_dec(MAX_DATETIME_WIDTH, Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 1

MariaDB patches in Debian
by Otto Kekäläinen 27 Aug '24

27 Aug '24

Hi! Could perhaps Tuukka or Daniel review the patches currently carried in Debian if some of them would be applicable to be imported or re-implemented directly upstream so they can be dropped in Debian? https://udd.debian.org/patches.cgi?src=mariadb&version=1%3A11.4.3-1 Thanks

1 0

Re: [MariaDB commits] [PATCH 1/6] MDEV-34696: do_gco_wait() completes too early on InnoDB dict stats updates
by andrei.elkin＠pp.inet.fi 23 Aug '24

23 Aug '24

Howdy Kristian. > Before doing mark_start_commit(), check that there is no pending deadlock > kill. If there is a pending kill, we won't commit (we will abort, roll back, > and retry). Then we should not mark the commit as started, since that could > potentially make the following GCO start too early, before we completed the > commit after the retry. > > This condition could trigger in some corner cases, where InnoDB would take > temporarily table/row locks that are released again immediately, not held > until the transaction commits. This happens with dict_stats updates and > possibly auto-increment locks. > > Such locks can be passed to thd_rpl_deadlock_check() and cause a deadlock > kill to be scheduled in the background. But since the blocking locks are > held only temporarily, they can be released before the background kill > happens. This way, the kill can be delayed until after mark_start_commit() > has been called. Thus we need to check the synchronous indication > rgi->killed_for_retry, not just the asynchroneous thd->killed. I think I understood what's going on, also thanks to a verbose Jira ticket description. > > Signed-off-by: Kristian Nielsen <knielsen(a)knielsen-hq.org> > --- > sql/rpl_parallel.cc | 20 ++++++++++++++++---- > sql/rpl_rli.cc | 17 +++++++++++++++++ > 2 files changed, 33 insertions(+), 4 deletions(-) > > diff --git a/sql/rpl_parallel.cc b/sql/rpl_parallel.cc > index 1cfdf96ee3b..9c4222d7817 100644 > --- a/sql/rpl_parallel.cc > +++ b/sql/rpl_parallel.cc > @@ -1450,11 +1450,23 @@ handle_rpl_parallel_thread(void *arg) > after mark_start_commit(), we have to unmark, which has at least a > theoretical possibility of leaving a window where it looks like all > transactions in a GCO have started committing, while in fact one > - will need to rollback and retry. This is not supposed to be possible > - (since there is a deadlock, at least one transaction should be > - blocked from reaching commit), but this seems a fragile ensurance, > - and there were historically a number of subtle bugs in this area. > + will need to rollback and retry. > + > + Normally this will not happen, since the kill is there to resolve a > + deadlock that is preventing at least one transaction from proceeding. > + One case it can happen is with InnoDB dict stats update, which can > + temporarily cause transactions to block each other, but locks are > + released immediately, they don't linger until commit. There could be > + other similar cases, there were historically a number of subtle bugs > + in this area. > + > + But once we start the commit, we can expect that no new lock > + conflicts will be introduced. So by handling any lingering deadlock > + kill at this point just before mark_start_commit(), we should be > + robust even towards spurious deadlock kills. > */ > + if (rgi->killed_for_retry != rpl_group_info::RETRY_KILL_NONE) > + wait_for_pending_deadlock_kill(thd, rgi); Assuming my understanding, please correct me if anything, I left a question on this block in https://github.com/MariaDB/server/commit/df0c36a354ffde2766ebed2615642c74b7… quoted further. ... this guard would not let a victim of an optimistic conflict proceed until it clears out itself it is such a victim. For instance in the binlog sequence like T_1, T_2, D_3 where T_2 is the victim, D_3 is a DDL, the victim T_2 won't anymore release the next gco member D_3 into its execution. But what if at this point T_2 is only going to be marked in rgi->killed_for_retry by T_1? That is it is apparently able, with this patch as well, having ...NONE killed status go through. T_1 would reach this very point later and when that is done before T_2 finds itself killed, T_1 would release D_3, and afterward T_2 would finally see itself KILLed into retry. Cheers, Andrei

2 4

Open PRs pending second approver (has already one)
by Otto Kekäläinen 23 Aug '24

23 Aug '24

Hi! There are couple PRs that have 1 approval, and are pending the second one. Would somebody have time to review them? In order of importance: https://github.com/MariaDB/server/pull/3401 "MDEV-34680 Asynchronous and Buffered Logging in MySQL Audit Plugin" https://github.com/MariaDB/server/pull/3406 (related but zero approvals: https://github.com/MariaDB/server/pull/3324 "MDEV-12182 Add client tcp port to audit plugin logs") https://github.com/MariaDB/server/pull/3208 "Refine OpenSSL test names and allow more complete OpenSSL version coverage" https://github.com/MariaDB/server/pull/3225 "MDEV-29955: add ${PCRE_LIBRARY_PATH} to link directories" Thanks, Otto

1 0

Re: 186aa192d1f: MDEV-34704 Quick mode produces the bug for mariadb client
by Sergei Golubchik 22 Aug '24

22 Aug '24

Hi, Sanja, ok to push, but see one comment below On Aug 22, Oleksandr Byelkin wrote: > revision-id: 186aa192d1f (mariadb-11.5.1-34-g186aa192d1f) > parent(s): 9e1923ca238 > author: Oleksandr Byelkin > committer: Oleksandr Byelkin > timestamp: 2024-08-05 16:16:09 +0200 > message: > > MDEV-34704 Quick mode produces the bug for mariadb client > > --quick-max-column-width parameter added to limit field > width in --quick mode. > > diff --git a/mysql-test/main/client.test b/mysql-test/main/client.test > new file mode 100644 > index 00000000000..59e687ff8b9 > --- /dev/null > +++ b/mysql-test/main/client.test > @@ -0,0 +1,33 @@ > +--echo # > +--echo # MDEV-34704: Quick mode produces the bug for mariadb client > +--echo # > + > + > +create table t1 (aaaaaaaaa char (5), aaaaa char (10), a char (127), b char(1)); > +insert into t1 values ("X", "X", "X", "X"); > + to make the result file more readable, I'd suggest to add here, like --echo mysql --table --quick ... --echo mysql --table --quick --quick-max-column-width=0 ... etc before every corresponding --exec > +--exec echo "select * from test.t1;" | $MYSQL --table --quick 2>&1 > + > +--exec echo "select * from test.t1;" | $MYSQL --table --quick --quick-max-column-width=0 2>&1 > + > +--exec echo "select * from test.t1;" | $MYSQL --table --quick --quick-max-column-width=10 2>&1 > + > +--exec echo "select * from test.t1;" | $MYSQL --table --quick --quick-max-column-width=20 2>&1 > + > +insert into t1 values ("01234", "0123456789", "01234567890123456789", "1"); > + > +--exec echo "select * from test.t1;" | $MYSQL --table --quick 2>&1 > + > + > +--exec echo "select * from test.t1;" | $MYSQL --table --quick --quick-max-column-width=0 2>&1 > + > +--exec echo "select * from test.t1;" | $MYSQL --table --quick --quick-max-column-width=10 2>&1 > + > +--exec echo "select * from test.t1;" | $MYSQL --table --quick --quick-max-column-width=20 2>&1 > + > +drop table t1; > + > +--echo # > +--echo # End of 10.7 tests > +--echo # As Marko said, use here only one line --echo # End of 10.7 tests otherwise merge mixes it up with --echo # --echo # MDEV-34704: Quick mode produces the bug for mariadb client --echo # Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

1 0

Suggestion: use 'main' as the development branch instead of an ever changing 11.x
by Otto Kekäläinen 22 Aug '24

22 Aug '24

Hi! Has the core developers had discussions about using branch name 'main' for development instead of switching to a new branch every 3 months? If all new features and other additions would always target the branch 'main' it would lower the barrier of entry for new contributors, and decrease the number of unnecessary rebases done by contributors. If the permanent development target was 'main', you would need to create new 11.x branches only at "freeze" when a new major version branch is cut for release. MariaDB is currently the only open source project I know that does *not* have a main/master/trunk branch and instead does these constant new branch names. I imagine this is not only annoying for contributors, but also the core developers who waste time asking contributors to rebase PRs every 3 months (see https://github.com/MariaDB/server/pull/2556 and https://github.com/MariaDB/server/pull/2671 as examples). - Otto

7 11

Re: [MariaDB commits] [PATCH 3/6] Fix sporadic test failure in rpl.rpl_create_drop_event
by Kristian Nielsen 22 Aug '24

22 Aug '24

Andrei Elkin <andrei.elkin(a)mariadb.com> writes: > Kristian Nielsen via developers <developers(a)lists.mariadb.org> writes: >> The already-started instance of the event happily continues to run after >> being dropped. I wonder if that's a bug, or if the documentation is >> wrong? > > To this question, I vaguely remember discussions about this matter which > must've concuded in that it's not a bug when the scheduler races ahead > of DROP EVENT to launch a piece of job, like INSERT in this case. Aha, I see. Following up on this, I found this comment in the code Stops the scheduler (again). Waits for acknowledgement from the scheduler that it has stopped - synchronous stopping. Already running events will not be stopped. If the user needs them stopped manual intervention is needed. So then the behaviour seen in the test case is expected, and just fixing the test to handle the possible race should be correct. I also see in the code that the thread running the events starts by loading the event definition from the system tables (mysql.event ...), db_repository->load_named_event(). So there should be no memory issues even if the event is dropped while an instance of it is running. Thanks! - Kristian.

1 0

Re: [MariaDB commits] [PATCH 3/6] Fix sporadic test failure in rpl.rpl_create_drop_event
by Kristian Nielsen 21 Aug '24

21 Aug '24

Andrei Elkin <andrei.elkin(a)mariadb.com> writes: >> My question is if it is ok and by design that the event can continue running >> after the scheduler is stopped and even after the event is dropped? That > Should not be ok. The docs https://mariadb.com/kb/en/drop-event/ > > are explicit > > Description > This statement drops the event named event_name. The event immediately ceases being active, and is deleted completely from the server. Thanks for the link, Andrei! Interesting, this doesn't seem to be how it works. This small test starts an event that runs for 3 seconds, but drops the event after 1 second: ----------------------------------------------------------------------- SET GLOBAL event_scheduler= OFF; CREATE TABLE t1(a INT); CREATE EVENT ev1 ON SCHEDULE EVERY 5 SECOND DO INSERT INTO t1 VALUES(SLEEP(3)); SET GLOBAL event_scheduler= ON; --sleep 1 SHOW FULL PROCESSLIST; SET GLOBAL event_scheduler= OFF; DROP EVENT ev1; SHOW FULL PROCESSLIST; --sleep 1 SHOW FULL PROCESSLIST; DROP TABLE t1; ----------------------------------------------------------------------- The already-started instance of the event happily continues to run after being dropped. I wonder if that's a bug, or if the documentation is wrong? And if it is a bug, if there are any use-after-free or similar memory issues with the event continuing to run after being dropped? Valgrind doesn't complain... ----------------------------------------------------------------------- SET GLOBAL event_scheduler= OFF; CREATE TABLE t1(a INT); CREATE EVENT ev1 ON SCHEDULE EVERY 5 SECOND DO INSERT INTO t1 VALUES(SLEEP(3)); Warnings: Warning 1105 Event scheduler is switched off, use SET GLOBAL event_scheduler=ON to enable it. SET GLOBAL event_scheduler= ON; SHOW FULL PROCESSLIST; Id User Host db Command Time State Info Progress 4 root localhost test Query 0 starting SHOW FULL PROCESSLIST 0.000 5 event_scheduler localhost NULL Daemon 1 Waiting for next activation NULL 0.000 6 root localhost test Connect 1 User sleep INSERT INTO t1 VALUES(SLEEP(3)) 0.000 SET GLOBAL event_scheduler= OFF; DROP EVENT ev1; SHOW FULL PROCESSLIST; Id User Host db Command Time State Info Progress 4 root localhost test Query 0 starting SHOW FULL PROCESSLIST 0.000 6 root localhost test Connect 1 User sleep INSERT INTO t1 VALUES(SLEEP(3)) 0.000 SHOW FULL PROCESSLIST; Id User Host db Command Time State Info Progress 4 root localhost test Query 0 starting SHOW FULL PROCESSLIST 0.000 6 root localhost test Connect 2 User sleep INSERT INTO t1 VALUES(SLEEP(3)) 0.000 DROP TABLE t1; ----------------------------------------------------------------------- - Kristian.

1 0

Re: [PATCH 1/6] MDEV-34696: do_gco_wait() completes too early on InnoDB dict stats updates
by Kristian Nielsen 20 Aug '24

20 Aug '24

Andrei Elkin <andrei.elkin(a)mariadb.com> writes: > Howdy Kristian. > I was going to review. Thanks Andrei! > Is the patch is any github branch? Looks so by the MDEV. It'd be good to > have it there for reviewing either. Yes, the entire patch series is in bb-10.5-knielsen: https://github.com/MariaDB/server/commits/bb-10.5-knielsen This particular issue is the commit: MDEV-34696: do_gco_wait() completes too early on InnoDB dict stats updates - Kristian.

1 0

Re: [MariaDB commits] [PATCH 3/6] Fix sporadic test failure in rpl.rpl_create_drop_event
by Kristian Nielsen 20 Aug '24

20 Aug '24

Hi Bar, I want to ask you about this fix of a sporadic failure of the test rpl.rpl_create_drop_event, which is authored by you. Not so much the test case fix, which should be good, but I want to ask someone who knows the event scheduler code better than me to be sure I am not merely hiding a real bug with the test case fix. What happens in the test case when it failed is that a new run of an event is started, and this run can continue while the event scheduler is stopped and even after the event itself is dropped: CREATE OR REPLACE EVENT ev1 ON SCHEDULE EVERY 1 SECOND DO INSERT INTO t1 VALUES (11); ... SET GLOBAL event_scheduler=off; DROP EVENT ev1; My question is if it is ok and by design that the event can continue running after the scheduler is stopped and even after the event is dropped? That this doesn't cause any problems like accessing freed memory or such? Or if you do not know, if you know who else I could ask? Thanks, - Kristian. Kristian Nielsen via commits <commits(a)lists.mariadb.org> writes: > Depending on timing, an extra event run could start just when the event > scheduler is shut down and delay running until after the table has been > dropped; this would cause the test to fail with a "table does not exist" > error in the log. > > Signed-off-by: Kristian Nielsen <knielsen(a)knielsen-hq.org> > --- > mysql-test/suite/rpl/t/rpl_create_drop_event.test | 6 ++++++ > 1 file changed, 6 insertions(+) > > diff --git a/mysql-test/suite/rpl/t/rpl_create_drop_event.test b/mysql-test/suite/rpl/t/rpl_create_drop_event.test > index 96a7e82d6f7..79bb0ffec90 100644 > --- a/mysql-test/suite/rpl/t/rpl_create_drop_event.test > +++ b/mysql-test/suite/rpl/t/rpl_create_drop_event.test > @@ -14,6 +14,12 @@ SET GLOBAL event_scheduler=on; > let $wait_condition= SELECT count(*)>0 FROM t1; > --source include/wait_condition.inc > SET GLOBAL event_scheduler=off; > +# If the time rolls to the next whole second just at this point, a new event > +# run may be scheduled. Wait for this to disappear, otherwise we see occasional > +# test failures if the table gets dropped before the extra event run completes. > +# Expect 5 connections: default, master, master1, server_1, binlog dump thread > +--let $wait_condition= SELECT COUNT(*) = 5 FROM INFORMATION_SCHEMA.PROCESSLIST; > +--source include/wait_condition.inc > SELECT DISTINCT a FROM t1; > DELETE FROM t1;

1 0

Re: [PATCH 1/6] MDEV-34696: do_gco_wait() completes too early on InnoDB dict stats updates
by Kristian Nielsen 20 Aug '24

20 Aug '24

Hi Brandon, Andrei, Does one of you want to review this patch for 10.5? It fixes a real (and serious) issue in parallel replication that is seen as sporadic failure of rpl.rpl_start_alter_4 in Buildbot. This is yet another occurrence of the problem where mark_start_commit() is done too early, this code unfortunately is fragile and has had a number of problems in the past. But I am quite confident about the patch, calling wait_for_pending_deadlock_kill() should be very safe, and should be correct here. The added assertion helps detect the problem more reliably and could also help catch any other remaining issues around mark_start_commit (though hopefully there are no more left now). - Kristian. Kristian Nielsen via commits <commits(a)lists.mariadb.org> writes: > Before doing mark_start_commit(), check that there is no pending deadlock > kill. If there is a pending kill, we won't commit (we will abort, roll back, > and retry). Then we should not mark the commit as started, since that could > potentially make the following GCO start too early, before we completed the > commit after the retry. > > This condition could trigger in some corner cases, where InnoDB would take > temporarily table/row locks that are released again immediately, not held > until the transaction commits. This happens with dict_stats updates and > possibly auto-increment locks. > > Such locks can be passed to thd_rpl_deadlock_check() and cause a deadlock > kill to be scheduled in the background. But since the blocking locks are > held only temporarily, they can be released before the background kill > happens. This way, the kill can be delayed until after mark_start_commit() > has been called. Thus we need to check the synchronous indication > rgi->killed_for_retry, not just the asynchroneous thd->killed. > > Signed-off-by: Kristian Nielsen <knielsen(a)knielsen-hq.org> > --- > sql/rpl_parallel.cc | 20 ++++++++++++++++---- > sql/rpl_rli.cc | 17 +++++++++++++++++ > 2 files changed, 33 insertions(+), 4 deletions(-) > > diff --git a/sql/rpl_parallel.cc b/sql/rpl_parallel.cc > index 1cfdf96ee3b..9c4222d7817 100644 > --- a/sql/rpl_parallel.cc > +++ b/sql/rpl_parallel.cc > @@ -1450,11 +1450,23 @@ handle_rpl_parallel_thread(void *arg) > after mark_start_commit(), we have to unmark, which has at least a > theoretical possibility of leaving a window where it looks like all > transactions in a GCO have started committing, while in fact one > - will need to rollback and retry. This is not supposed to be possible > - (since there is a deadlock, at least one transaction should be > - blocked from reaching commit), but this seems a fragile ensurance, > - and there were historically a number of subtle bugs in this area. > + will need to rollback and retry. > + > + Normally this will not happen, since the kill is there to resolve a > + deadlock that is preventing at least one transaction from proceeding. > + One case it can happen is with InnoDB dict stats update, which can > + temporarily cause transactions to block each other, but locks are > + released immediately, they don't linger until commit. There could be > + other similar cases, there were historically a number of subtle bugs > + in this area. > + > + But once we start the commit, we can expect that no new lock > + conflicts will be introduced. So by handling any lingering deadlock > + kill at this point just before mark_start_commit(), we should be > + robust even towards spurious deadlock kills. > */ > + if (rgi->killed_for_retry != rpl_group_info::RETRY_KILL_NONE) > + wait_for_pending_deadlock_kill(thd, rgi); > if (!thd->killed) > { > DEBUG_SYNC(thd, "rpl_parallel_before_mark_start_commit"); > diff --git a/sql/rpl_rli.cc b/sql/rpl_rli.cc > index 9d4e09a362c..8284b07f7ff 100644 > --- a/sql/rpl_rli.cc > +++ b/sql/rpl_rli.cc > @@ -2518,6 +2518,23 @@ rpl_group_info::unmark_start_commit() > > e= this->parallel_entry; > mysql_mutex_lock(&e->LOCK_parallel_entry); > + /* > + Assert that we have not already wrongly completed this GCO and signalled > + the next one to start, only to now unmark and make the signal invalid. > + This is to catch problems like MDEV-34696. > + > + The error inject rpl_parallel_simulate_temp_err_xid is used to test this > + precise situation, that we handle it gracefully if it somehow occurs in a > + release build. So disable the assert in this case. > + */ > +#ifndef DBUG_OFF > + bool allow_unmark_after_complete= false; > + DBUG_EXECUTE_IF("rpl_parallel_simulate_temp_err_xid", > + allow_unmark_after_complete= true;); > + DBUG_ASSERT(!gco->next_gco || > + gco->next_gco->wait_count > e->count_committing_event_groups || > + allow_unmark_after_complete); > +#endif > --e->count_committing_event_groups; > mysql_mutex_unlock(&e->LOCK_parallel_entry); > }

1 0

Re: [PATCH] MDEV-34481 optimize away waiting for owned by prepared xa non-unique index record
by Kristian Nielsen 15 Aug '24

15 Aug '24

Andrei Elkin <andrei.elkin(a)mariadb.com> writes: >> This is not the way to communicate back from engine to server. Instead, >> simply return a different error code than HA_ERR_LOCK_WAIT_TIMEOUT from >> InnoDB that you can check for here. You can maybe use one of the existing >> HA_ERR_* from include/my_base.h, or add a new one if necessary. > However, adding a new error is clearly more intrusive. The new error is completely handled inside the code you add for signalling the condition inside InnoDB locking code and handling it in replication row event apply. It will not be visible to users. So it is not intrusive to add a new one. Just be careful to not pick a number that conflicts with another entry in a later branch, if you do this in a GA release. Alternatively, you can use the existing HA_ERR_ROW_NOT_VISIBLE error, which is used in a somewhat similar way in storage/maria/ma_blockrec.c and nowhere else. But I would just add a new error for clarity. > And at the same time I thought a some sort of runtime state the slave > applier had always missed. So the intent of this struct is future-minded > as well. This is not the point. Even if the rgi->exec_flags already existed, it would still be wrong to use it for this. Here is the code where you are handling the condition: > + if (error == HA_ERR_LOCK_WAIT_TIMEOUT && !is_index_unique && > + (rgi->exec_flags & (1 << rpl_group_info::HIT_BUSY_INDEX))) > + { > + rgi->exec_flags &= ~(1 << rpl_group_info::HIT_BUSY_INDEX); > + skip_first_compare= true; So you _already_ are using a specific error code to signal the condition from InnoDB to replication, the error HA_ERR_LOCK_WAIT_TIMEOUT. But this error can also be thrown in other situations, so you need to add these extra conditions to distinguish. So essentially, you are using the wrong error code to signal the condition. Use a correct error code that uniquely identifies the condition that occured, and then you can avoid these extra conditions and simplify the code. > I would be fine with either method. Good, then please change it to return a proper error code. The only reason that you are using HA_ERR_LOCK_WAIT_TIMEOUT is that then you could do the quick change inside InnoDB to set its timeout value to 0, and this happens to then propagate the HA_ERR_LOCK_WAIT_TIMEOUT up to the callers without waiting. But that's not an appropriate way to design new code and APIs. Yes, it is frustrating when conceptually simple changes need extra effort, due to the need to work in an existing old and complex code base like MariaDB. But that is just the way it is, it is the same for all of us. - Kristian.

1 0

Can we get MariaDB GitHub CI to consistently be green?
by Otto Kekäläinen 10 Aug '24

10 Aug '24

Hi! Thanks to everyone who has worked on polishing the CI tests and integrations. Looking at e.g. 10.11 branch[1] I see that commit ccb7a1e[2] has a green checkmark next to it and all 15 CI jobs passed. I just wanted to check if everyone developing the MariaDB Server is committed to getting the CI to be consistently green? This means that GitHub rules need to be a bit more strict, not allowing any failing tests jobs at all, and developers and managers need to agree to stop adding new commits on any release branch if it can't be done without a fully passing CI run. Thanks to Daniel's review on latest failures we know these are at the moment recurring: * MDEV-25614 Galera test failure on GCF-354 included in MDEV-33073 always green buildbot * MDEV-33785 aarch64 macos encryption.create_or_replace * MDEV-33601 galera_3nodes.galera_safe_to_bootstrap test failing * MDEV-33786 galera_3nodes.galera_vote_rejoin_mysqldump mysql_shutdown failed at line 85: I see two approaches to get to consistently green CI: 1) Stop all development and focus on just fixing these, don't continue until CI is fully green, and once it is fully green make the GitHub branch protection settings one notch stricter to not allow any new commits unless the CI is fully green so it never regresses again. 2) Disable these tests and make the rules in GitHub branch protection one notch stricter right away, and not allow any new commits unless the CI is fully green ensuring no new recurring failures are introduced. - Otto [1] https://github.com/MariaDB/server/commits/10.11 [2] https://github.com/MariaDB/server/commit/ccb7a1e9a15e6a47aba97f9bdbfab2e4bf…

7 12

Re: [PATCH] MDEV-34481 optimize away waiting for owned by prepared xa non-unique index record
by Kristian Nielsen 01 Aug '24

01 Aug '24

Andrei Elkin <andrei.elkin(a)mariadb.com> writes: > Indeed a combination of the engine locking protocol and the slave retry > applies correctly to IDEMPOTENT mode. >> The optimistic parallel replication should work correctly with >> IDEMPOTENT, > Indeed. Ok good to know. It's definitely a tricky issue, but apparently no known bugs, at least to the two of us. As you saw, I did have to try a test case to feel confident it works ;-) - Kristian.

1 0

Re: [PATCH] MDEV-34481 optimize away waiting for owned by prepared xa non-unique index record
by Kristian Nielsen 31 Jul '24

31 Jul '24

Andrei Elkin <andrei.elkin(a)mariadb.com> writes: > Kristian Nielsen <knielsen(a)knielsen-hq.org> writes: >> You need to make sure that --slave_exec_mode=IDEMPOTENT is not used. >> Otherwise you can get silent data corruption. > > IDEMPOTENT is always risky. Here it would add up another bit, in the > parallel mode. What do you mean "IDEMPOTENT is always risky"? As far as I know, if the slave would replicate without error in STRICT, it will replicate exactly the same way in IDEMPOTENT. Only if there is an error in STRICT mode will the IDEMPOTENT mode do something different (ignore the error in some cases). So the risk is that we might fail to detect that the slave is diverged from the master, right? Note that diverged slave can be undetected even in STRICT mode in many cases. > Btw should IDEMPOTENT be disallowed in the parallel mode actually? No, why? IDEMPOTENT should work just as documented in all parallel replication modes. Or do you know of any bug with IDEMPOTENT and optimistic parallel replication that I don't know of? If so do you have a test case? The optimistic parallel replication should work correctly with IDEMPOTENT, since the second query that's ignoring the error should still be taking InnoDB row locks, which will conflict as normal with the first transaction and cause the second to be retried, this time without (ignored) error. I just tried with the below testcase, it seems to work. Only when you fiddle with the InnoDB locking like in your patch and ignore certain locks, do you get problems, and then you can end up with a later transaction completely ignoring a lock that is temporarily held by an earlier XA PREPARE (even though the corresponding XA COMMIT is also earlier). And this will result in incorrect replication and diverged slave, I think (I did not test). This is not "risky", this is incorrect behaviour and a serious bug. So it seems you will need to maybe give an error when replicating any XA transaction if slave_exec_mode=IDEMPOTENT? Or handle it some other way to avoid silently replicating incorrectly. As you know, my opinion is that this is just yet another symptom of the real issue of MDEV-32020, but I know you're going to ignore that. So I've given up caring about what happens to XA replication, as long as it doesn't negatively affect non-XA stuff. > I just don't see any its usefullness except creating a chaotic state. Why do you think it creates a "chaotic state"? - Kristian. ----------------------------------------------------------------------- --source include/have_innodb.inc --source include/have_binlog_format_row.inc --source include/master-slave.inc --connection master CALL mtr.add_suppression("Can't find record in 't1'"); ALTER TABLE mysql.gtid_slave_pos ENGINE=InnoDB; CREATE TABLE t1 (a INT PRIMARY KEY, b INT) ENGINE=InnoDB; CREATE TABLE t2 (a INT PRIMARY KEY) ENGINE=InnoDB; BEGIN; INSERT INTO t1 VALUES (1, 1); INSERT INTO t1 VALUES (2, 1); INSERT INTO t1 VALUES (3, 1); INSERT INTO t1 VALUES (4, 1); INSERT INTO t1 VALUES (5, 1); COMMIT; --sync_slave_with_master --source include/stop_slave.inc SET @old_exec_mode= @@GLOBAL.slave_exec_mode; SET GLOBAL slave_exec_mode=IDEMPOTENT; SET @old_parallel_mode= @@GLOBAL.slave_parallel_mode; SET GLOBAL slave_parallel_mode= optimistic; SET @old_threads= @@GLOBAL.slave_parallel_threads; SET GLOBAL slave_parallel_threads= 8; --source include/start_slave.inc # Block certain queries on the slave temporarily. BEGIN; INSERT INTO t2 VALUES (1), (2), (3); --connection master UPDATE t1 SET b=10 WHERE a=1; BEGIN; INSERT INTO t2 VALUES (1); # A DELETE to make room for a later INSERT, will be delayed a bit on slave. DELETE FROM t1 WHERE a=2; COMMIT; UPDATE t1 SET b=20 WHERE a=3; BEGIN; INSERT INTO t2 VALUES (2); # An INSERT to be deleted again shortly, will be delayed a bit on slave. INSERT INTO t1 VALUES (6,2); COMMIT; UPDATE t1 SET b=30 WHERE a=4; # This INSERT cannot run until after the earlier DELETE. INSERT INTO t1 VALUES (2, 1); # This DELETE cannot run until after the earlier INSERT. DELETE FROM t1 WHERE a=6; SELECT * FROM t1 ORDER BY a; --save_master_pos --connection slave --sleep 1 ROLLBACK; --sync_with_master SELECT * FROM t1 ORDER BY a; --source include/stop_slave.inc SET GLOBAL slave_exec_mode= @old_exec_mode; SET GLOBAL slave_parallel_mode= @old_parallel_mode; SET GLOBAL slave_parallel_threads= @old_threads; --source include/start_slave.inc --connection master DROP TABLE t1, t2; --source include/rpl_end.inc

1 0

Re: 1287c901: TLS/SSL changes (major rework)
by Sergei Golubchik 31 Jul '24

31 Jul '24

Hi, Georg, On Jul 31, Georg Richter wrote: > revision-id: 1287c901 (v3.4.0-5-g1287c901) > parent(s): 5386f1a3 > author: Georg Richter > committer: Georg Richter > timestamp: 2024-07-16 13:12:26 +0200 > message: > > TLS/SSL changes (major rework) > diff --git a/include/ma_common.h b/include/ma_common.h > index dfa96621..dc900b0d 100644 > --- a/include/ma_common.h > +++ b/include/ma_common.h > @@ -131,15 +131,9 @@ typedef struct st_mariadb_field_extension > MARIADB_CONST_STRING metadata[MARIADB_FIELD_ATTR_LAST+1]; /* 10.5 */ > } MA_FIELD_EXTENSION; > > -#if defined(HAVE_SCHANNEL) || defined(HAVE_GNUTLS) > -#define reset_tls_self_signed_error(mysql) \ > - do { \ > - free((char*)mysql->net.tls_self_signed_error); \ > - mysql->net.tls_self_signed_error= 0; \ > - } while(0) > -#else > -#define reset_tls_self_signed_error(mysql) \ > - do { \ > - mysql->net.tls_self_signed_error= 0; \ > +#ifdef HAVE_TLS > +#define reset_tls_error(mysql) \ > + do { \ > + mysql->net.tls_verify_status= 0; \ you've lost the original error message when you've changed char* pointer to the message to my_bool flag. > } while(0) > #endif > diff --git a/include/ma_tls.h b/include/ma_tls.h > index 23f53560..444ea4aa 100644 > --- a/include/ma_tls.h > +++ b/include/ma_tls.h > @@ -134,6 +141,7 @@ const char *ma_tls_get_cipher(MARIADB_TLS *ssl); > hash_type hash_type as defined in ma_hash.h > fp buffer for fingerprint > fp_len buffer length > + my_bool verify_period I don't see any 'my bool verify_period' in the list of arguments > > Returns: > actual size of finger print > @@ -150,7 +158,7 @@ unsigned int ma_tls_get_finger_print(MARIADB_TLS *ctls, uint hash_type, char *fp > int ma_tls_get_protocol_version(MARIADB_TLS *ctls); > const char *ma_pvio_tls_get_protocol_version(MARIADB_TLS *ctls); > int ma_pvio_tls_get_protocol_version_id(MARIADB_TLS *ctls); > -unsigned int ma_tls_get_peer_cert_info(MARIADB_TLS *ctls); > +unsigned int ma_tls_get_peer_cert_info(MARIADB_TLS *ctls, unsigned int size); is this part of the public API? > void ma_tls_set_connection(MYSQL *mysql); > > /* Function prototypes */ > diff --git a/include/mysql.h b/include/mysql.h > index 6c84a462..ba92527d 100644 > --- a/include/mysql.h > +++ b/include/mysql.h > @@ -448,6 +449,16 @@ typedef struct st_mysql_time > #define MYSQL_WAIT_EXCEPT 4 > #define MYSQL_WAIT_TIMEOUT 8 > > +#define MARIADB_TLS_VERIFY_OK 0 > +#define MARIADB_TLS_VERIFY_TRUST 1 > +#define MARIADB_TLS_VERIFY_HOST 2 we've agreed to swap two previous lines > +#define MARIADB_TLS_VERIFY_PERIOD 4 > +#define MARIADB_TLS_VERIFY_FINGERPRINT 8 is that right? you want to allow fingerprints even if the cert is expired? > +#define MARIADB_TLS_VERIFY_REVOKED 16 > +#define MARIADB_TLS_VERIFY_UNKNOWN 32 > +#define MARIADB_TLS_VERIFY_ERROR 128 /* last */ > + > + > typedef struct character_set > { > unsigned int number; /* character set number */ > diff --git a/libmariadb/ma_tls.c b/libmariadb/ma_tls.c > index 1bda7dcf..f6ea6661 100644 > --- a/libmariadb/ma_tls.c > +++ b/libmariadb/ma_tls.c > @@ -103,9 +103,54 @@ my_bool ma_pvio_tls_close(MARIADB_TLS *ctls) > return ma_tls_close(ctls); > } > > -int ma_pvio_tls_verify_server_cert(MARIADB_TLS *ctls) > +int ma_pvio_tls_verify_server_cert(MARIADB_TLS *ctls, unsigned int flags) > { > - return ma_tls_verify_server_cert(ctls); > + MYSQL *mysql; > + int rc; > + > + if (!ctls || !ctls->pvio || !ctls->pvio->mysql) > + return 0; > + > + mysql= ctls->pvio->mysql; > + > + /* Skip peer certificate verification */ > + if (ctls->pvio->mysql->options.extension->tls_allow_invalid_server_cert) > + { > + return 0; > + } > + > + rc= ma_tls_verify_server_cert(ctls, flags); > + > + /* Set error messages */ > + if (!mysql->net.last_errno) > + { > + if (mysql->net.tls_verify_status & MARIADB_TLS_VERIFY_PERIOD) > + my_set_error(mysql, CR_SSL_CONNECTION_ERROR, SQLSTATE_UNKNOWN, > + ER(CR_SSL_CONNECTION_ERROR), > + "Certificate not yet valid or expired"); > + else if (mysql->net.tls_verify_status & MARIADB_TLS_VERIFY_FINGERPRINT) > + my_set_error(mysql, CR_SSL_CONNECTION_ERROR, SQLSTATE_UNKNOWN, > + ER(CR_SSL_CONNECTION_ERROR), > + "Fingerprint validation of peer certificate failed"); > + else if (mysql->net.tls_verify_status & MARIADB_TLS_VERIFY_REVOKED) > + my_set_error(mysql, CR_SSL_CONNECTION_ERROR, SQLSTATE_UNKNOWN, > + ER(CR_SSL_CONNECTION_ERROR), > + "Certificate revoked"); > + else if (mysql->net.tls_verify_status & MARIADB_TLS_VERIFY_HOST) > + my_set_error(mysql, CR_SSL_CONNECTION_ERROR, SQLSTATE_UNKNOWN, > + ER(CR_SSL_CONNECTION_ERROR), > + "Hostname verification failed"); > + else if (mysql->net.tls_verify_status & MARIADB_TLS_VERIFY_UNKNOWN) > + my_set_error(mysql, CR_SSL_CONNECTION_ERROR, SQLSTATE_UNKNOWN, > + ER(CR_SSL_CONNECTION_ERROR), > + "Peer certificate verification failed"); > + else if (mysql->net.tls_verify_status & MARIADB_TLS_VERIFY_TRUST) > + my_set_error(mysql, CR_SSL_CONNECTION_ERROR, SQLSTATE_UNKNOWN, > + ER(CR_SSL_CONNECTION_ERROR), > + "Peer certificate is not trusted"); let's try to preserve the error as set by the ssl library here > + } > + > + return rc; > } > > const char *ma_pvio_tls_cipher(MARIADB_TLS *ctls) > diff --git a/plugins/auth/my_auth.c b/plugins/auth/my_auth.c > index faea968c..e7429fae 100644 > --- a/plugins/auth/my_auth.c > +++ b/plugins/auth/my_auth.c > @@ -382,14 +419,30 @@ static int send_client_reply_packet(MCPVIO_EXT *mpvio, > errno); > goto error; > } > + mysql->net.tls_verify_status = 0; > if (ma_pvio_start_ssl(mysql->net.pvio)) > goto error; > - if (mysql->net.tls_self_signed_error && > - (!mysql->passwd || !mysql->passwd[0] || !hashing(mpvio->plugin))) > - { > - /* cannot use auth to validate the cert */ > - set_error_from_tls_self_signed_error(mysql); > - goto error; > + > + verify_flags= MARIADB_TLS_VERIFY_PERIOD | MARIADB_TLS_VERIFY_REVOKED; > + if (have_fingerprint(mysql)) > + { > + verify_flags|= MARIADB_TLS_VERIFY_FINGERPRINT; > + } else { > + verify_flags|= MARIADB_TLS_VERIFY_TRUST | MARIADB_TLS_VERIFY_HOST; > + } > + > + if (ma_pvio_tls_verify_server_cert(mysql->net.pvio->ctls, verify_flags) > MARIADB_TLS_VERIFY_OK) > + { > + if (mysql->net.tls_verify_status > MARIADB_TLS_VERIFY_TRUST || > + (mysql->options.ssl_ca || mysql->options.ssl_capath)) > + goto error; > + > + if (is_local_connection(mysql->net.pvio)) I'd say this should be earlier. Like if (!is_local_connection(mysql->net.pvio)) if (have_fingerprint(mysql)) { verify_flags|= MARIADB_TLS_VERIFY_FINGERPRINT; } else { verify_flags|= MARIADB_TLS_VERIFY_TRUST | MARIADB_TLS_VERIFY_HOST; } > + { > + CLEAR_CLIENT_ERROR(mysql); > + } > + else if (!password_and_hashing(mysql, mpvio->plugin)) > + goto error; > } > } > #endif /* HAVE_TLS */ > @@ -769,8 +822,14 @@ retry: > auth_plugin= &dummy_fallback_client_plugin; > > /* can we use this plugin with this tls server cert ? */ > - if (mysql->net.tls_self_signed_error && !hashing(auth_plugin)) > - return set_error_from_tls_self_signed_error(mysql); > + if ((mysql->net.tls_verify_status & MARIADB_TLS_VERIFY_TRUST) && > + !password_and_hashing(mysql, auth_plugin)) > + { > + my_set_error(mysql, CR_SSL_CONNECTION_ERROR, SQLSTATE_UNKNOWN, > + ER(CR_SSL_CONNECTION_ERROR), > + "Certificate verification failure: The certificate is NOT trusted."); already commented about using library error message > + return 1; > + } > goto retry; > } > /* > @@ -782,7 +841,9 @@ retry: > if (ma_read_ok_packet(mysql, mysql->net.read_pos + 1, pkt_length)) > return -1; > > - if (!mysql->net.tls_self_signed_error) > + if (!mysql->net.tls_verify_status || > + ((mysql->net.tls_verify_status & MARIADB_TLS_VERIFY_TRUST) && > + is_local_connection(mysql->net.pvio))) Not needed, see above > return 0; > > assert(mysql->options.use_ssl); > diff --git a/libmariadb/secure/openssl.c b/libmariadb/secure/openssl.c > index 8231a244..b5b573a9 100644 > --- a/libmariadb/secure/openssl.c > +++ b/libmariadb/secure/openssl.c > @@ -459,24 +461,38 @@ error: > return NULL; > } > > -unsigned int ma_tls_get_peer_cert_info(MARIADB_TLS *ctls) > +unsigned int ma_tls_get_peer_cert_info(MARIADB_TLS *ctls, uint hash_size) > { > X509 *cert; > + unsigned int hash_alg; > SSL *ssl; > + char fp[129]; > > + switch (hash_size) { > + case 0: > + case 256: > + hash_alg= MA_HASH_SHA256; > + break; > + case 384: > + hash_alg= MA_HASH_SHA384; > + break; > + case 512: > + hash_alg= MA_HASH_SHA512; > + break; > + default: > + return 1; > + } > + > if (!ctls || !ctls->ssl) > return 1; > > - /* Did we already read peer cert information ? */ > - if (ctls->cert_info.version) > - return 0; > - > ssl= (SSL *)ctls->ssl; > > /* Store peer certificate information */ > + if (!ctls->cert_info.version) > + { when can it be already set? > if ((cert= SSL_get_peer_certificate(ssl))) > { > - char fp[33]; > #if OPENSSL_VERSION_NUMBER >= 0x10101000L > const ASN1_TIME *not_before= X509_get0_notBefore(cert), > *not_after= X509_get0_notAfter(cert); > @@ -714,9 +714,40 @@ my_bool ma_tls_close(MARIADB_TLS *ctls) > return rc; > } > > -int ma_tls_verify_server_cert(MARIADB_TLS *ctls) > +/** Check for possible errors, and store the result in net.tls_verify_status. > + verification will happen after handshake by ma_tls_verify_server_cert(). > + To retrieve all errors, this callback function returns always true. > + (By default OpenSSL stops verification after first error > +*/ > +static int ma_verification_callback(int preverify_ok __attribute__((unused)), X509_STORE_CTX *ctx) > { > - X509 *cert; > + SSL *ssl; > + > + int x509_err= X509_STORE_CTX_get_error(ctx); > + > + if ((ssl = X509_STORE_CTX_get_ex_data(ctx, SSL_get_ex_data_X509_STORE_CTX_idx()))) > + { > + MYSQL *mysql= (MYSQL *)SSL_get_app_data(ssl); > + > + if ((x509_err == X509_V_ERR_DEPTH_ZERO_SELF_SIGNED_CERT || > + x509_err == X509_V_ERR_SELF_SIGNED_CERT_IN_CHAIN)) > + mysql->net.tls_verify_status|= MARIADB_TLS_VERIFY_TRUST; > + else if (x509_err == X509_V_ERR_CERT_REVOKED) > + mysql->net.tls_verify_status|= MARIADB_TLS_VERIFY_REVOKED; > + else if (x509_err == X509_V_ERR_CERT_NOT_YET_VALID || > + x509_err == X509_V_ERR_CERT_HAS_EXPIRED) > + mysql->net.tls_verify_status|= MARIADB_TLS_VERIFY_PERIOD; > + else if (x509_err != X509_V_OK) > + mysql->net.tls_verify_status|= MARIADB_TLS_VERIFY_UNKNOWN; and here you need to store the error message in mysql->net. for example, else if (x509_err == X509_V_ERR_CERT_NOT_YET_VALID || x509_err == X509_V_ERR_CERT_HAS_EXPIRED) { if (mysql->net.tls_verify_status < MARIADB_TLS_VERIFY_PERIOD) mysql->net.tls_verify_error= X509_verify_cert_error_string(x509_err); mysql->net.tls_verify_status|= MARIADB_TLS_VERIFY_PERIOD; } or else if (x509_err == X509_V_ERR_CERT_NOT_YET_VALID || x509_err == X509_V_ERR_CERT_HAS_EXPIRED) { if (mysql->net.tls_verify_status < MARIADB_TLS_VERIFY_PERIOD) my_set_error(mysql, CR_SSL_CONNECTION_ERROR, SQLSTATE_UNKNOWN, ER(CR_SSL_CONNECTION_ERROR), X509_verify_cert_error_string(x509_err)); mysql->net.tls_verify_status|= MARIADB_TLS_VERIFY_PERIOD; } > + } > + > + /* continue verification */ > + return 1; > +} > + > +int ma_tls_verify_server_cert(MARIADB_TLS *ctls, unsigned int verify_flags) > +{ > + X509 *cert= NULL; > MYSQL *mysql; > SSL *ssl; > MARIADB_PVIO *pvio; > @@ -734,52 +765,97 @@ int ma_tls_verify_server_cert(MARIADB_TLS *ctls) > mysql= (MYSQL *)SSL_get_app_data(ssl); > pvio= mysql->net.pvio; > > + if (verify_flags & MARIADB_TLS_VERIFY_FINGERPRINT) > + { > + if (ma_pvio_tls_check_fp(ctls, mysql->options.extension->tls_fp, mysql->options.extension->tls_fp_list)) > + { > + mysql->net.tls_verify_status |= MARIADB_TLS_VERIFY_FINGERPRINT; > + return 1; > + } > + > + /* if certificates are valid and no revocation error occured, > + we can return */ > + if (!(mysql->net.tls_verify_status & MARIADB_TLS_VERIFY_PERIOD) && > + !(mysql->net.tls_verify_status & MARIADB_TLS_VERIFY_REVOKED)) you can check this before trying the fingerprint > + { > + mysql->net.tls_verify_status= MARIADB_TLS_VERIFY_OK; > + return 0; > + } > + } > + > + if (mysql->net.tls_verify_status & verify_flags) > + { > + return 1; > + } > + > + if (verify_flags & MARIADB_TLS_VERIFY_HOST) > + { > if (!mysql->host) > { > pvio->set_error(mysql, CR_SSL_CONNECTION_ERROR, SQLSTATE_UNKNOWN, > ER(CR_SSL_CONNECTION_ERROR), "Invalid (empty) hostname"); > - return 1; > + mysql->net.tls_verify_status|= MARIADB_TLS_VERIFY_HOST; > + return MARIADB_TLS_VERIFY_ERROR; > } > > if (!(cert= SSL_get_peer_certificate(ssl))) > { > pvio->set_error(mysql, CR_SSL_CONNECTION_ERROR, SQLSTATE_UNKNOWN, > ER(CR_SSL_CONNECTION_ERROR), "Unable to get server certificate"); > - return 1; > + mysql->net.tls_verify_status|= MARIADB_TLS_VERIFY_HOST; not sure it's MARIADB_TLS_VERIFY_HOST, may be MARIADB_TLS_VERIFY_UNKNOWN ? > + return MARIADB_TLS_VERIFY_ERROR; > } > + > #ifdef HAVE_OPENSSL_CHECK_HOST > if (X509_check_host(cert, mysql->host, strlen(mysql->host), 0, 0) != 1 > && X509_check_ip_asc(cert, mysql->host, 0) != 1) > + { > + mysql->net.tls_verify_status|= MARIADB_TLS_VERIFY_HOST; > goto error; > + } > #else > x509sn= X509_get_subject_name(cert); > > if ((cn_pos= X509_NAME_get_index_by_NID(x509sn, NID_commonName, -1)) < 0) > + { > + mysql->net.tls_verify_status|= MARIADB_TLS_VERIFY_HOST; > goto error; > + } > > if (!(cn_entry= X509_NAME_get_entry(x509sn, cn_pos))) > + { > + mysql->net.tls_verify_status|= MARIADB_TLS_VERIFY_HOST; > goto error; > + } > > if (!(cn_asn1 = X509_NAME_ENTRY_get_data(cn_entry))) > + { > + mysql->net.tls_verify_status|= MARIADB_TLS_VERIFY_HOST; > goto error; > + } > > cn_str = (char *)ASN1_STRING_data(cn_asn1); > > /* Make sure there is no embedded \0 in the CN */ > if ((size_t)ASN1_STRING_length(cn_asn1) != strlen(cn_str)) > + { > + mysql->net.tls_verify_status|= MARIADB_TLS_VERIFY_HOST; > goto error; > + } > > if (strcmp(cn_str, mysql->host)) > + { > + mysql->net.tls_verify_status|= MARIADB_TLS_VERIFY_HOST; > goto error; > + } > #endif > X509_free(cert); > - > + } > return 0; > error: > + if (cert) > X509_free(cert); > > - pvio->set_error(mysql, CR_SSL_CONNECTION_ERROR, SQLSTATE_UNKNOWN, > - ER(CR_SSL_CONNECTION_ERROR), "Validation of SSL server certificate failed"); > return 1; > } > Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

1 0

Re: [PATCH] MDEV-34481 optimize away waiting for owned by prepared xa non-unique index record
by Kristian Nielsen 31 Jul '24

31 Jul '24

Andrei Elkin <andrei.elkin(a)mariadb.com> writes: >> Also, are you aware that when you skip a record like this, you may be >> skipping the very record that needs to be changed? > Yet luckily this is covered by the BASE (of the patch) logics. > After(needless) scan the rest of duplicate key record set the third group > of events would have to retry, to success. (This is optimizable, but I You need to make sure that --slave_exec_mode=IDEMPOTENT is not used. Otherwise you can get silent data corruption. - Kristian.

1 0

Re: [PATCH] MDEV-34481 optimize away waiting for owned by prepared xa non-unique index record
by Kristian Nielsen 31 Jul '24

31 Jul '24

> From: Andrei <andrei.elkin(a)mariadb.com> > > This work partly implements a ROW binlog format MDEV-33455 part > that is makes non-unique-index-only table XA replication safe in RBR. > Existence of at least one non-null unique index has always guaranteed > safety (no hang to error). Hi Andrei, A couple early comments on your MDEV-34481 patch, about things that affect non-XA parts of the code and will need to be changed: > diff --git a/sql/sql_class.cc b/sql/sql_class.cc > index 6419a58fbe4..b2321cfd865 100644 > --- a/sql/sql_class.cc > +++ b/sql/sql_class.cc > @@ -5444,20 +5449,33 @@ thd_need_wait_reports(const MYSQL_THD thd) > transaction. > */ > extern "C" int > -thd_rpl_deadlock_check(MYSQL_THD thd, MYSQL_THD other_thd) > +thd_rpl_deadlock_check(MYSQL_THD thd, MYSQL_THD other_thd, > + bool is_other_prepared, bool is_index_unique, > + uint *flagged) > + > + /* > + Return to the caller the fact of the non-unique index wait lock > + conflicts with one of a prepared state transaction. > + */ > + if (!is_index_unique && rgi && rgi->is_row_event_execution()) { > + if (is_other_prepared && !other_thd) > + { > + rgi->exec_flags |= 1 << rpl_group_info::HIT_BUSY_INDEX; > + (*flagged)++; > + } > + } > + No, thd_rpl_deadlock_check() has a specific purpose related to handling deadlock between active transactions. Don't add this XA-specific and unrelated logic to this function. Instead, add a separate function that InnoDB can call in the specific case you need. That function then only needs the parameters `thd` and `flagged`, the others are not needed: extern "C" bool thd_xa_skip_lock_check(MYSQL_THD thd, uint *flagged) { return (thd->rgi_slave && thd->rgi_slave->us_row_event_execution()); } Then in InnoDB, call something like this: if (lock->trx->state == TRX_STATE_PREPARED && (!lock->index->is_unique() || lock->index->n_nullable)) count_non_unique_index_hit++; About the HIT_BUSY_INDEX, see here: > diff --git a/sql/log_event_server.cc b/sql/log_event_server.cc > index f3d01b7d9ed..166295ed263 100644 > --- a/sql/log_event_server.cc > +++ b/sql/log_event_server.cc > @@ -8149,11 +8154,20 @@ int Rows_log_event::find_row(rpl_group_info *rgi) > HA_READ_KEY_EXACT)))) > { > DBUG_PRINT("info",("no record matching the key found in the table")); > - if (error == HA_ERR_KEY_NOT_FOUND) > - error= row_not_found_error(rgi); > - table->file->print_error(error, MYF(0)); > - table->file->ha_index_end(); > - goto end; > + if (error == HA_ERR_LOCK_WAIT_TIMEOUT && !is_index_unique && > + (rgi->exec_flags & (1 << rpl_group_info::HIT_BUSY_INDEX))) > + { > + rgi->exec_flags &= ~(1 << rpl_group_info::HIT_BUSY_INDEX); > + skip_first_compare= true; > + } Here, you are effectively making a decision in the storage engine to skip this record which is locked by an XA PREPAREd transaction. But you are communicating this back to the server in a very convoluted way, by introducing the rgi->exec_flags and setting HIT_BUSY_INDEX. This is not the way to communicate back from engine to server. Instead, simply return a different error code than HA_ERR_LOCK_WAIT_TIMEOUT from InnoDB that you can check for here. You can maybe use one of the existing HA_ERR_* from include/my_base.h, or add a new one if necessary. if (error == HA_ERR_<something>) skip_first_compare= true; > diff --git a/sql/rpl_rli.h b/sql/rpl_rli.h > index 3f24481839b..a9ec44b58cb 100644 > --- a/sql/rpl_rli.h > +++ b/sql/rpl_rli.h > @@ -833,6 +833,15 @@ struct rpl_group_info > RETRY_KILL_KILLED > }; > uchar killed_for_retry; > + enum enum_exec_flags { > + HIT_BUSY_INDEX= 0 > + }; > + /* > + Being executed replication event's context. It could contain > + notification from engine such as attempts to lock already acquired > + index record. > + */ > + uint32 exec_flags; ... and then you don't need to add this. Also, are you aware that when you skip a record like this, you may be skipping the very record that needs to be changed? For example a case like this, with three transactions in order: XA START 'T1'; UPDATE my_table SET value=10 WHERE key=2 XA END 'T1'; XA PREPARE 'T1'; XA COMMIT 'T1'; XA START 'T2'; UPDATE my_table SET value=20 WHERE key=2; Transaction 'T2' may run speculatively in optimistic parallel replication after prepare of T1, but before commit of T1. Then it will skip the record key=2, IIUC. I didn't see any explanation of this case in the patch commit message or comments. Also I don't see this case tested in the test case, that needs to be added I believe. - Kristian.

1 0

Re: [MariaDB commits] [PATCH] MDEV-33856 New definition for Seconds_Behind_Master
by Kristian Nielsen 25 Jul '24

25 Jul '24

Hi Monty, Here is my review of the MDEV-33856 patch: > commit cc516f4d9913d732065c7abef983df42e5a8bab3 (origin/bb-11.6-MDEV-33856-seconds_behind_master) > Author: Monty <monty(a)mariadb.org> > Date: Tue May 14 23:47:59 2024 +0300 > > MDEV-33856 New definition for Seconds_Behind_Master If I understand the patch correctly, in introduces a new `Master_slave_time_diff` value as an alternative to the traditional Seconds_behind_master, with a clearer semantics. And does not change how Seconds_behind_master works. I think this is fine, and can help to preserve backwards compatibility. Just maybe mention this in the commit message, and/or change the bug title to "Alternative definition for Seconds_Behind_Master". > diff --git a/mysql-test/suite/perfschema/t/relaylog.test b/mysql-test/suite/perfschema/t/relaylog.test > index 12fc96a8b27..16f72efae98 100644 > --- a/mysql-test/suite/perfschema/t/relaylog.test > +++ b/mysql-test/suite/perfschema/t/relaylog.test > @@ -43,7 +43,7 @@ select > if (sum_number_of_bytes_read > 0, "MANY", "NONE") as SUM_NUMBER_OF_BYTES_READ, > if (sum_number_of_bytes_write > 0, "MANY", "NONE") as SUM_NUMBER_OF_BYTES_WRITE > from performance_schema.file_summary_by_instance > - where file_name like "%master-%" order by file_name; > + where file_name like "%master-%" and file_name not like "%sql/share%" order by file_name; > > select * from performance_schema.file_summary_by_instance > where file_name like "%slave-%" order by file_name; > @@ -96,7 +96,7 @@ sync_slave_with_master; > -- echo "============ Performance schema on slave ============" > > select * from performance_schema.file_summary_by_instance > - where file_name like "%master-%" order by file_name; > + where file_name like "%master-%" and file_name not like "%sql/share%" order by file_name; > > select > substring(file_name, locate("slave-", file_name)) as FILE_NAME, What is the reason for these changes (adding match on "%sql/share%" ? I don't understand the relation to the subject of the patch. > diff --git a/mysql-test/suite/rpl/t/master_last_event_time.inc b/mysql-test/suite/rpl/t/master_last_event_time.inc > new file mode 100644 > index 00000000000..0790d557a7b > --- /dev/null > +++ b/mysql-test/suite/rpl/t/master_last_event_time.inc > > +--connection server_3 > +--sleep 1 > +--source include/start_slave.inc > +--source include/sync_with_master_gtid.inc What is the purpose of --sleep 1 here? Usually it's wrong to have sleeps in the tests. Sometimes it is correct, but then there should be a comment explaining why a sleep is necessary (and sufficient) here. This sleep looks wrong, since there's a proper wait immediately after for the slave to sync up. > diff --git a/mysql-test/suite/rpl/t/master_last_event_time_stmt.test b/mysql-test/suite/rpl/t/master_last_event_time_stmt.test > new file mode 100644 > index 00000000000..fe0f16a3db3 > --- /dev/null > +++ b/mysql-test/suite/rpl/t/master_last_event_time_stmt.test > @@ -0,0 +1,84 @@ > +# > +# Statement specific tests for master_last_event_time > +# > +--source include/have_binlog_format_statement.inc > > +# Slave will replicate the above with a row event which will be very fast > +# compared to the master event. I don't understand this comment. Why will the slave replicate using a row event, when the test case sets binlog_format=statement? > diff --git a/sql/log.cc b/sql/log.cc > index 1cc0fa77f08..63d9c756911 100644 > --- a/sql/log.cc > +++ b/sql/log.cc > @@ -1898,10 +1898,20 @@ static inline int > binlog_commit_flush_xid_caches(THD *thd, binlog_cache_mngr *cache_mngr, > bool all, my_xid xid) > Xid_log_event end_evt(thd, xid, TRUE); > - return (binlog_flush_cache(thd, cache_mngr, &end_evt, all, TRUE, TRUE)); > + if (!thd->slave_thread && ! thd->user_time.val) > + { > + /* > + Ensure that on the master the event time is the time of commit, > + not the start of statement time. > + */ > + my_hrtime_t hrtime= my_hrtime(); > + end_evt.when= hrtime_to_my_time(hrtime); > + } > + DBUG_RETURN(binlog_flush_cache(thd, cache_mngr, &end_evt, all, TRUE, TRUE)); Ok, so correct me if I'm wrong, but I think the purpose here is: You want to have the time when the transaction ended (in case of long-running query). But for auto-committed query, there is no explicit COMMIT query, so the thd->start_time is the start of the transaction. So here you ensure that the XID_EVENT will have the time at start of commit in either case. If this is correct, probably could use a comment explaining this. Some comments: 1. It is better to check for thd->rgi_slave being non-NULL than for thd->slave_thread. The thd->rgi_slave is associated with the replication of some event or transaction. The thd->slave_thread just identifies the thread/connection. (This is mostly style/code clarity, as in current code the two values would track each other.) 2. Why the condition on thd->user_time.val? What is the situation where we are not replicating an event, but user_time is set and we do not want to put the current time in the XID_EVENT? 3. This handles an XID end event, which is for InnoDB transactions. But MyISAM/Aria use a different end event, a COMMIT query, and there are are few other cases as well. Don't you want consistently to have the end of the query as the time in all end events? Then this code should probably instead be in binlog_flush_cache() or perhaps write_transaction_to_binlog()? 4. Even given (3), there are still cases (eg. DDL) where there is no end event, and thus there is no event containing the time for the end of the operation on the master. Won't this cause some inconsistency in how the slave determines and displays its timings with this patch? If not, why not? > for (; has_xid && !entry.need_unlog && ha_info; ha_info= ha_info->next()) > { > if (ha_info->is_started() && ha_info->ht() != binlog_hton && > !ha_info->ht()->commit_checkpoint_request) > + { > entry.need_unlog= true; > + break; > + } > } Agree, this code always irritated me ;-) You could also move the condition (!entry.need_unlog && ha_info) out of the loop in an outer if () to not have to check it on every loop iteration (just an observation, entirely up to you, it's not something your patch introduced). > diff --git a/sql/log_event.cc b/sql/log_event.cc > index 3802f76efe0..706a4903eb0 100644 > --- a/sql/log_event.cc > +++ b/sql/log_event.cc > @@ -1104,10 +1116,12 @@ Log_event* Log_event::read_log_event(const uchar *buf, uint event_len, > switch(event_type) { > case QUERY_EVENT: > ev= new Query_log_event(buf, event_len, fdle, QUERY_EVENT); > + set_orig_exec_time_in_thd(((Query_log_event*) ev)->exec_time); > break; > case QUERY_COMPRESSED_EVENT: > ev= new Query_compressed_log_event(buf, event_len, fdle, > - QUERY_COMPRESSED_EVENT); > + QUERY_COMPRESSED_EVENT); > + set_orig_exec_time_in_thd(((Query_compressed_log_event*) ev)->exec_time); > @@ -964,6 +964,18 @@ Log_event* Log_event::read_log_event(IO_CACHE* file, > DBUG_RETURN(res); > } > > +/* > + Update thd->orig_exec_time > +*/ > + > +inline void set_orig_exec_time_in_thd(my_time_t exec_time) > +{ > +#if !defined(MYSQL_CLIENT) && defined(HAVE_REPLICATION) > + THD *thd= current_thd; > + if (likely(thd)) > + thd->orig_exec_time= exec_time; > +#endif > +} Using current_thd doesn't seem right here, and in fact it's not, as seen by this part of the patch where thd->orig_exec_time has to be set again for parallel replication: > diff --git a/sql/rpl_parallel.cc b/sql/rpl_parallel.cc > index 51ca3b183ec..d279761767a 100644 > --- a/sql/rpl_parallel.cc > +++ b/sql/rpl_parallel.cc > @@ -60,7 +60,10 @@ rpt_handle_event(rpl_parallel_thread::queued_event *qev, > rgi->future_event_relay_log_pos= qev->future_event_relay_log_pos; > strcpy(rgi->future_event_master_log_name, qev->future_event_master_log_name); > if (event_can_update_last_master_timestamp(ev)) > + { > rgi->last_master_timestamp= ev->when + ev->exec_time; > + thd->orig_exec_time= ev->exec_time; > + } So let's not set thd->orig_exec_time from Log_event::read_log_event(), it's redundant (for parallel replication), and logically setting it should be done when applying the event, not when reading the event. Instead, set the thd->orig_exec_time in exec_relay_log_event(), so we retain similarity with how replication is done in the parallel and non-parallel case. There is already a similar if (event_can_update_last_master_timestamp(ev)) in exec_relay_log_event(). (Also see comments below about moving this from THD to rpl_group_info to not increase THD just for this very specific case). > + > +/* > + Get the time when the event had been executed on the master. > + This works for both query events and load data events. > +*/ > + > +#if Q_EXEC_TIME_OFFSET != L_EXEC_TIME_OFFSET > +#error "Q_EXEC_TIME_OFFSET is not same as L_EXEC_TIME_OFFSET" > +#endif > + > +time_t query_event_get_time(const uchar *buf, > + const Format_description_log_event > + *description_event) > +{ > + time_t when= uint4korr(buf); > + buf+= description_event->common_header_len; > + return when + uint4korr(buf + Q_EXEC_TIME_OFFSET); Suggestion: Add an assert here: DBUG_ASSERT(LOG_EVENT_IS_QUERY(buf[EVENT_TYPE_OFFSET]) || LOG_EVENT_IS_LOAD_DATA(buf[EVENT_TYPE_OFFSET])); just to avoid someone mistakenly adding code later to call this for a wrong event type. > diff --git a/sql/sql_class.h b/sql/sql_class.h > index 2d671139a6c..05c9c4e8012 100644 > --- a/sql/sql_class.h > +++ b/sql/sql_class.h > @@ -3168,6 +3168,7 @@ class THD: public THD_count, /* this must be first */ > ulonglong start_utime, utime_after_lock, utime_after_query; > /* This can be used by handlers to send signals to the SQL level */ > ulonglong replication_flags; > + my_time_t orig_exec_time; // Exec time for last read binlog event > @@ -1348,8 +1346,14 @@ Query_log_event::Query_log_event(THD* thd_arg, const char* query_arg, > memset(&host, 0, sizeof(host)); > error_code= errcode; > > - end_time= my_time(0); > - exec_time = (ulong) (end_time - thd_arg->start_time); > + /* > + For slave threads, remember the original master exec time. > + This is needed to be able to calculate the master commit time. > + */ > + exec_time= ((thd->slave_thread) ? > + thd->orig_exec_time : > + (my_time(0) - thd_arg->start_time)); > + I think we can here avoid increasing the THD size, which is sad for a special case like this. Instead put the orig_exec_time into THD::rgi_slave, extending the struct rpl_group_info. Then the assignment above will be to rgi->orig_exec_time instead of thd->orig_exec_time, and the check here will be for thd->rgi_slave instead of thd->slave_thread. The rgi is the appropriate place for orig_exec_time, as it is a property of the transaction/event group being replicated. @@ -3171,146 +2856,170 @@ static const LEX_CSTRING msg_ignored= { STRING_WITH_LEN("Ignored") }; #endif > -static bool send_show_master_info_data(THD *thd, Master_info *mi, bool full, > - String *gtid_pos) > +inline void store_string_or_null(Field **field, const char *str) > +{ > > - mysql_mutex_unlock(&mi->run_lock); > +void store_master_info(THD *thd, Master_info *mi, TABLE *table, > > - protocol->store_string_or_null(mi->host, &my_charset_bin); > - protocol->store_string_or_null(mi->user, &my_charset_bin); > - protocol->store((uint32) mi->port); > - protocol->store((uint32) mi->connect_retry); > - protocol->store(mi->master_log_name, strlen(mi->master_log_name), > + store_string_or_null(field++, mi->host); > + store_string_or_null(field++, mi->user); > + (*field++)->store((uint32) mi->port); > + (*field++)->store((uint32) mi->connect_retry); > + (*field++)->store(mi->master_log_name, strlen(mi->master_log_name), So this patch refactors the code for SHOW SLAVE STATUS to use a shared code path with information_schema tables? That sounds like a good change. But for the future, please do such refactoring as a separate patch, you can see my patch series of pre-computing binlog checksums (MDEV-31273) for an example. Doing it together makes it very hard to review the actual code changes and see what is refactoring and what is new logic for master_slave_time_diff. > + if (unlikely(!rli->slave_timestamp) && Log_event::is_group_event(typ)) > + { > + /* > + First event for this slave. Assume that all the slave was up to date > + with the master just before the current event. > + */ > + rli->slave_timestamp= (time_t) ev->when + (time_t) ev->exec_time-1; I think the comment is meant to explain why there is the "-1" / minus-one in this expression, but it's not really clear exactly what is meant, so perhaps elaborate it a bit. Does it mean that master_slave_time_diff will start out as "1 second" just after the slave restarts? One way or the other, I do not have a strong opinion. But I want to give a general remark, from my experience with the troubles over the years with Seconds_Behind_Master. The problem with Seconds_Behind_Master was always that it didn't quite do what someone expected in some particular case, and then a new special case was added to try to accomodate. And after a number of iterations of this, we ended up with a semantics for Seconds_Behind_Master that no-one _really_ understands, and which still doesn't quite always do what people expect (which is in any case impossible if different people have different expectations). This "-1" here is clearly on the same road towards adding special cases. So my strong advice would be that, going forward, be _extremely_ conservative about modifying the semantics of master_slave_time_diff. The one good thing about the new status is that it has a (hopefully) clear semantics that can be documented (now including this "-1") and completely understood by users. Let's stick to that no matter who comes and wants just another small tweak, unless such tweaks actually make the semantics and documentation _simpler_. > + /* > + replay_log.description_event_for_exec can be null if the slave thread > + is getting killed > + */ > + if (LOG_EVENT_IS_QUERY((Log_event_type) buf[EVENT_TYPE_OFFSET]) || > + LOG_EVENT_IS_LOAD_DATA((Log_event_type) buf[EVENT_TYPE_OFFSET])) > + { > + time_t exec_time= query_event_get_time(buf, rli->relay_log. > + description_event_for_queue); > + set_if_bigger(rli->newest_master_timestamp, exec_time); > + } This confused me, because in Query_log_event, the exec_time is the duration of the query (a time interval), not an absolute time. So please use the name "end_time" for the variable instead of "exec_time", and consider renaming query_event_get_time() to query_event_get_end_time(). > > +static int make_slave_status_old_format(THD *thd, ST_SCHEMA_TABLE *schema_table) > +{ > + ST_FIELD_INFO *field_info= schema_table->fields_info; > + Name_resolution_context *context= &thd->lex->first_select_lex()->context; > + DBUG_ASSERT(thd->lex->sql_command == SQLCOM_SHOW_SLAVE_STAT); > + bool all_slaves= thd->lex->mi.show_all_slaves; > + ulonglong used_fields= ~0ULL; > + > + if (!all_slaves) > + { > + /* Remove 2 first fields and all fields above and including field 55 */ > + used_fields&= ~((1ULL << 0) | (1ULL << 1)); > + used_fields&= ((1ULL << 56)-1); Should there be some constant used for the magic numbers 55/56? And/or an assertion that this value is correct? Also, is the intention here that we will from here on never add more fields to legacy SHOW SLAVE STATUS? If so, there should probably be a comment about this; also see the test case mysql-test/suite/rpl/t/rpl_show_slave_status.test which has comments that might need updating. - Kristian

2 2

Moving forward from default-mysql-* packages in Debian
by Otto Kekäläinen 19 Jul '24

19 Jul '24

Hi! tl;dr: Anyone here interested in trying to push various open source projects in Debian and the upstreams to use mariadb-server directly instead of via default-mysql-server dependency? ## Situation Back in 2016 Debian introduced the virtual packages: - default-mysql-server - default-mysql-client - default-libmysqlclient-dev In Debian these have since pointed to MariaDB, and in Ubuntu they point to MySQL. Not much has happened since. There are only a handful packages in Debian that have declared an explicit dependency on MariaDB directly. Most packages continue to depend on default-mysql-* instead of MariaDB directly. As MariaDB and MySQL diverge, those packages will eventually end up depending on MySQL only unless they transition slowly to depend on MariaDB directly. It didn't happen organically, so it would need a push. Anyone here interested in taking up such a task? It requires talking to all the upstreams to officially support/adopt MariaDB going forward. ## Background - https://lists.debian.org/debian-devel-announce/2016/09/msg00000.html - https://lists.debian.org/debian-devel/2016/11/msg00832.html - https://wiki.debian.org/Teams/MySQL/default-mysql-server - https://tracker.debian.org/pkg/mysql-defaults ## Package lists # apt rdepends default-mysql-client default-mysql-client Reverse Depends: |Depends: audiolink |Depends: zoneminder |Depends: wordpress Recommends: tango-db |Depends: sqitch |Recommends: rsyslog-mysql |Depends: roundcube-mysql |Depends: rt5-db-mysql |Depends: rt4-db-mysql |Depends: redmine-mysql |Recommends: prelude-manager |Depends: postfixadmin |PreDepends: openstack-compute-node Depends: opendnssec-enforcer-mysql |Depends: ocsinventory-reports |Recommends: oar-server-mysql |Recommends: mysql-workbench |Suggests: mysql-sandbox Suggests: munin-plugins-core Suggests: motion Depends: libreoffice-canzeley-client Depends: lcmaps-plugins-jobrep-admin |Depends: kamailio-mysql-modules |Recommends: icinga2-ido-mysql Suggests: education-main-server |Depends: dbconfig-mysql |Suggests: cedar-backup3 |Depends: bacula-director-mysql |Depends: automysqlbackup # apt rdepends default-mysql-server default-mysql-server Reverse Depends: |Depends: python3-testing.mysqld (>= 1.0.2) |Depends: zoph |Recommends: zoneminder |Recommends: zabbix-server-mysql |Recommends: zabbix-proxy-mysql |Suggests: wordpress Suggests: trojan |Recommends: sympa |Recommends: spip |Suggests: sogo |Suggests: rsyslog-mysql Suggests: roundcube-mysql |Suggests: rt5-db-mysql |Suggests: rt4-db-mysql |Suggests: redmine-mysql |Suggests: adminer |Suggests: python3-mysqldb |Suggests: pwman3 |Suggests: prometheus-mysqld-exporter Recommends: postfix-gld |Suggests: phpmyadmin Suggests: pdns-backend-mysql Suggests: orthanc-mysql |Depends: openstack-cloud-services |Depends: openstack-cluster-installer |Suggests: ocsinventory-server |Suggests: oar-server-mysql |Recommends: mysqltuner |Suggests: mysql-sandbox |Recommends: mediawiki |Suggests: mailman3 |Suggests: mailman3-web Suggests: libreoffice-canzeley-client |Suggests: libreoffice-sdbc-mysql Suggests: inspircd Suggests: icingadb |Suggests: icinga2-ido-mysql Suggests: goval-dictionary Suggests: golang-github-ctdk-goiardi-dev Suggests: goiardi Suggests: digitemp |Depends: diaspora-common |Suggests: dbconfig-mysql Suggests: collectd-core |Recommends: cacti |Recommends: bacula-director-mysql |Recommends: audiolink # apt rdepends default-libmysqlclient-dev default-libmysqlclient-dev Reverse Depends: Depends: libboinc-app-dev Depends: libvtk9-dev Depends: urweb Depends: librust-mysqlclient-sys-dev Suggests: qtbase5-gles-dev Suggests: qtbase5-dev |Depends: postgresql-16-mysql-fdw Depends: libpoco-dev Recommends: newlisp |Suggests: mysql-sandbox Depends: libmysql-ocaml-dev Depends: libmysql++-dev Depends: libmailutils-dev Depends: kannel-dev Depends: libhoel-dev Suggests: libglpk40 Depends: libgdal-dev Suggests: fp-units-db-3.2.2 Suggests: fp-units-win-db-3.2.2 Depends: cl-sql-mysql

1 0

Re: [MariaDB commits] [PATCH] MDEV-15393 part 2. Fixes to 'gtid_slave_pos duplicate key errors after mysqldump restore' MDEV-34615 mysqldump wipes off pre-existing gtid slave state
by Kristian Nielsen 18 Jul '24

18 Jul '24

Andrei Elkin <andrei.elkin(a)mariadb.com> writes: >> The most common case is the provisioning of a slave with a complete dump >> from a master. In this case, it is required that the GTID position is set >> correctly across all domains. Merging the existing and new GTID position as >> attempted in this patch will break the provisioning of the slave. > > 'Will break' but merely for the technical (your pp1,2 to the concat) > reasons. I don't see what else can be a problem of MDEV-34615 part. No. What I mean is, consider some slave server that has an existing gtid_slave_pos='20-2-1000' say, domain_id=20. And then this server is for some reason re-provisioned (ie. all data overwritten) with a new mysqldump, say from a master with GTID pos '10-1-2000'. Now the correct GTID position matching the new dump is SET GLOBAL gtid_slave_pos='10-1-2000'; If you make the MDEV-34615 idea default, then the position for the new slave will be multi-domain '20-2-1000,10-1-2000', which is not correct and could cause problems later. Basically, only the user knows if the old @@gtid_slave_pos is meaningful and should be kept, or if it should be overwritten, mysqldump cannot reliably determine this. And changing the default will end up breaking things in some cases that are working now. The merging of GTID positions can be useful in the case where we are setting up a multi-source slave C by importing dumps from two master A and B, say. In this case the first import needs to overwrite any old gtid_slave_pos, and the second import should merge. Thus the need for the user to specify. I hope this explains the problem. - Kristian.

1 0

Re: [MariaDB commits] [PATCH] MDEV-15393 part 2. Fixes to 'gtid_slave_pos duplicate key errors after mysqldump restore' MDEV-34615 mysqldump wipes off pre-existing gtid slave state
by Kristian Nielsen 17 Jul '24

17 Jul '24

> commit 7a50135f3f908380e2dfb45e3f95c9fb10c01d62 > Author: Andrei <andrei.elkin(a)mariadb.com> > Date: Mon Jul 15 17:50:37 2024 +0300 > > MDEV-15393 part 2. Fixes to 'gtid_slave_pos duplicate key errors after mysqldump restore' > MDEV-34615 mysqldump wipes off pre-existing gtid slave state Hi Andrei, here my review of the patch, with a particular important issue to not break existing use cases and backwards compatibility, as well as some general issues/comments. > When mysqldump run on mysql system database it generates inserts sql commands > into mysql.gtid_slave_pos. After running the backup script those Here and below, I suggest better wording for text that is hard to understand or using wrong/unusual grammar. "When mysqldump is run to dump the `mysql` system database, it generates INSERT statements into the table mysql.gtid_slave_pos" > inserts did not produce an expected gtid state on slave. "did not produce the exspected gtid state" > In particular the maximum of mysql.gtid_slave_pos.sub_id did not make > into > rpl_global_gtid_slave_state.last_sub_id > a memory object that the table persists. I think you mean here "an in-memory object that is supposed to match the current state of the table". > And that was regardless of whether --gtid option in or out. "whether the --gtid option was specified by the user or not". > The fixes ensure the insert block of the dump script is followed > with a "summing-up" SET @global.gtid_slave_pos assignment. > To address MDEV-34615 the assignment makes sure a possibly > pre-existing gtid state (e.g reflecting another domain) remains as a > sub-state upon the dump installation. This change is not appropriate, as it breaks backwards compatibility and breaks a common use case. Certainly not in a GA release such as 10.5! The most common case is the provisioning of a slave with a complete dump from a master. In this case, it is required that the GTID position is set correctly across all domains. Merging the existing and new GTID position as attempted in this patch will break the provisioning of the slave. The existing supported way of not overwriting the slave's old @@GLOBAL.gtid_slave_pos is to use --master-data=2 --gtid. This will output the GTID position in the dump commented out, and the user can themselves (manually or through a script of their own) merge the positions as required for their setup and application. If you think it is important to have a facility in mysqldump to do merging automatically, then you should implement this as a new option. Such new option would need to go to the next development version (11.6 or 11.7 I suppose?). It could be --master-data = 5 or 6 for example (idea being bit 2 means "merge old and new position), or maybe some other way that doesn't require "magic" constants. I am commenting below on some problems with the implementation of the merging of old and new position. But note that these comments apply to a different patch that implement this as a new option. These changes must not be made to the existing default behaviour. > diff --git a/client/mysqldump.c b/client/mysqldump.c > index 1c9d92d6e09..0be890e2528 100644 > --- a/client/mysqldump.c > +++ b/client/mysqldump.c > @@ -5985,8 +5985,13 @@ static int dump_selected_tables(char *db, char **table_names, int tables) > } /* dump_selected_tables */ > > > +const char fmt_gtid_pos[] = "%sSET GLOBAL gtid_slave_pos=concat('%s', " > + "if(length(@@global.gtid_slave_pos) = 0, '', ','), " > + "@@global.gtid_slave_pos);\n"; Two problems with this: 1. This will result in a syntax error if the new position is empty and the old position is not (eg. SET GLOBAL gtid_slave_pos=',0-1-2'). 2. Unless I'm missing something obvious, this doesn't actually work to merge two different GTIDs with the same domain! It produces an error ER_DUPLICATE_GTID_DOMAIN. It seems there is a lack of coverage in the associated test case, if the actual merging of domains isn't even tested? SELECT @@GLOBAL.gtid_slave_pos; @@GLOBAL.gtid_slave_pos 0-1-2 SET GLOBAL gtid_slave_pos= CONCAT('0-1-10', ',', @@GLOBAL.gtid_slave_pos); rpl.tmp2 'mix' [ fail ] mysqltest: At line 12: query 'SET GLOBAL gtid_slave_pos= CONCAT('0-1-10', ',', @@GLOBAL.gtid_slave_pos)' failed: ER_DUPLICATE_GTID_DOMAIN (1943): GTID 0-1-2 and 0-1-10 conflict (duplicate domain id 0) > @@ -6054,8 +6059,11 @@ static int do_show_master_status(MYSQL *mysql_con, int consistent_binlog_pos, > fprintf(md_result_file, > "%sCHANGE MASTER TO MASTER_USE_GTID=slave_pos;\n", > comment_prefix); > - fprintf(md_result_file, > - "%sSET GLOBAL gtid_slave_pos='%s';\n", > + /* > + Defer print of SET gtid_slave_pos until after its placeholder > + table is guaranteed to have been dumped. > + */ > + sprintf(set_gtid_pos, fmt_gtid_pos, set_gtid_pos is a fixed size buffer, if I understand the patch correctly. This seems to mean that you have a buffer overflow here if the GTID position has too many domains? Note that it is not appropriate to simply use snprintf() instead, as this will merely truncate the output and give silent data corruption (or in this case probably a strange syntax error). Instead at least a proper error should be given (better to support arbitrary GTID position of course, but that might be a separate task if this is already an existing limitation of mysqldump, I did not check). > @@ -6181,10 +6189,8 @@ static int do_show_slave_status(MYSQL *mysql_con, int use_gtid, > mysql_free_result(slave); > return 1; > } > - print_comment(md_result_file, 0, > - "-- GTID position to start replication:\n"); > - fprintf(md_result_file, "%sSET GLOBAL gtid_slave_pos='%s';\n", > - gtid_comment_prefix, gtid_pos); > + /* similarly to do_show_master_status */ > + sprintf(set_gtid_pos, fmt_gtid_pos, gtid_comment_prefix, gtid_pos); I assume this is also a potential buffer overflow, didn't check closely. > @@ -6913,6 +6932,11 @@ int main(int argc, char **argv) > int exit_code; > int consistent_binlog_pos= 0; > int have_mariadb_gtid= 0; > + /* > + to hold SET @@global.gtid_slave_pos which is deferred to print > + until the function epilogue. > + */ > + char set_gtid_pos[3 + sizeof(fmt_gtid_pos) + MAX_GTID_LENGTH]= {0}; - Kristian.

1 0

'_' in numeric literals [MDEV-33228]
by Rohan Gupta 16 Jul '24

16 Jul '24

I was going through this issue https://jira.mariadb.org/browse/MDEV-34228, and noticed that during parsing we check if '_' is present in a token , if it does it is parsed as an identifier. My question here is to follow a SQL2023 standard of '_' in numeric literals, how would one go distinguishing the difference between numeric literals and identifiers ? mariadb documentation says that a identifier cannot start with a number until its in backticks `{identifier}`. Rohan Gupta

2 1

Re: d78376da69d: MDEV-34504 PURGE BINARY LOGS not working anymore
by Sergei Golubchik 10 Jul '24

10 Jul '24

Hi, Monty, On Jul 09, Michael Widenius wrote: > revision-id: d78376da69d (mariadb-11.4.2-11-gd78376da69d) > parent(s): 4ffeca643d5 > author: Michael Widenius > committer: Michael Widenius > timestamp: 2024-07-09 11:01:29 +0300 > message: > > MDEV-34504 PURGE BINARY LOGS not working anymore > > PURGE BINARY LOGS did not always purge binary logs. This commit fixes > some of the issues and adds notifications if a binary log cannot be > purged. > > User visible changes: > - 'PURGE BINARY LOG TO log_name' and 'PURGE BINARY LOGS BEFORE date' > worked differently. 'TO' ignored 'slave_connections_needed_for_purge' > while 'BEFORE' did not. Now both versions ignores the > 'slave_connections_needed_for_purge variable'. > - 'PURGE BINARY LOG..' commands now returns 'note' if a binary log cannot > be deleted like > Note 1375 Binary log 'master-bin.000004' is not purged because it is > the current active binlog > - Automatic binary log purges, based on date or size, will write a > note to the error log if a binary log matching the size or date > cannot yet be deleted. > - If 'slave_connections_needed_for_purge' is set from a config or > command line, it is set to 0 if Galera is enabled and 1 otherwise > (old default). This ensures that automatic binary log purge works > with Galera as before the addition of > 'slave_connections_needed_for_purge'. > If the variable is changed to 0, a warning will be printed to the error > log. > > Code changes: > - Added THD argument to several purge_logs related functions that needed > THD. > - Added 'interactive' options to purge_logs functions. This allowed > me to remove testing of sql_command == SQLCOM_PURGE. it's not really _ineractive_ but ok > - Changed purge_logs_before_date() to first check if log is applicable > before calling can_purge_logs(). This ensures we do not get a > notification for logs that does not match the remove criteria. > - MYSQL_BIN_LOG::can_purge_log() will write notifications to the user > or error log if a log cannot yet be removed. > - log_in_use() will return reason why a binary log cannot be removed. > - Moved checking of binlog_format for Galera to be after Galera is > initialized (The old check never worked). If Galera is enabled > we now change the binlog_format to ROW, with a warning, instead of > aborting the server. If this change happens, the binlog_format variable > will be marked with AUTO or FORCED, for information_schema.system_variables, > and a warning will be printed to the error log. What's "FORCE"? "AUTO" means that the value was automatically changed by the server, "FORCE" means the same, it looks redundant. And it's not the "source of the value", better to revert that part. > - Print also a warning if FLASHBACK changes the binlog_format to ROW. > Before this was done silently. > > diff --git a/sql/mysqld.cc b/sql/mysqld.cc > index 2b12001de9e..604eaadcc92 100644 > --- a/sql/mysqld.cc > +++ b/sql/mysqld.cc > @@ -5847,7 +5850,28 @@ int mysqld_main(int argc, char **argv) > #ifdef WITH_WSREP > wsrep_set_wsrep_on(nullptr); > if (WSREP_ON && wsrep_check_opts()) unireg_abort(1); > -#endif > + > + if (!opt_bootstrap && WSREP_PROVIDER_EXISTS && WSREP_ON) > + { > + if (global_system_variables.binlog_format != BINLOG_FORMAT_ROW) > + { > + sql_print_warning("Binlog_format changed to \"ROW\" because of Galera"); should be sql_print_information() > + global_system_variables.binlog_format= BINLOG_FORMAT_ROW; > + mark_binlog_format_used(binlog_format_used); > + } > + binlog_format_used= 1; > + if (!slave_connections_needed_for_purge_option_used) > + { > + slave_connections_needed_for_purge= > + internal_slave_connections_needed_for_purge= 0; > + mark_slave_connections_needed_for_purge_as_auto(); you don't need that, you can use SYSVAR_AUTOSIZE(internal_slave_connections_needed_for_purge, 0); > + sql_print_information( > + "slave_connections_needed_for_purge changed to 0 because " > + "of Galera. Change it to 1 or higher if this Galera node " > + "is also Master in a normal replication setup"); > + } > + } > +#endif /* WITH_WSREP */ > > #ifdef _WIN32 > /* > @@ -8219,7 +8243,9 @@ mysqld_get_one_option(const struct my_option *opt, const char *argument, > ((enum_slave_parallel_mode)opt_slave_parallel_mode); > break; > } > - > + case (int) OPT_SLAVE_CONNECTIONS_NEEDED_FOR_PURGE: > + slave_connections_needed_for_purge_option_used= 1; you don't need that, you can use if (!IS_SYSVAR_AUTOSIZE(&internal_slave_connections_needed_for_purge)) > + break; > case (int)OPT_BINLOG_IGNORE_DB: > { > binlog_filter->add_ignore_db(argument); > @@ -8714,18 +8740,14 @@ static int get_options(int *argc_ptr, char ***argv_ptr) > opt_bin_log= opt_bin_log_used= 1; > > /* Force format to row */ > + if (global_system_variables.binlog_format != BINLOG_FORMAT_ROW) > + { > + sql_print_warning("Binlog_format changed to \"ROW\" because of " > + "flashback"); sql_print_information > + global_system_variables.binlog_format= BINLOG_FORMAT_ROW; > + mark_binlog_format_used(binlog_format_used); > + } > binlog_format_used= 1; > - global_system_variables.binlog_format= BINLOG_FORMAT_ROW; > - } > - > - if (!opt_bootstrap && WSREP_PROVIDER_EXISTS && WSREP_ON && > - global_system_variables.binlog_format != BINLOG_FORMAT_ROW) > - { > - > - WSREP_ERROR ("Only binlog_format = 'ROW' is currently supported. " > - "Configured value: '%s'. Please adjust your configuration.", > - binlog_format_names[global_system_variables.binlog_format]); > - return 1; > } Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 1

GitHub account owners - please +1 these MariaDB mention PRs
by Otto Kekäläinen 10 Jul '24

10 Jul '24

Hi! I assume everyone on this mailing list has a GitHub account. Could you please +1 these PRs that add a mention of MariaDB in the documentation of various projects that currently only mention MySQL? Open each link below and press the thumbs up icon in the lower left corner of the PR description: https://github.com/bookshelf/bookshelf/pull/2129 https://github.com/ponyorm/pony/pull/708 https://github.com/metabase/metabase/issues/40325 https://github.com/SeaQL/seaql.github.io/pull/116 I am asking this because there are a lot of database tools and libraries out there that work with both MySQL and MariaDB, but they only mention MySQL, which is a pity, because the users of those tools might actually switch from MariaDB to MySQL simply out of confusion/fear that the tool supports only MySQL, which is almost never true. One of those rare cases is Javascript/Node.js Drizzle ORM which needs changes to support MariaDB. There is already a PR for that, you could +1 it as well: https://github.com/drizzle-team/drizzle-orm/pull/1692 The Rust Diesel ORM has a similar situation going on: https://github.com/diesel-rs/diesel/issues/1882 https://github.com/diesel-rs/diesel/pull/3964 However most projects have absolutely no MySQL-specific things and work with MariaDB, and just needed the docs updated, like was done in these cases: https://github.com/mysql-net/MySqlConnector/pull/1460 https://github.com/rails/rails/pull/51330 https://github.com/prisma/docs/pull/5706 https://github.com/gregstoll/gallery2/pull/168 https://github.com/coleifer/peewee/pull/2858

3 5

MariaDB 11.4 build consumers too much disk space - solved by not shipping the embedded server in Debian anymore
by Otto Kekäläinen 09 Jul '24

09 Jul '24

Hi all! While preparing 11.4.2 for upload to Debian[1] I noticed that builders fail on lack of disk space[2]. Below are some listings of build artifacts sorted by size. I solved this for now by stopping[3] to ship the embedded server in Debian, I don't think it had many users anyway. Leaving this here for visibility/comments though. [1] https://salsa.debian.org/mariadb-team/mariadb-server/-/merge_requests/88 [2] https://salsa.debian.org/otto/mariadb-server/-/jobs/5793989 [3] https://salsa.debian.org/mariadb-team/mariadb-server/-/merge_requests/88/di… ± du -shc * | sort -hr 12G total 7,8G builddir 3,7G debian 234M mysql-test 95M storage 30M sql 17M strings 5,1M plugin ± find debian/tmp/ -type f -exec du -b {} \; | sort -n | tail -n 25 8938376 debian/tmp/usr/bin/myisampack 8956744 debian/tmp/usr/bin/mariadb-test 9441288 debian/tmp/usr/bin/myisamchk 9661400 debian/tmp/usr/bin/mariadb-client-test 9725048 debian/tmp/usr/bin/mariadb-binlog 9894072 debian/tmp/usr/lib/mysql/plugin/ha_spider.so 10277896 debian/tmp/usr/lib/mysql/plugin/ha_connect.so 10622344 debian/tmp/usr/bin/aria_s3_copy 11875712 debian/tmp/usr/bin/aria_dump_log 12048792 debian/tmp/usr/bin/aria_ftdump 12079744 debian/tmp/usr/bin/aria_pack 12814096 debian/tmp/usr/bin/aria_read_log 12943168 debian/tmp/usr/bin/aria_chk 25659064 debian/tmp/usr/lib/mysql/plugin/ha_mroonga.so 138399064 debian/tmp/usr/bin/sst_dump 143275720 debian/tmp/usr/lib/mysql/plugin/ha_rocksdb.so 145093496 debian/tmp/usr/bin/mariadb-ldb 214442664 debian/tmp/usr/bin/test-connect-t 214822248 debian/tmp/usr/bin/mariadb-embedded 214849432 debian/tmp/usr/lib/x86_64-linux-gnu/libmariadbd.so.19 215011504 debian/tmp/usr/bin/mariadb-test-embedded 216148632 debian/tmp/usr/bin/mariadb-client-test-embedded 248523904 debian/tmp/usr/sbin/mariadbd 256329048 debian/tmp/usr/bin/mariadb-backup 622299760 debian/tmp/usr/lib/x86_64-linux-gnu/libmariadbd.a ± find builddir -type f -exec du -b {} \; | sort -n | tail -n 25 18278408 builddir/storage/rocksdb/librocksdb_tools.a 25227146 builddir/sql/libwsrep.a 25298746 builddir/sql/libwsrep_provider.a 25614608 builddir/storage/mroonga/vendor/groonga/lib/libgroonga.a 25659064 builddir/storage/mroonga/ha_mroonga.so 92889078 builddir/storage/perfschema/libperfschema_embedded.a 97604058 builddir/storage/perfschema/libperfschema.a 125637396 builddir/storage/innobase/libinnobase_embedded.a 131904864 builddir/storage/innobase/libinnobase.a 138399064 builddir/storage/rocksdb/sst_dump 143275720 builddir/storage/rocksdb/ha_rocksdb.so 145093496 builddir/storage/rocksdb/mariadb-ldb 189426944 builddir/unittest/sql/my_json_writer-t 189648448 builddir/unittest/sql/explain_filename-t 214442664 builddir/unittest/embedded/test-connect-t 214822248 builddir/libmysqld/examples/mariadb-embedded 214849432 builddir/libmysqld/libmariadbd.so.19 215011504 builddir/libmysqld/examples/mariadb-test-embedded 216148632 builddir/libmysqld/examples/mariadb-client-test-embedded 248523904 builddir/sql/mariadbd 256329048 builddir/extra/mariabackup/mariadb-backup 337460022 builddir/libmysqld/libsql_embedded.a 380429390 builddir/sql/libsql.a 511486028 builddir/storage/rocksdb/librocksdblib.a 622299760 builddir/libmysqld/libmariadbd.a

1 2

Re: [MariaDB commits] [PATCH] MDEV-34504 PURGE BINARY LOGS not working anymore
by Kristian Nielsen 09 Jul '24

09 Jul '24

Hi Monty, Here is my review of the patch for MDEV-34504. Looks good, just a bunch of small suggestions: > From: Monty <monty(a)mariadb.org> > > PURGE BINARY LOGS did not always purge binary logs. This commit fixes > some of the issues and adds notifications if a binary log cannot be > purged. > diff --git a/mysql-test/suite/binlog/r/binlog_index.result b/mysql-test/suite/binlog/r/binlog_index.result > index 9dfda71f9a7..2d2363a7fec 100644 > --- a/mysql-test/suite/binlog/r/binlog_index.result > +++ b/mysql-test/suite/binlog/r/binlog_index.result > @@ -30,6 +30,7 @@ flush logs; > flush logs; > *** must be a warning master-bin.000001 was not found *** > Warnings: > +Note 1375 Binary log 'master-bin.000004' is not purged because it is the current active binlog Wording: "it is the active binlog" or "it is the currently active binlog". > +Warnings: > +Note 1375 Binary log 'master-bin.000004' is not purged because it is in use by an active XID transaction A slightly better explanation is: "because it may be needed for crash recovery". This more directly explains to the user why the file cannot be removed yet (and what happens if the user removes the file anyway through the file system). It's true that this is currently related mostly to xa transactions, but the details are more complex. It's not really "active" transactions (the transactions are already committed at this point, it's the InnoDB redo log that may not yet be synced to disk). And there may be other reasons in the future (I think maybe binlog rotation already can be a reason, too). > --- a/mysql-test/suite/binlog/t/binlog_flush_binlogs_delete_domain.test > +++ b/mysql-test/suite/binlog/t/binlog_flush_binlogs_delete_domain.test > @@ -91,6 +91,8 @@ while ($domain_cnt) > FLUSH BINARY LOGS; > --let $purge_to_binlog= query_get_value(SHOW MASTER STATUS, File, 1) > --eval PURGE BINARY LOGS TO '$purge_to_binlog' > +--replace_column 2 # > +SHOW BINARY LOGS; Better use --source include/show_binary_logs.inc here (for consistency with other tests, it does the same thing). > diff --git a/sql/log.cc b/sql/log.cc > index c27c4f3353b..1384fb0b3e7 100644 > --- a/sql/log.cc > +++ b/sql/log.cc > @@ -4791,8 +4791,8 @@ int MYSQL_BIN_LOG::purge_first_log(Relay_log_info* rli, bool included) > > @@ -4902,7 +4903,6 @@ int MYSQL_BIN_LOG::purge_logs(const char *to_log, > log_info.log_file_name); > goto err; > } > - > if (find_next_log(&log_info, 0) || exit_loop) > break; > } Accidental deletion of empty line. > @@ -5428,18 +5435,23 @@ int MYSQL_BIN_LOG::real_purge_logs_by_size(ulonglong binlog_pos) > -MYSQL_BIN_LOG::can_purge_log(const char *log_file_name_arg) > +MYSQL_BIN_LOG::can_purge_log(THD *thd, const char *log_file_name_arg, > + bool interactive) > { > - THD *thd= current_thd; // May be NULL at startup > @@ -5464,8 +5479,7 @@ MYSQL_BIN_LOG::can_purge_log(const char *log_file_name_arg) > purge_sending_new_binlog_file= sending_new_binlog_file; > } > if ((res= log_in_use(log_file_name_arg, > - (is_relay_log || > - (thd && thd->lex->sql_command == SQLCOM_PURGE)) ? > + (is_relay_log || interactive) ? > 0 : slave_connections_needed_for_purge))) It's great that you avoid using current_thd here. In fact, with this change, thd is no longer used at all in can_purge_log(), so you don't need to pass it in as an argument any longer. (I'm guessing this is a left-over from an earlier version of the patch, before you introduced the `interactive` argument, which I also think is very good). So please remove the `thd` argument of can_purge_log() ... > @@ -5338,6 +5339,7 @@ int MYSQL_BIN_LOG::real_purge_logs_by_size(ulonglong binlog_pos) > MY_STAT stat_area; > char to_log[FN_REFLEN]; > ulonglong found_space= 0; > + THD *thd= current_thd; ... and then you can also remove this call of current_thd. > @@ -5473,9 +5487,39 @@ MYSQL_BIN_LOG::can_purge_log(const char *log_file_name_arg) > waiting_for_slave_to_change_binlog= 1; > strmake(purge_binlog_name, log_file_name_arg, > sizeof(purge_binlog_name)-1); > + if (res == 1) > + reason= "it is in use by a slave thread"; > + else > + reason= "less than 'slave_connections_needed_for_purge' slaves has " > + "processed it"; > + goto error; Grammar: s/has/have/: "less than 'slave_connections_needed_for_purge' slaves have processed it" > @@ -5847,7 +5850,28 @@ int mysqld_main(int argc, char **argv) > #ifdef WITH_WSREP > wsrep_set_wsrep_on(nullptr); > if (WSREP_ON && wsrep_check_opts()) unireg_abort(1); > -#endif > + > + if (!opt_bootstrap && WSREP_PROVIDER_EXISTS && WSREP_ON) > + { > + if (global_system_variables.binlog_format != BINLOG_FORMAT_ROW) > + { > + sql_print_warning("Binlog_format changed to \"ROW\" because of Galera"); > + global_system_variables.binlog_format= BINLOG_FORMAT_ROW; > + mark_binlog_format_used(binlog_format_used); > + } This change seems unrelated to MDEV-34504. Not critical I think, but for the future it's best to put such changes in a separate commit. (The change itself is good, I think.) > @@ -1303,13 +1312,19 @@ Sys_slave_connections_needed_for_purge( > "slave_connections_needed_for_purge", > "Minimum number of connected slaves required for automatic binary " > "log purge with max_binlog_total_size, binlog_expire_logs_seconds " > - "or binlog_expire_logs_days.", > + "or binlog_expire_logs_days. Default is 0 when Galera is enabled and 1 " > + "otherwise.", Since Galera is #ifdef, I guess this should be something like: "or binlog_expire_logs_days. " #ifdef WSREP "Default is 0 when Galera is enabled and 1 otherwise.", #else "Defaults to 1.", #endif > diff --git a/sql/sys_vars.h b/sql/sys_vars.h > new file mode 100644 > index 00000000000..8f4eac38cd0 > --- /dev/null > +++ b/sql/sys_vars.h > @@ -0,0 +1,17 @@ > +/* Copyright (c) 2024, MariaDB Corporation. > + > + This program is free software; you can redistribute it and/or modify > + it under the terms of the GNU General Public License as published by > + the Free Software Foundation; version 2 of the License. Hm. It's not copyrighted by MariaDB Corporation solely, there are other copyright holders. If you want to have a copyright line before the GPL header, you could maybe say "Copyright (c) 2024, MariaDB Corporation and others", or "Parts of this file Copyright (c) 2024, MariaDB Corporation". Or just omit it. - Kristian.

1 0

Re: [MariaDB commits] [PATCH] MDEV-33487 Feature non-blocking binlog
by Kristian Nielsen 08 Jul '24

08 Jul '24

Here is my review of the MDEV-33487 / PR#3087 patch. First, some general remarks. This patch addresses a specific bottleneck when binlogging huge transactions (eg. multi-gigabyte table updates logged in ROW mode). The root cause is a fundamental limitation in the current binlog format that requires all transactions to be binlogged as a single event group, using consecutive bytes in a single binlog file. Thus, while writing the large transaction to the binlog during commit, all other transaction commits will have to wait before they can commit. Eventually, we should move towards an enhanced binlog format that doesn't have this limitation, so that transactions can be partially binlogged during execution, avoiding this bottlenect. There are a number of other problems caused by this limitation that are not addressed by this patch. Changing the binlog format is a complex task not easily done, so this patch should be seen as a temporary solution to one specific bottleneck, until if/when this underlying limitation can hopefully be removed. Below my detailed review of the code in the patch. This patch touches core code of the binlog group commit mechanism, which is complex code with a number of tricky issues to consider, and very important to keep correct and maintainable. I have tried to give concrete suggestions on ways to write the code, as well as some necessary extra tests. I think if MariaDB PLC wants this patch to be merged, they need to assign a core developer to take responsibility for it to ensure that any problems that turn up later will be handled in a reasonable time frame (eg. prioritise time to work with the author to debug and get fixes reviewed and merged). > From: poempeng <poempeng(a)tencent.com> > > During the commit stage of a large transaction, it may cost too much > time to write binlog cache and block all subsequent transactions for > a long time. One possible solution of this issue is to write the binlog > of the large transaction to a temporary file and then rename it to the > next new binlog file. It's not really the cost of writing (which is similar with and without the patch), the problem is doing costly I/O under the global mutex LOCK_log that blocks other commits. I suggest making that clear from the commit message, for example: Very large transactions (eg. gigabytes) can stall other transactions for a long time because the data is written while holding LOCK_log, which blocks other commits from binlogging. One possible solution to this issue is to write a large transaction to its own binlog file outside of holding LOCK_log, and then forcing a binlog rotate after obtaining LOCK_log, simply renaming the new binlog file in place. > diff --git a/sql/log.cc b/sql/log.cc > index 460cefea47b..9de1c6118cd 100644 > --- a/sql/log.cc > +++ b/sql/log.cc > @@ -74,6 +74,9 @@ > #include <utility> // pair > #endif > > +#include <atomic> > +#include <chrono> These should not be needed after changes mentioned below. > @@ -3727,7 +3730,10 @@ bool MYSQL_BIN_LOG::open(const char *log_name, > enum cache_type io_cache_type_arg, > ulong max_size_arg, > bool null_created_arg, > - bool need_mutex) > + bool need_mutex, > + const char *file_to_rename, > + my_off_t file_size, > + group_commit_entry *entry) > + if (file_to_rename) > + { > + if (write_gtid_and_skip_event(file_size, entry)) > + goto err; > + } > + I don't think you need to write these here, inside MYSQL_BIN_LOG::open(). Instead you can do it in the caller after open() returns, right? Then you can avoid the extra arguments file_size and entry, and the conditional here. > @@ -6826,7 +6850,6 @@ MYSQL_BIN_LOG::write_gtid_event(THD *thd, bool standalone, > thd->variables.server_id= global_system_variables.server_id; > } > #endif > - Avoid unrelated changes in the diff like this one (to simplify later merges and history search). > @@ -7849,11 +7872,18 @@ int Event_log::write_cache_raw(THD *thd, IO_CACHE *cache) > - mysql_mutex_assert_owner(&LOCK_log); > + > + IO_CACHE *out_file= f; > + if (likely(f == nullptr)) > + { > + mysql_mutex_assert_owner(&LOCK_log); > + out_file= get_log_file(); > + } Don't add another conditional here. Instead just pass in the IO_CACHE to use as a parameter in each call site, eg. make the `f` parameter mandatory, not optional. > @@ -8581,7 +8612,23 @@ MYSQL_BIN_LOG::queue_for_group_commit(group_commit_entry *orig_entry) > bool > MYSQL_BIN_LOG::write_transaction_to_binlog_events(group_commit_entry *entry) > { > - int is_leader= queue_for_group_commit(entry); > + int is_leader; > + if (unlikely(entry->cache_mngr->trx_cache.get_byte_position() >= > + non_blocking_binlog_threshold) || > + DBUG_IF("non_blocking_binlog_ignore_cache_size")) > + { > + if (!can_use_non_blocking_binlog(entry) || > + write_non_blocking_binlog(entry)) > + goto original_commit_path; > + /* thread using non-blocking binlog is treated as a single group */ > + is_leader= 1; > + entry->non_blocking_log= true; > + entry->next= nullptr; > + goto group_commit_leader; > + } > +original_commit_path: > + > + is_leader= queue_for_group_commit(entry); Ok, so when we force a binlog rotation due to binlog_non_blocking_threshold, we do this by itself, not as part of a group commit. I think that is sensible. The way this gets integrated in the existing code is not very nice, with all these extra goto's and conditionals on entry->non_blocking_log. Can we instead make the decision on binlog_non_blocking_threshold in the function MYSQL_BIN_LOG::write_transaction_to_binlog()? And then call into MYSQL_BIN_LOG::trx_group_commit_leader() directly, skipping the group commit code with `is_leader` in write_transaction_to_binlog_events(). That should greatly simplify the logic. Probably the trx_group_commit_leader() needs to be refactored a bit, split in two, where the first part is dealing with grabbing the group commit queue, while only the second part is called in the binlog_non_blocking_threshold case, to do the actual binlogging. You might also consider using a separate function for the binlogging in the binlog_non_blocking_threshold case, extracting pieces from trx_group_commit_leader() into smaller shared functions; it depends on what gives the cleanest and simplest code, and is best decided as part of the refactoring of trx_group_commit_leader(). Another thing related to is this: How does the code ensure that commit ordering (on the slave) is preserved? This is the wait_for_prior_commit() mechanism, and is needed to ensure on the slave that the GTIDs are binlogged in the right order. This is normally handled by queue_for_group_commit(), which is skipped in the binlog_non_blocking_threshold case. It was not clear to me from the code how this gets handled. This probably also needs a test case to test that binlog order will be correct on the slave when multiple transactions commit in parallel (eg. optimistic parallel replication), and several of them use the binlog_non_blocking_threshold case. > @@ -12459,6 +12519,7 @@ mysql_bin_log_commit_pos(THD *thd, ulonglong *out_pos, const char **out_file) > } > #endif /* INNODB_COMPATIBILITY_HOOKS */ > > +mysql_rwlock_t binlog_checksum_rwlock; > > static void > binlog_checksum_update(MYSQL_THD thd, struct st_mysql_sys_var *var, > @@ -12468,6 +12529,7 @@ binlog_checksum_update(MYSQL_THD thd, struct st_mysql_sys_var *var, > bool check_purge= false; > ulong UNINIT_VAR(prev_binlog_id); > > + mysql_rwlock_wrlock(&binlog_checksum_rwlock); > mysql_mutex_lock(mysql_bin_log.get_log_lock()); > if(mysql_bin_log.is_open()) > { Ok, so you introduce a new lock to protect changing binlog_checksum_options. This is otherwise protected by LOCK_log, but we don't want to be holding that while writing the bin transaction, obviously. But did you check all places where binlog_checksum_options is accessed? Your patch doesn't use the new binlog_checksum_rwlock anywhere except in binlog_checksum_update(). There might be some places that currently use LOCK_log to protect the access, and could instead use the new binlog_checksum_rwlock (reducing contention on LOCK_log). Or did you check already, and there was no such opportunity? > +bool MYSQL_BIN_LOG::can_use_non_blocking_binlog(group_commit_entry *entry) > +{ > + DBUG_ASSERT(entry->cache_mngr->trx_cache.get_byte_position() >= > + non_blocking_binlog_threshold || > + DBUG_IF("non_blocking_binlog_ignore_cache_size")); > + THD *thd= entry->thd; > + if (unlikely(!is_open()) || encrypt_binlog || > + !entry->cache_mngr->stmt_cache.empty() || > + entry->cache_mngr->trx_cache.has_incident() || thd->slave_thread || > + thd->wait_for_commit_ptr) Oh. So you disable this on the slave. Is that really necessary? Ok, on the slave, in many cases the following transactions in any case are blocked from committing before the large one, so I see that this could be acceptable. It does seem a very special-purpose use case though. So if this is kept disabled for slave threads, it needs to be very clearly documented, and there should be a comment here as well in the code about why this is done. > +std::string MYSQL_BIN_LOG::generate_random_file_name() "generate_tmp_binlog_file_name" ? > + if (unlikely(!binlog_dir_inited.load(std::memory_order_acquire))) > + { > + char dev[FN_REFLEN]; > + size_t dev_length; > + mysql_mutex_lock(&LOCK_log); > + if (!binlog_dir_inited.load(std::memory_order_relaxed) && name != nullptr) This is not necessary. Just initialize what you need when the binlog is initialized, just like other binlog initialization, avoiding the check for binlog_dir_inited. See for example init_server_components() in sql/mysqld.cc > + std::string temp_file_name; Normally we don't use std::string in the server source, as it makes it harder to control the memory allocation and/or to avoid dynamic memory allocation. In this case we would normally use a fixed-size buffer of length FN_REFLEN, or the class String from sql/sql_string.h with some fixed stack-allocated initial buffer for the common case of not too long name, to avoid a dynamic memory allocation. > + temp_file_name.reserve(FN_REFLEN); > + auto now_in_sys= std::chrono::system_clock::now().time_since_epoch(); > + auto now_in_ms= > + std::chrono::duration_cast<std::chrono::milliseconds>(now_in_sys) > + .count(); You don't need to add this millisecond timestamp, do you? The file name will already be unique from temp_bin_counter? > + auto count= temp_bin_counter.fetch_add(1); This by default uses std::memory_order_seq_cst, which is overly restrictive. You can just use class Atomic_counter from include/my_counter.h to get a simple atomic counter with std::memory_order_relaxed semantics. > + temp_file_name.append(binlog_dir); > + temp_file_name.append("_temp_bin_"); > + temp_file_name.append(std::to_string(now_in_ms)); > + temp_file_name.push_back('_'); > + temp_file_name.append(std::to_string(count)); > +template <typename F= std::function<void()>> class Scoped_guard > +{ > +public: > + Scoped_guard(F f); > + ~Scoped_guard(); > + > +private: > + F func; > +}; Hm. Yes, this is nice, but it is not good if each new feature implements its own specific class like this. So either this should go somewhere shared where it can be used in the future by other code (did you check that there isn't already something similar that can be used, in the MariaDB source code or in the C++ standard library?), preferably as a separate commit. Or alternatively stay with the existing old-fascioned style for error handling used elsewhere in sql/log.cc. > + size_t skip_event_len= non_blocking_binlog_reserved_size - skip_event_start; > + skip_event_buf= > + (uchar *) my_malloc(PSI_INSTRUMENT_ME, skip_event_len, MYF(MY_ZEROFILL)); > + generate_skip_event(skip_event_buf, skip_event_len, skip_event_start, > + entry->thd); > + if (my_b_write(&log_file, skip_event_buf, skip_event_len) != 0) > + return true; Do we really need to generate this skip event? It seems quite hacky to have a random dummy event in the middle of the binlog. The size you need to reserve should be possible to pre-calculate: - Format_description_log_event - Binlog_checkpoint_event - Gtid_list_log_event - Gtid_log_event The only issue should be the Gtid_list_log_event. This can grow in size, but only if a new GTID domain id or server id gets added during the writing of the temporary file. This is very unlikely to happen, and the code can check for if this happens and in this case fall back to writing the data to the binlog file normally under LOCK_log. This way, we can avoid this strange skip event. > + if (temp_file_name.size() >= FN_REFLEN) > + { > + sql_print_warning("The name of temp file '%s' is too long!", > + temp_file_name.c_str()); > + return true; This will go in the server log for the user/DBA to wonder about. So add a bit of context to the message so that it is clear that it is about the temporary file used for binlog_non_blocking_threshold. > + thd->push_internal_handler(&dummy_error_handler); Hm. Why do you need to do this? If this is really needed, then it needs a good comment explaning why. But I suspect you might not need this at all. Is it to avoid assertions when an error is handled on the file operations? If you don't want mysql_file_open() and so to set an error with my_error() (because you want to handle the error yourself somehow), then just don't pass MY_WME as a flag to the function. > + if (unlikely(generate_new_name(new_name, name, 0))) > + abort(); > + new_name_ptr= new_name; > + /* > + We log the whole file name for log file as the user may decide > + to change base names at some point. > + */ > + Rotate_log_event r(new_name + dirname_length(new_name), 0, > + LOG_EVENT_OFFSET, 0); > + if (unlikely(write_event(&r, alg))) > + abort(); > + if (unlikely(flush_io_cache(&log_file) != 0)) > + abort(); No, don't crash the server like this. Errors during writing binlog are indeed very tricky. But it shouldn't crash the server. In the worst case, we may be left with a corrup binlog, but the error should be handled and reported to the user/DBA, similar to how it is done in other parts of the code. How does this code work with respect to the binlog checkpoint mechanism? The binlog checkpoint mechanism is a way to ensure that sufficient binlog files are kept to ensure correct crash recovery, in an asynchroneous way that does not require syncing the InnoDB redo log at binlog rotation. It uses the struct xid_count_per_binlog to keep track of which binlogs are active, and uses this to write binlog checkpoint events as appropriate. There are some tricky concurrency issues around this mechanism, so it is important to consider carefully how this will be handled in different cases by the code. For example, if the binlog checkpoint gets delayed and spans multiple binlog files including one or more generated by this patch; or what happens if RESET MASTER is run in parallel, etc. I don't see any changes related to xid_count_per_binlog in the patch. Is this because this is all handled correctly by MYSQL_BIN_LOG::open()? I would like to see a comment somewhere explaining how this is working. Also I think there needs to be some test cases testing this, eg. delayed binlog checkpointing when several binlog_non_blocking_threshold binlog rotations are running in parallel. > diff --git a/sql/log.h b/sql/log.h > index fc5209d1922..af2b56da802 100644 > --- a/sql/log.h > +++ b/sql/log.h > @@ -19,6 +19,7 @@ > +private: > + // reserved size for format_description_log_event, gtid_list_log_event ... > + static const my_off_t non_blocking_binlog_reserved_size= 10 * 1024 * 1024; Oh, 10 *megabyte* default for the dummy event? That's surely excessive? But hopefully we can avoid the dummy event completely as suggested above. > +static Sys_var_ulonglong Sys_var_non_blocking_binlog_threshold( > + "non_blocking_binlog_threshold", > + "If the binlog size of a transaction exceeds this value, we write it to a " > + "new temporary file and rename it to the next binlog file.", > + GLOBAL_VAR(non_blocking_binlog_threshold), CMD_LINE(OPT_ARG), > + VALID_RANGE(256 * 1024 * 1024, ULONGLONG_MAX), DEFAULT(ULONGLONG_MAX), > + BLOCK_SIZE(1)); Maybe name it so that it starts with "binlog_", eg. binlog_non_blocking_threshold? I think the VALID_RANGE should be extended, there is no need to forbid users to set it low for testing purposes etc. (even though I agree normally a large value would be strongly recommended). Suggestion for better description: If the binlog size in bytes of a transaction exceeds this value, force a binlog rotation and write the transaction to its own binlog file. This can reduce stalls of other `commits when binlogging very large transactions, at the cost of creating extra binlog files smaller than the configured --max-binlog-size. - Kristian.

1 0

Question regarding underscore in Numeric Literals
by Rohan Gupta 08 Jul '24

08 Jul '24

https://jira.mariadb.org/browse/MDEV-34228 I am trying my hands at this issue, which is a new feature introduced in SQL2023 and have been unable to determine if the underscore should only be part of queries, or it should be part of the users query's outputs. Also I have been trying to fiddle, around the parser in sql_lex.cc. Any pointers or guidance on how to approach this? Rohan Gupta

2 1

How to debug build/project?
by Rohan Gupta 05 Jul '24

05 Jul '24

I followed https://mariadb.org/get-involved/getting-started-for-developers/get-code-bu… and am able to run the server and mariadb client. Right now i am trying to debug the project after building and running it but havent been successful. I am trying the issue https://jira.mariadb.org/browse/MDEV-34228 In SQL 2023 we have the use of underscore with numeric literals. Any pointers on how to approach this will be appreciated. Been trying to fiddle around numericliteral.h file. Thank You

3 2

PRs with one approval pending to get the second one
by Otto Kekäläinen 01 Jul '24

01 Jul '24

Hi! Here are a couple of PRs that have already one approval, but have been pending to get the second approval for some weeks. Could any devs here spare some review time and add their approval so the PRs can be merged? (or shout out if they object them being merged) https://github.com/MariaDB/server/pull/3301 https://github.com/MariaDB/server/pull/3261 https://github.com/MariaDB/server/pull/3245 https://github.com/MariaDB/server/pull/3244 https://github.com/MariaDB/server/pull/3225 https://github.com/MariaDB/server/pull/2961

1 0

UDFs and character sets
by Jaco Kroon 25 Jun '24

25 Jun '24

Hi, It's a known issue that strings to and from UDF functions are a pain (https://dev.mysql.com/blog-archive/a-tale-of-udfs-with-character-sets/, amongst others). Most recently I ran into an issue trying to create a UDF to assist with IPv6 manipulations (in it's current, not working as ex form at https://github.com/jkroonza/uls-mysql/blob/master/src/uls_inet6.c) The two UDF functions there is intended to find the "network" (first) and "broadcast" (last) address of a range, with the intention to add functions to perform other manipulations (like adding some offset to an IPv6, or even add two IPv6's together, eg, to find the "next" network for 2c0f:f720::/40 you can add 0:0:100:: to 2c0f:f720::). The referenced works great when you get values from inet6 column to pass to the function (it gets transformed into textual presentation before being passed to the UDF) eg: MariaDB [test]> select uls_inet6_network_address('::ffff:192.168.1.123', 120); +--------------------------------------------------------+ | uls_inet6_network_address('::ffff:192.168.1.123', 120) | +--------------------------------------------------------+ | ::ffff:192.168.1.0 | +--------------------------------------------------------+ 1 row in set (0.001 sec) However, given this: MariaDB [test]> create table t(t inet6 not null, primary key(t)); Query OK, 0 rows affected (0.024 sec) MariaDB [test]> insert into t values('::ffff:192.168.1.15'); Query OK, 1 row affected (0.014 sec) MariaDB [test]> select * from t where uls_inet6_network_address('::ffff:192.168.1.123', 120) < t; Empty set, 1 warning (0.009 sec) MariaDB [test]> show warnings; +---------+------+---------------------------------------------+ | Level | Code | Message | +---------+------+---------------------------------------------+ | Warning | 1292 | Incorrect inet6 value: '::ffff:192.168.1.0' | +---------+------+---------------------------------------------+ 1 row in set (0.000 sec) I started to dig, to find this: MariaDB [test]> select charset(uls_inet6_network_address('::ffff:192.168.1.123', 120)); +-----------------------------------------------------------------+ | charset(uls_inet6_network_address('::ffff:192.168.1.123', 120)) | +-----------------------------------------------------------------+ | binary | +-----------------------------------------------------------------+ 1 row in set (0.000 sec) Then I played a bit and found that if I can rig it to the returned string has 16 characters I don't get an error; eg: MariaDB [test]> select * from t where uls_inet6_network_address('::ff:f:ffff:ffff', 120) > t; +---------------------+ | t | +---------------------+ | ::ffff:192.168.1.15 | +---------------------+ 1 row in set (0.001 sec) Based on this I'm inferring that: 1. Contextually the difference between BINARY() and CHAR() types is the character set (ie, binary is implemented as a kind of characters set for strings, or perhaps strings are binaries with a specific character set). 2. The inet6 storage column (which I now realize is simply BINARY(16), with a transform applied to transform strings to and from the BINARY() format, I think the technical term in the code is "fixed binary storage storage"). 3. When string (character) data gets sent *to* an inet6 column a "character set conversion" is performed, and as is the case with UDFs, it's already BINARY, so no conversion is actually performed, and since the BINARY length is now *not* 16 bytes, we get the above warning, and thus comparisons are as if with a NULL value, and always false. I've been contemplating possible "solutions" to the problem without breaking backwards compatibility, and it's tricky. The below aims at a simple way of specifying the return "character set" of a string return in more detail. And possibly even looking at the character sets for parameters. MySQL have started down a path (as per the tale above) whereby it's using some other mechanism to give indications about character sets. I don't like the particular mechanism, but it could work, looks like it does only character set though, in which case for my example above, I could just force both input and output to "latin1" so that MySQL/MariaDB is forced to convert between that and INET6 again. I think we can do better. I'm not convinced their change will permit backwards compatibility - ie, any UDF that uses those functions will require them and cannot opt to operate in a degraded mode. Not sure that's even a desirable thing to have though ... so perhaps copying what they're doing here is the way to go. My ideas: Option 1. Simply permit specifying a more precise string type during function creation. For example, instead of: MariaDB [uls]> CREATE FUNCTION uls_inet6_last_address RETURNS STRING SONAME 'uls_inet6.so'; Do this: MariaDB [uls]> CREATE FUNCTION uls_inet6_last_address RETURNS INET6 SONAME 'uls_inet6.so'; And this is permitted because INET6 is a specialization of BINARY, which is really what MySQL passes to/from UDF functions anyway. The downside is this will only work for returns. This would, work for my use case, since from INET6 => UDF the decoder is called anyway, so I do get the text presentation on function entry, and now I can now I just need to return the binary form of INET6. If sent back to the client it gets converted by the server to This eliminates at least *one* of the two pointless conversions to/from binary format. Option 2. A mechanism to pass the character set/binary type upon entry and exit. What I'm thinking here is a non-backwards compatible change, potentially. Or use the *extension pointer somehow. I'm still having some trouble navigating the codebase to figure out exactly how the void *extension pointers in UDF_ARGS and UDF_INIT structs are used. But I'm thinking they can be (ab)used to somehow pass an additional structure, which may be how 3 is achieved perhaps? To be more precise, the current UDF_ARGS looks like: 47 typedef struct UDF_ARGS { 48 unsigned int arg_count; /**< Number of arguments */ 49 enum Item_result *arg_type; /**< Pointer to item_results */ 50 char **args; /**< Pointer to argument */ 51 unsigned long *lengths; /**< Length of string arguments */ 52 char *maybe_null; /**< Set to 1 for all maybe_null args */ 53 char **attributes; /**< Pointer to attribute name */ 54 unsigned long *attribute_lengths; /**< Length of attribute arguments */ 55 void *extension; 56 } UDF_ARGS; I can't find any source files that references both the UDF_ARGS type and references this extension field. As such I'm hoping I can safely assume that in current implementations this will always be NULL. If so it becomes possible to turn this into a pointer-sized version field, or struct size field, ie something like: union { void *extension; size_t argsize; /* one alternative */ long api_version; /* another alternative */ }; for argsize this way it becomes possible to check that this field is >= sizeof(UDF_ARGS) against which the UDF was compiled. This *feels* flakey, but should be fine. If a user wants more fine-grained compile backwards compat, then the user can check that all fields (further than those listed) are positioned inside the structure. Another option is simply to have extension be a pointer to some UDF_ARGS_EXPANDED struct, which really is a pointer back to UDF_ARGS which is really something like: typedef struct UDF2_ARGS { ... everything from above, except void* extension becomes: void *UDF2_ARGS extension; /* this points to self */ size_t argsize; /* or long api_version */ bool udf2_supported; /* set to false by the server, if the UDF supports this, set to true */ const char ** string_arg_types; /* in _init this will be passed as the raw type, eg, INET6 if the input is an INET6 field, can be set in _init to have the server cast the type, ie, perform some conversion */ ... additional fields; } UDF2_ARGS; So in my example, if udf2_supported comes back as false after _init, then current behaviour is maintained, and INET6 will be converted to string, and return strings are assumed to be BINARY (current behaviour). If however, udf2_supported, then my code could set string_args_types[0] = "inet6"; - which the engine knows is a BINARY(16) type, and use the INET6 conversion code to convert to and from BINARY(16) as needed. Option 3. Some other more formal UDF API versioning/api definition/scheme. So call this UDFv2, and make this extensible so that features can be "negotiated", or that the UDF itself can even "fail to load" in case where it requires support from the running server that's not provided. The idea here would be that a symbol like "mariadb_udf_init" is exported, passing it some structure that defines the capabilities of the server it's running on, the module can then decline to load, or proceed, but limit itself based on functionality available kind of thing. I'm guessing option 2 is also a way of achieving this. And may be sufficient. I do however like the idea of having a {func}_load and {func}_unload call (some of our PBKDF functions would also benefit from this in that we can load the openssl algorithms once at startup instead of every time on _init. There may be other complications though, but still, this could be one suggestion for v2. I'm happy to write a more formal specification based off of the informal specification, and I'm inclined to attempt option 2 above. I just honestly have no idea what's all REALLY involved, and where to find the various bits and pieces of code. So I guess the initial steps would be: 1. Add a _load function which is invoked at *load* time, passing server and version information. Not sure what else could be useful, intension is to initialise the UDF for use as needed. UDFs that want to maintain backwards compatible would need to track if this was indeed called or not, and if not, act accordingly, or if it can't be backwards compatible without this, error out in _init. Suggestions as to parameters to pass would be great, I suggest we use a struct again with a size field as it's first field to ensure that we can add extra fields at a later stage, safely. Eg: struct UDF_LOADINFO { size_t load_size; const char* server; struct { uint16_t major, minor, patch; /* 10, 6, 17 - for example what I've got runing currently */ const char* extra; /* MariaDB-log, the stuff after the dash from SELECT version(); */ } version; } UDF_LOADINFO; And so we will call "bool (*func_load)(const UDF_LOADINFO*)" IFF it exist. 2. Add an _unload function "void (*func_unload)()", purpose of which is to clean up after _load. Note: It's possible to use __attribute__((constructor)) and destructor functions, but my experience is that they're messy, and if you have multiple functions in a UDF you can't know which ones you're being loaded for. 3. Expand for UDF2_ARGS as explained in option 2 above, as well as UDF2_INIT. 4. Ability to flag functions as *deterministic* (same return for same arguments) rather than merely const or not const. This way SQL can optimize calls in the same manner as for stored functions. I'm not going to dig into that, but happy to add the flag. 5. Document and flag this as UDFv2 as and when it hits formal release. Kind regards, Jaco

2 7

Re: 98ebe0a3afc: MDEV-19123 Change default charset from latin1 to utf8mb4
by Sergei Golubchik 17 Jun '24

17 Jun '24

Hi, Alexander, No comments about actual code changes, only about tests. Please, see below On Jun 12, Alexander Barkov wrote: > revision-id: 98ebe0a3afc (mariadb-11.5.1-12-g98ebe0a3afc) > parent(s): 186a30de58b > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2024-06-11 14:17:11 +0400 > message: > > MDEV-19123 Change default charset from latin1 to utf8mb4 > > Changing the default server character set from latin1 to utf8mb4. > diff --git a/mysql-test/main/column_compression.test b/mysql-test/main/column_compression.test > --- a/mysql-test/main/column_compression.test > +++ b/mysql-test/main/column_compression.test > @@ -9,20 +9,20 @@ let $typec= BLOB COMPRESSED; > let $typeu= BLOB; > --source column_compression.inc > > -let $typec= TEXT COMPRESSED; > -let $typeu= TEXT; > +let $typec= TEXT COMPRESSED CHARACTER SET latin1; > +let $typeu= TEXT CHARACTER SET latin1; why? > --source column_compression.inc > > let $typec= VARBINARY(10000) COMPRESSED; > let $typeu= VARBINARY(10000); > --source column_compression.inc > > -let $typec= VARCHAR(10000) COMPRESSED; > -let $typeu= VARCHAR(10000); > +let $typec= VARCHAR(10000) COMPRESSED CHARACTER SET latin1; > +let $typeu= VARCHAR(10000) CHARACTER SET latin1; > --source column_compression.inc > > let $typec= TEXT COMPRESSED CHARSET ucs2; > -let $typeu= TEXT; > +let $typeu= TEXT CHARACTER SET latin1; > --source column_compression.inc > > SET column_compression_zlib_wrap=DEFAULT; > diff --git a/mysql-test/main/create.result b/mysql-test/main/create.result > --- a/mysql-test/main/create.result > +++ b/mysql-test/main/create.result > @@ -506,9 +506,9 @@ FROM t1; > SHOW CREATE TABLE t2; > Table Create Table > t2 CREATE TABLE `t2` ( > - `ifnull(c_tinytext, CAST('yet another binary data' AS BINARY))` tinyblob DEFAULT NULL, > - `ifnull(c_text, CAST('yet another binary data' AS BINARY))` blob DEFAULT NULL, > - `ifnull(c_mediumtext, CAST('yet another binary data' AS BINARY))` mediumblob DEFAULT NULL, > + `ifnull(c_tinytext, CAST('yet another binary data' AS BINARY))` blob DEFAULT NULL, > + `ifnull(c_text, CAST('yet another binary data' AS BINARY))` mediumblob DEFAULT NULL, > + `ifnull(c_mediumtext, CAST('yet another binary data' AS BINARY))` longblob DEFAULT NULL, looks wrong UPD: there were more changes like this below, I didn't comment on them > `ifnull(c_longtext, CAST('yet another binary data' AS BINARY))` longblob DEFAULT NULL > ) ENGINE=MyISAM DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_uca1400_ai_ci > DROP TABLE t2; > diff --git a/mysql-test/main/ctype_utf8_def_upgrade.result b/mysql-test/main/ctype_utf8_def_upgrade.result > --- a/mysql-test/main/ctype_utf8_def_upgrade.result > +++ b/mysql-test/main/ctype_utf8_def_upgrade.result > @@ -61,23 +61,23 @@ t1 CREATE TABLE `t1` ( > PRIMARY KEY (`Host`,`Db`) > ) ENGINE=MyISAM DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_uca1400_ai_ci COMMENT='Host privileges; Merged with database privileges' > DROP TABLE t1; > -SET @@character_set_database=DEFAULT; > +SET @@collation_database=DEFAULT; why? > # Now do the same, but doing 'ALTER DATABASE' to create the db.opt file, > # instead of setting variables directly. > # Emulate a pre-4.1 database without db.opt > SHOW CREATE DATABASE db1; > Database Create Database > -db1 CREATE DATABASE `db1` /*!40100 DEFAULT CHARACTER SET utf8mb3 COLLATE utf8mb3_uca1400_ai_ci */ > +db1 CREATE DATABASE `db1` /*!40100 DEFAULT CHARACTER SET utf8mb3 COLLATE utf8mb3_general_ci */ > USE db1; > -SELECT @@character_set_database, 'taken from defaults' AS comment; > -@@character_set_database comment > -utf8mb3 taken from defaults > +SELECT @@collation_database, 'taken from defaults' AS comment; > +@@collation_database comment > +utf8mb3_general_ci taken from defaults > USE test; > ALTER DATABASE db1 DEFAULT CHARACTER SET latin1; > USE db1; > -SELECT @@character_set_database, 'taken from db.opt' AS comment; > -@@character_set_database comment > -latin1 taken from db.opt > +SELECT @@collation_database, 'taken from db.opt' AS comment; > +@@collation_database comment > +latin1_swedish_ci taken from db.opt > SELECT COUNT(*) FROM t1; > ERROR HY000: Got error 190 "Incompatible key or row definition between the MariaDB .frm file and the information in the storage engine. You can try REPAIR TABLE ... USE_FRM possibly followed by ALTER TABLE ... FORCE or dump and restore the table to fix this" from storage engine MyISAM > REPAIR TABLE t1 USE_FRM; > diff --git a/mysql-test/main/endspace.test b/mysql-test/main/endspace.test > --- a/mysql-test/main/endspace.test > +++ b/mysql-test/main/endspace.test > @@ -13,7 +13,7 @@ drop table if exists t1; > # Test MyISAM tables. > # > > -create table t1 (text1 varchar(32) not NULL, KEY key1 (text1)); > +create table t1 (text1 varchar(32) not NULL, KEY key1 (text1)) charset=latin1; why is that? > insert into t1 values ('teststring'), ('nothing'), ('teststring\t'); > check table t1; > select * from t1 ignore key (key1) where text1='teststring' or > diff --git a/mysql-test/main/func_compress.test b/mysql-test/main/func_compress.test > --- a/mysql-test/main/func_compress.test > +++ b/mysql-test/main/func_compress.test > @@ -1,5 +1,9 @@ > -- source include/have_compress.inc > -- source include/have_normal_zlib.inc > + > +--source include/test_db_charset_latin1.inc same as in column_compression.test ? > + > + > # > # Test for compress and uncompress functions: > # > diff --git a/mysql-test/main/long_unique.result b/mysql-test/main/long_unique.result > --- a/mysql-test/main/long_unique.result > +++ b/mysql-test/main/long_unique.result > @@ -41,7 +41,7 @@ Ignored NO > > MyISAM file: DATADIR/test/t1 > Record format: Packed > -Character set: latin1_swedish_ci (8) > +Character set: ? (0) huh? > Data records: 10 Deleted blocks: 0 > Recordlength: 12 > > diff --git a/mysql-test/main/mysqlbinlog_row_compressed.result b/mysql-test/main/mysqlbinlog_row_compressed.result > --- a/mysql-test/main/mysqlbinlog_row_compressed.result > +++ b/mysql-test/main/mysqlbinlog_row_compressed.result > @@ -40,9 +40,10 @@ SET @@session.sql_mode=#/*!*/; > SET @@session.auto_increment_increment=1, @@session.auto_increment_offset=1/*!*/; > /*!\C latin1 *//*!*/; > SET @@session.character_set_client=latin1,@@session.collation_connection=8,@@session.collation_server=#/*!*/; > +SET @@session.character_set_collations='utf8mb3=utf8mb3_uca1400_ai_ci,ucs2=ucs2_uca1400_ai_ci,utf8mb4=utf8mb4_uca1400_ai_ci,utf16=utf16_uca1400_ai_ci,utf32=utf32_uca1400_ai_ci'/*!*/; why did this appear? > SET @@session.lc_time_names=0/*!*/; > SET @@session.collation_database=DEFAULT/*!*/; > -CREATE TABLE t1 (pk INT PRIMARY KEY, f1 INT, f2 INT, f3 TINYINT, f4 MEDIUMINT, f5 BIGINT, f6 INT, f7 INT, f8 char(1)) > +CREATE TABLE t1 (pk INT PRIMARY KEY, f1 INT, f2 INT, f3 TINYINT, f4 MEDIUMINT, f5 BIGINT, f6 INT, f7 INT, f8 char(1)) CHARSET=latin1 > /*!*/; > # at <pos> > #<date> server id 1 end_log_pos # CRC32 XXX GTID 0-1-2 ddl thread_id=TID > diff --git a/mysql-test/main/sp.result b/mysql-test/main/sp.result > --- a/mysql-test/main/sp.result > +++ b/mysql-test/main/sp.result > @@ -6800,6 +6800,8 @@ DROP FUNCTION IF EXISTS f2; > # > > CREATE FUNCTION f1() RETURNS VARCHAR(65525) RETURN 'Hello'; > +Warnings: > +Note 1246 Converting column '' from VARCHAR to TEXT this could use latin1, I suppose > > CREATE FUNCTION f2() RETURNS TINYINT RETURN 1; > > diff --git a/mysql-test/suite/binlog_encryption/rpl_special_charset.opt b/mysql-test/suite/binlog_encryption/rpl_special_charset.opt > --- a/mysql-test/suite/binlog_encryption/rpl_special_charset.opt > +++ b/mysql-test/suite/binlog_encryption/rpl_special_charset.opt > @@ -1 +1 @@ > ---character-set-server=utf16 > +--character-set-server=utf16 --collation-server=utf16_general_ci why? > diff --git a/mysql-test/suite/federated/assisted_discovery.test b/mysql-test/suite/federated/assisted_discovery.test > --- a/mysql-test/suite/federated/assisted_discovery.test > +++ b/mysql-test/suite/federated/assisted_discovery.test > @@ -38,7 +38,7 @@ create table t1 ( > d varchar(4096) not null, > primary key (a), > key (b,c,d(255)) > -); > +) CHARSET=latin1; why? > show create table t1; > > connection master; > diff --git a/mysql-test/suite/funcs_1/r/is_columns_innodb.result b/mysql-test/suite/funcs_1/r/is_columns_innodb.result > --- a/mysql-test/suite/funcs_1/r/is_columns_innodb.result > +++ b/mysql-test/suite/funcs_1/r/is_columns_innodb.result > @@ -739,6 +740,9 @@ WHERE table_schema LIKE 'test%' > AND CHARACTER_OCTET_LENGTH / CHARACTER_MAXIMUM_LENGTH <> 1 > ORDER BY CHARACTER_SET_NAME, COLLATION_NAME, COL_CML; > COL_CML DATA_TYPE CHARACTER_SET_NAME COLLATION_NAME > +4.0000 char utf8mb4 utf8mb4_uca1400_ai_ci > +4.0000 enum utf8mb4 utf8mb4_uca1400_ai_ci > +4.0000 set utf8mb4 utf8mb4_uca1400_ai_ci I suppose this changed the intention of the test > Warnings: > Warning 1365 Division by 0 > Warning 1365 Division by 0 > diff --git a/mysql-test/suite/gcol/r/innodb_virtual_index.result b/mysql-test/suite/gcol/r/innodb_virtual_index.result > --- a/mysql-test/suite/gcol/r/innodb_virtual_index.result > +++ b/mysql-test/suite/gcol/r/innodb_virtual_index.result > @@ -114,7 +114,7 @@ KEY `vbidxcol` (`vbidxcol`), > KEY `a_2` (`a`,`vbidxcol`), > KEY `vbidxcol_2` (`vbidxcol`), > FULLTEXT KEY `ftsic` (`c`,`b`) > -) ENGINE=InnoDB; > +) ENGINE=InnoDB CHARSET=latin1; why latin1 everywhere in this file? > Warnings: > Note 1831 Duplicate index `vbidxcol_2`. This is deprecated and will be disallowed in a future release > ALTER TABLE ibstd_08 ADD COLUMN nc07006 BIGINT AUTO_INCREMENT NOT NULL , ADD KEY auto_nc07006(nc07006); > diff --git a/mysql-test/suite/innodb/r/online_table_rebuild.result b/mysql-test/suite/innodb/r/online_table_rebuild.result > --- a/mysql-test/suite/innodb/r/online_table_rebuild.result > +++ b/mysql-test/suite/innodb/r/online_table_rebuild.result > @@ -10,7 +10,7 @@ INSERT INTO t1 VALUES(2, repeat('b', 100), repeat('a', 100)); > COMMIT; > SET DEBUG_SYNC="now SIGNAL dml_commit"; > connection default; > -ERROR 23000: Duplicate entry 'bbbbbbbbbb' for key 'f2' > +ERROR 23000: Duplicate entry 'bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb' for key 'f2' why? > connection default; > SET DEBUG_SYNC="inplace_after_index_build SIGNAL dml_start WAIT_FOR dml_commit"; > ALTER TABLE t1 ADD PRIMARY KEY(f1); > diff --git a/mysql-test/suite/innodb/t/alter_primary_key.test b/mysql-test/suite/innodb/t/alter_primary_key.test > --- a/mysql-test/suite/innodb/t/alter_primary_key.test > +++ b/mysql-test/suite/innodb/t/alter_primary_key.test > @@ -1,6 +1,9 @@ > --source innodb_default_row_format.inc > --source include/have_debug.inc > --source include/have_debug_sync.inc > +--disable_query_log > +--source include/test_db_charset_latin1.inc > +--enable_query_log you don't disable query log in other tests, why here? > > --echo # > --echo # MDEV-23244 ALTER TABLE…ADD PRIMARY KEY fails to flag > diff --git a/mysql-test/suite/perfschema/t/short_option_1-master.opt b/mysql-test/suite/perfschema/t/short_option_1-master.opt > --- a/mysql-test/suite/perfschema/t/short_option_1-master.opt > +++ b/mysql-test/suite/perfschema/t/short_option_1-master.opt > @@ -1 +1 @@ > --a -Cutf8 -W1 > +-a -Cutf8 -W1 --collation-server=utf8_general_ci was it necessary? > diff --git a/mysql-test/suite/rpl/include/rpl_charset.inc b/mysql-test/suite/rpl/include/rpl_charset.inc > --- a/mysql-test/suite/rpl/include/rpl_charset.inc > +++ b/mysql-test/suite/rpl/include/rpl_charset.inc > @@ -145,4 +145,11 @@ sync_slave_with_master; > --echo # > > > +connection master; > +SET GLOBAL collation_server=utf8mb4_uca1400_ai_ci; > +SET SESSION collation_server=utf8mb4_uca1400_ai_ci; > +connection slave; > +SET GLOBAL collation_server=utf8mb4_uca1400_ai_ci; > +SET SESSION collation_server=utf8mb4_uca1400_ai_ci; why? > + > --source include/rpl_end.inc > diff --git a/mysql-test/suite/sys_vars/inc/secure_timestamp_func.inc b/mysql-test/suite/sys_vars/inc/secure_timestamp_func.inc > --- a/mysql-test/suite/sys_vars/inc/secure_timestamp_func.inc > +++ b/mysql-test/suite/sys_vars/inc/secure_timestamp_func.inc > @@ -58,7 +58,7 @@ set time_zone='+00:00'; > set timestamp=1234567890.101112; > select @@timestamp, now(6); > > -create table t1 (b varchar(20), a timestamp(6) default current_timestamp(6)); > +create table t1 (b varchar(20), a timestamp(6) default current_timestamp(6)) charset=latin1; why? charset shouldn't affect secure_timestamp functionality (same in all other secure_timestamp tests) > insert t1 (b) values ('replicated'); > sync_slave_with_master; > create trigger t1rbr before insert on t1 for each row set new.a=now(6); > diff --git a/storage/connect/mysql-test/connect/r/bson.result b/storage/connect/mysql-test/connect/r/bson.result > --- a/storage/connect/mysql-test/connect/r/bson.result > +++ b/storage/connect/mysql-test/connect/r/bson.result > @@ -321,7 +321,7 @@ WHO CHAR(12), > WEEK INT(2) JPATH='$.WEEK[2].NUMBER', > WHAT CHAR(32) JPATH='$.WEEK[2].EXPENSE[*].WHAT', > AMOUNT DOUBLE(8,2) JPATH='$.WEEK[2].EXPENSE[*].AMOUNT') > -ENGINE=CONNECT TABLE_TYPE=BSON FILE_NAME='expense.json'; > +ENGINE=CONNECT CHARSET=latin1 TABLE_TYPE=BSON FILE_NAME='expense.json'; is it necessary? are there any connect/bson tests for utf8mb4? > SELECT * FROM t4; > WHO WEEK WHAT AMOUNT > Joe 5 Beer 14.00 > diff --git a/storage/connect/mysql-test/connect/r/endian.result b/storage/connect/mysql-test/connect/r/endian.result > --- a/storage/connect/mysql-test/connect/r/endian.result > +++ b/storage/connect/mysql-test/connect/r/endian.result > @@ -10,7 +10,7 @@ birth DATE NOT NULL FIELD_FORMAT='L', > id CHAR(5) NOT NULL FIELD_FORMAT='L2', > salary DOUBLE(9,2) NOT NULL DEFAULT 0.00 FIELD_FORMAT='LF', > dept INT(4) NOT NULL FIELD_FORMAT='L2' > -) ENGINE=CONNECT TABLE_TYPE=BIN BLOCK_SIZE=5 FILE_NAME='Testbal.dat'; > +) ENGINE=CONNECT CHARSET=latin1 TABLE_TYPE=BIN BLOCK_SIZE=5 FILE_NAME='Testbal.dat'; or any connect utf8mb4 tests whatsoever? > SELECT * FROM t1; > fig name birth id salary dept > 5500 ARCHIBALD 1980-01-25 3789 4380.50 318 > diff --git a/storage/rocksdb/mysql-test/rocksdb/r/checksum_table_live.result b/storage/rocksdb/mysql-test/rocksdb/r/checksum_table_live.result > --- a/storage/rocksdb/mysql-test/rocksdb/r/checksum_table_live.result > +++ b/storage/rocksdb/mysql-test/rocksdb/r/checksum_table_live.result > @@ -1,7 +1,7 @@ > DROP TABLE IF EXISTS t1,t2; > -CREATE TABLE t1 (a INT PRIMARY KEY, b CHAR(8)) ENGINE=rocksdb CHECKSUM=1; > +CREATE TABLE t1 (a INT PRIMARY KEY, b CHAR(8)) ENGINE=rocksdb CHECKSUM=1 CHARSET=latin1; again. was it necessary? are there any rocksdb tests with utf8mb4? > INSERT INTO t1 (a,b) VALUES (1,'a'),(2,'b'); > -CREATE TABLE t2 (a INT PRIMARY KEY, b CHAR(8)) ENGINE=rocksdb CHECKSUM=1; > +CREATE TABLE t2 (a INT PRIMARY KEY, b CHAR(8)) ENGINE=rocksdb CHECKSUM=1 CHARSET=latin1; > CHECKSUM TABLE t1; > Table Checksum > test.t1 4259194219 Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 3

Review of MDEV-30073 Wrong result on 2nd execution of PS for query
by Michael Widenius 12 Jun '24

12 Jun '24

Review of MDEV-30073 Wrong result on 2nd execution of PS for query with NOT EXISTS Bye the way, nice to see that this patch close so many different MDEVs! Please add a list of all the MDEV's closed by this patch in the commit message! MDEV-23828: query with EXISTS over subquery using mergeable derived table / view executed in PS/SP for the second time MDEV-30396: 2nd execution of query with EXISTS subquery over view / DT MDEV-31175: 2nd execution of SELECT with mergeable view / DT and aggregate function MDEV-33081: 2-nd execution of query with forced materialization of a mergeable view because of too many base tables MDEV-31269: 2nd execution of query with HAVING containing IN predicand over subquery with EXISTS in WHERE clause MDEV-31281: 2nd execution of query with IN subquery whose left operand is aggregate function or aggregate function MDEV-31178: 2 execution of query with conversion from IN predicate to IN subquery applied > index 446154e517b..d474a39e228 100644 > --- a/mysql-test/main/func_group.test > +++ b/mysql-test/main/func_group.test > @@ -1268,12 +1268,14 @@ set @@optimizer_switch='materialization=on,in_to_exists=off,semijoin=off'; > --echo # 1) Test that subquery materialization is setup for query with > --echo # premature optimize() exit due to "Impossible WHERE" > --echo # > +--disable_ps2_protocol > SELECT MIN(t2.pk) > FROM t2 JOIN t1 ON t1.pk=t2.pk > WHERE 'j' > HAVING ('m') IN ( > SELECT v > FROM t2); > +--enable_ps2_protocol Why this change? Does this produce wrong results after your patch? At least I cannot see anything in the query that would disallow double execution. If double execution does not work anymore for this one, please add a note about this in the commit message or in the test case. > diff --git a/mysql-test/main/mysqltest_ps.test b/mysql-test/main/mysqltest_ps.test > index c5a332f691f..853ce8bd38e 100644 > --- a/mysql-test/main/mysqltest_ps.test > +++ b/mysql-test/main/mysqltest_ps.test > @@ -32,3 +32,4 @@ select 1 + "2 a"; > create table t (a int primary key, b blob default ''); > select a, (2*a) AS a from t group by a; > drop table t; > + > Remove the change to the above. No need to touch this test case in this commit > diff --git a/mysql-test/main/order_by.test b/mysql-test/main/order_by.test > index c1e0c318283..7ca9a8f8def 100644 > --- a/mysql-test/main/order_by.test > +++ b/mysql-test/main/order_by.test > @@ -90,7 +90,9 @@ drop table t1; > create table t1 (i int); > insert into t1 values(1),(2),(1),(2),(1),(2),(3); > select distinct i from t1; > +--disable_ps2_protocol > select distinct i from t1 order by rand(5); > +--enable_ps2_protocol > Why disable ps2_protocol here? > diff --git a/mysql-test/main/type_date.test b/mysql-test/main/type_date.test > index 3566e427d50..5f17a893d53 100644 > --- a/mysql-test/main/type_date.test > +++ b/mysql-test/main/type_date.test > @@ -601,7 +601,7 @@ SHOW CREATE TABLE t2; > DROP TABLE t2; > DROP VIEW v1; > DROP TABLE t1; > - > +# --enable_ps2_protocol The above looks wrong. Probably leftover from some testing. Please revert the change to this file > diff --git a/mysql-test/main/type_newdecimal.test b/mysql-test/main/type_newdecimal.test > index 366199e8aa8..7223b7b44f0 100644 > --- a/mysql-test/main/type_newdecimal.test > +++ b/mysql-test/main/type_newdecimal.test > @@ -1929,7 +1929,9 @@ drop table t1; > SELECT AVG(DISTINCT 0.00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001); > --enable_view_protocol > CREATE TABLE t1 AS SELECT NULL AS v1; > +--disable_ps2_protocol > SELECT 1 FROM t1 GROUP BY v1 ORDER BY AVG ( from_unixtime ( '' ) ) ; > +--enable_ps2_protocol > DROP TABLE t1; Why the change? > diff --git a/mysql-test/main/win.test b/mysql-test/main/win.test > index cba6b0fd7af..00c79425fc0 100644 > --- a/mysql-test/main/win.test > +++ b/mysql-test/main/win.test > @@ -2770,6 +2770,7 @@ drop table t1; > CREATE TABLE t1 ( a char(25), b text); > INSERT INTO t1 VALUES ('foo','bar'); > > +--disable_ps2_protocol > SELECT > SUM(b) OVER (PARTITION BY a), > ROW_NUMBER() OVER (PARTITION BY b) > @@ -2777,6 +2778,7 @@ FROM t1 > GROUP BY > LEFT((SYSDATE()), 'foo') > WITH ROLLUP; > +--enable_ps2_protocol > drop table t1; Why ? > diff --git a/mysql-test/suite/federated/federatedx_create_handlers.test b/mysql-test/suite/federated/federatedx_create_handlers.test > index 8e504590282..b2884397333 100644 > --- a/mysql-test/suite/federated/federatedx_create_handlers.test > +++ b/mysql-test/suite/federated/federatedx_create_handlers.test > @@ -94,6 +94,7 @@ DEFAULT CHARSET=latin1; > INSERT INTO federated.t3 VALUES > ('yyy'), ('www'), ('yyy'), ('xxx'), ('www'), ('yyy'), ('www'); > > +--sorted_result > SELECT * > FROM federated.t3, (SELECT * FROM federated.t1 WHERE id > 3) t > WHERE federated.t3.name=t.name; > I assume this has nothing to do with the patch, but just a random failure you fixed? If that is the case, please add a comment about this file change in the commit message or move this change to another commit. > --- a/sql/item.cc > +++ b/sql/item.cc > @@ -3027,7 +3027,11 @@ Item* Item_ref::build_clone(THD *thd) > unlikely(!(copy->ref= (Item**) alloc_root(thd->mem_root, > sizeof(Item*)))) || > unlikely(!(*copy->ref= (* ref)->build_clone(thd)))) > + { > + if (copy) > + copy->ref= 0; > return 0; > + } > return copy; > } Ok code, but I would have prefered this a bit more simple approach: Item* Item_ref::build_clone(THD *thd) { Item_ref *copy= (Item_ref *) get_copy(thd); if (likely(copy)) { if (unlikely(!(copy->ref= (Item**) alloc_root(thd->mem_root, sizeof(Item*)))) || unlikely(!(*copy->ref= (* ref)->build_clone(thd)))) { copy->ref= 0; return 0; } } return copy; } > @@ -5823,22 +5827,48 @@ Item_field::fix_outer_field(THD *thd, Field **from_field, Item **reference) > select->group_list.elements && > (place == SELECT_LIST || place == IN_HAVING)) > { > - Item_outer_ref *rf; > /* > - If an outer field is resolved in a grouping select then it > - is replaced for an Item_outer_ref object. Otherwise an > - Item_field object is used. > - The new Item_outer_ref object is saved in the inner_refs_list of > - the outer select. Here it is only created. It can be fixed only > - after the original field has been fixed and this is done in the > - fix_inner_refs() function. > - */ > - ; > - if (!(rf= new (thd->mem_root) Item_outer_ref(thd, context, this))) > - return -1; > - thd->change_item_tree(reference, rf); > - select->inner_refs_list.push_back(rf, thd->mem_root); > - rf->in_sum_func= thd->lex->in_sum_func; > + This Item_field represents a reference to a column in some > + outer select. Wrap this Item_field item into an Item_outer_ref > + object if we are at the first execution of the query or at > + processing the prepare command for it. Make this wrapping > + permanent for the first query execution so that it could be used > + for next executions of the query. > + Add the wrapper item to the list st_select_lex::inner_refs_list > + of the select against which this Item_field has been resolved. > + */ > + bool is_first_execution= thd->is_first_query_execution(); > + if (is_first_execution || > + thd->stmt_arena->is_stmt_prepare()) The above is the same as: ((stmt_arena->is_conventional() || stmt_arena->state != STMT_EXECUTED) && !stmt_arena->is_stmt_prepare()) || stmt_arena->is_stmt_prepare()) => ((stmt_arena->state == STMT_CONVENTIONAL_EXECUTION || stmt_arena->state != STMT_EXECUTED) && stmt_arena->state != STMT_INITIALIZED) || stmt_arena->state == STMT_INITIALIZED)) => (stmt_arena->state != Query_arena::STMT_EXECUTED && stmt_arena->state != Query_arena::STMT_INITIALIZED) || stmt_arena->state == Query_arena::STMT_INITIALIZED => stmt_arena->state != Query_arena::STMT_EXECUTED Is it not better to use the above as it is shorter and easier to understand as there are less hidden abstractions? Is STMT_EXECUTED safe to use? We don't set state to STMT_EXECUTE in case of an error during execute. If the error is temporary (depending on data or out of space during execution) then state will be left in STMT_INITIALIZED. Don't we have a problem in this case for the next execution? In 11.5 we can trivially simulate an error in almost any query by setting the tmp disk quota to a very low value. Could be fixed by adding a new state STMT_OPTIMIZED that tells us that all optimizations rewrites has been done and then we could check for STMT_OPTIMIZED || STMT_EXECUTED or force a re-prepare if the previous execution returned an error (may be can set state to STMT_ERROR in this case)? Another options is for a new prepare in the case the last query ended with an error. > + /* > + If an outer field is resolved in a grouping select then it > + is replaced for an Item_outer_ref object. Otherwise an > + Item_field object is used. > + The new Item_outer_ref object is saved in the inner_refs_list of > + the outer select. Here it is only created. It can be fixed only > + after the original field has been fixed and this is done in the > + fix_inner_refs() function. > + */ > + if (!(rf= new (thd->mem_root) Item_outer_ref(thd, context, this))) > + rc= -1; > + select->inner_refs_list.push_back(rf, thd->mem_root); > + if (is_first_execution) > + *reference= rf; > + else > + thd->change_item_tree(reference, rf); > + if (arena) > + thd->restore_active_arena(arena, &backup); > + if (rc) > + return rc; > + rf->in_sum_func= thd->lex->in_sum_func; > + } > } I don't like that we use 'rf' when we know it is 0. We can fix that with the following code change that also removes the 'rc' variable (shorter): { Query_arena *arena=0, backup; Item_outer_ref *rf; bool is_first_execution; if ((is_first_execution= thd->is_first_query_execution()) && !thd->is_conventional()) arena= thd->activate_stmt_arena_if_needed(&backup); /* If an outer field is resolved in a grouping select then it is replaced for an Item_outer_ref object. Otherwise an Item_field object is used. The new Item_outer_ref object is saved in the inner_refs_list of the outer select. Here it is only created. It can be fixed only after the original field has been fixed and this is done in the fix_inner_refs() function. */ if (!(rf= new (thd->mem_root) Item_outer_ref(thd, context, this))) { select->inner_refs_list.push_back(rf, thd->mem_root); if (is_first_execution) *reference= rf; else thd->change_item_tree(reference, rf); } if (arena) thd->restore_active_arena(arena, &backup); if (!rf) // Memory allocation failed return -1 rf->in_sum_func= thd->lex->in_sum_func; } Now code has exactly the same amount of jump instructions as your code, but it is shorter, avoids the usage of 'ref' and does not need 'rc'. Note that in comparing to the original code, we are creating an arena for the states: STMT_INITIALIZED, STMT_INITIALIZED_FOR_SP= 1, STMT_PREPARED= 2, STMT_ERROR= -1 This is probably ok, just wanted to ensure you are aware of this. Would it be enough/safer to just test for SMT_PREPARED to see if an arena should be used? (Just wondering) > /* > A reference is resolved to a nest level that's outer or the same as > @@ -5938,7 +5968,7 @@ Item_field::fix_outer_field(THD *thd, Field **from_field, Item **reference) > else if (ref != not_found_item) > { > Item *save; > - Item_ref *rf; > + Item_ref *rf= NULL;; remove extra ; > /* Should have been checked in resolve_ref_in_select_and_group(). */ > DBUG_ASSERT(*ref && (*ref)->is_fixed()); > @@ -5950,35 +5980,64 @@ Item_field::fix_outer_field(THD *thd, Field **from_field, Item **reference) <cut> > + This Item_field represents a reference to a column in some outer select. > + Wrap this Item_field item into an object of the Item_ref class or a class > + derived from Item_ref we are at the first execution of the query or at > + processing the prepare command for it. Make this wrapping permanent for > + the first query execution so that it could be used for next executions > + of the query. The type of the wrapper depends on the place where this > + item field occurs or on type of the query. If it is in the the having clause > + we use a wrapper of the Item_ref type. Otherwise we use a wrapper either > + of Item_direct_ref type or of Item_outer_ref. The latter is used for > + grouping queries. Such a wrapper is always added to the list inner_refs_list > + for the select against which the outer reference has been resolved. > */ > + bool is_first_execution= thd->is_first_query_execution(); > + if (is_first_execution || > + thd->stmt_arena->is_stmt_prepare()) > + { Replace with if (thd->stmt_arena->state != Query_arena::STMT_EXECUTED) { bool is_first_execution= thd->is_first_query_execution(); > + int rc= 0; > + Query_arena *arena, backup; > + arena= 0; > + if (is_first_execution) > + arena= thd->activate_stmt_arena_if_needed(&backup); > + rf= (place == IN_HAVING ? > + new (thd->mem_root) > + Item_ref(thd, context, ref, table_name, > + field_name, alias_name_used) : > + (!select->group_list.elements ? > + new (thd->mem_root) > + Item_direct_ref(thd, context, ref, table_name, > + field_name, alias_name_used) : > + new (thd->mem_root) > + Item_outer_ref(thd, context, ref, table_name, > + field_name, alias_name_used))); > + *ref= save; > + if (!rf) > + rc= -1; > + if (!rc && place != IN_HAVING && select->group_list.elements) > + { > + outer_context->select_lex->inner_refs_list.push_back( > + (Item_outer_ref*)rf, thd->mem_root); > + ((Item_outer_ref*)rf)->in_sum_func= thd->lex->in_sum_func; > + } > + if (is_first_execution) > + *reference= rf; > + else > + thd->change_item_tree(reference, rf); > + if (arena) > + thd->restore_active_arena(arena, &backup); > + if (rc) > + return rc; Pleace do same change as in previous case by using an 'if' instead of setting rc. > We can not "move" aggregate function in the place where > @@ -6003,22 +6062,46 @@ Item_field::fix_outer_field(THD *thd, Field **from_field, Item **reference) > this, (Item_ident*)*reference, false); > if (last_checked_context->select_lex->having_fix_field) > { > + This Item_field represents a reference to a column in having clause > + of some outer select. Wrap this Item_field item into an Item_ref object > + if we are at the first execution of the query or at processing the > + prepare command for it. Make this wrapping permanent for the first query > + execution so that it could be used for next executions of the query. > */ > - DBUG_ASSERT(!rf->fixed); // Assured by Item_ref() > - if (rf->fix_fields(thd, reference) || rf->check_cols(1)) > - return -1; > + Item_ref *rf; > + bool is_first_execution= thd->is_first_query_execution(); > + if (is_first_execution || > + thd->stmt_arena->is_stmt_prepare()) Fix test, as in other cases. > + { > + int rc= 0; > + Query_arena *arena, backup; > + arena= 0; > + if (is_first_execution) > + arena= thd->activate_stmt_arena_if_needed(&backup); > + rf= new (thd->mem_root) Item_ref(thd, context, > + (*from_field)->table->s->db, > + Lex_cstring_strlen((*from_field)-> > + table->alias.c_ptr()), > + field_name); > + if (!rf) > + rc= -1; > + if (is_first_execution) > + *reference= rf; > + else > + thd->change_item_tree(reference, rf); > + if (arena) > + thd->restore_active_arena(arena, &backup); > + if (rc) > + return rc; > + /* > + rf is Item_ref => never substitute other items (in this case) > + during fix_fields() => we can use rf after fix_fields() > + */ > + DBUG_ASSERT(!rf->fixed); // Assured by Item_ref() > + if (rf->fix_fields(thd, reference) || rf->check_cols(1)) > + return -1; > + } same here. Change if (!rf) rc= -1; ... if (arena) thd->restore_active_arena(arena, &backup); to if (rc) { .... } if (arena) thd->restore_active_arena(arena, &backup); > @@ -8321,6 +8404,9 @@ bool Item_ref::fix_fields(THD *thd, Item **reference) > goto error; > } > + if ((*ref)->fix_fields_if_needed(thd, reference)) > + goto error; > + > set_properties(); Add a comment when this is needed (ie, when reference <> 0 and it is not fixed. I tried to come up with a simple comment but failed :( I assume this is at least true for the prepare phase of a prepared statement. > > if ((*ref)->check_cols(1)) > @@ -9282,8 +9368,21 @@ bool Item_direct_view_ref::fix_fields(THD *thd, Item **reference) > bitmap_set_bit(fld->table->read_set, fld->field_index); > } > } > - else if ((*ref)->fix_fields_if_needed(thd, ref)) > - return TRUE; > + else > + { > + if (thd->is_noninitial_query_execution()) replace (thd->is_noninitial_query_execution()) with thd->is_stmt_executed() See later comments in definition of is_noninitial_query_execution() > + { > + bool rc= false; > + bool save_wrapper= thd->lex->current_select->no_wrap_view_item; > + thd->lex->current_select->no_wrap_view_item= TRUE; > + rc= (*ref)->fix_fields_if_needed(thd, ref); > + thd->lex->current_select->no_wrap_view_item= save_wrapper; > + if (rc) > + return TRUE; > + } > + else if ((*ref)->fix_fields_if_needed(thd, ref)) > + return TRUE; > + } Do we need no_wrap_view_item anymore (we probably do, but it is not obvious)? The original code that introduced no_wrap_view_item used it to test things in sql_base.cc but that code does not exist anymore. The only usage we have of it now is in item.cc: if (!thd->lex->current_select->no_wrap_view_item && thd->lex->in_sum_func && thd->lex->in_sum_func->nest_level == select->nest_level) set_if_bigger(thd->lex->in_sum_func->max_arg_level, select->nest_level); There is no code comment what the above code is supposed to do. Could you PLEASE add a comment about what is the purpose o the above code, especially why the nest_level is not applicable to no_wrap_view_items ? I have verified that we need !thd->lex->current_select->no_wrap_view_item here. We get crashes in main.derived_view if remove the test. > diff --git a/sql/item_cmpfunc.cc b/sql/item_cmpfunc.cc > index f8959cfd2bf..cc9c81608ca 100644 > --- a/sql/item_cmpfunc.cc > +++ b/sql/item_cmpfunc.cc > @@ -4588,7 +4588,9 @@ bool Item_func_in::value_list_convert_const_to_int(THD *thd) > */ > if (arg[0]->type() != Item::NULL_ITEM && > !convert_const_to_int(thd, field_item, &arg[0])) > - all_converted= false; > + all_converted= false; > + else > + block_transform_into_subq= true; > } if (all_converted) m_comparator.set_handler(&type_handler_slonglong); The above looks strange. You are setting block_transform_into_subq to true in the case of IN (1,1) but not setting it for ("abb",NULL); I assume the idea is that if if convert_const_to_int() changes anything, then we block conversion of the IN to sub queries as our code cannot convert back to use the original code in this case? (as the translation to use subqueries is permanent). Here is a test cases the sets block_transform_into_subq to true: create table t1 (a bigint); insert into t1 select seq from seq_1_to_100; select benchmark(1,1); select * from t1 where a in ('1','2','3'); drop table t1; Is there not a better way to do this ? - Make all translations of strings to integers permantently in the statement arena - Keep the derived table around for next execution (which would be a nice speedup). If nothing can be done to improve things, we should ae least move setting block_transform_into_subq to the end of the function and fix that we only block the transformation for prepared statements: > if (all_converted) > m_comparator.set_handler(&type_handler_slonglong); -> if (all_converted) { m_comparator.set_handler(&type_handler_slonglong); if (thd->is_first_query_execution()) block_transform_into_subq= 1; } ---- > diff --git a/sql/item_cmpfunc.h b/sql/item_cmpfunc.h > index 465625f69bf..1404fa30d6b 100644 > --- a/sql/item_cmpfunc.h > +++ b/sql/item_cmpfunc.h > @@ -2401,6 +2401,7 @@ class Item_func_in :public Item_func_opt_neg, > { > return check_argument_types_like_args0(); > } > + bool block_transform_into_subq; Please add a comment about the purpose of this variable, when it should be used and how it is used. As far as I can see, this set in the case where an IN() requires a item conversion in which case we have to block it for prepared statements as we cannot roll it back. > diff --git a/sql/item_subselect.cc b/sql/item_subselect.cc > index ff84e6b7bf9..4dcd1f3ae9d 100644 > --- a/sql/item_subselect.cc > +++ b/sql/item_subselect.cc > @@ -104,6 +104,16 @@ void Item_subselect::init(st_select_lex *select_lex, > unit->item= this; > engine->change_result(this, result, FALSE); > } > + else if ((unit->item->substype() == ALL_SUBS || > + unit->item->substype() == ANY_SUBS) && > + (((Item_in_subselect*) unit->item)-> > + test_set_strategy(SUBS_MAXMIN_INJECTED) || > + ((Item_in_subselect*) unit->item)-> > + test_set_strategy(SUBS_MAXMIN_ENGINE))) > + { > + unit->item= this; > + engine->change_result(this, result, FALSE); > + } > else > { > /* Better to combine the test with the previous test as the code inside the {} is identical? Note that I understand the above code to 100%, so adding a comment what the purpose of this code would be great. Also change the function 'test_set_strategy()' to 'test_choosen_strategy(SUBS_MAXMIN_INJECTED | SUBS_MAXMIN_INJECTED)' This will make the code shorter and easier to read. Later in the review there is a new definition for test_choosen_strategy(). This is very similar to test_strategy(), but also ensure that the strategy is chosen. > @@ -200,15 +211,6 @@ void Item_in_subselect::cleanup() > > void Item_allany_subselect::cleanup() > { > - /* > - The MAX/MIN transformation through injection is reverted through the > - change_item_tree() mechanism. Revert the select_lex object of the > - query to its initial state. > - */ > - for (SELECT_LEX *sl= unit->first_select(); > - sl; sl= sl->next_select()) > - if (test_set_strategy(SUBS_MAXMIN_INJECTED)) > - sl->with_sum_func= false; > Item_in_subselect::cleanup(); > } I assume the above is now removed becasue MAX/MIN optimization changes are now permanent in Item_allany_subselect::transform_into_max_min(JOIN *join). Please add an explanation for the above change to the commit message and also add a note that thanks to this change we don't need to cleanup for MAX/MIN optimization. > @@ -497,6 +505,9 @@ void Item_subselect::recalc_used_tables(st_select_lex *new_parent, > Ref_to_outside *upper; > DBUG_ENTER("recalc_used_tables"); > > + if (!unit->thd->is_first_query_execution()) > + DBUG_VOID_RETURN; With prepared statements, the optimizer plan can change for each query. How come we don't need to recalc_used_tables() just because we have done one execution before? > @@ -2143,11 +2155,11 @@ bool Item_allany_subselect::transform_into_max_min(JOIN *join) > } > if (upper_item) > upper_item->set_sum_test(item); > - thd->change_item_tree(&select_lex->ref_pointer_array[0], item); > + select_lex->ref_pointer_array[0]= item; > { > List_iterator<Item> it(select_lex->item_list); > it++; > - thd->change_item_tree(it.ref(), item); > + it.replace(item); > } Looks like we are now are doing the MAX/MIN transformation permanently. > +uint st_select_lex_unit::ub_eq_for_exists2_in() > +{ > + if (derived && derived->is_materialized_derived()) > + return 0; > + st_select_lex *sl= first_select(); > + if (sl->next_select()) > + return 0; > + uint n= sl->select_n_eq; > + for (st_select_lex_unit *unit= sl->first_inner_unit(); > + unit; > + unit= unit->next_unit()) > + { > + n+= unit->ub_eq_for_exists2_in(); > + } > + return n; > +} Please use a more clear function name. 'ub' is not clear. Please also add a function comment to explain the purpose if this function as I do not understand the purpose of it. The function is is used here: bool Item_exists_subselect::fix_fields(THD *thd, Item **ref) { DBUG_ENTER("Item_exists_subselect::fix_fields"); st_select_lex *sel= unit->first_select(); sel->select_n_reserved= unit->ub_eq_for_exists2_in(); But it's purpose is not that clear or what it trying to calculate. For example, if you have a query: SELECT (a=1) + (b=2) + (c=3) select_n_eq will be 3. In case of SELECT (a = 1 or b = 1) or (a=2 or b=4) select_n_eq will be 4 I don't see how either value can be of any use for the optimizer. Why not instead calculate the number of = operations in the top level Item_comp_and() ? You could have the counter in Item_comp_and of how many '=' comparisons there are inside it. This would give you an exact number of = that may be needed, compared to the current code. Note find_inner_outer_equalities() is only appending items from the top and level or alternatively 0 or 1 element for other items. > +bool Item_singlerow_subselect::test_set_strategy(uchar strategy_arg) > +{ > + DBUG_ASSERT(strategy_arg == SUBS_MAXMIN_INJECTED || > + strategy_arg == SUBS_MAXMIN_ENGINE); > + return ((strategy & SUBS_STRATEGY_CHOSEN) && > + (strategy & ~SUBS_STRATEGY_CHOSEN) == strategy_arg); > +} Change to the following so that we can check all bits at once, like in test_strategy(). bool Item_singlerow_subselect::test_chosen_strategy(uint strategy_arg) { DBUG_ASSERT(strategy_arg && !(strategy_arg & ~(SUBS_MAXMIN_INJECTED | SUBS_MAXMIN_ENGINE))); return ((strategy & SUBS_STRATEGY_CHOSEN) && (strategy & strategy_arg); } > diff --git a/sql/item_subselect.h b/sql/item_subselect.h > index d3572e1abe7..9e318e74bab 100644 > --- a/sql/item_subselect.h > +++ b/sql/item_subselect.h > @@ -302,6 +302,7 @@ class Item_singlerow_subselect :public Item_subselect > { > protected: > Item_cache *value, **row; > + uchar strategy; Change to uint16 to ensure that we can add more bits later without having to change a lot of code (thanks to alignment, the storage required does not change if we use uint16) > diff --git a/sql/item_sum.cc b/sql/item_sum.cc > index 4cf403c1618..e1a32185ad1 100644 > --- a/sql/item_sum.cc > +++ b/sql/item_sum.cc > @@ -93,10 +93,15 @@ bool Item_sum::init_sum_func_check(THD *thd) > thd->lex->in_sum_func= this; > nest_level= thd->lex->current_select->nest_level; > ref_by= 0; > + if (!thd->is_noninitial_query_execution() || > + (thd->active_stmt_arena_to_use()->state == > + Query_arena::STMT_SP_QUERY_ARGUMENTS)) > + { > aggr_level= -1; > aggr_sel= NULL; > max_arg_level= -1; > max_sum_func_level= -1; > + } Please fix the indentation. Please add a comment for the above test. Here is a suggestion: /* Item_sum objects are persistent and does not have to be updated for pre-executed prepared statements (stmt_arena->state == EXECUTED) except when setting variables for stored procedures (state == Query_arena::STMT_SP_QUERY_ARGUMENTS)) */ > outer_fields.empty(); > return FALSE; > } > @@ -409,7 +414,7 @@ bool Item_sum::register_sum_func(THD *thd, Item **ref) > sl= sl->master_unit()->outer_select() ) > sl->master_unit()->item->get_with_sum_func_cache()->set_with_sum_func(); > } > - if (aggr_sel) > + if (aggr_sel && aggr_sel != thd->lex->current_select ) > thd->lex->current_select->mark_as_dependent(thd, aggr_sel, NULL); Remove extra space before && and before ) Please add comment under which circumstances this is true: aggr_sel != thd->lex->current_select I would like to understand why this was not needed before. > +++ b/sql/opt_subselect.cc > @@ -962,8 +962,13 @@ bool JOIN::transform_max_min_subquery() > if (!subselect || (subselect->substype() != Item_subselect::ALL_SUBS && > subselect->substype() != Item_subselect::ANY_SUBS)) > DBUG_RETURN(0); > - DBUG_RETURN(((Item_allany_subselect *) subselect)-> > - transform_into_max_min(this)); > + bool rc= false; > + Query_arena *arena, backup; > + arena= thd->activate_stmt_arena_if_needed(&backup); > + rc= (((Item_allany_subselect *) subselect)->transform_into_max_min(this)); > + if (arena) > + thd->restore_active_arena(arena, &backup); > + DBUG_RETURN(rc); > } Please update the comment for this function to make it clear that the changes to use min/max are now permanent. > > @@ -1889,7 +1894,7 @@ static bool convert_subq_to_sj(JOIN *parent_join, Item_in_subselect *subq_pred) > with thd->change_item_tree > */ > Item_func_eq *item_eq= > - new (thd->mem_root) Item_func_eq(thd, left_exp_orig, > + new (thd->mem_root) Item_func_eq(thd, left_exp, > subq_lex->ref_pointer_array[0]); > if (!item_eq) > goto restore_tl_and_exit; I assume this was a bug in the old code? I checked the history and this line has changed back and between left_exp_orig and left_exp over time. The documentation for left_expr_orig is: /* Important for PS/SP: left_expr_orig is the item that left_expr originally pointed at. That item is allocated on the statement arena, while left_expr could later be changed to something on the execution arena. */ Item *left_expr_orig; As an experiment I added left_exp_orig back and did run the test that you had changed in your commit with valgrind. All test passed. Do you have any test that shows that left_exp is the right value here or do you have any other proof why left_exp is right ? > @@ -6535,6 +6540,10 @@ bool JOIN::choose_subquery_plan(table_map join_tables) > if (is_in_subquery()) > { > in_subs= unit->item->get_IN_subquery(); > + if (in_subs->test_set_strategy(SUBS_MAXMIN_INJECTED)) > + return false; > + if (in_subs->test_set_strategy(SUBS_MAXMIN_ENGINE)) > + return false; > if (in_subs->create_in_to_exists_cond(this)) > return true; You can replace the above two changed lines with: if (in_subs->test_chosen_strategy(SUBS_MAXMIN_INJECTED | SUBS_MAXMIN_INJECTED)) > diff --git a/sql/sql_array.h b/sql/sql_array.h > index b6de1b18d78..b9fdef0623c 100644 > --- a/sql/sql_array.h > +++ b/sql/sql_array.h > @@ -39,7 +39,10 @@ template <typename Element_type> class Bounds_checked_array > > Bounds_checked_array(Element_type *el, size_t size_arg) > : m_array(el), m_size(size_arg) > - {} > + { > + if (!m_size) > + m_array= NULL; > + } Strange that one would call it with m_size() with el != 0 and m_size == 0. Sounds like a bug in the caller. I would change the above to be a DBUG_ASSERT instead. > void reset() { m_array= NULL; m_size= 0; } > > diff --git a/sql/sql_base.cc b/sql/sql_base.cc > index f1fd0053a82..a6cdd218c44 100644 > --- a/sql/sql_base.cc > +++ b/sql/sql_base.cc > @@ -5783,10 +5783,7 @@ find_field_in_view(THD *thd, TABLE_LIST *table_list, > { > if (!my_strcasecmp(system_charset_info, field_it.name()->str, name)) > { > - // in PS use own arena or data will be freed after prepare > - if (register_tree_change && > - thd->stmt_arena->is_stmt_prepare_or_first_stmt_execute()) > - arena= thd->activate_stmt_arena_if_needed(&backup); > + arena= thd->activate_stmt_arena_if_needed(&backup); Why this change? Will it not use statement area for a lot of cases where it did not before? like for state == STMT_EXECUTED ? - This was discussed on slack, but we did not have time to reach a conclusion. > @@ -5805,11 +5802,23 @@ find_field_in_view(THD *thd, TABLE_LIST *table_list, > the replacing item. > */ > if (*ref && !(*ref)->is_autogenerated_name()) > + { > + arena=0; > + if (thd->is_first_query_execution()) > + arena= thd->activate_stmt_arena_if_needed(&backup); > item->set_name(thd, (*ref)->name); > + if (arena) > + thd->restore_active_arena(arena, &backup); Item::set_name() does never allocate any memory, so we don't need arena here. (I checked all code on the sql level). > + } > + if (item != *ref) > + { > + if (register_tree_change && > + !thd->is_first_query_execution()) The above is same as: register_tree_change && !(stmp_arena->state != STMT_INITIALIZED && stmt_arena->state != Query_arena::STMT_EXECUTED) => register_tree_change && (stmp_arena->state == STMT_INITIALIZED || stmt_arena->state == STMT_EXECUTED)) Is the above correct and the intention of this cde? > + thd->change_item_tree(ref, item); > + else > + *ref= item; > + } You can remove the else before *ref. It is faster to update *ref than handle the jump. I am still a bit concerned that we set *ref=item when register_tree_change != 0 as our code comments says that if register_tree_change is set, then *ref will be removed before next execution and any tries to acces it can result in a core dump. > @@ -7639,10 +7648,14 @@ bool setup_fields(THD *thd, Ref_ptr_array ref_pointer_array, > TODO: remove it when (if) we made one list for allfields and > ref_pointer_array > */ > - if (!ref_pointer_array.is_null()) > + if (thd->is_first_query_execution() || > + thd->stmt_arena->is_stmt_prepare()) The new test is same as (see earlier examples): thd->stmt_arena->state != Query_arena::STMT_EXECUTED Please replace the test with the above or replace it with is_stmt_exceuted() <cut> > /* > @@ -7682,17 +7695,6 @@ bool setup_fields(THD *thd, Ref_ptr_array ref_pointer_array, > ref[0]= item; > ref.pop_front(); > } > - /* > - split_sum_func() must be called for Window Function items, see > - Item_window_func::split_sum_func. > - */ > - if (sum_func_list && > - ((item->with_sum_func() && item->type() != Item::SUM_FUNC_ITEM) || > - item->with_window_func)) > - { > - item->split_sum_func(thd, ref_pointer_array, *sum_func_list, > - SPLIT_SUM_SELECT); > - } I assume this is moved part of this from setup_fields to JOIN::prepare(). Any particular reason why this move was required ? Just curious. Still, it would be good to explain this in the commit message. (One reason is that it will allow us to remove sum_func_list from the arguments to setup_fields) I don't see any big problem with the code as it is JOIN::prepare() that is calling both setup_fields() and split_sum_func(). It will be a bit slower as we have to traverse the field_list twice, but that is not a high cost. Note that if we go with the above code change, you should also remove sum_func_list from the arguments as the above code was the only place it was used. Also, please add back the comment to the moved code: /* split_sum_func() must be called for Window Function items, see Item_window_func::split_sum_func. */ > diff --git a/sql/sql_class.cc b/sql/sql_class.cc > index 7913f1a40d2..19f1ec9728e 100644 > --- a/sql/sql_class.cc > +++ b/sql/sql_class.cc > @@ -3959,12 +3962,16 @@ void Query_arena::free_items() > Item *next; > DBUG_ENTER("Query_arena::free_items"); > /* This works because items are allocated on THD::mem_root */ > + for (next= free_list; next; next= next->next) > + { > + next->cleanup(); > + } Why not call cleanup in the following loop instead of looping over the elements twice (as we did before)? Is there some cases when we cannot do cleanup as part of delete_self()? If yes, please add comment in the code why we have to traverse the list twice as otherwise it is very likely some developer will try to optimize the first loop away while looking at the code. If this is the case, is this not a problem also for free_items() ? > diff --git a/sql/sql_class.h b/sql/sql_class.h > index 7e5d9ac96e3..2f05b2dbd03 100644 > --- a/sql/sql_class.h > +++ b/sql/sql_class.h > @@ -2569,6 +2569,20 @@ class THD: public THD_count, /* this must be first */ > return static_cast<PSI_thread*>(my_atomic_loadptr((void*volatile*)&m_psi)); > } > > + inline bool is_first_query_execution() > + { > + return (stmt_arena->is_conventional() || > + stmt_arena->state != Query_arena::STMT_EXECUTED) && > + !stmt_arena->is_stmt_prepare(); (stmt_arena->is_conventional() || stmt_arena->state != Query_arena::STMT_EXECUTED) && !stmt_arena->is_stmt_prepare() => (stmt_arena->state == STMT_CONVENTIONAL_EXECUTION || stmt_arena->state != STMT_EXECUTED) && stmp_arena->state != STMT_INITIALIZED) => stmt_arena->state != STMT_INITIALIZED && stmt_arena->state != STMT_EXECUTED Simpler and easier to understand. > + } > + > + inline bool is_noninitial_query_execution() > + { > + return !stmt_arena->is_conventional() && > + stmt_arena->state == Query_arena::STMT_EXECUTED && > + !stmt_arena->is_stmt_prepare() ; Please add comment to explain this function and when to use it. (name of is_first_query_execution() is clear, but this one is not) Anyway: !stmt_arena->is_conventional() && stmt_arena->state == Query_arena::STMT_EXECUTED && !stmt_arena->is_stmt_prepare() ; => stmt_arena->state != STMT_CONVENTIONAL_EXECUTION && stmt_arena->state == STMT_EXECUTED && stmt_arena->state != STMT_INITIALIZED => stmt_arena->state == STMT_EXECUTED Better to renamed it to is_stmt_executed() and move it to where we define is_stmt_prepare() in class THD and class Query_arena > diff --git a/sql/sql_lex.cc b/sql/sql_lex.cc > index b669925a263..d7b7e98a6e4 100644 > --- a/sql/sql_lex.cc > +++ b/sql/sql_lex.cc > @@ -3601,6 +3607,8 @@ ulong st_select_lex::get_table_join_options() > > uint st_select_lex::get_cardinality_of_ref_ptrs_slice(uint order_group_num_arg) > { > + if (card_of_ref_ptrs_slice) > + return card_of_ref_ptrs_slice; I assume this is just a cache and the code should work also without it? If yes, please add a comment about that. > if (!((options & SELECT_DISTINCT) && !group_list.elements)) > hidden_bit_fields= 0; > > @@ -3620,20 +3628,23 @@ uint st_select_lex::get_cardinality_of_ref_ptrs_slice(uint order_group_num_arg) > order_group_num * 2 + > hidden_bit_fields + > fields_in_window_functions; > + card_of_ref_ptrs_slice= n; > return n; > } Does caching card_of_ref_ptrs_slice really help ? (The function is quite trivial) We call the function in one place, setup_ref_array() It is called once per query (For selectr in JOIN::prepare()) I think having the variable and testing and updating clearing it for each normal query will cost us more than not having it. This assumes we have more normal queries than we have prepared statements. I would suggest we remove the variable. > -bool st_select_lex::setup_ref_array(THD *thd, uint order_group_num) > +static > +bool setup_ref_ptrs_array(THD *thd, Ref_ptr_array *ref_ptrs, uint n_elems) > { > + if (!ref_ptrs->is_null()) > + { > + if (ref_ptrs->size() >= n_elems) > + return false; > + } > Item **array= static_cast<Item**>( > thd->active_stmt_arena_to_use()->alloc(sizeof(Item*) * n_elems)); > if (likely(array != NULL)) > + *ref_ptrs= Ref_ptr_array(array, n_elems); > return array == NULL; > } I assume the change is that the new code will alloocate a new array if the old array was not big enough. That is good. Howevever it would be good to know when this can happen. For ref_pointer_array this should be impossible as we cache the value and it cannot change for two iterations. Is this reallocate needed for the save_ref_ptrs array? Please add a comment to explain in which case the old array was not big enough It would also be good to have the code for setup_ref_ptr_array, setup_ref_array and save_ref_ptrs_after_persistent_rewrites and save_ref_ptrs_if_needed close to each other as these are very similar and very related. > + > +bool st_select_lex::setup_ref_array(THD *thd, uint order_group_num) > +{ > + /* > + We have to create array in prepared statement memory if it is a > + prepared statement > + */ > + uint slice_card= !thd->is_noninitial_query_execution() ? > + get_cardinality_of_ref_ptrs_slice(order_group_num) : > + card_of_ref_ptrs_slice; > + uint n_elems= slice_card * 5; > + DBUG_ASSERT(n_elems % 5 == 0); You can remove the DBUG_ASSERT. It is obvious that this is always correct (for this code). > + return setup_ref_ptrs_array(thd, &ref_pointer_array, n_elems); > +} With the cache, why don't have to test for !thd->is_noninitial_query_execution(). This should be good enough: uint slice_card= get_cardinality_of_ref_ptrs_slice(order_group_num); With the cache, it will return at once after the first call. As the cache is updated on first run, the answer should always be the same is in your original code. > +bool st_select_lex::save_ref_ptrs_after_persistent_rewrites(THD *thd) > +{ > + uint n_elems= card_of_ref_ptrs_slice; > + if (setup_ref_ptrs_array(thd, &save_ref_ptrs, n_elems)) > + return true; > + if (!ref_pointer_array.is_null()) > + join->copy_ref_ptr_array(save_ref_ptrs, join->ref_ptr_array_slice(0)); > + return false; > +} Please add to function start: /* Ensure that setup_ref_array has been called */ DBUG_ASSERT(card_of_ref_ptrs_slice); It looks like all calls to setup_ref_ptrs_array() are using the same value for all future calls. If this is the case, why did you change setup_ref_ptrs_array() to check if number of elements has grown? If this is not needed, remove the check and add an assert() to ensure that the elements does not grow over time. > +bool st_select_lex::save_ref_ptrs_if_needed(THD *thd) > +{ > + if (thd->is_first_query_execution()) > + { > + if (save_ref_ptrs_after_persistent_rewrites(thd)) > + return true; > + } > + return false; > +} The above if is true in case of stmt_arena->state != STMT_INITIALIZED && stmt_arena->state != STMT_EXECUTED We call save_ref_prts_if_needed in JOIN::prepare() and subselect_single_select_engine::prepare(), mysql_select() and st_select_lex_unit::prepare_join I assume this is ok. However, I did find some inconsistently in how this functioin is called. JOIN::prepare() calls it on errors before returning (for most cases but not all). subselect_single_select_engine::prepare() & mysql_select() calls it if join->prepare() returns error, but not otherwise. st_select_lex_unit::prepare_join() calls it always after prepare. I would like to know when we MUST call save_ref_prts_if_needed(). Do we have to call it in case JOIN::prepare returns 0 ? It would be easier if things would be done this way: - JOIN::prepare() would always call it in case of error if there is any need to call it. - We would not have to call it in subselect_single_select_engine::prepare() or mysql_select() - We would remove the call in st_select_lex_unit::prepare_join() or alternative call it only if join->prepare() returns 0. (Depending on what is correct here) > void st_select_lex_unit::print(String *str, enum_query_type query_type) > { > if (with_clause) > @@ -4958,7 +5007,7 @@ bool st_select_lex::optimize_unflattened_subqueries(bool const_only) > } > if ((res= inner_join->optimize())) > return TRUE; > - if (!inner_join->cleaned) > + if (inner_join->thd->is_first_query_execution()) > sl->update_used_tables(); > sl->update_correlated_cache(); > is_correlated_unit|= sl->is_correlated; Fix intendation. Why is cleaned not good enough here? cleaned means that the join structure cannot be used anymore. Why not do: if (!inner_join->cleaned && inner_join->thd->is_first_query_execution()) Should be safer. > diff --git a/sql/sql_lex.h b/sql/sql_lex.h > index 3ab50d4aaa8..59029057a23 100644 > --- a/sql/sql_lex.h > +++ b/sql/sql_lex.h > @@ -1045,6 +1045,8 @@ class st_select_lex_unit: public st_select_lex_node { > > bool can_be_merged(); > > + uint ub_eq_for_exists2_in(); > + > friend class st_select_lex; > }; > > @@ -1176,6 +1178,7 @@ class st_select_lex: public st_select_lex_node > uint curr_tvc_name; > /* true <=> select has been created a TVC wrapper */ > bool is_tvc_wrapper; > + uint fields_added_by_fix_inner_refs; You can remove the above variable as it is not used anywhere. It is set at: sql_select.cc:1499: select_lex->fields_added_by_fix_inner_refs= sql_lex.cc:3119: fields_added_by_fix_inner_refs= 0; but not used anywhere. > /* > Needed to correctly generate 'PRIMARY' or 'SIMPLE' for select_type column > of EXPLAIN > @@ -1201,6 +1204,8 @@ class st_select_lex: public st_select_lex_node > > /// Array of pointers to top elements of all_fields list > Ref_ptr_array ref_pointer_array; > + Ref_ptr_array save_ref_ptrs; > + uint card_of_ref_ptrs_slice; Please add uint variables together with other uint variables. Otherwise we may loose 4 byte for each variable because of alignment. > > /* > number of items in select_list and HAVING clause used to get number > @@ -1220,6 +1225,7 @@ class st_select_lex: public st_select_lex_node > uint order_group_num; > /* reserved for exists 2 in */ > uint select_n_reserved; > + uint select_n_eq; Please add a descripton for the above variable. (Name does not tell me what it is used for). > /* > it counts the number of bit fields in the SELECT list. These are used when DISTINCT is > converted to a GROUP BY involving BIT fields. > @@ -1295,6 +1301,8 @@ class st_select_lex: public st_select_lex_node > case of an error during prepare the PS is not created. > */ > uint8 changed_elements; // see TOUCHED_SEL_* > + uint8 save_uncacheable; > + uint8 save_master_uncacheable; The above two variables can be removed as these are only assigned to but never used for anything. > /* TODO: add foloowing first_* to bitmap above */ > bool first_natural_join_processing; > bool first_cond_optimization; > diff --git a/sql/sql_parse.cc b/sql/sql_parse.cc > index 5db53b6f2b8..83569207a0c 100644 > --- a/sql/sql_parse.cc > +++ b/sql/sql_parse.cc > @@ -4956,11 +4956,24 @@ mysql_execute_command(THD *thd) > if ((res= multi_delete_precheck(thd, all_tables))) > break; > > + if (thd->stmt_arena->is_stmt_prepare()) > + { > + if (add_item_to_list(thd, new (thd->mem_root) Item_null(thd))) > + goto error; > + } > + else if (thd->is_first_query_execution()) > + { The above tests for stmt_arena->state != STMT_INITIALIZED && stmt_arena->state != STMT_EXECUTED While .. != STMT_EXECUTED would be enough as it is tested on if above. No big deal, just and observarion > + if (select_lex->item_list.elements != 0) > + select_lex->item_list.empty(); > + bool rc= false; You don't have to set it to false here. > + Query_arena *arena= 0, backup; > + arena= thd->activate_stmt_arena_if_needed(&backup); > + rc= add_item_to_list(thd, new (thd->stmt_arena->mem_root) Item_null(thd)); > + if (arena) > + thd->restore_active_arena(arena, &backup); > + if (rc) > + goto error; > + } > diff --git a/sql/sql_select.cc b/sql/sql_select.cc > index a7b84bbfe3b..27eb466b47c 100644 > --- a/sql/sql_select.cc > +++ b/sql/sql_select.cc > @@ -551,6 +551,19 @@ fix_inner_refs(THD *thd, List<Item> &all_fields, SELECT_LEX *select, > the found_in_group_by field of the references from the list. > */ > List_iterator_fast <Item_outer_ref> ref_it(select->inner_refs_list); > + > + if (thd->is_noninitial_query_execution()) > + { > + while ((ref= ref_it++)) > + { > + if (ref->found_in_select_list) > + continue; > + int el= all_fields.elements; > + all_fields.push_front(ref_pointer_array[el], thd->mem_root); > + } > + return false; > + } Please move the comment before this loop after the loop as it refers to the next loop. Also please add a short comment why the purpose of this loop is. Will this not cause the ref_pointer_array to grow for each prepared statement call (after the first one)? Looking at the other parts of your patch, I thought that the size and content of ref_pointer_array[] will be fixed after the first call. (or saved in save_ref_ptrs). Adding something to ref_pointer_array in case of STMT_EXECUTED sounds like it is going against the above principles. > + bool is_first_execution= thd->is_first_query_execution(); > + if (is_first_execution || > + thd->stmt_arena->is_stmt_prepare()) The above can be replaced with if (thd->stmt_area->state != STMT_EXECUTED) It a bit confusing that we have defined: inline bool is_stmt_execute() const { return state == STMT_PREPARED || state == STMT_EXECUTED; } as it would be been logical to have inline bool is_stmt_execute() const { return state == STMT_EXECUTED; } Adding a function is_stmt_executed() would easily be confused with is_stmt_execute(). Maybe we should rename is_stmt_execute to is_stmt_preared_or_executed() and add is_stmt_executed() ? > @@ -1336,18 +1367,13 @@ JOIN::prepare(TABLE_LIST *tables_init, COND *conds_init, uint og_num, > DBUG_RETURN(-1); > } > > - if (thd->lex->current_select->first_cond_optimization) > - { > - if ( conds && ! thd->lex->current_select->merged_into) > - select_lex->select_n_reserved= conds->exists2in_reserved_items(); > - else > - select_lex->select_n_reserved= 0; > - } I asume this is now what is calcualated in Item_exists_subselect::fix_fields() and not needed here anymore? > @@ -1463,6 +1489,38 @@ JOIN::prepare(TABLE_LIST *tables_init, COND *conds_init, uint og_num, > if (res) > DBUG_RETURN(res); > > + if (thd->is_first_query_execution() || > + thd->lex->is_ps_or_view_context_analysis()) > + { > + if (fix_inner_refs(thd, all_fields, select_lex, ref_ptrs)) > + DBUG_RETURN(-1); Remove the above outside the if as the code is identical also for the else part. > + if (thd->is_first_query_execution() && > + all_fields.elements > select_lex->item_list.elements) > + select_lex->fields_added_by_fix_inner_refs= > + select_lex->inner_refs_list.elements; You can remove the above 4 lines as fields_added_by_fix_inner_refs is not used anywhere. > + } > + else > + { > + if (fix_inner_refs(thd, all_fields, select_lex, ref_ptrs)) > + DBUG_RETURN(-1); > + select_lex->uncacheable= select_lex->save_uncacheable; > + select_lex->master_unit()->uncacheable= select_lex->save_master_uncacheable; you can remove the above as neither variable is used. > + } > + The above means you can replace the above query block with: if (fix_inner_refs(thd, all_fields, select_lex, ref_ptrs)) DBUG_RETURN(-1); > @@ -1624,6 +1681,7 @@ JOIN::prepare(TABLE_LIST *tables_init, COND *conds_init, uint og_num, > err: > delete procedure; /* purecov: inspected */ > procedure= 0; > + select_lex->save_ref_ptrs_if_needed(thd); > DBUG_RETURN(-1); /* purecov: inspected */ > } ok. Please check if there is any other case in JOIN::prepare where we return <> 0 where we should call save_ref_ptrs_if_needed(thd). By the way, I am not sure why we would save_ref_ptrs_if_needed() at all in case of errors. The state will in this case not be set to STMT_EXECUTED and thus the state should not be used by the next queryt. Please add a comment why we need the result in case of an error. > // Update used tables after all handling derived table procedures > - select_lex->update_used_tables(); > + if (select_lex->first_cond_optimization) > + select_lex->update_used_tables(); Why first_cond_optimization() instead of state != STMT_EXECUTED ? > - if (transform_max_min_subquery()) > + if (select_lex->first_cond_optimization && transform_max_min_subquery()) > DBUG_RETURN(1); /* purecov: inspected */ > > if (select_lex->first_cond_optimization) Please combine the above two ifs > @@ -2042,6 +2100,11 @@ JOIN::optimize_inner() <cut> > + select_lex->save_uncacheable= select_lex->uncacheable; > + select_lex->save_master_uncacheable= select_lex->master_unit()->uncacheable; Remove as not used. > @@ -4832,6 +4895,7 @@ mysql_select(THD *thd, TABLE_LIST *tables, List<Item> &fields, COND *conds, > if ((err= join->prepare(tables, conds, og_num, order, false, group, > having, proc_param, select_lex, unit))) > { > + select_lex->save_ref_ptrs_if_needed(thd); Can probably be removed. See previous comment > @@ -4858,6 +4922,7 @@ mysql_select(THD *thd, TABLE_LIST *tables, List<Item> &fields, COND *conds, > if ((err= join->prepare(tables, conds, og_num, order, false, group, having, > proc_param, select_lex, unit))) > { > + select_lex->save_ref_ptrs_if_needed(thd); > goto err; Can probably be removed. See previous comment > @@ -25172,14 +25253,33 @@ find_order_in_list(THD *thd, Ref_ptr_array ref_pointer_array, <cut> > - from_field= find_field_in_tables(thd, (Item_ident*) order_item, tables, > - NULL, &view_ref, IGNORE_ERRORS, FALSE, > - FALSE); > + if (!thd->is_noninitial_query_execution()) > + { > + from_field= find_field_in_tables(thd, (Item_ident*) order_item, tables, > + NULL, &view_ref, IGNORE_ERRORS, FALSE, > + FALSE); > + order->view_ref= view_ref; > + order->from_field= from_field; > + } > + else > + { > + if (order->view_ref) > + { Fix indentation > + view_ref= order->view_ref; > + from_field= order->from_field; > + } > + else > + { Fix indentation > + from_field= find_field_in_tables(thd, (Item_ident*) order_item, > + tables, NULL, &view_ref, > + IGNORE_ERRORS, FALSE, FALSE); > + } > + } > if (!from_field) > from_field= (Field*) not_found_field; > } > @@ -25232,6 +25332,12 @@ find_order_in_list(THD *thd, Ref_ptr_array ref_pointer_array, > if (found_item != not_found_item) > { > order->item= &ref_pointer_array[all_fields.elements-1-counter]; > + if (!thd->is_first_query_execution() && > + !thd->lex->is_ps_or_view_context_analysis()) > + { > + if ((*order->item)->fix_fields_if_needed_for_order_by(thd, order->item)) > + return TRUE; > + } The above code is new. Why do need this when we did not need it before? Please add a comment for the block above. > order->in_field_list= 0; > return FALSE; > } > diff --git a/sql/sql_union.cc b/sql/sql_union.cc > index c3c4198439a..5fb31ccbe46 100644 > --- a/sql/sql_union.cc > +++ b/sql/sql_union.cc > @@ -1114,6 +1114,8 @@ bool st_select_lex_unit::prepare_join(THD *thd_arg, SELECT_LEX *sl, > thd_arg->lex->proc_list.first), > sl, this); > > + sl->save_ref_ptrs_if_needed(thd); You may be able to remove the above, look at other comments. > @@ -2784,6 +2786,8 @@ bool st_select_lex::cleanup() > > if (join) > { > + if (join->thd->stmt_arena->is_stmt_prepare()) > + inner_refs_list.empty(); > List_iterator<TABLE_LIST> ti(leaf_tables); > TABLE_LIST *tbl; > while ((tbl= ti++)) > @@ -2813,7 +2817,6 @@ bool st_select_lex::cleanup() > continue; > error= (bool) ((uint) error | (uint) lex_unit->cleanup()); > } > - inner_refs_list.empty(); > exclude_from_table_unique_test= FALSE; > hidden_bit_fields= 0; > DBUG_RETURN(error); The above patch looks a bit strange. We are deleting the join in this function. Isn't the inner_refs_list depending on the join struct in any way? One effect of the above is that we are resetting inner_refs_list only later when excecuting the next query in st_select_lex::init_select() for anything else than STMT_INITIALIZED. I tried reverting the fix and I did get errors. I assume this is because we now only update inner_refs_list once and permantently during first execution? It would be good to add some kind of comment in the commit message about this change. Why only is_stmt_prepare and not also for STMT_CONVENTIONAL_EXECUTION ? Regards, Monty

1 0

Re: 175adef14ef: MDEV-32537 due to Linux, restrict thread name 15 characters, also in PS.
by Sergei Golubchik 11 Jun '24

11 Jun '24

Hi, Vladislav, On Jun 08, Vladislav Vaintroub wrote: > revision-id: 175adef14ef (mariadb-11.0.1-288-g175adef14ef) > parent(s): 4ffb583d72f > author: Vladislav Vaintroub > committer: Vladislav Vaintroub > timestamp: 2024-01-25 09:57:52 +0100 > message: > > MDEV-32537 due to Linux, restrict thread name 15 characters, also in PS. > > Rename some threads to workaround this restrictions, > e.g "rpl_parallel_thread"->"rpl_parallel", > "slave_background" -> "slave_bg" etc. I know I suggested that, but I'm getting a second thought now, sorry. This would break compatibility with performance schema tools, whatever they might be. I don't know if there are any, though, so may be it's not a concern. Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 2

Re: 4ffb583d72f: MDEV-32537 Validate my_thread_set_name parameter against performance schema names.
by Sergei Golubchik 11 Jun '24

11 Jun '24

Hi, Vladislav, On Jun 08, Vladislav Vaintroub wrote: > revision-id: 4ffb583d72f (mariadb-11.0.1-287-g4ffb583d72f) > parent(s): 7d863d3388d > author: Vladislav Vaintroub > committer: Vladislav Vaintroub > timestamp: 2024-01-20 11:26:21 +0100 > message: > > MDEV-32537 Validate my_thread_set_name parameter against performance schema names. > > In order to be consistent with PSI force thread name to be the same as > last part of thread_class_name e.g thread with PSI name > "thread/sql/main" must be called "main" . Due to Linux restriction on > thread name length ( 15 chars), full name can't be used. Does it have to be that way? I thought that mysql_thread_create() could automatically name the thread, why would we need to do it manually? Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 2

Re: MariaDB topic on GitHub
by Ian Gilfillan 11 Jun '24

11 Jun '24

Hi! Thanks for the suggestions. We'll take a look on our side at adding topics and improving the repo metadata. On Tue, Mar 19, 2024 at 8:55 AM Ian Gilfillan <ian(a)mariadb.org> wrote: > Hi! > > Thanks for the suggestions. We'll take a look on our side at adding topics > and improving the repo metadata. > > On Tue, Mar 19, 2024 at 5:20 AM Otto Kekäläinen <otto(a)kekalainen.net> > wrote: > >> Hi! >> >> I was browsing the GitHub topic 'mariadb' at >> https://github.com/topics/mariadb and to my surprise MariaDB itself >> isn't in that list at all. This seems to be due to >> https://github.com/MariaDB/server not having any topics defined at >> all. >> >> I recommend you add in the GitHub settings of >> https://github.com/MariaDB/server at least the topic 'mariadb', and >> while at it, also add link to mariadb.org. >> >> More ideas on how to improve the look and features of the GitHub page >> can be found by browsing projects like https://github.com/pulsar-edit >> and https://github.com/prisma which have all their GitHub account >> metadata polished. >> >> - Otto >> >

3 8

Re: 7d863d3388d: MDEV-32537 Name threads to improve debugging experience and diagnostics.
by Sergei Golubchik 08 Jun '24

08 Jun '24

Hi, Vladislav, I'd prefer you use my_thread_set_name() everywhere, even in linux- or windows-only code. It's not performance critical, and it'd be much easier to grep to thread names if they all are set by the same function. Otherwise ok. On Jun 08, Vladislav Vaintroub wrote: > revision-id: 7d863d3388d (mariadb-11.0.1-286-g7d863d3388d) > parent(s): 8bf9f218556 > author: Vladislav Vaintroub > committer: Vladislav Vaintroub > timestamp: 2024-01-18 20:35:54 +0100 > message: > > MDEV-32537 Name threads to improve debugging experience and diagnostics. > > Use SetThreadDescription/pthread_setname_np to give threads a name. > > diff --git a/mysys/my_thr_init.c b/mysys/my_thr_init.c > index 2e8decd7d06..4271935472e 100644 > --- a/mysys/my_thr_init.c > +++ b/mysys/my_thr_init.c > @@ -192,6 +192,35 @@ my_bool my_thread_global_init(void) > return 0; > } > > +#ifdef _WIN32 > +#define MAX_THREAD_NAME 256 > +#elif defined(__linux__) > +#define MAX_THREAD_NAME 16 > +#elif defined(__FreeBSD__) || defined(__OpenBSD__) > +#include <pthread_np.h> > +#endif > + > +void my_thread_set_name(const char *name) > +{ > +#ifdef _WIN32 > + wchar_t wname[MAX_THREAD_NAME]; > + wname[0]= 0; > + MultiByteToWideChar(CP_UTF8, 0, name, -1, wname, MAX_THREAD_NAME); > + SetThreadDescription(GetCurrentThread(), wname); > +#elif defined __linux__ > + char shortname[MAX_THREAD_NAME]; > + snprintf(shortname, MAX_THREAD_NAME, "%s", name); > + pthread_setname_np(pthread_self(), shortname); > +#elif defined __NetBSD__ > + pthread_setname_np(pthread_self(), "%s", (void *) name); > +#elif defined __FreeBSD__ || defined __OpenBSD__ > + pthread_set_name_np(pthread_self(), name); > +#elif defined __APPLE__ > + pthread_setname_np(name); > +#else > + (void) name; > +#endif > +} > > /** > End the mysys thread system. Called when ending the last thread > diff --git a/tpool/aio_linux.cc b/tpool/aio_linux.cc > index 507c6b9264f..f39b1e9b457 100644 > --- a/tpool/aio_linux.cc > +++ b/tpool/aio_linux.cc > @@ -93,6 +94,7 @@ class aio_linux final : public aio > > static void getevent_thread_routine(aio_linux *aio) > { > + pthread_setname_np(pthread_self(), "my_getevents"); > /* > We collect events in small batches to hopefully reduce the > number of system calls. > diff --git a/tpool/aio_win.cc b/tpool/aio_win.cc > index b44f705bd1e..930bab88777 100644 > --- a/tpool/aio_win.cc > +++ b/tpool/aio_win.cc > @@ -92,6 +92,7 @@ class tpool_generic_win_aio : public aio > > static void aio_completion_thread_proc(tpool_generic_win_aio* aio) > { > + SetThreadDescription(GetCurrentThread(), L"aio_completion_thread_proc"); > aio->completion_thread_work(); > } > > Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

1 0

Re: [PATCH] MDEV-21322: Report slave progress to the master
by Kristian Nielsen 04 Jun '24

04 Jun '24

Kristian Nielsen via commits <commits(a)lists.mariadb.org> writes: > From: Brandon Nesterenko <brandon.nesterenko(a)mariadb.com> > > This patch extends the command `SHOW REPLICA HOSTS` with three > columns: Hi Brandon, please find my review of MDEV-21322 below. > This patch extends the command `SHOW REPLICA HOSTS` with three > columns: > > 1) Gtid_State_Sent. This represents that latest GTIDs sent to the > replica in each domain. It will always be populated, regardless of > the semi-sync status (i.e. asynchronous connections will still > update this column with the latest GTID state sent to the replica). > > 2) Gtid_State_Ack. For semi-synchronous connections (only), this > column represents the last GTID in each domain that the replica has > acknowledged. Use "Pos" instead of "State", eg. Gtid_Pos_Sent and Gtid_Pos_Ack. To be consistent with eg. gtid_binlog_pos, which has last GTID per domain_id; vs. gtid_binlog_state, which has GTID per every (domain_id, server_id) combination. > @@ -2272,6 +2292,30 @@ send_event_to_slave(binlog_send_info *info, Log_event_type event_type, > return "Failed to run hook 'after_send_event'"; > } > > + if (info->thd->slave_info) > + { > + strncpy(info->thd->slave_info->gtid_state_sent.log_file, > + info->log_file_name + info->dirlen, > + strlen(info->log_file_name) - info->dirlen); > + info->thd->slave_info->gtid_state_sent.log_pos= pos; > + } Isn't there missing synchronisation here against concurrent access from another thread? I guess we need LOCK_thd_data around the strncpy(). Maybe we can avoid taking LOCK_thd_data except in the uncommon case where the gile name changes. It would be best to avoid a mutex lock/unlock for _every_ event sent to a slave. The position doesn't need locking I think, you can use an atomic std::memory_order_relaxed mostly for documentation if you like. > @@ -125,8 +114,11 @@ int THD::register_slave(uchar *packet, size_t packet_length) > if (check_access(this, PRIV_COM_REGISTER_SLAVE, any_db.str, NULL,NULL,0,0)) > return 1; > if (!(si= (Slave_info*)my_malloc(key_memory_SLAVE_INFO, sizeof(Slave_info), > - MYF(MY_WME)))) > + MYF(MY_WME|MY_ZEROFILL)))) > return 1; > + memset(si->gtid_state_sent.log_file, '\0', FN_REFLEN); > + memset(si->gtid_state_ack.log_file, '\0', FN_REFLEN); I think it's good to add the MY_ZEROFILL here, but then the two memset() are redundant and can be removed. > @@ -1235,6 +1251,14 @@ int Repl_semi_sync_master::update_sync_header(THD* thd, unsigned char *packet, > *need_sync= sync; > > l_end: > + if (is_on()) > + { > + thd->slave_info->sync_status= > + sync ? thd->slave_info->sync_status= > + Slave_info::SYNC_STATE_SEMI_SYNC_ACTIVE > + : thd->slave_info->sync_status= > + Slave_info::SYNC_STATE_SEMI_SYNC_STALE; > + } There's something wierd here, double assignment of thd->slave_info->sync_status, probably you just meant: thd->slave_info->sync_status= sync ? Slave_info::SYNC_STATE_SEMI_SYNC_ACTIVE : Slave_info::SYNC_STATE_SEMI_SYNC_STALE; More importantly, update_sync_header() is called just _before_ writing the event on the socket to the connected slave. Is that really the right place to do so? Shouldn't the status be updated only _after_ the event has been sent? Or does it not matter, because the actual TCP sending is asynchroneous anyway? In that case, should add a comment explaining why updating the status here is ok. > --- a/mysql-test/suite/rpl/r/rpl_mixed_ddl_dml.result > +++ b/mysql-test/suite/rpl/r/rpl_mixed_ddl_dml.result > @@ -11,8 +11,8 @@ n > 2002 > connection master; > show slave hosts; > -Server_id Host Port Master_id > -2 127.0.0.1 9999 1 > +Server_id Host Port Master_id Gtid_State_Sent Gtid_State_Ack Sync_Status > +2 127.0.0.1 9999 1 0-1-2 Asynchronous Just a remark that this output could be non-deterministic unless the test guarantees that the state will be stable at this point. So please check that the test will produce stable output; or if you already did so, then great! > +--echo # state, and Gtid_State_Sent/Ack should start, and only Gtid_State_Sent Typo, s/should start/should start empty/ ? > +--connection server_2 > +--source include/stop_slave.inc > +set global rpl_semi_sync_slave_enabled = 1; > +--source include/start_slave.inc > + > +--connection server_1 > +let $status_var= Rpl_semi_sync_master_clients; > +let $status_var_value= 1; > +source include/wait_for_status_var.inc; > + > +--replace_result $SLAVE_MYPORT SLAVE_PORT $SERVER_MYPORT_3 SLAVE2_PORT $DEFAULT_MASTER_PORT DEFAULT_PORT > +SHOW REPLICA HOSTS; Hm, won't this output be non-deterministic? From a quick look in the code, Rpl_semi_sync_master_clients is incremented from the start of mysql_binlog_send(), but the state is only set from update_sync_header() when the first event is sent to the slave. So won't there be a (small) window where the state has not yet switched to ACTIVE when this SHOW SLAVE HOSTS runs? So better wait for the state to become ACTIVE instead. Or for the expected value in Gtid_Pos_Sent, where that is non-empty. Also, better use SHOW SLAVE HOSTS here, for consistency with rest of test suite. > +--let $nextval= `SELECT max(a)+1 from t1` > +--eval insert into t1 values ($nextval) Is there a reason here to use separate --let and --eval? And not just: INSERT INTO t1 SELECT max(a)+1 FROM t1; > +--echo # Ensuring master gtid_binlog_pos matches Gtid_State_Ack > +--let $gtid_ack= query_get_value(show slave hosts, Gtid_State_Ack, 1) > +if (`SELECT strcmp("$master_gtid","$gtid_ack") != 0`) > +{ > + --echo # Master gtid_binlog_pos: $master_gtid > + --echo # Gtid_State_Ack: $gtid_ack > + --die Master's gtid_binlog_pos should match Gtid_State_Ack, but doesn't Again, sync_with_master_gtid.inc on the slave will not guarantee that the master saw the ack yet, even though it will be a rare race when it doesn't happen. So you'd need to wait for the expected value here with wait_condition.inc or somilar instead, I think. Probably the same in subsequent parts of the test case. > +--echo # > +--echo # 21322.5: When connecting a new slave (server_id 3) which initially has > +--echo # semi-sync disabled, SHOW SLAVE HOSTS on the master should show its > +--echo # Sync_Status as asynchronous (while server_id 2 is still semi-sync > +--echo # active). Sorry, it's a very long and complex test case, I didn't have the time today to carefully read and check every detail in it after this point. Please carefully check all the cases for possible races like described above. Replication tests are notorious for having non-deterministic failures (as I'm sure you've experienced first-hand), and it's important to try to avoid that as much as possible. A general remark on the test, the output of Gtid_Pos_Sent and Gtid_Pos_Ack are full GTID positions (multiple domain_id). But there is no test of having more than one domain in the output. I think you need to extend the test to test at least one case where there are multiple domains. And also need to test when the GTID state in the binlog has multiple server_id (though maybe the test case already does this, I didn't check myself). - Kristian.

2 3

Re: eba212af92d: MDEV-9101 Limit size of created disk temporary files and tables
by Sergei Golubchik 08 May '24

08 May '24

Hi, Monty, On Apr 18, Michael Widenius wrote: > commit eba212af92d > Author: Michael Widenius <monty(a)mariadb.org> > Date: Thu Mar 14 17:59:00 2024 +0100 > > diff --git a/sql/json_table.cc b/sql/json_table.cc > index 6713e796eb7..da5ab0c450a 100644 > --- a/sql/json_table.cc > +++ b/sql/json_table.cc > @@ -734,7 +734,8 @@ bool Create_json_table::finalize(THD *thd, TABLE *table, > > table->db_stat= HA_OPEN_KEYFILE; > if (unlikely(table->file->ha_open(table, table->s->path.str, O_RDWR, > - HA_OPEN_TMP_TABLE | HA_OPEN_INTERNAL_TABLE))) > + HA_OPEN_TMP_TABLE | HA_OPEN_INTERNAL_TABLE | > + HA_OPEN_SIZE_TRACKING))) shouldn't HA_OPEN_INTERNAL_TABLE always imply HA_OPEN_SIZE_TRACKING automatically? > DBUG_RETURN(true); > > table->set_created(); > diff --git a/storage/maria/ma_statrec.c b/storage/maria/ma_statrec.c > index d8a8b0a05d7..ff0c6c7a617 100644 > --- a/storage/maria/ma_statrec.c > +++ b/storage/maria/ma_statrec.c > @@ -45,6 +45,13 @@ my_bool _ma_write_static_record(MARIA_HA *info, const uchar *record) > my_errno=HA_ERR_RECORD_FILE_FULL; > return(2); > } > + if (info->s->tracked && are there temporary files that you don't want to track? are there non-temporary files that you want to track? > + _ma_update_tmp_file_size(&info->s->track_data, > + MY_ALIGN(info->s->state.state.data_file_length+ > + info->s->base.pack_reclength, > + MARIA_TRACK_INCREMENT_SIZE))) > + return(2); > + > if (info->opt_flag & WRITE_CACHE_USED) > { /* Cash in use */ > if (my_b_write(&info->rec_cache, record, > diff --git a/storage/maria/ma_page.c b/storage/maria/ma_page.c > index 5881456a69a..0877c966d1c 100644 > --- a/storage/maria/ma_page.c > +++ b/storage/maria/ma_page.c > @@ -417,6 +417,13 @@ my_off_t _ma_new(register MARIA_HA *info, int level, > DBUG_RETURN(HA_OFFSET_ERROR); > } > share->state.state.key_file_length+= block_size; > + if (info->s->tracked && > + _ma_update_tmp_file_size(&share->track_index, > + share->state.state.key_file_length)) shouldn't you do it before share->state.state.key_file_length is increased? Currently it seems that you're failing the operation, but the file will stay increased same pattern everywhere else > + { > + mysql_mutex_unlock(&share->intern_lock); > + DBUG_RETURN(HA_OFFSET_ERROR); > + } > /* Following is for not transactional tables */ > info->state->key_file_length= share->state.state.key_file_length; > mysql_mutex_unlock(&share->intern_lock); > diff --git a/storage/maria/ma_delete_all.c b/storage/maria/ma_delete_all.c > index f355d0da3e8..ed2087d3091 100644 > --- a/storage/maria/ma_delete_all.c > +++ b/storage/maria/ma_delete_all.c > @@ -132,9 +132,16 @@ int maria_delete_all_rows(MARIA_HA *info) > if (_ma_flush_table_files(info, MARIA_FLUSH_DATA|MARIA_FLUSH_INDEX, > FLUSH_IGNORE_CHANGED, FLUSH_IGNORE_CHANGED) || > mysql_file_chsize(info->dfile.file, 0, 0, MYF(MY_WME)) || > - mysql_file_chsize(share->kfile.file, share->base.keystart, 0, MYF(MY_WME))) > + mysql_file_chsize(share->kfile.file, share->base.keystart, 0, > + MYF(MY_WME))) should be > 0 here, shouldn't it? Both for kfile and dfile. > goto err; > > + if (info->s->tracked) > + { > + _ma_update_tmp_file_size(&info->s->track_data, 0); > + _ma_update_tmp_file_size(&info->s->track_index, share->base.keystart); but this shouldn't update the size if mysql_file_chsize returned -1 > + } > + > if (_ma_initialize_data_file(share, info->dfile.file)) > goto err; > > diff --git a/sql/uniques.cc b/sql/uniques.cc > index 36725e80a6b..44d93bb0f88 100644 > --- a/sql/uniques.cc > +++ b/sql/uniques.cc > @@ -720,7 +720,7 @@ bool Unique::merge(TABLE *table, uchar *buff, size_t buff_size, > /* Open cached file for table records if it isn't open */ > if (! my_b_inited(outfile) && > open_cached_file(outfile, mysql_tmpdir, TEMP_PREFIX, DISK_CHUNK_SIZE, > - MYF(MY_WME))) > + MYF(MY_WME | MY_TRACK | MY_TRACK_WITH_LIMIT))) as far as I can see, open_cached_file is only used for temporary files. it should apply MY_TRACK | MY_TRACK_WITH_LIMIT internally, it'll be less error-prone > return 1; > > bzero((char*) &sort_param,sizeof(sort_param)); > diff --git a/storage/maria/ma_write.c b/storage/maria/ma_write.c > index 9a6859bfd4d..7404db84067 100644 > --- a/storage/maria/ma_write.c > +++ b/storage/maria/ma_write.c > @@ -386,6 +387,11 @@ int maria_write(MARIA_HA *info, const uchar *record) > } > } > } > + else if (my_errno == HA_ERR_LOCAL_TMP_SPACE_FULL || > + my_errno == HA_ERR_GLOBAL_TMP_SPACE_FULL) > + { > + filepos= HA_OFFSET_ERROR; /* Avoid write_record_abort() */ why avoid? HA_ERR_LOCAL_TMP_SPACE_FULL is like EQUOT or ENOSPC. why should be masked as HA_OFFSET_ERROR, and not be a fatal error? > + } > else > fatal_error= 1; > > diff --git a/storage/maria/ma_close.c b/storage/maria/ma_close.c > index 7441e29a97b..47935d61b88 100644 > --- a/storage/maria/ma_close.c > +++ b/storage/maria/ma_close.c > @@ -14,7 +14,7 @@ > along with this program; if not, write to the Free Software > Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1335 USA */ > > -/* close a isam-database */ > +/* close an Aria table */ at last :) it's what 27 years old? > /* > TODO: > We need to have a separate mutex on the closed file to allow other threads > diff --git a/sql/sql_window.cc b/sql/sql_window.cc > index e0c7cef1a9e..4959c7833a1 100644 > --- a/sql/sql_window.cc > +++ b/sql/sql_window.cc > @@ -2922,15 +2922,16 @@ bool compute_window_func(THD *thd, > if (unlikely(thd->is_error() || thd->is_killed())) > break; > > - > /* Return to current row after notifying cursors for each window > function. */ > - tbl->file->ha_rnd_pos(tbl->record[0], rowid_buf); > + if (tbl->file->ha_rnd_pos(tbl->record[0], rowid_buf)) > + return true; how can it fail? > } > > /* We now have computed values for each window function. They can now > be saved in the current row. */ > - save_window_function_values(window_functions, tbl, rowid_buf); > + if (save_window_function_values(window_functions, tbl, rowid_buf)) > + return true; > > rownum++; > } > diff --git a/mysql-test/main/status.test b/mysql-test/main/status.test > index acf29f35b9e..b6be046a9ac 100644 > --- a/mysql-test/main/status.test > +++ b/mysql-test/main/status.test > @@ -244,9 +244,9 @@ let $rnd_next = `show global status like 'handler_read_rnd_next'`; > let $tmp_table = `show global status like 'Created_tmp_tables'`; > show status like 'com_show_status'; > show status like 'hand%write%'; > -show status like '%tmp%'; > +show status like '%_tmp%'; Doesn't look like a very meaningful change. Did you mean '%\_tmp%' ? > show status like 'hand%write%'; > -show status like '%tmp%'; > +show status like '%_tmp%'; > show status like 'com_show_status'; > let $rnd_next2 = `show global status like 'handler_read_rnd_next'`; > let $tmp_table2 = `show global status like 'Created_tmp_tables'`; > diff --git a/mysys/mf_cache.c b/mysys/mf_cache.c > index 2fec59f4e70..712187f6181 100644 > --- a/mysys/mf_cache.c > +++ b/mysys/mf_cache.c > @@ -43,7 +43,7 @@ my_bool open_cached_file(IO_CACHE *cache, const char* dir, const char *prefix, > cache->file_name=0; > cache->buffer=0; /* Mark that not open */ > if (!init_io_cache(cache, -1, cache_size, WRITE_CACHE, 0L, 0, > - MYF(cache_myflags | MY_NABP))) > + MYF(cache_myflags | MY_NABP | MY_TRACK))) Why not MY_TRACK_WITH_LIMIT ? > { > DBUG_RETURN(0); > } > diff --git a/sql/sql_select.cc b/sql/sql_select.cc > index 6f2797b6b39..324f7679e8f 100644 > --- a/sql/sql_select.cc > +++ b/sql/sql_select.cc > @@ -22275,7 +22275,8 @@ bool open_tmp_table(TABLE *table) > int error; > if (unlikely((error= table->file->ha_open(table, table->s->path.str, O_RDWR, > HA_OPEN_TMP_TABLE | > - HA_OPEN_INTERNAL_TABLE)))) > + HA_OPEN_INTERNAL_TABLE | > + HA_OPEN_SIZE_TRACKING)))) shouldn't HA_OPEN_TMP_TABLE or HA_OPEN_INTERNAL_TABLE imply HA_OPEN_SIZE_TRACKING? > { > table->file->print_error(error, MYF(0)); /* purecov: inspected */ > table->db_stat= 0; > diff --git a/sql/handler.cc b/sql/handler.cc > index fec473d97bb..a05b7476724 100644 > --- a/sql/handler.cc > +++ b/sql/handler.cc > @@ -443,9 +444,12 @@ int ha_init_errors(void) > /* Zerofill it to avoid uninitialized gaps. */ remove the comment, if you no longer zerofill > if (! (handler_errmsgs= (const char**) my_malloc(key_memory_handler_errmsgs, > HA_ERR_ERRORS * sizeof(char*), > - MYF(MY_WME | MY_ZEROFILL)))) > + MYF(MY_WME)))) > return 1; > > + /* Copy default handler error messages */ > + memcpy(handler_errmsgs, handler_error_messages, HA_ERR_ERRORS * sizeof(char*)); > + > /* Set the dedicated error messages. */ > SETMSG(HA_ERR_KEY_NOT_FOUND, ER_DEFAULT(ER_KEY_NOT_FOUND)); > SETMSG(HA_ERR_FOUND_DUPP_KEY, ER_DEFAULT(ER_DUP_KEY)); > diff --git a/storage/maria/maria_def.h b/storage/maria/maria_def.h > index 705562eb795..b256466f1f0 100644 > --- a/storage/maria/maria_def.h > +++ b/storage/maria/maria_def.h > @@ -778,6 +785,14 @@ typedef struct st_maria_share > myf write_flag; > enum data_file_type data_file_type; > enum pagecache_page_type page_type; /* value depending transactional */ > + > + /* > + tracked will cause lost bytes (not aligned) but this is ok as it is always > + used with tmp_file_tracking if set I don't understand your explanation why "this is ok" > + */ > + my_bool tracked; /* Tracked table (always internal) */ > + struct tmp_file_tracking track_data,track_index; > + > /** > if Checkpoint looking at table; protected by close_lock or THR_LOCK_maria > */ > diff --git a/include/my_sys.h b/include/my_sys.h > index af509c7df45..0df12963ab9 100644 > --- a/include/my_sys.h > +++ b/include/my_sys.h > @@ -178,6 +180,17 @@ uchar *my_large_malloc(size_t *size, myf my_flags); > void my_large_free(void *ptr, size_t size); > void my_large_page_truncate(size_t *size); > > +/* Tracking tmp file usage */ > + > +struct tmp_file_tracking > +{ > + ulonglong last_position; I had to read how "last_position" is used to realize that it's in fact previous file_size value. Why not to call it previous_file_size? Or recorded_file_size ? Or old_file_size? it's not a "position" > + ulonglong file_size; > +}; > + > +typedef int (*TMPFILE_SIZE_CB)(struct tmp_file_tracking *track, int no_error); > +extern TMPFILE_SIZE_CB update_tmp_file_size; > + > #ifdef _WIN32 > extern BOOL my_obtain_privilege(LPCSTR lpPrivilege); > #endif > diff --git a/sql/sql_class.h b/sql/sql_class.h > index c1efb49f6da..42eee9913ec 100644 > --- a/sql/sql_class.h > +++ b/sql/sql_class.h > @@ -1070,12 +1074,22 @@ typedef struct system_status_var > */ > > #define last_system_status_var questions > -#define last_cleared_system_status_var local_memory_used > + > +/* Parameters to set_status_var_init() */ > + > +#define STATUS_OFFSET(A) offsetof(STATUS_VAR,A) > +/* Clear as part of flush */ > +#define clear_up_to_tmp_space_used STATUS_OFFSET(tmp_space_used) > +/* Clear as part of startup */ > +#define clear_up_to_memory_used STATUS_OFFSET(local_memory_used) > +/* Full initialization. Note that global_memory_used is updated early! */ > +#define clear_up_to_global_memory_used STATUS_OFFSET(global_memory_used) > +#define last_restored_status_var clear_up_to_tmp_space_used those are bad names. and the fact that you need to rename these defines and change all code that uses them - this kind of shows they're not good. when there will be a new variable, clear_up_to_tmp_space_used can become clear_up_to_new_variable and everything will need to be changed again. better use names that don't depend on individual variables. last_restored_status_var and last_cleared_system_status_var are good. Try to split variables logically. First there are additive variables - global value is the sum of all local values. They're cleared, they're added up tp global, etc. Then there are non-additive, not cleared, not added - like local_memory_used and tmp_space_used, for example. And global_memory_used is special because it's initialized early, right? so, may be defines based on these properties would be easier to understand. because now I'm totally at loss when to use clear_up_to_tmp_space_used, when to use clear_up_to_memory_used, when clear_up_to_global_memory_used and when last_restored_status_var. > + > > /** Number of contiguous global status variables */ > -constexpr int COUNT_GLOBAL_STATUS_VARS= int(offsetof(STATUS_VAR, > - last_system_status_var) / > - sizeof(ulong)) + 1; > +constexpr int COUNT_GLOBAL_STATUS_VARS= > + int(STATUS_OFFSET(last_system_status_var) /sizeof(ulong)) + 1; > > /* > Global status variables > diff --git a/sql/filesort.cc b/sql/filesort.cc > index abcd4127a0b..df4508b635a 100644 > --- a/sql/filesort.cc > +++ b/sql/filesort.cc > @@ -1082,6 +1078,11 @@ static ha_rows find_all_keys(THD *thd, Sort_param *param, SQL_SELECT *select, > DBUG_RETURN(num_records); > > err: > + if (!quick_select) > + { > + (void) file->extra(HA_EXTRA_NO_CACHE); > + file->ha_rnd_end(); > + } why? > sort_form->column_bitmaps_set(save_read_set, save_write_set); > DBUG_RETURN(HA_POS_ERROR); > } /* find_all_keys */ > diff --git a/sql/mysqld.cc b/sql/mysqld.cc > index dc4a955aa79..4727b7b5413 100644 > --- a/sql/mysqld.cc > +++ b/sql/mysqld.cc > @@ -1947,6 +1950,10 @@ static void mysqld_exit(int exit_code) > if (exit_code == 0 || opt_endinfo) > SAFEMALLOC_REPORT_MEMORY(0); > } > + if (global_tmp_space_used) > + fprintf(stderr, "Warning: tmp_space not freed: %llu\n", > + (ulonglong) global_tmp_space_used); This is confusing. almost all (if not all) temp space is automatically freed by the OS when the process exits, so this warning will only scare the user about lost space when in fact no space is lost, it's only some space was not accounted for inside MariaDB. I suggest either make this warning debug-only or rephrase it like "Note: %llu tmp_space was lost and will be freed automatically when the process exits" > + > DBUG_LEAVE; > #ifdef _WIN32 > my_report_svc_status(SERVICE_STOPPED, exit_code, 0); > @@ -7571,6 +7636,7 @@ SHOW_VAR status_vars[]= { > {"Tc_log_page_size", (char*) &tc_log_page_size, SHOW_LONG_NOFLUSH}, > {"Tc_log_page_waits", (char*) &tc_log_page_waits, SHOW_LONG}, > #endif > + {"tmp_space_used", (char*) offsetof(STATUS_VAR, tmp_space_used), SHOW_LONGLONG_STATUS}, first letter - upper case > #ifdef HAVE_POOL_OF_THREADS > {"Threadpool_idle_threads", (char *) &show_threadpool_idle_threads, SHOW_SIMPLE_FUNC}, > {"Threadpool_threads", (char *) &show_threadpool_threads, SHOW_SIMPLE_FUNC}, > diff --git a/sql/log.cc b/sql/log.cc > index 460cefea47b..a9bc3758dd5 100644 > --- a/sql/log.cc > +++ b/sql/log.cc > @@ -2388,6 +2384,8 @@ bool Event_log::check_write_error(THD *thd) > case ER_TRANS_CACHE_FULL: > case ER_STMT_CACHE_FULL: > case ER_ERROR_ON_WRITE: > + case EE_LOCAL_TMP_SPACE_FULL: > + case EE_GLOBAL_TMP_SPACE_FULL: not a good idea to mix EE_ OS-like errors and ER_ SQL layer errors > case ER_BINLOG_LOGGING_IMPOSSIBLE: > checked= TRUE; > break; > @@ -6650,7 +6651,8 @@ Event_log::flush_and_set_pending_rows_event(THD *thd, Rows_log_event* event, > { > set_write_error(thd, is_transactional); > if (check_cache_error(thd, cache_data) && > - stmt_has_updated_non_trans_table(thd)) > + (stmt_has_updated_non_trans_table(thd) || > + !is_transactional)) why? > cache_data->set_incident(); > delete pending; > cache_data->set_pending(NULL); > @@ -7868,8 +7877,16 @@ int Event_log::write_cache(THD *thd, binlog_cache_data *cache_data) > DBUG_RETURN(res ? ER_ERROR_ON_WRITE : 0); > } > > - if (reinit_io_cache(cache, READ_CACHE, 0, 0, 0)) > + /* > + Allow flush of transaction logs to temporary go over the tmp space limit > + as we do not want the commit to fail > + */ > + cache->myflags&= ~MY_TRACK_WITH_LIMIT; > + res= reinit_io_cache(cache, READ_CACHE, 0, 0, 0); > + cache->myflags|= MY_TRACK_WITH_LIMIT; you also remove MY_TRACK_WITH_LIMIT inside reinit_io_cache(), so this is redundant. Or it's redundant inside reinit_io_cache(), either way, no point in doing it twice > + if (res) > DBUG_RETURN(ER_ERROR_ON_WRITE); > + > /* Amount of remaining bytes in the IO_CACHE read buffer. */ > size_t log_file_pos; > uchar header_buf[LOG_EVENT_HEADER_LEN]; > diff --git a/mysys/mf_iocache.c b/mysys/mf_iocache.c > index 4ee1331bdb3..6214057f9e3 100644 > --- a/mysys/mf_iocache.c > +++ b/mysys/mf_iocache.c > @@ -73,6 +74,50 @@ int (*_my_b_encr_read)(IO_CACHE *info,uchar *Buffer,size_t Count)= 0; > int (*_my_b_encr_write)(IO_CACHE *info,const uchar *Buffer,size_t Count)= 0; > > > +static inline my_bool tmp_file_track(IO_CACHE *info, ulonglong file_size) > +{ > + if ((info->myflags & (MY_TRACK | MY_TRACK_WITH_LIMIT)) && > + update_tmp_file_size) > + { > + if (info->tracking.file_size < file_size) > + { > + int error; > + info->tracking.file_size= file_size; > + if ((error= update_tmp_file_size(&info->tracking, > + !(info->myflags & > + MY_TRACK_WITH_LIMIT)))) > + { > + if (info->myflags & MY_WME) > + my_error(my_errno, MYF(0)); > + info->error= -1; > + return 1; > + } > + } > + } > + return 0; > +} > + > +my_bool io_cache_tmp_file_track(IO_CACHE *info, ulonglong file_size) > +{ > + return tmp_file_track(info, file_size); > +} > + > + > +void end_tracking_io_cache(IO_CACHE *info) what does "end tracking IO_CACHE" mean? I read it as "end tracking of this IO_CACHE, it is no longer tracked after this function call" - but it's not what it is doing. so what how do you read this function name? > +{ > + if ((info->myflags & (MY_TRACK | MY_TRACK_WITH_LIMIT)) && > + info->tracking.file_size) > + { > + info->tracking.file_size= 0; > + update_tmp_file_size(&info->tracking, 1); > + } > +} > + > +void truncate_io_cache(IO_CACHE *info) > +{ > + if (my_chsize(info->file, 0, 0, MYF(MY_WME)) <= 0) > + end_tracking_io_cache(info); not quite. If my_chsize() returns -1, you should not do info->tracking.file_size= 0, because file size wasn't actually changed. > +} > > static void > init_functions(IO_CACHE* info) Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 1

Re: 0d6dc52f43d: MDEV-30046 Refactor write_record and fix idempotent replication
by Sergei Golubchik 04 May '24

04 May '24

Hi, Nikita, On Apr 29, Nikita Malyavin wrote: > revision-id: 0d6dc52f43d (mariadb-10.5.24-175-g0d6dc52f43d) > parent(s): 652e9ee405e > author: Nikita Malyavin > committer: Nikita Malyavin > timestamp: 2024-04-27 21:08:21 +0200 > message: > > MDEV-30046 Refactor write_record and fix idempotent replication > > Idempotent write_row works same as REPLACE: if there is a duplicating > record in the table, then it will be deleted and re-inserted, with the > same update optimization. > > The code in Rows:log_event::write_row was basically copy-pasted from > write_record. > > What's done: > REPLACE operation was unified across replication and sql. It is now > representred as a Write_record class, that holds the whole state, and allows > re-using some resources in between the row writes. > > Replace, IODKU and single insert implementations are split across different > methods, reluting in a much cleaner code. > > The entry point is preserved as a single Write_record::write_record() call. > The implementation to call is chosen on the constructor stage. > > This allowed several optimizations to be done: > 1. The table key list is not iterated for every row. We find last unique key in > the order of checking once and preserve it across the rows. See last_uniq_key(). > 2. ib_handler::referenced_by_foreign_key acquires a global lock. This call was > done per row as well. Not all the table config that allows optimized replace is > folded into a single boolean field can_optimize. All the fields to check are > even stored in a single register on a 64-bit platform. > 3. DUP_REPLACE and DUP_UPDATE cases now have one less level of indirection > 4. modified_non_trans_tables is checked and set only when it's really needed. > 5. Obsolete bitmap manipulations are removed. > > Also: > * Unify replace initialization step across implementations: > add prepare_for_replace and finalize_replace > * alloca is removed in favor of mem_root allocation. This memory is reused > across the rows. > * An rpl-related callback is added to the replace branch, meaning that an extra > check is made per row replace even for the common case. It can be avoided with > templates if considered a problem. > > diff --git a/mysql-test/main/long_unique_bugs_replication.test b/mysql-test/main/long_unique_bugs_replication.test > index 9c44d13e6a5..f2225cde101 100644 > --- a/mysql-test/main/long_unique_bugs_replication.test > +++ b/mysql-test/main/long_unique_bugs_replication.test > + > +--echo # > +--echo # End of 10.4 tests > +--echo # ... > --echo # > --echo # End of 10.4 tests > --echo # may be 10.5? > diff --git a/sql/log_event_server.cc b/sql/log_event_server.cc > index 38e8b062f18..b5810eeaf7b 100644 > --- a/sql/log_event_server.cc > +++ b/sql/log_event_server.cc > @@ -5705,10 +5708,10 @@ int Rows_log_event::do_apply_event(rpl_group_info *rgi) > &m_cols_ai : &m_cols); > bitmap_intersect(table->write_set, after_image); > > - this->slave_exec_mode= slave_exec_mode_options; // fix the mode > - > + COPY_INFO copy_info; > + Write_record write_record; > // Do event specific preparations > - error= do_before_row_operations(rli); > + error= do_before_row_operations(rgi, &copy_info, &write_record); You create write_record on the stack and save it inside this->m_write_record 1. why not to have it permanently a part of Rows_log_event ? 2. if you'll keep it on the stack, please, reset m_write_record=0 before returning from ::do_apply_event(). And may be even do m_write_record= &write_record here, not in ::do_before_row_operations() it'll be a bit easier to see how a local variable is stored and then freed and that it doesn't actually leave the scope. > > /* > Bug#56662 Assertion failed: next_insert_id == 0, file handler.cc > @@ -7132,6 +7148,40 @@ Write_rows_log_event::do_before_row_operations(const Slave_reporting_capability > m_table->mark_auto_increment_column(true); > } > > + if (slave_exec_mode == SLAVE_EXEC_MODE_IDEMPOTENT && > + (m_table->file->ha_table_flags() & HA_DUPLICATE_POS || > + m_table->s->long_unique_table)) > + error= m_table->file->ha_rnd_init_with_error(0); I'd put this inside if (slave_exec_mode == SLAVE_EXEC_MODE_IDEMPOTENT) about ~40 lines above the place where you put it now. > + > + if (!error) > + { > + bzero(copy_info, sizeof *copy_info); > + copy_info->handle_duplicates= > + slave_exec_mode == SLAVE_EXEC_MODE_IDEMPOTENT ? > + DUP_REPLACE : DUP_ERROR; simply copy_info->handle_duplicates= thd->lex->duplicates; > + copy_info->table_list= m_table->pos_in_table_list; > + > + int (*callback)(void *, void*)= NULL; > + if (!get_flags(COMPLETE_ROWS_F)) > + { > + /* > + If row is incomplete we will use the record found to fill > + missing columns. > + */ > + callback= [](void *e, void* r)->int { > + auto rgi= static_cast<rpl_group_info*>(r); > + auto event= static_cast<Write_rows_log_event*>(e); > + return event->incomplete_record_callback(rgi); > + }; > + } > + new (write_record) Write_record(thd, m_table, copy_info, > + m_vers_from_plain && > + m_table->versioned(VERS_TIMESTAMP), > + m_table->triggers && do_invoke_trigger(), > + NULL, callback, this, rgi); please, use write_record->init(...) here. And in all similar places below, I won't comment on them separately. > + m_write_record= write_record; > + } > + > return error; > } > > @@ -7173,7 +7223,15 @@ Write_rows_log_event::do_after_row_operations(const Slave_reporting_capability * > { > m_table->file->print_error(local_error, MYF(0)); > } > - return error? error : local_error; > + int rnd_error= 0; > + if (m_table->file->inited) > + { > + DBUG_ASSERT(slave_exec_mode == SLAVE_EXEC_MODE_IDEMPOTENT); > + DBUG_ASSERT(m_table->file->ha_table_flags() & HA_DUPLICATE_POS || > + m_table->s->long_unique_table); DBUG_ASSERT(m_table->file->inited == handler::RND); > + rnd_error= m_table->file->ha_rnd_end(); > + } > + return error? error : local_error ? local_error : rnd_error; > } > > bool Rows_log_event::process_triggers(trg_event_type event, > diff --git a/sql/sql_insert.cc b/sql/sql_insert.cc > index ff71361493a..e410efda121 100644 > --- a/sql/sql_insert.cc > +++ b/sql/sql_insert.cc > @@ -677,6 +678,30 @@ Field **TABLE::field_to_fill() > return triggers && triggers->nullable_fields() ? triggers->nullable_fields() : field; > } > > +int prepare_for_replace(TABLE *table, enum_duplicates handle_duplicates, > + bool ignore) > +{ > + bool create_lookup_handler= handle_duplicates != DUP_ERROR; why do you initialize create_lookup_handler here and then immediately repeat it two lines below? :) less confusing would be bool create_lookup_handler= handle_duplicates != DUP_ERROR || ignore; if (create_lookup_handler) { > + if (ignore || handle_duplicates != DUP_ERROR) > + { > + create_lookup_handler= true; > + table->file->extra(HA_EXTRA_IGNORE_DUP_KEY); > + if (table->file->ha_table_flags() & HA_DUPLICATE_POS || > + table->s->long_unique_table) > + { > + if (table->file->ha_rnd_init_with_error(false)) > + return 1; > + } > + } > + return table->file->prepare_for_insert(create_lookup_handler); > +} > + > +int finalize_replace(TABLE *table, enum_duplicates handle_duplicates, > + bool ignore) > +{ DBUG_ASSERT(m_table->file->inited != handler::INDEX); > + return table->file->inited ? table->file->ha_rnd_end() : 0; > +} > + > > /** > INSERT statement implementation > @@ -1728,31 +1745,74 @@ int mysql_prepare_insert(THD *thd, TABLE_LIST *table_list, > } > > > /* Check if there is more uniq keys after field */ this comment isn't correct anymore, is it? on the other hand, the function is small and the name is clear, I think you can simply delete the comment, if you'd like. > +static ushort get_last_unique_key(TABLE *table, KEY *key_info) > +{ > + for (uint key= table->s->keys; key; --key) > + if (key_info[key-1].flags & HA_NOSAME) > + return key-1; > + return MAX_KEY; > +} > > +/** > + An update optimization can be done in REPLACE only when certain criterial met. > + In particular, if the "error key" was the last one to check. If not, there can > + be other keys that can fail the uniqueness check after we will delete the row. > + > + The problem arises with the order of the key checking. It is divided in two > + parts: > + 1. Higher-level handler::ha_write_row goes first. It checks for long unique > + and period overlaps (the latter is not supported by REPLACE yet). better "(when the latter will be supported by REPLACE)" this way the comment will be less wrong and misleading when we'll finally implement this support and but forget to update the comment :) > + 2. Engine usual unique index check. > + However, the order of the keys in TABLE_SHARE::key_info is: > + * first, engine unique keys > + * then long unique, which is not seen by engine > + WITHOUT OVERLAPS key has no particular order > + see sort_keys in sql_table.cc > + > + So we need to wrap this order. > + @return last unique key to check for duplication, or MAX_KEY if none. > + */ > -static int last_uniq_key(TABLE *table, const KEY *key, uint keynr) > +ushort Write_record::get_last_unique_key() const > { > + if (info->handle_duplicates != DUP_REPLACE) > + return MAX_KEY; // Not used. > + if (!table->s->key_info) > + return MAX_KEY; > /* > When an underlying storage engine informs that the unique key > conflicts are not reported in the ascending order by setting > the HA_DUPLICATE_KEY_NOT_IN_ORDER flag, we cannot rely on this > information to determine the last key conflict. > > The information about the last key conflict will be used to > do a replace of the new row on the conflicting row, rather > than doing a delete (of old row) + insert (of new row). > > Hence check for this flag and disable replacing the last row > - by returning 0 always. Returning 0 will result in doing > + by returning MAX_KEY always. Returning MAX_KEY will result in doing > a delete + insert always. > */ > if (table->file->ha_table_flags() & HA_DUPLICATE_KEY_NOT_IN_ORDER) > - return 0; > + return MAX_KEY; > > - while (++keynr < table->s->keys) > - if (key[keynr].flags & HA_NOSAME) > - return 0; > - return 1; > + /* > + Note, that TABLE_SHARE and TABLE see long uniques differently: > + - TABLE_SHARE sees as HA_KEY_ALG_LONG_HASH and HA_NOSAME > + - TABLE sees as usual non-unique indexes > + */ > + return > + table->key_info[0].flags & HA_NOSAME ? > + /* > + We have an in-engine unique, which should be checked last. > + Go through the TABLE list, which knows nothing about long uniques. > + */ > + ::get_last_unique_key(table, table->key_info) : > + /* > + We can only have long uniques. > + Go through the TABLE_SHARE list, where long uniques can be found. > + */ > + ::get_last_unique_key(table, table->s->key_info); > } > > > @@ -1786,449 +1846,511 @@ int vers_insert_history_row(TABLE *table) > } > > /* > - Write a record to table with optional deleting of conflicting records, > - invoke proper triggers if needed. > - > - SYNOPSIS > - write_record() > - thd - thread context > - table - table to which record should be written > - info - COPY_INFO structure describing handling of duplicates > - and which is used for counting number of records inserted > - and deleted. > - sink - result sink for the RETURNING clause > - > - NOTE > - Once this record will be written to table after insert trigger will > - be invoked. If instead of inserting new record we will update old one > - then both on update triggers will work instead. Similarly both on > - delete triggers will be invoked if we will delete conflicting records. > - > - Sets thd->transaction.stmt.modified_non_trans_table to TRUE if table which > - is updated didn't have transactions. > - > - RETURN VALUE > - 0 - success > - non-0 - error > - > -int write_record(THD *thd, TABLE *table, COPY_INFO *info, select_result *sink) > -{ > - int error, trg_error= 0; > - char *key=0; > - MY_BITMAP *save_read_set, *save_write_set; > - ulonglong prev_insert_id= table->file->next_insert_id; > - ulonglong insert_id_for_cur_row= 0; > - ulonglong prev_insert_id_for_cur_row= 0; > - DBUG_ENTER("write_record"); > + Retrieve the old (duplicated) row into record[1] to be able to > + either update or delete the offending record. We either: > + > + - use ha_rnd_pos() with a row-id (available as dup_ref) to the > + offending row, if that is possible (MyISAM and Blackhole, also long unique > + and application-time periods), or else > + > + - use ha_index_read_idx_map() with the key that is duplicated, to > + retrieve the offending row. > */ > +int Write_record::locate_dup_record() > +{ > + handler *h= table->file; > + int error= 0; > + if (h->ha_table_flags() & HA_DUPLICATE_POS || h->lookup_errkey != (uint)-1) > + { > + DBUG_PRINT("info", ("Locating offending record using rnd_pos()")); > + > + error= h->ha_rnd_pos(table->record[1], h->dup_ref); > + if (unlikely(error)) > + { > + DBUG_PRINT("info", ("rnd_pos() returns error %d",error)); > + h->print_error(error, MYF(0)); > + } > + } > + else > + { > + DBUG_PRINT("info", > + ("Locating offending record using ha_index_read_idx_map")); > > + if (h->lookup_handler) > + h= h->lookup_handler; Why? > + > + error= h->extra(HA_EXTRA_FLUSH_CACHE); > + if (unlikely(error)) > + { > + DBUG_PRINT("info",("Error when setting HA_EXTRA_FLUSH_CACHE")); > + return my_errno; > + } > > - info->records++; > - save_read_set= table->read_set; > - save_write_set= table->write_set; > + if (unlikely(key == NULL)) > + { > + key= static_cast<uchar*>(thd->alloc(table->s->max_unique_length)); > + if (key == NULL) > + { > + DBUG_PRINT("info",("Can't allocate key buffer")); > + return ENOMEM; > + } > + } > > - DBUG_EXECUTE_IF("rpl_write_record_small_sleep_gtid_100_200", > + key_copy(key, table->record[0], table->key_info + key_nr, 0); > + > + error= h->ha_index_read_idx_map(table->record[1], key_nr, key, > + HA_WHOLE_KEY, HA_READ_KEY_EXACT); why did you revert commit 5e345281e3599c79 here? > + if (unlikely(error)) > { > - if (thd->rgi_slave && (thd->rgi_slave->current_gtid.seq_no == 100 || > - thd->rgi_slave->current_gtid.seq_no == 200)) > - my_sleep(20000); > - }); > - if (info->handle_duplicates == DUP_REPLACE || > - info->handle_duplicates == DUP_UPDATE) > + DBUG_PRINT("info", ("index_read_idx() returns %d", error)); > + h->print_error(error, MYF(0)); > + } > + } > + > + if (unlikely(error)) > + return error; > + > + if (table->vfield) > + { > + /* > + We have not yet called update_virtual_fields() > + in handler methods for the just read row in record[1]. > + */ > + table->move_fields(table->field, table->record[1], table->record[0]); > + error= table->update_virtual_fields(table->file, VCOL_UPDATE_FOR_REPLACE); > + table->move_fields(table->field, table->record[0], table->record[1]); > + } > + return error; > +} > + > +/** > + REPLACE is defined as either INSERT or DELETE + INSERT. If > + possible, we can replace it with an UPDATE, but that will not > + work on InnoDB if FOREIGN KEY checks are necessary. > + > + Suppose that we got the duplicate key to be a key that is not > + the last unique key for the table and we perform an update: > + then there might be another key for which the unique check will > + fail, so we're better off just deleting the row and trying to insert again. > + > + Additionally we don't use ha_update_row in following cases: > + * when triggers should be invoked, as it may spoil the intermedite NEW_ROW > + representation > + * when we have referenced keys, as there can be different ON DELETE/ON UPDATE > + actions > +*/ > +int Write_record::replace_row(ha_rows *inserted, ha_rows *deleted) > +{ > + int error; > + while (unlikely(error=table->file->ha_write_row(table->record[0]))) > { > - while (unlikely(error=table->file->ha_write_row(table->record[0]))) > + error= prepare_handle_duplicate(error); > + if (unlikely(error)) > + return restore_on_error(); > + > + if (incomplete_records_cb && (error= incomplete_records_cb(arg1, arg2))) This callback is strange. I'd rather inherit Write_record_rpl from Write_record and put rpl-specific logic there. Also you could avoid a switch on INSERT/REPLACE/ODKU if you'd use inheritance instead. But the switch is less strange than the callback :) > + return restore_on_error(); > + > + if (can_optimize && key_nr == last_unique_key) > { > - uint key_nr; > - /* > - If we do more than one iteration of this loop, from the second one the > - row will have an explicit value in the autoinc field, which was set at > - the first call of handler::update_auto_increment(). So we must save > - the autogenerated value to avoid thd->insert_id_for_cur_row to become > - 0. > - */ > - if (table->file->insert_id_for_cur_row > 0) > - insert_id_for_cur_row= table->file->insert_id_for_cur_row; > - else > - table->file->insert_id_for_cur_row= insert_id_for_cur_row; > - bool is_duplicate_key_error; > - if (table->file->is_fatal_error(error, HA_CHECK_ALL)) > - goto err; > - is_duplicate_key_error= > - table->file->is_fatal_error(error, HA_CHECK_ALL & ~HA_CHECK_DUP); > - if (!is_duplicate_key_error) > - { > - /* > - We come here when we had an ignorable error which is not a duplicate > - key error. In this we ignore error if ignore flag is set, otherwise > - report error as usual. We will not do any duplicate key processing. > - */ > - if (info->ignore) > - { > - table->file->print_error(error, MYF(ME_WARNING)); > - goto after_trg_or_ignored_err; /* Ignoring a not fatal error */ > - } > - goto err; > - } > - if (unlikely((int) (key_nr = table->file->get_dup_key(error)) < 0)) > - { > - error= HA_ERR_FOUND_DUPP_KEY; /* Database can't find key */ > - goto err; > - } > - DEBUG_SYNC(thd, "write_row_replace"); > + DBUG_PRINT("info", ("Updating row using ha_update_row()")); > + if (table->versioned(VERS_TRX_ID)) > + table->vers_start_field()->store(0, false); > > - /* Read all columns for the row we are going to replace */ > - table->use_all_columns(); > - /* > - Don't allow REPLACE to replace a row when a auto_increment column > - was used. This ensures that we don't get a problem when the > - whole range of the key has been used. > - */ > - if (info->handle_duplicates == DUP_REPLACE && table->next_number_field && > - key_nr == table->s->next_number_index && insert_id_for_cur_row > 0) > - goto err; > - if (table->file->has_dup_ref()) > + error= table->file->ha_update_row(table->record[1], table->record[0]); > + > + if (likely(!error)) > + ++*deleted; > + else if (error != HA_ERR_RECORD_IS_THE_SAME) > + return on_ha_error(error); > + > + if (versioned) > { > - DBUG_ASSERT(table->file->inited == handler::RND); > - if (table->file->ha_rnd_pos(table->record[1],table->file->dup_ref)) > - goto err; > + store_record(table, record[2]); > + error= vers_insert_history_row(table); > + restore_record(table, record[2]); > + if (unlikely(error)) > + return on_ha_error(error); > } > + > + break; > + } > + else > + { > + DBUG_PRINT("info", ("Deleting offending row and trying to write" > + " new one again")); > + > + auto *trg = table->triggers; > + if (use_triggers && trg->process_triggers(table->in_use, TRG_EVENT_DELETE, > + TRG_ACTION_BEFORE, TRUE)) > + return restore_on_error(); > + if (!versioned) > + error = table->file->ha_delete_row(table->record[1]); > else > { > - if (table->file->extra(HA_EXTRA_FLUSH_CACHE)) /* Not needed with NISAM */ > - { > - error=my_errno; > - goto err; > - } > - > - if (!key) > - { > - if (!(key=(char*) my_safe_alloca(table->s->max_unique_length))) > - { > - error=ENOMEM; > - goto err; > - } > - } > - key_copy((uchar*) key,table->record[0],table->key_info+key_nr,0); > - key_part_map keypart_map= (1 << table->key_info[key_nr].user_defined_key_parts) - 1; > - if ((error= (table->file->ha_index_read_idx_map(table->record[1], > - key_nr, (uchar*) key, > - keypart_map, > - HA_READ_KEY_EXACT)))) > - goto err; > + store_record(table, record[2]); > + restore_record(table, record[1]); > + table->vers_update_end(); > + error = table->file->ha_update_row(table->record[1], table->record[0]); > + restore_record(table, record[2]); > } > - if (table->vfield) > + > + if (unlikely(error)) > + return on_ha_error(error); > + ++*deleted; > + > + if (use_triggers) > { > - /* > - We have not yet called update_virtual_fields(VOL_UPDATE_FOR_READ) > - in handler methods for the just read row in record[1]. > - */ > - table->move_fields(table->field, table->record[1], table->record[0]); > - int verr = table->update_virtual_fields(table->file, VCOL_UPDATE_FOR_REPLACE); > - table->move_fields(table->field, table->record[0], table->record[1]); > - if (verr) > - goto err; > + error= trg->process_triggers(table->in_use, TRG_EVENT_DELETE, > + TRG_ACTION_AFTER, TRUE); > + if (unlikely(error)) > + return restore_on_error(); > } > - if (info->handle_duplicates == DUP_UPDATE) > - { > - int res= 0; > - /* > - We don't check for other UNIQUE keys - the first row > - that matches, is updated. If update causes a conflict again, > - an error is returned > - */ > - DBUG_ASSERT(table->insert_values != NULL); > - store_record(table,insert_values); > - restore_record(table,record[1]); > - table->reset_default_fields(); > + } > + } > > - /* > - in INSERT ... ON DUPLICATE KEY UPDATE the set of modified fields can > - change per row. Thus, we have to do reset_default_fields() per row. > - Twice (before insert and before update). > - */ > - DBUG_ASSERT(info->update_fields->elements == > - info->update_values->elements); > - if (fill_record_n_invoke_before_triggers(thd, table, > - *info->update_fields, > - *info->update_values, > - info->ignore, > - TRG_EVENT_UPDATE)) > - goto before_trg_err; > - > - bool different_records= (!records_are_comparable(table) || > - compare_record(table)); > - /* > - Default fields must be updated before checking view updateability. > - This branch of INSERT is executed only when a UNIQUE key was violated > - with the ON DUPLICATE KEY UPDATE option. In this case the INSERT > - operation is transformed to an UPDATE, and the default fields must > - be updated as if this is an UPDATE. > - */ > - if (different_records && table->default_field) > - table->evaluate_update_default_function(); > - > - /* CHECK OPTION for VIEW ... ON DUPLICATE KEY UPDATE ... */ > - res= info->table_list->view_check_option(table->in_use, info->ignore); > - if (res == VIEW_CHECK_SKIP) > - goto after_trg_or_ignored_err; > - if (res == VIEW_CHECK_ERROR) > - goto before_trg_err; > - > - table->file->restore_auto_increment(prev_insert_id); > - info->touched++; > - if (different_records) > - { > - if (unlikely(error=table->file->ha_update_row(table->record[1], > - table->record[0])) && > - error != HA_ERR_RECORD_IS_THE_SAME) > - { > - if (info->ignore && > - !table->file->is_fatal_error(error, HA_CHECK_ALL)) > - { > - if (!(thd->variables.old_behavior & > - OLD_MODE_NO_DUP_KEY_WARNINGS_WITH_IGNORE)) > - table->file->print_error(error, MYF(ME_WARNING)); > - goto after_trg_or_ignored_err; > - } > - goto err; > - } > + /* > + If more than one iteration of the above loop is done, from > + the second one the row being inserted will have an explicit > + value in the autoinc field, which was set at the first call of > + handler::update_auto_increment(). This value is saved to avoid > + thd->insert_id_for_cur_row becoming 0. Use this saved autoinc > + value. > + */ > + if (table->file->insert_id_for_cur_row == 0) > + table->file->insert_id_for_cur_row = insert_id_for_cur_row; > > - if (error != HA_ERR_RECORD_IS_THE_SAME) > - { > - info->updated++; > - if (table->versioned() && > - table->vers_check_update(*info->update_fields)) > - { > - if (table->versioned(VERS_TIMESTAMP)) > - { > - store_record(table, record[2]); > - if ((error= vers_insert_history_row(table))) > - { > - info->last_errno= error; > - table->file->print_error(error, MYF(0)); > - trg_error= 1; > - restore_record(table, record[2]); > - goto after_trg_or_ignored_err; > - } > - restore_record(table, record[2]); > - } > - info->copied++; > - } > - } > - else > - error= 0; > - /* > - If ON DUP KEY UPDATE updates a row instead of inserting > - one, it's like a regular UPDATE statement: it should not > - affect the value of a next SELECT LAST_INSERT_ID() or > - mysql_insert_id(). Except if LAST_INSERT_ID(#) was in the > - INSERT query, which is handled separately by > - THD::arg_of_last_insert_id_function. > - */ > - prev_insert_id_for_cur_row= table->file->insert_id_for_cur_row; > - insert_id_for_cur_row= table->file->insert_id_for_cur_row= 0; > - trg_error= (table->triggers && > - table->triggers->process_triggers(thd, TRG_EVENT_UPDATE, > - TRG_ACTION_AFTER, TRUE)); > - info->copied++; > - } > + return after_insert(inserted); > +} > > - /* > - Only update next_insert_id if the AUTO_INCREMENT value was explicitly > - updated, so we don't update next_insert_id with the value from the > - row being updated. Otherwise reset next_insert_id to what it was > - before the duplicate key error, since that value is unused. > - */ > - if (table->next_number_field_updated) > - { > - DBUG_ASSERT(table->next_number_field != NULL); > > - table->file->adjust_next_insert_id_after_explicit_value(table->next_number_field->val_int()); > - } > - else if (prev_insert_id_for_cur_row) > - { > - table->file->restore_auto_increment(prev_insert_id_for_cur_row); > - } > - goto ok; > +int Write_record::insert_on_duplicate_update(ha_rows *inserted, > + ha_rows *updated) > +{ > + int res= table->file->ha_write_row(table->record[0]); > + if (likely(!res)) > + return after_insert(inserted); > + > + res=prepare_handle_duplicate(res); > + if (unlikely(res)) > + return restore_on_error(); > + > + ulonglong prev_insert_id_for_cur_row= 0; > + /* > + We don't check for other UNIQUE keys - the first row > + that matches, is updated. If update causes a conflict again, > + an error is returned > + */ > + DBUG_ASSERT(table->insert_values != NULL); > + store_record(table,insert_values); > + restore_record(table,record[1]); > + table->reset_default_fields(); > + > + /* > + in INSERT ... ON DUPLICATE KEY UPDATE the set of modified fields can > + change per row. Thus, we have to do reset_default_fields() per row. > + Twice (before insert and before update). > + */ > + DBUG_ASSERT(info->update_fields->elements == > + info->update_values->elements); > + if (fill_record_n_invoke_before_triggers(thd, table, > + *info->update_fields, > + *info->update_values, > + info->ignore, > + TRG_EVENT_UPDATE)) > + return restore_on_error(); > + > + bool different_records= (!records_are_comparable(table) || > + compare_record(table)); > + /* > + Default fields must be updated before checking view updateability. > + This branch of INSERT is executed only when a UNIQUE key was violated > + with the ON DUPLICATE KEY UPDATE option. In this case the INSERT > + operation is transformed to an UPDATE, and the default fields must > + be updated as if this is an UPDATE. > + */ > + if (different_records && table->default_field) > + table->evaluate_update_default_function(); > + > + /* CHECK OPTION for VIEW ... ON DUPLICATE KEY UPDATE ... */ > + res= info->table_list->view_check_option(table->in_use, info->ignore); > + if (unlikely(res)) > + { > + if (res == VIEW_CHECK_ERROR) > + return restore_on_error(); > + DBUG_ASSERT(res == VIEW_CHECK_SKIP); > + return 0; > + } > + > + table->file->restore_auto_increment(prev_insert_id); > + info->touched++; > + if (different_records) > + { > + int error= table->file->ha_update_row(table->record[1], table->record[0]); > + > + if (unlikely(error) && error != HA_ERR_RECORD_IS_THE_SAME) > + { > + if (info->ignore && !table->file->is_fatal_error(error, HA_CHECK_ALL)) why not is_fatal_error(error) ? > + { > + if (!(thd->variables.old_behavior & > + OLD_MODE_NO_DUP_KEY_WARNINGS_WITH_IGNORE)) > + table->file->print_error(error, MYF(ME_WARNING)); > + return 0; > } > - else /* DUP_REPLACE */ > + return on_ha_error(error); > + } > + > + if (error != HA_ERR_RECORD_IS_THE_SAME) > + { > + ++*updated; > + if (table->versioned() && > + table->vers_check_update(*info->update_fields)) > { > - /* > - The manual defines the REPLACE semantics that it is either > - an INSERT or DELETE(s) + INSERT; FOREIGN KEY checks in > - InnoDB do not function in the defined way if we allow MySQL > - to convert the latter operation internally to an UPDATE. > - We also should not perform this conversion if we have > - timestamp field with ON UPDATE which is different from DEFAULT. > - Another case when conversion should not be performed is when > - we have ON DELETE trigger on table so user may notice that > - we cheat here. Note that it is ok to do such conversion for > - tables which have ON UPDATE but have no ON DELETE triggers, > - we just should not expose this fact to users by invoking > - ON UPDATE triggers. > - > - Note, TABLE_SHARE and TABLE see long uniques differently: > - - TABLE_SHARE sees as HA_KEY_ALG_LONG_HASH and HA_NOSAME > - - TABLE sees as usual non-unique indexes > - */ > - bool is_long_unique= table->s->key_info && > - table->s->key_info[key_nr].algorithm == > - HA_KEY_ALG_LONG_HASH; > - if ((is_long_unique ? > - /* > - We have a long unique. Test that there are no in-engine > - uniques and the current long unique is the last long unique. > - */ > - !(table->key_info[0].flags & HA_NOSAME) && > - last_uniq_key(table, table->s->key_info, key_nr) : > - /* > - We have a normal key - not a long unique. > - Test is the current normal key is unique and > - it is the last normal unique. > - */ > - last_uniq_key(table, table->key_info, key_nr)) && > - !table->file->referenced_by_foreign_key() && > - (!table->triggers || !table->triggers->has_delete_triggers())) > - { > - if (table->versioned(VERS_TRX_ID)) > - { > - bitmap_set_bit(table->write_set, table->vers_start_field()->field_index); > - table->file->column_bitmaps_signal(); > - table->vers_start_field()->store(0, false); > - } > - if (unlikely(error= table->file->ha_update_row(table->record[1], > - table->record[0])) && > - error != HA_ERR_RECORD_IS_THE_SAME) > - goto err; > - if (likely(!error)) > - { > - info->deleted++; > - if (!table->file->has_transactions()) > - thd->transaction->stmt.modified_non_trans_table= TRUE; > - if (table->versioned(VERS_TIMESTAMP)) > - { > - store_record(table, record[2]); > - error= vers_insert_history_row(table); > - restore_record(table, record[2]); > - if (unlikely(error)) > - goto err; > - } > - } > - else > - error= 0; // error was HA_ERR_RECORD_IS_THE_SAME > - /* > - Since we pretend that we have done insert we should call > - its after triggers. > - */ > - goto after_trg_n_copied_inc; > - } > - else > + if (versioned) > { > - if (table->triggers && > - table->triggers->process_triggers(thd, TRG_EVENT_DELETE, > - TRG_ACTION_BEFORE, TRUE)) > - goto before_trg_err; > - > - if (!table->versioned(VERS_TIMESTAMP)) > - error= table->file->ha_delete_row(table->record[1]); > - else > - { > - store_record(table, record[2]); > - restore_record(table, record[1]); > - table->vers_update_end(); > - error= table->file->ha_update_row(table->record[1], > - table->record[0]); > - restore_record(table, record[2]); > - } > + store_record(table, record[2]); > + error= vers_insert_history_row(table); > + restore_record(table, record[2]); > if (unlikely(error)) > - goto err; > - if (!table->versioned(VERS_TIMESTAMP)) > - info->deleted++; > - else > - info->updated++; > - if (!table->file->has_transactions_and_rollback()) > - thd->transaction->stmt.modified_non_trans_table= TRUE; > - if (table->triggers && > - table->triggers->process_triggers(thd, TRG_EVENT_DELETE, > - TRG_ACTION_AFTER, TRUE)) > - { > - trg_error= 1; > - goto after_trg_or_ignored_err; > - } > - /* Let us attempt do write_row() once more */ > + return on_ha_error(error); > } > + ++*inserted; > } > } > - > - /* > - If more than one iteration of the above while loop is done, from > - the second one the row being inserted will have an explicit > - value in the autoinc field, which was set at the first call of > - handler::update_auto_increment(). This value is saved to avoid > - thd->insert_id_for_cur_row becoming 0. Use this saved autoinc > - value. > - */ > - if (table->file->insert_id_for_cur_row == 0) > - table->file->insert_id_for_cur_row= insert_id_for_cur_row; > - > /* > - Restore column maps if they where replaced during an duplicate key > - problem. > + If ON DUP KEY UPDATE updates a row instead of inserting > + one, it's like a regular UPDATE statement: it should not > + affect the value of a next SELECT LAST_INSERT_ID() or > + mysql_insert_id(). Except if LAST_INSERT_ID(#) was in the > + INSERT query, which is handled separately by > + THD::arg_of_last_insert_id_function. > */ > - if (table->read_set != save_read_set || > - table->write_set != save_write_set) > - table->column_bitmaps_set(save_read_set, save_write_set); > + prev_insert_id_for_cur_row= table->file->insert_id_for_cur_row; > + insert_id_for_cur_row= table->file->insert_id_for_cur_row= 0; > + > + ++*inserted; // Conforms the older behavior; > + > + if (use_triggers > + && table->triggers->process_triggers(thd, TRG_EVENT_UPDATE, > + TRG_ACTION_AFTER, TRUE)) > + return restore_on_error(); > } > - else if (unlikely((error=table->file->ha_write_row(table->record[0])))) > + > + /* > + Only update next_insert_id if the AUTO_INCREMENT value was explicitly > + updated, so we don't update next_insert_id with the value from the > + row being updated. Otherwise reset next_insert_id to what it was > + before the duplicate key error, since that value is unused. > + */ > + if (table->next_number_field_updated) > + { > + DBUG_ASSERT(table->next_number_field != NULL); > + > + table->file->adjust_next_insert_id_after_explicit_value( > + table->next_number_field->val_int()); > + } > + else if (prev_insert_id_for_cur_row) > + { > + table->file->restore_auto_increment(prev_insert_id_for_cur_row); > + } > + > + return send_data(); > +} > + > +bool Write_record::is_fatal_error(int error) > +{ > + return error == HA_ERR_LOCK_DEADLOCK || > + error == HA_ERR_LOCK_WAIT_TIMEOUT || These errors were only in rpl code, not anywhere in sql_insert.cc. I don't think it's wrong to check for them everywhere consistently, but does it change the behavior? Can you create a test case for INSERT before your changes that comes to thd->is_fatal_error() with HA_ERR_LOCK_DEADLOCK or HA_ERR_LOCK_WAIT_TIMEOUT? What will be the behavior before and after your change? > + table->file->is_fatal_error(error, HA_CHECK_ALL); > +} > + > +int Write_record::single_insert(ha_rows *inserted) > +{ > + DBUG_EXECUTE_IF("rpl_write_record_small_sleep_gtid_100_200", > + { > + uint64 seq= thd->rgi_slave ? thd->rgi_slave->current_gtid.seq_no : 0; > + if (seq == 100 || seq == 200) > + my_sleep(20000); > + }); > + > + int error= table->file->ha_write_row(table->record[0]); > + if (unlikely(error)) > { > DEBUG_SYNC(thd, "write_row_noreplace"); > - if (!info->ignore || > - table->file->is_fatal_error(error, HA_CHECK_ALL)) > - goto err; > + if (!info->ignore || is_fatal_error(error)) > + return on_ha_error(error); > if (!(thd->variables.old_behavior & > OLD_MODE_NO_DUP_KEY_WARNINGS_WITH_IGNORE)) > table->file->print_error(error, MYF(ME_WARNING)); > table->file->restore_auto_increment(prev_insert_id); > - goto after_trg_or_ignored_err; > + return 0; > } > + return after_insert(inserted); > +} > > -after_trg_n_copied_inc: > - info->copied++; > - thd->record_first_successful_insert_id_in_cur_stmt(table->file->insert_id_for_cur_row); > - trg_error= (table->triggers && > - table->triggers->process_triggers(thd, TRG_EVENT_INSERT, > - TRG_ACTION_AFTER, TRUE)); > +/** > + Write a record to table with optional deleting of conflicting records, > + invoke proper triggers if needed. > + > + @note > + Once this record will be written to table after insert trigger will > + be invoked. If instead of inserting new record we will update old one > + then both on update triggers will work instead. Similarly both on > + delete triggers will be invoked if we will delete conflicting records. > > -ok: > + Sets thd->transaction.stmt.modified_non_trans_table to TRUE if table which > + is updated didn't have transactions. > + > + @return > + 0 - success > + non-0 - error > +*/ > +int Write_record::write_record() > +{ > + ha_rows inserted= 0, updated= 0; > + prev_insert_id= table->file->next_insert_id; > + info->records++; > + ignored_error= false; > + int res= 0; > + switch(info->handle_duplicates) > + { > + case DUP_ERROR: > + res= single_insert(&inserted); > + break; > + case DUP_UPDATE: > + res= insert_on_duplicate_update(&inserted, &updated); > + info->updated+= updated; > + break; > + case DUP_REPLACE: > + res= replace_row(&inserted, &updated); > + if (versioned) > + info->updated+= updated; > + else > + info->deleted+= updated; > + } > + > + info->copied+= inserted; > + if (inserted || updated) > + notify_non_trans_table_modified(); > + return res; > +} > + > + > +int Write_record::prepare_handle_duplicate(int error) > +{ > /* > - We send the row after writing it to the table so that the > - correct values are sent to the client. Otherwise it won't show > - autoinc values (generated inside the handler::ha_write()) and > - values updated in ON DUPLICATE KEY UPDATE. > + If we do more than one iteration of the replace loop, from the second one the > + row will have an explicit value in the autoinc field, which was set at > + the first call of handler::update_auto_increment(). So we must save > + the autogenerated value to avoid thd->insert_id_for_cur_row to become > + 0. > */ > - if (sink && sink->send_data(thd->lex->returning()->item_list) < 0) > - trg_error= 1; > + if (table->file->insert_id_for_cur_row > 0) > + insert_id_for_cur_row= table->file->insert_id_for_cur_row; > + else > + table->file->insert_id_for_cur_row= insert_id_for_cur_row; > + > + if (is_fatal_error(error)) > + return on_ha_error(error); > > -after_trg_or_ignored_err: > - if (key) > - my_safe_afree(key,table->s->max_unique_length); > + bool is_duplicate_key_error= > + table->file->is_fatal_error(error, HA_CHECK_ALL & ~HA_CHECK_DUP); > + if (!is_duplicate_key_error) > + { > + /* > + We come here when we had an ignorable error which is not a duplicate > + key error. In this we ignore error if ignore flag is set, otherwise > + report error as usual. We will not do any duplicate key processing. > + */ > + if (info->ignore) > + { > + table->file->print_error(error, MYF(ME_WARNING)); > + ignored_error= true; > + return 1; /* Ignoring a not fatal error */ > + } > + return on_ha_error(error); > + } > + > + key_nr = table->file->get_dup_key(error); > + > + if (unlikely(key_nr == std::numeric_limits<decltype(key_nr)>::max())) Well, the handler returns (uint)-1, please compare with it, instead of inventing numerous other ways of archieving the same. And why did you decide to use ushort for the key number? We pretty much never use ushort, key number is uint everywhere. I agree that ushort is enough size-wise, but let's rather be consistent. > + return on_ha_error(HA_ERR_FOUND_DUPP_KEY); /* Database can't find key */ > + > + DEBUG_SYNC(thd, "write_row_replace"); > + > + /* Read all columns for the row we are going to replace */ > + table->use_all_columns(); > + /* > + Don't allow REPLACE to replace a row when an auto_increment column > + was used. This ensures that we don't get a problem when the > + whole range of the key has been used. > + */ > + if (info->handle_duplicates == DUP_REPLACE && table->next_number_field && > + key_nr == table->s->next_number_index && insert_id_for_cur_row > 0) > + return on_ha_error(error); > + > + error= locate_dup_record(); > + if (error) > + return on_ha_error(error); > + > + return 0; > +} > + > +inline > +void Write_record::notify_non_trans_table_modified() > +{ > if (!table->file->has_transactions_and_rollback()) > thd->transaction->stmt.modified_non_trans_table= TRUE; ok, as you like. I'm personally not a great fan of one-liners that are used only once. > - DBUG_RETURN(trg_error); > +} > > -err: > +inline > +int Write_record::on_ha_error(int error) > +{ > info->last_errno= error; > table->file->print_error(error,MYF(0)); > - > -before_trg_err: > + DBUG_PRINT("info", ("Returning error %d", error)); > + return restore_on_error(); > +} > + > +inline > +int Write_record::restore_on_error() > +{ > table->file->restore_auto_increment(prev_insert_id); > - if (key) > - my_safe_afree(key, table->s->max_unique_length); > - table->column_bitmaps_set(save_read_set, save_write_set); > - DBUG_RETURN(1); > + return ignored_error ? 0 : 1; > +} > + > +inline > +int Write_record::after_insert(ha_rows *inserted) > +{ > + ++*inserted; > + thd->record_first_successful_insert_id_in_cur_stmt(table->file->insert_id_for_cur_row); > + return after_ins_trg() || send_data(); > +} > + > +inline > +int Write_record::after_ins_trg() > +{ > + int res= 0; > + if (use_triggers) > + res= table->triggers->process_triggers(thd, TRG_EVENT_INSERT, > + TRG_ACTION_AFTER, TRUE); > + return res; > +} > + > +inline > +int Write_record::send_data() > +{ > + /* > + We send the row after writing it to the table so that the > + correct values are sent to the client. Otherwise it won't show > + autoinc values (generated inside the handler::ha_write()) and > + values updated in ON DUPLICATE KEY UPDATE. > + */ > + return sink ? sink->send_data(thd->lex->returning()->item_list) < 0 : 0; > } > > > + > /****************************************************************************** > Check that there aren't any null_fields > ******************************************************************************/ > @@ -4816,18 +4934,8 @@ select_create::prepare(List<Item> &_values, SELECT_LEX_UNIT *u) > > restore_record(table,s->default_values); // Get empty record > thd->cuted_fields=0; > - bool create_lookup_handler= info.handle_duplicates != DUP_ERROR; > - if (info.ignore || info.handle_duplicates != DUP_ERROR) > - { > - create_lookup_handler= true; > - table->file->extra(HA_EXTRA_IGNORE_DUP_KEY); > - if (table->file->ha_table_flags() & HA_DUPLICATE_POS) > - { > - if (table->file->ha_rnd_init_with_error(0)) > - DBUG_RETURN(1); > - } > - } > - table->file->prepare_for_insert(create_lookup_handler); > + if (prepare_for_replace(table, info.handle_duplicates, info.ignore)) ok. strange that select_create::prepare doesn't call select_insert::prepare > + DBUG_RETURN(1); > if (info.handle_duplicates == DUP_REPLACE && > (!table->triggers || !table->triggers->has_delete_triggers())) > table->file->extra(HA_EXTRA_WRITE_CAN_REPLACE); > diff --git a/sql/sql_parse.cc b/sql/sql_parse.cc > index 5db53b6f2b8..7ce4850f534 100644 > --- a/sql/sql_parse.cc > +++ b/sql/sql_parse.cc > @@ -4814,7 +4815,8 @@ mysql_execute_command(THD *thd) > &lex->value_list, > lex->duplicates, > lex->ignore, > - result))) > + result, > + &write))) Why do you need Write_record here on the stack? Wouldn't it be more logical to have it in select_insert ? > { > if (lex->analyze_stmt) > ((select_result_interceptor*)sel_result)->disable_my_ok_calls(); Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 1

Re: 652e9ee405e: MDEV-30046 wrong row targeted with "insert ... on duplicate" and "replace"
by Sergei Golubchik 29 Apr '24

29 Apr '24

Hi, Nikita, Good fix. Ok to push, thanks On Apr 29, Nikita Malyavin wrote: > revision-id: 652e9ee405e (mariadb-10.5.24-174-g652e9ee405e) > parent(s): 3d417476256 > author: Nikita Malyavin > committer: Nikita Malyavin > timestamp: 2024-04-27 21:08:21 +0200 > message: > > MDEV-30046 wrong row targeted with "insert ... on duplicate" and "replace" > > When HA_DUPLICATE_POS is not supported, the row to replace was navigated by > ha_index_read_idx_map, which uses only hash to navigate. > > Suchwise, given a hash collision it may choose an incorrect row. > > handler::position would be correct and very convenient to use here. > The code is updated to set dup_ref by a handler independently of engine > capabilities, when extra lookup is made (for long unique or something else, > for example WITHOUT OVERLAPS) > Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

1 1

SOUNDEX sources
by lucyfrost＠tutanota.com 25 Apr '24

25 Apr '24

I'm developing a project which uses soundex matching (alongside other options more sophisticated than that). This is custom software with a mariad back end. MariaDB soundex built-in produces codes of apparently arbitary length; the usual is 4 characters. I've come to the conclusion that the best way forward is to mimic Maria's soundex in my code. I cannot find a description of the algorithm Maria uses though: the only source code I can find is in mf_soundex.c, and that is explicitly limited to 4 characters. But I have installed triggers which generate codes such as A4152636125262 Where is the soundex function found? Anyone know?

3 4

3 PRs pending second approver
by Otto Kekäläinen 25 Apr '24

25 Apr '24

Hi all with permission to review / add approval, This is ready to be merged into 11.5 and already has approval from Andrew, just needs one more approval: https://github.com/MariaDB/server/pull/2671 This has also one approval from Andrew, pending a second to be merged: https://github.com/MariaDB/server/pull/2169 Likewise, approved by Andrew, pending second: https://github.com/MariaDB/server/pull/2516 Additionally, this has been open for since January without any reviews: https://github.com/MariaDB/server/pull/2985 This was updated a month ago based on feedback, but hasn't gotten a re-review since: https://github.com/MariaDB/server/pull/3114 Pending re-review: https://github.com/MariaDB/server/pull/2556 - Otto

3 2

Re: [MariaDB commits] [PATCH] MDEV-33602: Sporadic test failure in rpl.rpl_gtid_stop_start
by Kristian Nielsen 25 Apr '24

25 Apr '24

Andrei Elkin <andrei.elkin(a)mariadb.com> writes: >>> --- a/sql/rpl_rli.cc >>> +++ b/sql/rpl_rli.cc >>> @@ -1030,7 +1030,7 @@ void Relay_log_info::inc_group_relay_log_pos(ulonglong log_pos, >>> rgi->last_master_timestamp > last_master_timestamp) >>> last_master_timestamp= rgi->last_master_timestamp; >>> } >>> - else >>> + else if (!mi->using_gtid || seen_gtid_log_list_event) >>> { >>> /* Non-parallel case. */ >>> group_relay_log_pos= event_relay_log_pos; >> >> Agree, this looks fine. >> The dump thread should always be sending the gtid_list event. And yes, the >> log_pos of the events received until then are not useful to set for the >> slave. >> >> Right, so this seems a good approach for a fix. >> >> Maybe I should push the workaround to 10.5 to help the effort to remove >> sporadic failures, and then file a separate bug about the underlying problem >> and your proposed solution? > > Sounds reasonable. Thank you! Filed as https://jira.mariadb.org/browse/MDEV-33996 - Kristian.

1 0

Re: [MariaDB commits] [PATCH] MDEV-33602: Sporadic test failure in rpl.rpl_gtid_stop_start
by Kristian Nielsen 24 Apr '24

24 Apr '24

Andrei Elkin <andrei.elkin(a)mariadb.com> writes: >> 1. I have not so far been able to reproduce the failure, so this patch is >> assumed to fix the sporadic failure, but I have not been able to verify that >> it does. > > I verified your analysis to confirm. Thanks for checking! >> But on the other hand this will not solve the fundamental issue, that until the >> GTID mode connect is completed, we will in the general case not have a valid >> corresponding old-style position. So maybe the current behaviour is >> ok? > Sure but it looks fixes are not overly difficult. > > As it's Gtid_list_log_event::log_pos that makes the file:pos (old-style coordinates) > state be valid why won't be keep the coordinates intact until the event > has been arrived/processed? Say that situation is remembered in `RLI::seen_gtid_log_list_event`. > Then for instance in the serial case the fixes would look like > > --- a/sql/rpl_rli.cc > +++ b/sql/rpl_rli.cc > @@ -1030,7 +1030,7 @@ void Relay_log_info::inc_group_relay_log_pos(ulonglong log_pos, > rgi->last_master_timestamp > last_master_timestamp) > last_master_timestamp= rgi->last_master_timestamp; > } > - else > + else if (!mi->using_gtid || seen_gtid_log_list_event) > { > /* Non-parallel case. */ > group_relay_log_pos= event_relay_log_pos; Agree, this looks fine. The dump thread should always be sending the gtid_list event. And yes, the log_pos of the events received until then are not useful to set for the slave. Right, so this seems a good approach for a fix. Maybe I should push the workaround to 10.5 to help the effort to remove sporadic failures, and then file a separate bug about the underlying problem and your proposed solution? Thanks, - Kristian.

1 0

Re: [MariaDB commits] [PATCH] MDEV-33602: Sporadic test failure in rpl.rpl_gtid_stop_start
by Kristian Nielsen 23 Apr '24

23 Apr '24

Hi Andrei, Brandon, What do you think of this patch to fix sporadic failure of rpl.rpl_gtid_stop_start in buildbot, MDEV-33602? The test fails relatively frequent, so would be good to get it fixed. Two issues: 1. I have not so far been able to reproduce the failure, so this patch is assumed to fix the sporadic failure, but I have not been able to verify that it does. 2. The question is if this will be just hiding a real bug? When the GTID-mode slave connects, it receives a few events from the start of the binlog (Gtid_list, Binlog_checkpoint, etc), and these briefly update the old-style position to a wrong value until the fake Gtid_list is received with the correct position. One might argue that this should be avoided. But on the other hand this will not solve the fundamental issue, that until the GTID mode connect is completed, we will in the general case not have a valid corresponding old-style position. So maybe the current behaviour is ok? - Kristian. Kristian Nielsen via commits <commits(a)lists.mariadb.org> writes: > The test could fail with a duplicate key error because switching to non-GTID > mode could start at the wrong old-style position. The position could be > wrong when the previous GTID connect was stopped before receiving the fake > GTID list event which gives the old-style position corresponding to the GTID > connected position. > > Work-around by injecting an extra event and syncing the slave before > switching to non-GTID mode. > > Signed-off-by: Kristian Nielsen <knielsen(a)knielsen-hq.org> > --- > .../suite/rpl/r/rpl_gtid_stop_start.result | 4 ++++ > mysql-test/suite/rpl/t/rpl_gtid_stop_start.test | 16 ++++++++++++++++ > 2 files changed, 20 insertions(+) > > diff --git a/mysql-test/suite/rpl/r/rpl_gtid_stop_start.result b/mysql-test/suite/rpl/r/rpl_gtid_stop_start.result > index ae0050c353a..e8633cd45bb 100644 > --- a/mysql-test/suite/rpl/r/rpl_gtid_stop_start.result > +++ b/mysql-test/suite/rpl/r/rpl_gtid_stop_start.result > @@ -111,6 +111,10 @@ a > 6 > 7 > *** MDEV-4486: Allow to start old-style replication even if mysql.gtid_slave_pos is unavailable > +connection server_1; > +INSERT INTO t1 VALUES (8); > +DELETE FROM t1 WHERE a=8; > +connection server_2; > connection server_2; > include/stop_slave.inc > CHANGE MASTER TO master_use_gtid= no; > diff --git a/mysql-test/suite/rpl/t/rpl_gtid_stop_start.test b/mysql-test/suite/rpl/t/rpl_gtid_stop_start.test > index b5ff294908b..032ebb77d1e 100644 > --- a/mysql-test/suite/rpl/t/rpl_gtid_stop_start.test > +++ b/mysql-test/suite/rpl/t/rpl_gtid_stop_start.test > @@ -173,6 +173,22 @@ SELECT * FROM t1 ORDER BY a; > > --echo *** MDEV-4486: Allow to start old-style replication even if mysql.gtid_slave_pos is unavailable > > +# In GTID mode, the old-style replication position is also updated. But during > +# GTID connect, the old-style position is not known until receiving the fake > +# GTID list event, which contains the required position value. If we happened > +# to stop the slave above before this fake GTID list event, the test could fail > +# with duplicate key errors due to switching to non-GTID mode at a wrong > +# position too far back in the binlog. > +# > +# Work-around this by injecting an extra dummt event and syncing the slave to > +# it, ensuring the old-style position will be updated. > +--connection server_1 > +INSERT INTO t1 VALUES (8); > +DELETE FROM t1 WHERE a=8; > +--save_master_pos > +--connection server_2 > +--sync_with_master > + > --connection server_2 > --source include/stop_slave.inc > CHANGE MASTER TO master_use_gtid= no;

1 0

MDEV-27512, --slave-skip-errors, and optimistic parallel replication
by Kristian Nielsen 18 Apr '24

18 Apr '24

Hi Brandon, I only now got to check on MDEV-27512. This bug seems to be about using --slave_skip_errors=ALL with optimistic parallel replication. I saw your patch for the bug. While the patch by itself is perhaps ok, I think it's only fixing a minor thing, and not addressing the real problem, let me explain my thinking: Now, --slave-skip-errors is obviously a dangerous option, but presumably the use case is to make the slave continue at all costs in case an unexpected error occurs, rather than stop the slave and break replication. But optimistic parallel replication by design can introduce many different transient errors due to conflicts, that are then handled by rolling back and retrying the offending transactions. These errors are *expected*, and they do *not* cause slave to stop or replication to break. However, it appears that --slave_skip_errors also affects such transient errors due to optimistic parallel replication, and will cause any such transaction to be silently ignored! This must be very wrong, it will cause massive replication divergence. It seems to me this is the real bug here. When an error is encountered during optimistic apply of a transaction (is_parallel_retry_error() returns true, eg. rgi->speculation == SPECULATE_OPTIMISTIC, then this error should _not_ be subject to --slave-skip-errors. The transaction should be rolled back as normal, and wait_for_prior_commit() done. Then after setting rgi->speculation = SPECULATE_WAIT and retrying, if we still get the error, --slave-skip-errors can apply. I put together a quick test that seems to show this behaviour, included below. This tests replicates correctly without replication stopping with error. But running it with --mysqld=--slave-skip-errors=all, it replicates incorrectly, skipping lots of transactions. The test is somewhat contrieved, but I think it shows the real problem, that --slave-skip-errors can randomly cause transactions to be skipped or not depending on if optimistic parallel replication triggers a matching transient error or not. So in summary, it looks like there is a real problem here, that optimistic parallel replication is not working correctly with --slave-skip-errors, transient errors incorrectly causes conflicts to skip transactions rather than retrying them. This will cause replication to diverge even when no real errors occur. And the patch for MDEV-27512 doesn't seem to address this issue. So I think MDEV-27512 should be re-opened, what do you think? - Kristian. ----------------------------------------------------------------------- --source include/have_innodb.inc --source include/have_binlog_format_row.inc --source include/master-slave.inc --connection master ALTER TABLE mysql.gtid_slave_pos ENGINE=InnoDB; CREATE TABLE t1 (a INT PRIMARY KEY, b INT) ENGINE=InnoDB; INSERT INTO t1 VALUES (1,NULL), (2,2), (3,NULL), (4,4), (5, NULL), (6, 6); --sync_slave_with_master --source include/stop_slave.inc CHANGE MASTER TO master_use_gtid=slave_pos; SET @old_timeout= @@GLOBAL.innodb_lock_wait_timeout; SET GLOBAL innodb_lock_wait_timeout= 5; SET @old_parallel= @@GLOBAL.slave_parallel_threads; SET @old_mode= @@GLOBAL.slave_parallel_mode; SET GLOBAL slave_parallel_mode= aggressive; SET GLOBAL slave_parallel_threads= 20; --connection master UPDATE t1 SET b=b+1 WHERE a=6; --disable_query_log let $i= 0; while ($i < 40) { eval UPDATE t1 SET b=b+1 WHERE a=2; inc $i; } --enable_query_log SELECT * FROM t1 ORDER BY a; --save_master_pos --connection slave1 # Block first worker, and recursively pause all following workers that get # temporary errors before they can retry. BEGIN; SELECT * FROM t1 WHERE a=6 FOR UPDATE; --connection slave # Cause initial row not found error. SET STATEMENT sql_log_bin=0 FOR UPDATE t1 SET a=7 WHERE a=2; --source include/start_slave.inc --sleep 2 # Now following workers should be waiting for prior commit before retrying. # Remove the row not found error. SET STATEMENT sql_log_bin=0 FOR UPDATE t1 SET a=2 WHERE a=7; --connection slave1 ROLLBACK; --connection slave --sync_with_master SELECT * FROM t1 ORDER BY a; # Cleanup --connection slave --source include/stop_slave.inc SET GLOBAL innodb_lock_wait_timeout= @old_timeout; SET GLOBAL slave_parallel_threads= @old_parallel; SET GLOBAL slave_parallel_mode= @old_mode; --source include/start_slave.inc --connection default DROP TABLE t1; --source include/rpl_end.inc

2 2

Feedback plugin status?
by Otto Kekäläinen 16 Apr '24

16 Apr '24

Hi! I was about to merge a long overdue MR in Debian to enable the Feedback plugin (https://salsa.debian.org/mariadb-team/mariadb-server/-/merge_requests/16) However I noticed all the links at https://mariadb.org/about/#usage-statistics are broken. The wiki page at https://mariadb.com/kb/en/feedback-plugin/ says that the plugin is 'currently not working'. What does this mean? What is the path to get it working again? Is there a need for somebody to rewrite the stats server part? Thanks, - Otto

3 4

MariaDB/MySQL wire protocol ownership and MariaDB Connector C source code hosting?
by Otto Kekäläinen 13 Apr '24

13 Apr '24

Hi! The MariaDB wire protocol is an interplay between the MariaDB Server and the MariaDB Connector C. However, only the server is in the MariaDB Foundation's GitHub account at https://github.com/mariadb/server, and the Connector C is in the MariaDB PLC account at https://github.com/mariadb-corporation/mariadb-connector-c. Is there perhaps some plan to hand over the Connector C to the MariaDB Foundation? The reason I am asking is that the connector does not seem equally "true open source" as the server is, making changes that require coordination in both server and connector (e.g. https://github.com/mariadb-corporation/mariadb-connector-c/pull/227) kind of out-of-scope for the MariaDB Foundation, which seems counterproductive. If/when wire protocol changes might be made and changes to both server and libmariadb coordinated in an open "working group" like suggested in https://github.com/MariaDB/server/pull/2684, it might be very challenging for the Foundation or the open source community to drive it if the PLC staff member maintaining the connector does not participate. Also, seems the PR reviews at https://github.com/mariadb-corporation/mariadb-connector-c/pulls have been stalled since 2022. MariaDB Foundation staff are funded to facilitate reviews, but can't really be *responsible* for reviewing them as long as it is not a Foundation repository and review policies are not owned by the Foundation. It would just make a lot of sense if such a central piece of the MariaDB Server would be in the Foundation's GitHub account. The MariaDB Connector C / libmaraidb is already published and distributed by the MariadB Foundation, so moving the source code hosting over shouldn't big such a big thing, right? - Otto

3 2

7 PRs pending second approver - can somebody do it?
by Otto Kekäläinen 11 Apr '24

11 Apr '24

Hi Sergei, Andrew or anyone else who wants to help, These PRs have already been approved by one reviewer 2-8 weeks ago and are pending the second reviewer to approve/merge. These indicate that Sergei's review is requested, but I guess anybody could be the second reviewer: https://github.com/MariaDB/server/pull/2169 https://github.com/MariaDB/server/pull/2671 The indicate that Andrewi's review is requested, but I guess anybody could be the second reviewer: https://github.com/MariaDB/server/pull/3100 https://github.com/MariaDB/server/pull/3120 https://github.com/MariaDB/server/pull/3125 https://github.com/MariaDB/server/pull/3142 https://github.com/MariaDB/server/pull/3154 I am sending this to the mailing list as well just to raise general awareness that while Andrew and Daniel are putting in a lot of effort to review "external" contributions, it is not enough and even approved PRs get stalled. Other core contributors should perhaps get into the habit of reviewing/approving PRs as well. If there was a rule in github.com/MariaDB/server that prevents any commits from being merged unless it has two approvals, it would both help improve the code quality AND also force core contributors to spend more time adding approvals on GitHub and thus automatically increase the likelihood that there are no one-approval PRs around for weeks and weeks without getting the second approval. - Otto

3 2

MariaDB supports DBdeployer and MySqlJS, but they don't support MariaDB?
by Otto Kekäläinen 09 Apr '24

09 Apr '24

Hi! While digging into the CI situation I noticed that https://buildbot.mariadb.org/ advertises: "We run other ecosystem tests for: * DBdeployer * MySqlJS" However, when I browse https://github.com/mysqljs/mysql and https://github.com/datacharmer/dbdeployer there are zero mentions of MariaDB. Just wondering is anybody working to get them to mention MariaDB in their docs? If they are tested as part of the MariaDB CI, surely they should reciprocate and at least tell in their README that MariaDB is also supported. Maybe somebody from the MariaDB Foundation could send a PR to update their README, so they don't need to do anything else than just accept it. This was done for example in https://github.com/mysql-net/MySqlConnector/pull/1460/files by Daniel Lenski, and now that project mentions both MySQL and MariaDB. - Otto

3 2

Safe use of sprint/printf/fprintf in MariaDB code base
by Otto Kekäläinen 08 Apr '24

08 Apr '24

Hi all, Reading about the xz-utils backdoor authors submission of converting safe_fprintf() to fprintf() in libarchive[1] presumably in order to introduce intentional vulnerability reminded me that the MariaDB code base still has a plenty unsafe sprint/printf/fprintf use that can easily be found with scanners such as Flawfinder[2] and cppcheck[3]. There are currently 6 merge requests open by two authors (CC'd) to fix some of these issues[4]. Could we please have some more attention on these by the core contributors? If core contributors are not happy with the submissions, could you perhaps write your own safe functions (there are already some in m_string.h[5]) like many other projects seem to have (also libarchive had[6]) and then ask all contributors to use them consistently? Use of specific memory safe functions could also be mandated via the coding standards[7]. [1] https://github.com/libarchive/libarchive/pull/1609 [2] https://github.com/MariaDB/server/blob/11.5/.gitlab-ci.yml#L461-L489 [3] https://github.com/MariaDB/server/blob/11.5/.gitlab-ci.yml#L522-L554 [4] https://github.com/MariaDB/server/pulls?q=is%3Apr+is%3Aopen+sprintf+ [5] https://github.com/MariaDB/server/blob/11.5/include/m_string.h [6] https://github.com/libarchive/libarchive/blob/6110e9c82d8ba830c3440f36b9904… [7] https://github.com/MariaDB/server/blob/11.5/CODING_STANDARDS.md

2 3

question about Revision #f24038b851
by ShawnYan 04 Apr '24

04 Apr '24

hi, there, when I check release notes, https://mariadb.com/kb/en/mariadb-10419-changelog/ <https://mariadb.com/kb/en/mariadb-10419-changelog/ > I notice this commit record, Revision #f24038b851 2021-03-07 14:06:01 +0100 mark Aria allocations for temp tables as MY_THREAD_SPECIFIC but, it can not find in the jira system, can't learn more detail/background about this commit, could anyone help with it, may we know which task refer to this commit? https://github.com/MariaDB/server/commit/f24038b851 <https://github.com/MariaDB/server/commit/f24038b851 > Thank you. hi, there, when I check release notes, https://mariadb.com/kb/en/mariadb-10419-changelog/ <https://mariadb.com/kb/en/mariadb-10419-changelog/ > I notice this commit record, Revision #f24038b851 2021-03-07 14:06:01 +0100 mark Aria allocations for temp tables as MY_THREAD_SPECIFIC but, it can not find in the jira system, can't learn more detail about this commit, could anyone help with it, may we know which task refer to this commit? https://github.com/MariaDB/server/commit/f24038b851 <https://github.com/MariaDB/server/commit/f24038b851 > Thank you.

2 1

Re: [PATCH 0/2] Suggestion for smaller fix to XA parallel replication performance, MDEV-31949
by Kristian Nielsen 19 Mar '24

19 Mar '24

Andrei Elkin <andrei.elkin(a)mariadb.com> writes: > In normal cases, like that of not so many duplicate xids, the size > of maybe_active_xid must remain constrained because of "continuous" > relocation of elements from the tail to replace garbage (normally found > in head indexes). > Nevertheless as the array can be dynamically re-allocated to reflect some > some spike in the load, I suggest in > > origin/review__knielsen_xa_sched_minimal_fix > > to take care of that. I think your suggestion in review__knielsen_xa_sched_minimal_fix is fine. I'm not sure exactly how large the array can grow in pathological cases. As you wrote, in most normal cases we would expect it to not grow that much. But shrinking it if necessary surely cannot hurt. Thanks, - Kristian.

1 0

MariaDB topic on GitHub
by Otto Kekäläinen 19 Mar '24

19 Mar '24

Hi! I was browsing the GitHub topic 'mariadb' at https://github.com/topics/mariadb and to my surprise MariaDB itself isn't in that list at all. This seems to be due to https://github.com/MariaDB/server not having any topics defined at all. I recommend you add in the GitHub settings of https://github.com/MariaDB/server at least the topic 'mariadb', and while at it, also add link to mariadb.org. More ideas on how to improve the look and features of the GitHub page can be found by browsing projects like https://github.com/pulsar-edit and https://github.com/prisma which have all their GitHub account metadata polished. - Otto

1 0

Re: [PATCH] MDEV-33551: Semi-sync Wait Point AFTER_COMMIT Slow on Workloads with Heavy Concurrency
by Kristian Nielsen 18 Mar '24

18 Mar '24

> From: Brandon Nesterenko <brandon.nesterenko(a)mariadb.com> > > When using semi-sync replication with > rpl_semi_sync_master_wait_point=AFTER_COMMIT, the performance of the > primary can significantly reduce compared to AFTER_SYNC's > performance for workloads with many concurrent users executing > transactions. This is because all connections on the primary share Hi Brandon, Here is my second round review of the MDEV-33551 patch. I think these comments of mine from first round still need to be addressed: > > A bit of care must be taken about lifetime of Tranx_node and THD > > respectively (in case one goes away but not the other). But the whole > > > +void Repl_semi_sync_master::await_all_slave_replies() > > > + wait_result= cond_timewait(front.thd, &abstime); > > > > I'm wondering, is it safe here to wait on front.thd->COND_wakeup_ready? > > Isn't there a possibility that this might go away during the wait? I tried a simple testcase (attached) which disconnects the session thread while await_all_slave_replies() is running during shutdown. As I expected, I could see the COND_wakeup_ready of the session thread being de-allocated while await_all_slave_replies() is still waking up from that condition. That cannot be correct. Similarly, it seems clearly possible that a session thread is killed and exits while its entry is still present in the m_active_tranxs list, which will similarly lead to signalling a de-allocated COND_wakeup_ready. When a thread aborts the wait before the entry is removed, I think it should simply look up its entry (can use the m_trx_htb hash table, or even just iterate the list) and set its THD to NULL (or remove the entry, whatever is simpler). For the await_all_slave_replies() problem, I think the whole wait-at-shutdown is much too complex as it is now. The whole is_thd_awaiting_semisync_ack(), n_threads_awaiting_ack, special-case not killing threads waiting for semi-sync-ack, and now introducing unmark_thd_awaiting_ack. And it still has the bug waiting on a COND_wakeup_ready of another THD that can disappear during the wait. I think all of that should just go away, the wait in await_all_slave_replies() should be done much simpler with a dedicated condition variable to wait for. Eg. await_all_slave_replies() just does a mysql_cond_wait(COND_binlog_send) under LOCK_binlog until the last binlog position is ack'ed, and this condition is then signalled from report_reply_binlog() (either always, or await_all_slave_replies() can set a flag to request signal). Or have the shutdown thread enqueue its THD at the end of the m_active_tranxs list. Let's avoid all this complex interaction between SHUTDOWN WAIT FOR ALL SLAVES and session threads. BTW, COND_binlog_send is currently unused in your patch. Below detailed comments on individual parts of the patch: > --- a/sql/mysqld.cc > +++ b/sql/mysqld.cc > @@ -1750,18 +1750,12 @@ static void close_connections(void) > - if (shutdown_wait_for_slaves && repl_semisync_master.get_master_enabled()) > + if (shutdown_wait_for_slaves && repl_semisync_master.get_master_enabled() && > + repl_semisync_master.sync_get_master_wait_sessions()) Do you need to check sync_get_master_wait_sessions() > 0 here? Why not wait unconditionally for the last binlog position sent? > --- a/sql/semisync_master.cc > +++ b/sql/semisync_master.cc > @@ -68,6 +68,14 @@ static ulonglong timespec_to_usec(const struct timespec *ts) > return (ulonglong) ts->tv_sec * TIME_MILLION + ts->tv_nsec / TIME_THOUSAND; > } > > +int signal_waiting_transaction(THD *waiting_thd, const char *binlog_file, > + my_off_t binlog_pos) > +{ > + if (waiting_thd->is_awaiting_semisync_ack) > + mysql_cond_signal(&waiting_thd->COND_wakeup_ready); The check on is_awaiting_semi_sync_ack seems wrong here. Normally, the THD will be waiting, otherwise it would not be in the list in the first place, so faster with an occasional redundant signal than a conditional always. And if the THD is not waiting, it's not safe to access it anyway. Instead, as mentioned above, have a check for thd != NULL here, and let a non-waiting thread clear the THD in its entry before it releases the LOCK_binlog. > -int Repl_semi_sync_master::cond_timewait(struct timespec *wait_time) > +int Repl_semi_sync_master::cond_timewait(THD *thd, struct timespec *_wait_time) > + if (_wait_time == NULL) > + { > + create_timeout(&gen_wait_time, NULL); > + real_wait_time= &gen_wait_time; > + } > + else > + { > + real_wait_time= _wait_time; > + } Don't pass in booleans that are always hard-coded true or false in the caller. Instead create two different functions, one that takes a time_spec, and one that does not (the latter creating it and calling the former). > @@ -533,7 +559,8 @@ void Repl_semi_sync_master::remove_slave() > Signal transactions waiting in commit_trx() that they do not have to > wait anymore. > */ > - cond_broadcast(); > + m_active_tranxs->clear_active_tranx_nodes(NULL, NULL, > + signal_waiting_transaction); This does not compile on my machine (the second NULL is passed to my_off_t position), is that a last-minute change that was never tested? Or NULL is defined differently on your machine, so it doesn't fail to compile? BTW, question about the old code: Why didn't the old code clear the m_active_tranxs list? Was it a bug? Or was it cleared elsewhere before (and if so where)? > -void Repl_semi_sync_master::await_slave_reply() > +void Repl_semi_sync_master::await_all_slave_replies() > + The lock is taken per iteration rather than over the whole loop to allow > + more opportunity for the user threads to handle their ACKs. That sounds like a mis-understanding. The lock is always released by mysql_cond_timedwait() in each iteration of the loop, no need to release/take it again for just a couple instructions. > + while (TRUE) > + { > + lock(); > + if (m_active_tranxs->is_empty() || !get_master_enabled() || !is_on()) > + { > + unlock(); > + break; > + } > + wait_result= m_active_tranxs->for_front( > + [](THD *thd, const char *binlog_file, my_off_t binlog_pos) -> int { > + return (thd->is_awaiting_semisync_ack) > + ? repl_semisync_master.cond_timewait(thd, NULL) > + : 0; > + }); As I wrote at the top, it is not safe to wait on the COND_wakeup_ready of another THD, since that THD might go away during the wait. Instead, let's do a specific wait for the end position here, either on COND_binlog or by enqueing an extra THD at the end. (And btw, in your code, why wait for the _front_ of the list and not the _rear_?) > diff --git a/sql/semisync_master.h b/sql/semisync_master.h > index 99f46869354..1eb690b572b 100644 > --- a/sql/semisync_master.h > +++ b/sql/semisync_master.h > > + * The pre_delete_hook parameter is a function pointer that will be invoked > + * for each Active_tranx node, in order, from m_trx_front to log_file_name, > + * e.g. to signal their wakeup condition. Repl_semi_sync_binlog::LOCK_binlog > + * is held while this is invoked. "from m_trx_front to log_file_name", I don't understand? (Typo s/log_file_name/m_trx_rear/ ?) > +/* > + Element in Repl_semi_sync_master::wait_queue to preserve the state of a > + transaction waiting for an ACK. > +*/ > +typedef struct _semisync_wait_trx { > + const char *binlog_name; > + my_off_t binlog_pos; > + THD *thd; > +} semisync_wait_trx_t; > + This is not longer used (should be removed). > * Input: (the transaction events' ending binlog position) > * trx_wait_binlog_name - (IN) ending position's file name > * trx_wait_binlog_pos - (IN) ending position's file offset > + * unmark_thd_awaiting_ack - (IN) Whether or not to unmark > + * thd->is_awaiting_semisync_ack after receiving > + * an ACK from the replica. If using wait_point > + * AFTER_SYNC with binlog_group_commit, only the > + * last transaction written to binlog in the > + * group should negate the boolean, because the > + * same thread (i.e. leader thread) will wait for > + * all transaction ACKs. Negating it too early > + * would break SHUTDOWN WAIT FOR ALL SLAVES, as > + * that is the condition tested to bypass killing > + * threads in phase 1. In all other situations, > + * the boolean should be negated immediately. That seems unnecessarily convoluted. I think the real problem here is that write_tranx_in_binlog() sets the is_awaiting_semi_sync_ack flag on the wrong THD, on the waiter_thd instead of the trx_thd. But once we fix the await_all_slave_replies(), do we even need the is_awaiting_semi_sync_ack flag any more? If so, why? If not, then this point becomes moot anyway. - Kristian. --source include/have_innodb.inc --source include/have_debug.inc --source include/have_binlog_format_mixed.inc --source include/master-slave.inc --connection master CREATE TABLE t1 (a INT PRIMARY KEY, b INT) ENGINE=InnoDB; INSERT INTO t1 VALUES (1,0); --sync_slave_with_master source include/stop_slave.inc; SET @old_semi_enabled= @@GLOBAL.rpl_semi_sync_slave_enabled; SET GLOBAL rpl_semi_sync_slave_enabled = 1; --source include/start_slave.inc --connection master SET GLOBAL rpl_semi_sync_master_enabled= 1; SET GLOBAL rpl_semi_sync_master_timeout= 5000; let $status_var= Rpl_semi_sync_master_clients; let $status_var_value= 1; source include/wait_for_status_var.inc; show status like 'Rpl_semi_sync_master_status'; show status like 'Rpl_semi_sync_master_clients'; --connection slave SET @old_dbug= @@GLOBAL.debug_dbug; SET GLOBAL debug_dbug="+d,simulate_delay_semisync_slave_reply"; --connect(con1, localhost, root,,) --connect(con2, localhost, root,,) --connection con1 send INSERT INTO t1 VALUES (2,0); --sleep 0.3 --disconnect con1 --connection default --write_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect wait EOF #let $status_var= Rpl_semi_sync_master_wait_sessions; #let $status_var_value= 1; #source include/wait_for_status_var.inc; --sleep 0.1 --connection con2 SHUTDOWN WAIT FOR ALL SLAVES; --disconnect con2 --connection default --source include/wait_until_disconnected.inc --connection master --source include/wait_until_disconnected.inc --connection server_1 --source include/wait_until_disconnected.inc --connection default --append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect restart EOF --enable_reconnect --source include/wait_until_connected_again.inc --connection master --enable_reconnect --source include/wait_until_connected_again.inc --connection server_1 --enable_reconnect --source include/wait_until_connected_again.inc --save_master_pos --connection slave SET GLOBAL debug_dbug= @old_dbug; SET GLOBAL rpl_semi_sync_slave_enabled= @old_semi_enabled; --source include/stop_slave.inc --source include/start_slave.inc --sync_with_master --connection default DROP TABLE t1; --source include/rpl_end.inc

2 3

Re: e6cacca0b39: MDEV-23729 MDEV-32218 INFORMATION_SCHEMA table for user data
by Sergei Golubchik 15 Mar '24

15 Mar '24

Hi, Nikita, no test for FLUSH PRIVILEGES no test for a user with manually expired password doesn't compile in embedded hostname 'neo' is replaced, but not everywhere I'll try to fix it myself, but no guarantees On Mar 15, Nikita Malyavin wrote: > revision-id: e6cacca0b39 (mariadb-11.4.1-10-ge6cacca0b39) > parent(s): 929c2e06aae > author: Nikita Malyavin > committer: Nikita Malyavin > timestamp: 2024-03-15 15:32:45 +0100 > message: > > MDEV-23729 MDEV-32218 INFORMATION_SCHEMA table for user data > > * A new table INFORMATION_SCHEMA.USERS is introduced. > * It stores auxiliary user data > * An unprivileged user can access their own data, and that is the main > difference with what mysql.global_priv provides > * The fields are currently: USER, PASSWORD_ERRORS, PASSWORD_EXPIRATION_TIME > * If password_errors is ignored for the user, PASSWORD_ERRORS is NULL > * PASSWORD_EXPIRATION_TIME is a timestamp with exact point in time, calculated > from password_last_changed and password_lifetime (i.e. days) stored for the user > > diff --git a/sql/sql_acl.cc b/sql/sql_acl.cc > index 14450a5a610..a87b04f3d39 100644 > --- a/sql/sql_acl.cc > +++ b/sql/sql_acl.cc ... > +int fill_users_schema_table(THD *thd, TABLE_LIST *tables, COND *cond) > +{ > + int res= 0; > +#ifndef NO_EMBEDDED_ACCESS_CHECKS > + bool see_whole_table= check_access(thd, SELECT_ACL, "mysql", NULL, NULL, > + true, true) == 0; > + TABLE *table= tables->table; > + > + if (!see_whole_table) > + { > + mysql_mutex_lock(&acl_cache->lock); > + ACL_USER *cur_user= find_user_exact(thd->security_ctx->priv_host, > + thd->security_ctx->priv_user); > + if (!cur_user) > + { > + mysql_mutex_unlock(&acl_cache->lock); > + my_error(ER_INVALID_CURRENT_USER, MYF(0)); Hmm. Questionable. I'd just return an empty result set here, I_S tables normally don't throw errors. But this is an exceptional situation and I cannot say that an error is wrong here, let's keep it your way. > + return 1; > + } > + > + res= fill_users_schema_record(thd, table, cur_user); > + mysql_mutex_unlock(&acl_cache->lock); > + return res; > + } > + > + mysql_mutex_lock(&acl_cache->lock); > + for (size_t i= 0; res == 0 && i < acl_users.elements; i++) > + { > + ACL_USER *user= dynamic_element(&acl_users, i, ACL_USER*); > + res= fill_users_schema_record(thd, table, user); > + } > + mysql_mutex_unlock(&acl_cache->lock); > +#endif > + return res; > +} > + Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

1 0

Re: [PATCH 0/2] Suggestion for smaller fix to XA parallel replication performance, MDEV-31949
by Kristian Nielsen 13 Mar '24

13 Mar '24

Andrei Elkin <andrei.elkin(a)mariadb.com> writes: > which makes a worker with dependent pieces of job be *effectively* placed at the > tail of the available worker ('fifo') queue. Indeed. And this BTW is why a simple sliding window of size 2*workers is not enough, because in principle a worker can sit in the fifo for many transactions without being scheduled, if there are many dependency restrictions. I coded the thing with "generations" as a quick way to handle this and have a way to clean out old dependency information. This could of course be improved in many ways. Maybe remember the two last XID for every worker and have a hash table mapping each XID to its worker, or something. But this is just a PoC. > No you are not. Thanks for the catch! To excuse myself it was supposed > to be a poc that I committed hastedly. Yes yes, no problem, I understand, it's good to do poc. I was just momentarily confused, but the intent is clear, thanks. - Kristian.

1 0

Re: a473db9e4fe: MDEV-25829 Change default collation to utf8mb4_1400_ai_ci
by Sergei Golubchik 13 Mar '24

13 Mar '24

Hi, Alexander, On Mar 08, Alexander Barkov wrote: > commit a473db9e4fe > Author: Alexander Barkov <bar(a)mariadb.com> > Date: Thu Nov 2 14:16:09 2023 +0400 > > MDEV-25829 Change default collation to utf8mb4_1400_ai_ci "utf8mb4_uca1400_ai_ci". It's still not too late to rename the MDEV and change commits to match. I'm not reviewing InnoDB changes, I hope Marko did > > Step#3 The main patch > > diff --git a/mysql-test/include/ctype_utf8mb4.inc b/mysql-test/include/ctype_utf8mb4.inc > index 436b0f2782f..73011cef81d 100644 > --- a/mysql-test/include/ctype_utf8mb4.inc > +++ b/mysql-test/include/ctype_utf8mb4.inc > @@ -58,11 +58,11 @@ select CONVERT(_koi8r' > # "a\0" < "a" > # "a\0" < "a " > > -SELECT 'a' = 'a '; > -SELECT 'a\0' < 'a'; > -SELECT 'a\0' < 'a '; > -SELECT 'a\t' < 'a'; > -SELECT 'a\t' < 'a '; > +SELECT 'a' = 'a ' collate utf8mb4_general_ci; > +SELECT 'a\0' < 'a' collate utf8mb4_general_ci; > +SELECT 'a\0' < 'a ' collate utf8mb4_general_ci; > +SELECT 'a\t' < 'a' collate utf8mb4_general_ci; > +SELECT 'a\t' < 'a ' collate utf8mb4_general_ci; 1. what will happen in utf8mb4_uca1400_ai_ci? 2. if you specifically want this file to test utf8mb4_general_ci, you can do SET NAMES, cannot you? Once for the whole file. Or set character_set_collations > > # > # The same for binary collation > diff --git a/mysql-test/main/ctype_euckr.result b/mysql-test/main/ctype_euckr.result > index 9e030f9cc6d..8de90f89a26 100644 > --- a/mysql-test/main/ctype_euckr.result > +++ b/mysql-test/main/ctype_euckr.result > @@ -2244,7 +2244,7 @@ FE80 > DELETE FROM t2 WHERE a='?'; > ALTER TABLE t2 ADD u VARCHAR(1) CHARACTER SET utf8, ADD a2 VARCHAR(1) CHARACTER SET euckr; > UPDATE IGNORE t2 SET u=a, a2=u; > -SELECT s as unassigned_code FROM t2 WHERE u='?'; > +SELECT s as unassigned_code FROM t2 WHERE u=binary'?'; why? > unassigned_code > A2E8 > A2E9 > diff --git a/mysql-test/main/ctype_like_range.result b/mysql-test/main/ctype_like_range.result > index e5e5e61126b..fa694b0c250 100644 > --- a/mysql-test/main/ctype_like_range.result > +++ b/mysql-test/main/ctype_like_range.result > @@ -284,7 +284,7 @@ id name val > 32 mn 63616161616161616161616161616161 > 32 mx 63616161616161616161616161616161 > 32 sp -------------------------------- > -ALTER TABLE t1 MODIFY a VARCHAR(32) CHARACTER SET utf8; > +ALTER TABLE t1 MODIFY a VARCHAR(32) CHARACTER SET utf8 COLLATE utf8_general_ci; do you have tests for ctype_like_range with utf8mb3_uca1400_ai_ci ? > INSERT INTO t1 (a) VALUES (_ucs2 0x0425),(_ucs2 0x045F); > INSERT INTO t1 (a) VALUES (_ucs2 0x2525),(_ucs2 0x5F5F); > SELECT * FROM v1; > diff --git a/mysql-test/main/ctype_utf8.test b/mysql-test/main/ctype_utf8.test > index a875fe51f3a..2f56d10bccf 100644 > --- a/mysql-test/main/ctype_utf8.test > +++ b/mysql-test/main/ctype_utf8.test > @@ -24,7 +24,7 @@ drop database if exists mysqltest; > --disable_warnings > drop table if exists t1,t2; > --enable_warnings > -set names utf8; > +set names utf8 collate utf8_general_ci; why not to change the test to work with utf8mb3_uca1400_ai_ci? > > select left(_utf8 0xD0B0D0B1D0B2,1); > select right(_utf8 0xD0B0D0B2D0B2,1); > diff --git a/mysql-test/main/dyncol.test b/mysql-test/main/dyncol.test > index 493e9b3842d..ba302861e03 100644 > --- a/mysql-test/main/dyncol.test > +++ b/mysql-test/main/dyncol.test > @@ -10,10 +10,10 @@ > --echo # column create > --echo # > select hex(COLUMN_CREATE(1, NULL AS char character set utf8)); > -select hex(COLUMN_CREATE(1, "afaf" AS char character set utf8)); > -select hex(COLUMN_CREATE(1, 1212 AS char character set utf8)); > -select hex(COLUMN_CREATE(1, 12.12 AS char character set utf8)); > -select hex(COLUMN_CREATE(1, 99999999999999999999999999999 AS char character set utf8)); > +select hex(COLUMN_CREATE(1, "afaf" AS char character set utf8 collate utf8_general_ci)); > +select hex(COLUMN_CREATE(1, 1212 AS char character set utf8 collate utf8_general_ci)); > +select hex(COLUMN_CREATE(1, 12.12 AS char character set utf8 collate utf8_general_ci)); > +select hex(COLUMN_CREATE(1, 99999999999999999999999999999 AS char character set utf8 collate utf8_general_ci)); why? > select hex(COLUMN_CREATE(1, NULL AS unsigned int)); > select hex(COLUMN_CREATE(1, 1212 AS unsigned int)); > select hex(COLUMN_CREATE(1, 7 AS unsigned int)); > diff --git a/mysql-test/main/func_json.result b/mysql-test/main/func_json.result > index ab35822a6ce..fce2caa2842 100644 > --- a/mysql-test/main/func_json.result > +++ b/mysql-test/main/func_json.result > @@ -414,6 +414,13 @@ select json_object('foo', json_unquote(json_object('bar', c)),'qux', c) as fld f > fld > {"foo": "{\"bar\": \"abc\"}", "qux": "abc"} > {"foo": "{\"bar\": \"def\"}", "qux": "def"} > +create table t2 as select json_object('foo', json_unquote(json_object('bar', c)),'qux', c) as fld from t1 limit 0; > +show create table t2; > +Table Create Table > +t2 CREATE TABLE `t2` ( > + `fld` varchar(39) DEFAULT NULL > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 COLLATE=latin1_swedish_ci > +drop table t2; why do you test here? > drop table t1; > select json_object("a", json_object("b", "abcd")); > json_object("a", json_object("b", "abcd")) > diff --git a/mysql-test/main/func_regexp_pcre.test b/mysql-test/main/func_regexp_pcre.test > index 77f5af6e0ff..7cf9e8007de 100644 > --- a/mysql-test/main/func_regexp_pcre.test > +++ b/mysql-test/main/func_regexp_pcre.test > @@ -23,17 +23,17 @@ SELECT 'Ã ' RLIKE '\\x{00C0}' COLLATE utf8_bin; > SELECT 'Ã' RLIKE '\\x{00C0}' COLLATE utf8_bin; > > # Checking how (?i) and (?-i) affect case sensitivity > -CREATE TABLE t1 (s VARCHAR(10) CHARACTER SET utf8); > +CREATE TABLE t1 (s VARCHAR(10) CHARACTER SET utf8 COLLATE utf8_general_ci); why? > INSERT INTO t1 VALUES ('a'),('A'); > -CREATE TABLE t2 (p VARCHAR(10) CHARACTER SET utf8); > +CREATE TABLE t2 (p VARCHAR(10) CHARACTER SET utf8 COLLATE utf8_general_ci); > INSERT INTO t2 VALUES ('a'),('(?i)a'),('(?-i)a'),('A'),('(?i)A'),('(?-i)A'); > SELECT s,p,s RLIKE p, s COLLATE utf8_bin RLIKE p FROM t1,t2 ORDER BY BINARY s, BINARY p; > DROP TABLE t1,t2; > > > # Checking Unicode character classes > -CREATE TABLE t1 (ch VARCHAR(22)) CHARACTER SET utf8; > -CREATE TABLE t2 (class VARCHAR(32)) CHARACTER SET utf8; > +CREATE TABLE t1 (ch VARCHAR(22)) CHARACTER SET utf8 COLLATE utf8_general_ci; > +CREATE TABLE t2 (class VARCHAR(32)) CHARACTER SET utf8 COLLATE utf8_general_ci; and here > INSERT INTO t1 VALUES ('Ð¯'),('Î£'),('A'),('Ã'); > INSERT INTO t1 VALUES ('Ñ'),('Ï'),('a'),('Ã '); > INSERT INTO t1 VALUES ('ã'),('ê°·'),('à¶´'); > diff --git a/mysql-test/main/mrr_icp_extra.result b/mysql-test/main/mrr_icp_extra.result > index 8f6ee88acc6..d7cde103288 100644 > --- a/mysql-test/main/mrr_icp_extra.result > +++ b/mysql-test/main/mrr_icp_extra.result > @@ -754,7 +754,7 @@ t1 CREATE TABLE `t1` ( > KEY `t` (`t`(5)) > ) ENGINE=MyISAM DEFAULT CHARSET=latin1 COLLATE=latin1_swedish_ci > drop table t1; > -create table t1 (v char(10) character set utf8); > +create table t1 (v char(10) character set utf8 collate utf8_general_ci); why here? > show create table t1; > Table Create Table > t1 CREATE TABLE `t1` ( > diff --git a/mysql-test/main/myisam.test b/mysql-test/main/myisam.test > index 1a20f97a54f..4a283737122 100644 > --- a/mysql-test/main/myisam.test > +++ b/mysql-test/main/myisam.test > @@ -1039,7 +1039,7 @@ create table t1 (v varchar(65536)); > show create table t1; > drop table t1; > set statement sql_mode = 'NO_ENGINE_SUBSTITUTION' for > -create table t1 (v varchar(65530) character set utf8); > +create table t1 (v varchar(65530) character set utf8 collate utf8_general_ci); why? collation doesn't matter here, does it? > show create table t1; > drop table t1; > > diff --git a/mysql-test/main/order_by.result b/mysql-test/main/order_by.result > index 274f29e34dc..85059fb3370 100644 > --- a/mysql-test/main/order_by.result > +++ b/mysql-test/main/order_by.result > @@ -4142,7 +4142,7 @@ drop table t1; > # > # MDEV-21922: Allow packing addon fields even if they don't honour max_length_for_sort_data > # > -create table t1 (a varchar(200) character set utf8, b int); > +create table t1 (a varchar(200) character set utf8 collate utf8_general_ci, b int); why? I think I see too many forced utf8_general_ci in tests and it's difficult to believe that the collation actually matters in all these cases. > insert into t1 select seq, seq from seq_1_to_10; > select * from t1 order by a; > a b > diff --git a/sql/item_jsonfunc.cc b/sql/item_jsonfunc.cc > index 508ea9f644e..5731ad1f04b 100644 > --- a/sql/item_jsonfunc.cc > +++ b/sql/item_jsonfunc.cc > @@ -842,7 +842,7 @@ String *Item_func_json_quote::val_str(String *str) > bool Item_func_json_unquote::fix_length_and_dec(THD *thd) > { > collation.set(&my_charset_utf8mb3_general_ci, > - DERIVATION_COERCIBLE, MY_REPERTOIRE_ASCII); > + DERIVATION_CAST, MY_REPERTOIRE_ASCII); why? I guess that's why you have json_unquote test above. > max_length= args[0]->max_length; > set_maybe_null(); > return FALSE; Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 1

Re: b657bc75f90: MDEV-25829 Change default collation to utf8mb4_1400_ai_ci
by Sergei Golubchik 13 Mar '24

13 Mar '24

Hi, Alexander, On Mar 08, Alexander Barkov wrote: > revision-id: b657bc75f90 (mariadb-11.0.1-262-gb657bc75f90) > parent(s): 8e5f41b3453 > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2023-11-15 10:58:03 +0400 > message: > > MDEV-25829 Change default collation to utf8mb4_1400_ai_ci > > Step#2 - Adding a new collation derivation level for CAST and CONVERT. > > Now character string cast functions: > - CAST(string_expr AS CHAR) > - CONVERT(expr USING charset_name) > > have a new collation derivation level between: > > - string literals > - utf8 metadata functions, e.g. user() and database() > > Before the change these cast functions had collation derivation equal > to table columns, which caused more illegal mix of collation conflicts. > > Note, binary string cast functions: > - CAST(string_expr AS BINARY) > - CONVERT(expr USING binary) > did not change their collation derivation. Why not? Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 1

Re: 8e5f41b3453: MDEV-25829 Change default collation to utf8mb4_1400_ai_ci
by Sergei Golubchik 13 Mar '24

13 Mar '24

Hi, Alexander, On Mar 08, Alexander Barkov wrote: > revision-id: 8e5f41b3453 (mariadb-11.0.1-261-g8e5f41b3453) > parent(s): 9e457cbe501 > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2023-11-15 09:37:26 +0400 > message: > > MDEV-25829 Change default collation to utf8mb4_1400_ai_ci > > Step#1 - Changing collation derivation for string user variables > from EXPLICIT to COERCIBLE. from IMPLICIT, oherwise ok to push. and I still think user variable could remember the derivation. It's definitely not a part of MDEV-25829, but if you agree that it should, let's create an MDEV for that. Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 1

Re: [PATCH 0/2] Suggestion for smaller fix to XA parallel replication performance, MDEV-31949
by Kristian Nielsen 12 Mar '24

12 Mar '24

Andrei Elkin <andrei.elkin(a)mariadb.com> writes: > I reviewed your branch to agree with the idea of xid conflicts > handling and its implementation. > > I however could not understand the need of the refactoring part. > In order to track the xid dependency couple functions and and > rpl_parallel_entry::maybe_active_xid > a sort of a sliding window - that the 2nd commit of your branch > introduced must be sufficient to my analysis. > Of course it needs to hold worker indexes of active xid:s. > > This observation led me to create a review branch > > origin/review__knielsen_xa_sched_minimal_fix > > [where origin git@github.com:MariaDB/server.git] > which is a somewhat light elaboration over the 2nd commit of your branch. The reason for the refactoring part is to allow to distribute the transactions as evenly as possible over the worker threads, given the scheduling constraints imposed by XA dependencies. Let's say we have 5 worker threads, and transactions T1..T15. If there are no scheduling constraints, we can schedule evenly like this: W1: T1 T6 T11 W2: T2 T7 T12 W3: T3 T8 T13 W4: T4 T9 T14 W5: T5 T10 T15 Now suppose we need to schedule T6 on the same worker as T4, and T8 on the same worker as T3. The most even way to distribute the transactions is now this: W1: T1 T7 T12 W2: T2 T9 T14 W3: T3 T8 T13 W4: T4 T6 T11 W5: T5 T10 T15 You see, we try to always have 5 consecutive transactions T_i, T_{i+1}, ..., T+{i+4} scheduled on 5 different worker threads, except when this is not possible due to dependency restrictions. But this most-even scheduling requires to change the scheduling order of threads from the original sequential 1,2,3,4,5. You see, the last transactions T11..T15 are scheduled on worker threads in order: W4, W1, W3, W2, W5. The refactoring patch introduces thread_sched_fifo to hold the current scheduling order of worker threads. Whenever we have to schedule out-of-order due to a scheduling dependency, we put the scheduled worker at the end of this fifo, to preserve even scheduling for the next N transactions. In the original scheduling code, we never scheduled workers out-of-order, so the scheduling order was always cyclic W1, W2, W3, W4, W5, W1, ..., and a simple incrementing counter was sufficient. But this is no longer sufficient when out-of-order scheduling occurs. In the example, if we use a simple counter and try to schedule on worker ((i+1) mod N) after worker (i), then we get the following uneven scheduling: W1: T1 T11 W2: T2 T12 W3: T3 T8 T13 W4: T4 T6 T9 T14 W5: T5 T7 T10 T15 Because T6 and T8 have particular scheduling requirements, they cause the scheduling to skip workers. The result is that W4 and W5 get too many transactions, while W1 and W2 get too few, and parallelism is reduced. I hope this explain the reasoning. > Could you please have a look at it? If I understand your patch correctly, it schedules using a simple incrementing counter when there are no dependency requirements, and so would suffer from the uneven scheduling in the above example. It also looks like there's a mistake in the code: > + if ((idx= check_xa_xid_dependency(&gtid_ev->xid)) < (uint32) -1) > + { > + /* > + A previously scheduled event group with the same XID might still be > + active in a worker, so schedule this event group in the same worker > + to avoid a conflict. > + */ > + } > + else > + { > + /* Record this XID now active. */ > + xid_active_generation *a= > + (xid_active_generation *)alloc_dynamic(&maybe_active_xid); > + if (!a) > + return NULL; > + ++idx; > + if (idx >= rpl_thread_max) > + idx= 0; If check_xa_xid_dependency() returns "no dependency", then idx is set to (uint32)-1 and the else {...} branch is executed. This does idx++, which increments the (uint32)-1 to 0. So it looks to me that _all_ transactions are scheduled on worker 0, or am I missing something? This is a simple mistake, it's an easy fix to not assign idx when check_xa_xid_dependency() returns "no dependency". But the code would still suffer from uneven scheduling as in the example, IIUC. So there's a good reason to introduce the thread_sched_fifo as done in my refactoring patch. This though still leaves the limitation in my minimal patch that XA PREPARE xid1 cannot group commit with XA COMMIT xid1. The impact of this will depend on how many transactions on average appear between an XA PREPARE and its associated XA COMMIT, as well as on how expensive fsync() is relative to the cost of each transaction. - Kristian.

1 0

Re: 9e5d4dfc49b: MDEV-23729 MDEV-32218 INFORMATION_SCHEMA table for user login data
by Sergei Golubchik 11 Mar '24

11 Mar '24

Hi, Nikita, On Mar 10, Nikita Malyavin wrote: > revision-id: 9e5d4dfc49b (mariadb-11.4.1-10-g9e5d4dfc49b) > parent(s): 929c2e06aae > author: Nikita Malyavin > committer: Nikita Malyavin > timestamp: 2024-02-29 17:24:27 +0100 > message: > > MDEV-23729 MDEV-32218 INFORMATION_SCHEMA table for user login data > > * A new table INFORMATION_SCHEMA.LOGON is introduced. There were many cases where we lack an INFORMATION_SCHEMA with all user accounts. Forcing users to select from mysql.user is a shame. Let's use the chance and create a table USERS. It doesn't need more columns now than what you've created, but we'll likely create more in the future. > * Upon idea, it stores auxiliary user data related to login/security/resources > * An unprivileged user can access their own data, and that is the main > difference with what mysql.global_priv provides exactly! > * The fields are currently: USER, WRONG_PASSWORD_ATTEMPTS, EXPIRATION_TIME I wanted to write about splitting USER into USER and HOST, but indeed it's just USER everywhere else in INFORMATION_SCHEMA (e.g. DEFINER and GRANTEE columns). And functions like USER() and CURRENT_USER() don't split either. So, you're right, let's consistently use user@host everywhere. > diff --git a/mysql-test/main/information_schema_stats.result b/mysql-test/main/information_schema_stats.result > index 352bcbab823..e38788872fa 100644 > --- a/mysql-test/main/information_schema_stats.result > +++ b/mysql-test/main/information_schema_stats.result ... > +connect(localhost,naughty_user,wrong_passwd,test,16000,/home/nik/mariadb/bld/mysql-test/var/tmp/mysqld.1.sock); here and below - you forgot to replace the path with MASTER_MYSOCK > diff --git a/mysql-test/main/information_schema_stats.test b/mysql-test/main/information_schema_stats.test > index fd5171c3fb4..dbcabe45965 100644 > --- a/mysql-test/main/information_schema_stats.test > +++ b/mysql-test/main/information_schema_stats.test > @@ -47,3 +47,69 @@ select * from information_schema.index_statistics where table_schema='test' and > select * from information_schema.table_statistics where table_schema='test' and table_name='just_a_test'; > set global userstat=@save_userstat; > --enable_ps2_protocol > + > +--echo # > +--echo # MDEV-23729 INFORMATION_SCHEMA Table info. about user locked due to > +--echo # max_password_errors > +--echo # > +--echo # MDEV-32218 message to notify end-user N-days prior the password get > +--echo # expired > +--echo # I don't see how you test "info about user locked due to max_password_errors". This is the reason for implementing this new table, it has to be tested. At least add select * from information_schema.logon where wrong_password_attemps >= @max_password_errors; and show it it's empty at first and then not empty. > diff --git a/sql/sql_acl.cc b/sql/sql_acl.cc > index 14450a5a610..4d199bf0e61 100644 > --- a/sql/sql_acl.cc > +++ b/sql/sql_acl.cc > @@ -12956,6 +12956,87 @@ int fill_schema_column_privileges(THD *thd, TABLE_LIST *tables, COND *cond) > #endif > } > > +namespace Show > +{ > + ST_FIELD_INFO users_fields_info[] = > + { > + Column("USER", Userhost(), NOT_NULL), > + Column("WRONG_PASSWORD_ATTEMPTS", SLonglong(), NULLABLE), @@max_password_errors is documented as "If there is more than this number of failed connect attempts due to invalid password, user will be blocked from further connections until FLUSH_PRIVILEGES" Let's use call incorrect password consistently everywhere. It could be "bad password", "wrong password", "incorrect password", "invalid password", but it should be the same everywhere. I personally think that "invalid" implies some kind if validity check, e.g. not shorter than 10 characters, not equal to the username, etc and a password that doesn't satisfy these validity rules is "invalid". Trying to login with an incorrect password is a different kind of error, a password can be valid but wrong. May be, ask Ian and/or Daniel about it? And then don't forget to update @@max_password_errors help text to match. Also, you don't have a test case for FLUSH PRIVILEGES. > + Column("EXPIRATION_TIME", SLonglong(), NULLABLE), PASSWORD_EXPIRATION_TIME, the user account itself does not expire. > + CEnd() > + }; > +}; > + > +static bool ignore_max_password_errors(const ACL_USER *acl_user); > + > +static int fill_logon_schema_record(THD *thd, TABLE * table, ACL_USER *user) > +{ > + ulonglong lifetime= user->password_lifetime < 0 > + ? default_password_lifetime > + : user->password_lifetime; > + > + bool ignore_password_errors= ignore_max_password_errors(user); why? I think it's still useful to show how many wrong password attempts were there for an account even if it doesn't get blocked. > + bool ignore_expiration_date= lifetime == 0; > + > + /* Skip user if nothing to show */ > + if (ignore_password_errors && ignore_expiration_date) > + return 0; > + > + Grantee_str grantee(user->user.str, safe_str(user->host.hostname)); > + table->field[0]->store(grantee, strlen(grantee), system_charset_info); > + if (ignore_password_errors) > + { > + table->field[1]->set_null(); > + } > + else > + { > + table->field[1]->set_notnull(); > + table->field[1]->store(user->password_errors); > + } > + if (ignore_expiration_date) > + { > + table->field[2]->set_null(); > + } > + else > + { > + table->field[2]->set_notnull(); > + table->field[2]->store(user->password_last_changed > + + user->password_lifetime * 3600 * 24, true); I think it'd be more generally useful to show password_last_changed and expiration period separately. > + } > + > + return schema_table_store_record(thd, table); > +} > + > +int fill_logon_schema_table(THD *thd, TABLE_LIST *tables, COND *cond) > +{ > + int res= 0; > +#ifndef NO_EMBEDDED_ACCESS_CHECKS > + bool see_whole_table= check_global_access(thd, PROCESS_ACL, true) == 0; I don't think PROCESS_ACL is a very logical choice here. And there's nothing better as far as I can see. May be let's just do if (check_access(thd, SELECT_ACL, "mysql", NULL, NULL, 1, 1)) ? There are many checks like that in e.g. sql_parse.cc > + > + TABLE *table= tables->table; > + > + if (!see_whole_table) > + { > + mysql_mutex_lock(&acl_cache->lock); > + ACL_USER *cur_user= find_user_exact(thd->security_ctx->priv_host, > + thd->security_ctx->priv_user); 1. cur_user can be NULL if someone dropped it while there was an active connection for this user 2. add a test for it > + > + res= fill_logon_schema_record(thd, table, cur_user); > + mysql_mutex_unlock(&acl_cache->lock); > + return res; > + } > + > + mysql_mutex_lock(&acl_cache->lock); > + for (size_t i= 0; res == 0 && i < acl_users.elements; i++) > + { > + ACL_USER *user= dynamic_element(&acl_users, i, ACL_USER*); > + res= fill_logon_schema_record(thd, table, user); > + } > + mysql_mutex_unlock(&acl_cache->lock); > +#endif > + return res; > +} > + Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 2

Backup MariaDB Databases for Suprema BioStar 2 Door Access System
by Turritopsis Dohrnii Teo En Ming 09 Mar '24

09 Mar '24

Subject: Backup MariaDB Databases for Suprema BioStar 2 Door Access System Good day from Singapore, On 7 March 2024 Thursday, when I was installing new self-signed SSL certificate for the door access system for a law firm in Singapore, I notice that Suprema BioStar 2 also uses the open source MariaDB database server. If I did not remember wrongly, there are 3 databases. However, the entire platform is a Windows environment, not Linux. How can I backup all the databases in the MariaDB database server? Thank you. Regards, Mr. Turritopsis Dohrnii Teo En Ming Targeted Individual in Singapore Blogs: https://tdtemcerts.blogspot.com https://tdtemcerts.wordpress.com GIMP also stands for Government-Induced Medical Problems.

1 0

Re: be66e975ecf: MDEV-31340 Remove MY_COLLATION_HANDLER::strcasecmp()
by Sergei Golubchik 08 Mar '24

08 Mar '24

Hi, Alexander, Ok to push, thanks for the great refactoring! I've looked at the diff of this and previous commit excluding InnoDB. On Mar 08, Alexander Barkov wrote: > revision-id: be66e975ecf (mariadb-11.0.1-282-gbe66e975ecf) > parent(s): c0c1c80346b > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2024-01-19 11:04:02 +0400 > message: > > MDEV-31340 Remove MY_COLLATION_HANDLER::strcasecmp() > Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

1 0

64-bit time_t support in MariaDB?
by Otto Kekäläinen 07 Mar '24

07 Mar '24

Hello! Does MariaDB already fully support 64-bit time_t (the year 2038 problem)? MariaDB 10.11 was today uploaded to Debian experimental without any 64-bit time_t support changes, just a library name change[1] to test that it works with the Debian 64-bit time_t tooling as Debian is currently undergoing a transition to it[2]. So far MariaDB built successfully and passed the main MTR suite on 11 architectures[3]. Monty mentioned that some work is in progress, and indeed I found his commits in two branches [4,5] but looking at the commits and their link to buildbot it seems they are not passing the CI. I also found one MDEV[6] to make TIMESTAMP use the whole 32-bit unsigned range. Are there any other efforts in progress? Does MTR main suite include tests for 64-bit time support? Should MTR automatically currently fail if toolchain time is 64-bit? [1] https://salsa.debian.org/mariadb-team/mariadb-server/-/commit/063109a306a01… [2] https://lists.debian.org/debian-devel-announce/2024/02/msg00000.html [3] https://buildd.debian.org/status/package.php?p=mariadb&suite=experimental [4] https://github.com/MariaDB/server/commits/bb-11.4-timestamp/ [5] https://github.com/MariaDB/server/commits/bb-11.4-monty/ [6] https://jira.mariadb.org/browse/MDEV-32188

2 6

Re: [PATCH 2/2] MDEV-33551: Semi-sync Wait Point AFTER_COMMIT Slow on Workloads with Heavy Concurrency
by Kristian Nielsen 06 Mar '24

06 Mar '24

> commit c51636f254703602f6f6e2e4a260e607e737b9c1 (origin/10.6-MDEV-33551) > Author: Brandon Nesterenko <brandon.nesterenko(a)mariadb.com> > Date: Mon Feb 26 07:01:47 2024 -0700 > > MDEV-33551: Semi-sync Wait Point AFTER_COMMIT Slow on Workloads with Heavy Concurrency First some overall comments. The approach of waking up each thread on its own condition seems good to me. Monty had a different suggestion, which is to have only a single ack sent per group commit; this could also be a possibility, but it seems orthogonal to your fix (ie. both could make sense). Monty also had the idea to move the AFTER_COMMIT sync point into the group commit code, though that changes the semantics of AFTER_COMMIT somewhat, especially with --innodb-flush-log-at-trx-commit=3 (or non-InnoDB storage engine). So in this review I'll focus on keeping your overall approach the same. Your patch introduces a new wait_queue with the list of waiting threads. But there is already a very similar queue, m_active_tranxs. It looks to me as if the m_active_tranx could easily be extended so it can be used for this, and avoid introducing another queue. Your wait_queue is pushed in write_tranx_in_binlog() which already does m_active_tranxs->insert_tranx_node(), and it is popped in report_reply_binlog() which also does m_active_tranxs->clear_active_tranx_nodes(). So add some info to struct Tranx_node, maybe a THD * is all that is needed? A bit of care must be taken about lifetime of Tranx_node and THD respectively (in case one goes away but not the other). But the whole m_active_tranxs is already protected by a lock, and there is a hash for looking up specific position, so I think a THD that wants to abort its wait can simply clear the THD * from its Tranx_node or something like that to ensure the semisync ack will not try to wake up a THD that no longer exists. You'll need to see how this will affect the code for shutdown (await_all_slave_replies()) where it needs to wait for all acks (IIUC). But I hope we can be pragmatic here and find something simple that works. Another general thing related to this: > 3) Repl_semi_sync_master::commit_trx() no longer loops to await > its specific ACK. It waits once, and will either fail from > timeout, or receive its ACK. We still need the loop to wait. From `man pthread_cond_wait`: "When using condition variables there is always a Boolean predicate in‐ volving shared variables associated with each condition wait that is true if the thread should proceed. Spurious wakeups from the pthread_cond_timedwait() or pthread_cond_wait() functions may occur. Since the return from pthread_cond_timedwait() or pthread_cond_wait() does not imply anything about the value of this predicate, the predi‐ cate should be re-evaluated upon such return." This affects the code in the patch in some places, detailed comments below. > 2) The time when thd::is_awaiting_semi_sync_ack is set is moved > to at binlogging time, to ensure transactions which have been > binlogged and queued up to await an ACK are not killed, > and are still waited on. I don't understand this. Why shouldn't a session waiting for ACK be killable? On the contrary, such a wait can take potentially a long time, seems important that it can be killed? I don't fully understand the existing code for this, maybe the patch is correct and I'm just looking for an explanation why it is correct like this. Or maybe this is a limitation of the existing code, and your patch doesn't change it (so having the ability to kill a waiting thread could be added in a different patch)? > diff --git a/mysql-test/suite/rpl/t/rpl_semi_sync_cond_var_per_thd.test b/mysql-test/suite/rpl/t/rpl_semi_sync_cond_var_per_thd.test > index f8fa0a99d9c..ebe15a4ca32 100644 > --- a/mysql-test/suite/rpl/t/rpl_semi_sync_cond_var_per_thd.test > +++ b/mysql-test/suite/rpl/t/rpl_semi_sync_cond_var_per_thd.test > @@ -22,6 +22,7 @@ > --source include/master-slave.inc > > --connection master > +call mtr.add_suppression("Got an error reading communication packets"); Is this suppression needed? If so, why? > diff --git a/sql/mysqld.cc b/sql/mysqld.cc > index e224871795e..b315edc091c 100644 > --- a/sql/mysqld.cc > +++ b/sql/mysqld.cc > @@ -1750,18 +1750,12 @@ static void close_connections(void) > /* > If we are waiting on any ACKs, delay killing the thread until either an ACK > is received or the timeout is hit. > - while (waiting_threads-- > 0) > - repl_semisync_master.await_slave_reply(); > + repl_semisync_master.await_all_slave_replies(); Just curious here, why did you change this code? Was it just because now there is no general COND_binlog_send to wait on? Or was it fix a bug not directly related to the main issue? I'm asking because if it makes things simpler, I think it's fine to keep the COND_binlog_send and signal it in addition to the individual thread-specific condition (in fact your patch doesn't seem to remove it though it looks as if it's no longer used). On the other hand, this old code in close_connections() that simply calls await_slave_reply() N times (N = sync_get_master_wait_sessions()) looks quite wrong to me, looping for a specific number of times makes no sense. So fixing it is good in any case. > diff --git a/sql/semisync_master.cc b/sql/semisync_master.cc > index 0eaf0f0e0e2..f421e924a61 100644 > --- a/sql/semisync_master.cc > +++ b/sql/semisync_master.cc > @@ -502,16 +502,33 @@ void Repl_semi_sync_master::unlock() > > void Repl_semi_sync_master::cond_broadcast() > { > - mysql_cond_broadcast(&COND_binlog_send); > + while (!wait_queue.empty()) > + { > + semisync_wait_trx_t next_waiter= wait_queue.front(); Maybe this function should now have a better name than cond_broadcast() (wakeup_ready_tranx() or something?). But this code will probably change in any case if the wait_queue is folded into m_active_tranxs. > -int Repl_semi_sync_master::cond_timewait(struct timespec *wait_time) > +int Repl_semi_sync_master::cond_timewait(THD *thd, struct timespec *wait_time) I think this function now makes little sense, clearer just to call directly mysql_cond_wait(thd->COND_wakeup_ready) at the caller, what do you think? > @@ -695,12 +712,36 @@ int Repl_semi_sync_master::report_reply_binlog(uint32 server_id, > + lock(); > + while (!wait_queue.empty()) > + { > + semisync_wait_trx_t next_waiter= wait_queue.front(); > > - cond_broadcast(); > + cmp= Active_tranx::compare(m_reply_file_name, m_reply_file_pos, > + next_waiter.binlog_name, next_waiter.binlog_pos); > + if (cmp >= 0) > + { > + DBUG_PRINT("semisync", ("%s: signal thread %llu.", > + "Repl_semi_sync_master::report_reply_binlog", > + next_waiter.thd->thread_id)); > + mysql_cond_signal(&next_waiter.thd->COND_wakeup_ready); > + wait_queue.pop(); This code is very similar to Active_tranx::clear_active_tranx_nodes(), so hopefully the two can be integrated and wait_queue becomes unnecessary. > @@ -779,7 +821,20 @@ int Repl_semi_sync_master::report_binlog_update(THD* thd, const char *log_file, > strcpy(log_info->log_file, log_file + dirname_length(log_file)); > log_info->log_pos = log_pos; > > - return write_tranx_in_binlog(log_info->log_file, log_pos); > + /* > + THD arg depends on wait point mode. If after storage engine commit, the > + individual connection threads will perform the wait for semi-sync ACKt, > + thd is the thread of the user connection thread. > + If it is after binlog sync, the binlog leader thread will perform the > + semi-sync waits on behalf of the grouped transaction (which at this > + point, we (current_thd) are the leader). If using binlog_group_commit, > + thd is the thread of the user connection thread. > + */ > + thd_to_cond_wait= ((wait_point() == SEMI_SYNC_MASTER_WAIT_POINT_AFTER_STORAGE_COMMIT) > + ? thd > + : current_thd); This condition is complex. It's good that you added the detailed comment explaining what's going on, but I think it would be better to avoid it completely. Why not just pass in the value of thd_to_cond_wait explicitly from the caller? As an aside, we have this code in trx_group_commit_leader(): for (current= queue; current != NULL; current= current->next) { current->error= repl_semisync_master.wait_after_sync(current->cache_mngr-> last_commit_pos_file, current->cache_mngr-> last_commit_pos_offset); It seems silly to wait for every binlog position to be acked her one by one, why not just wait for the last one? repl_semisync_master.wait_after_sync(last_in_queue->cache_mngr->last_commit_pos_file, ...); But while related, it's perhaps not directly part of the problem you're addressing in your patch, so up to you if you want to include this or leave it for a possible later patch. > @@ -852,12 +907,11 @@ int Repl_semi_sync_master::commit_trx(const char* trx_wait_binlog_name, > - if (!get_master_enabled() || !is_on()) > + if (!get_master_enabled() || !is_on() || thd_killed(thd)) > goto l_end; > - while (is_on() && !thd_killed(thd)) > - { > - /* We have to check these again as things may have changed */ > - if (!rpl_semi_sync_master_clients && !rpl_semi_sync_master_wait_no_slave) > - { > - aborted= 1; > - break; > - } As described earlier, because of the (at least theoretical) possibility of spurious wakeup, we should keep the while loop here and the checks. > @@ -1253,6 +1294,8 @@ int Repl_semi_sync_master::write_tranx_in_binlog(const char* log_file_name, > } > else > { > + wait_queue.push({log_file_name, log_file_pos, thd}); Hopefully this can be instead folded into m_active_tranxs->insert_tranx_node() which is called just a bit below. > -void Repl_semi_sync_master::await_slave_reply() > +void Repl_semi_sync_master::await_all_slave_replies() > + while (TRUE) > + { > + lock(); > + if (wait_queue.empty() || !get_master_enabled() || !is_on()) > + { > + unlock(); > + break; > + } > + front= wait_queue.front(); > create_timeout(&abstime, NULL); > - cond_timewait(&abstime); > - > -end: > + wait_result= cond_timewait(front.thd, &abstime); I'm wondering, is it safe here to wait on front.thd->COND_wakeup_ready? Isn't there a possibility that this might go away during the wait? A simple way to avoid this could be just to keep the original global COND_binlog_send and wait for that here (and signal both that and thd->COND_wakeup_ready when waking up). IIUC, this code will only run in a single thread during shutdown, so probably fine to have some extra unnecessary wakeups here. > diff --git a/sql/semisync_master.h b/sql/semisync_master.h > index 99f46869354..6de0333d4ce 100644 > --- a/sql/semisync_master.h > +++ b/sql/semisync_master.h > @@ -21,6 +21,7 @@ > +/* > + Element in Repl_semi_sync_master::wait_queue to preserve the state of a > + transaction waiting for an ACK. > +*/ > +typedef struct _semisync_wait_trx { > + const char *binlog_name; > + my_off_t binlog_pos; > + THD *thd; > +} semisync_wait_trx_t; > + std::queue<semisync_wait_trx_t> wait_queue; Let's see if we can avoid this, and use the existing m_active_tranxs queue instead. diff --git a/sql/semisync_master.cc b/sql/semisync_master.cc index 8cc721e5737..0eaf0f0e0e2 100644 --- a/sql/semisync_master.cc +++ b/sql/semisync_master.cc @@ -979,6 +979,14 @@ int Repl_semi_sync_master::commit_trx(const char* trx_wait_binlog_name, { rpl_semi_sync_master_trx_wait_num++; rpl_semi_sync_master_trx_wait_time += wait_time; + /* + Assert we have either recieved our ACK; or have timed out and are + awoken in an off state. + */ + DBUG_ASSERT(!get_master_enabled() || !is_on() || thd->is_killed() || + 0 <= Active_tranx::compare( + m_reply_file_name, m_reply_file_pos, + trx_wait_binlog_name, trx_wait_binlog_pos)); } } } I think it's a bit much to crash the server here (even if only debug build), because of the possibility of spurious wakeups. It's good to have the test case though, maybe put the assertion inside DBUG_EXECUTE_IF()? I'd make it just log an error in the error log which the test case could flag (to avoid crashing), but that's a matter of style I guess and up to you. - Kristian.

2 2

Re: 95f2a07d323: MDEV-33582 Add more warnings to be able to better diagnose network issues
by Sergei Golubchik 05 Mar '24

05 Mar '24

Hi, Michael, On Mar 04, Michael Widenius wrote: > revision-id: 95f2a07d323 (mariadb-10.6.17-30-g95f2a07d323) > parent(s): 33dcf8155b4 > author: Michael Widenius > committer: Michael Widenius > timestamp: 2024-03-04 14:52:03 +0200 > message: > > MDEV-33582 Add more warnings to be able to better diagnose network issues > > Warnings are added to net_server.cc when > global_system_variables.log_warnings >= 4. > > When the above condition holds then: > - All communication errors from net_serv.cc is also written to the > general log. "to the error log" > - In case of a of not being able to read or write a packet, a more detailed "of a of" ? > error is given. > > Other things: > - vio_close() is now first marking the socket closed before closing it. > The reason for this is to ensure that the connection that gets a read > error can check if the reason was that the socket was closed. > - Added option to NET to avoid calling my_error() when there is no data > to be read on socket. This is only used in semi sync master to speed > up handling zero size packages that seams to happen from time to time. "seems" I don't understand why it happens and whether it's normal or not, and if normal, why it's normal only for semisync and not normal otherwise. > - Add a new state to vio to be able to detect if vio is acive, shutdown or > closed. > - Testing of the new warnings is done in rpl_get_lock.test > - Suppress some of the new warnings in mtr to allow one to run some of > the tests with -mysqld=--log-warnings=4. All test in the 'rpl' suite > can now be run with this option. > - Ensure that global.log_warnings are restored at test end in a way > that allows one to use mtr --mysqld=--log-warnings=4. > > diff --git a/mysql-test/mariadb-test-run.pl b/mysql-test/mariadb-test-run.pl > index 4b3d323760d..eabfe1171cc 100755 > --- a/mysql-test/mariadb-test-run.pl > +++ b/mysql-test/mariadb-test-run.pl > @@ -4486,6 +4486,11 @@ sub extract_warning_lines ($$) { > qr/Slave I\/0: Master command COM_BINLOG_DUMP failed/, > qr/Error reading packet/, > qr/Lost connection to MariaDB server at 'reading initial communication packet'/, > + qr/Could not read packet:.* state: [2-3] /, > + qr/Could not read packet:.* errno: 104 /, > + qr/Could not write packet:.* errno: 32 /, > + qr/Could not write packet:.* errno: 104 /, > + qr/Connection was killed/, why are these errors "normal" and can be ignored? > qr/Failed on request_dump/, > qr/Slave: Can't drop database.* database doesn't exist/, > qr/Slave: Operation DROP USER failed for 'create_rout_db'/, > diff --git a/mysql-test/suite/rpl/include/rpl_extra_col_master.test b/mysql-test/suite/rpl/include/rpl_extra_col_master.test > index 02645a577f5..512118f91cd 100644 > --- a/mysql-test/suite/rpl/include/rpl_extra_col_master.test > +++ b/mysql-test/suite/rpl/include/rpl_extra_col_master.test > @@ -59,6 +59,10 @@ > #VARCHAR(M) > # > > +--disable_query_log > +call mtr.add_suppression("Could not read packet:.* errno: 11"); > +--enable_query_log error 11? "Resource temporarily unavailable"? What does it mean in this context? > + > --let $_saved_conn= $CURRENT_CONNECTION > > let $binformat = `SHOW VARIABLES LIKE '%binlog_format%'`; > diff --git a/sql/semisync_master_ack_receiver.cc b/sql/semisync_master_ack_receiver.cc > index 9320c7fc3e6..1bb462ef934 100644 > --- a/sql/semisync_master_ack_receiver.cc > +++ b/sql/semisync_master_ack_receiver.cc > @@ -256,6 +256,12 @@ void Ack_receiver::run() > thd->security_ctx->skip_grants(); > thd->set_command(COM_DAEMON); > init_net(&net, net_buff, REPLY_MESSAGE_MAX_LENGTH); > + /* > + Don't generate error message in case of no data on socket. Will speed > + up my_net_read() slightly in case of NET_READ_INTERRUPTED. > + */ > + net.no_errors_for_no_data= 1; do you mean, it's for performance reasons? > + Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 1

Re: ea9a05f0821: This commit is about improving repair of tables
by Sergei Golubchik 04 Mar '24

04 Mar '24

Hi, Michael, First, instead of situationally fixing repair code here and there, let's take a step back and see how repair works. I've walked the whole REPAIR code line by line, including all storage engines, and that's a simplified description: * if USE_FRM was specified * < FRM_VER_TRUE_VARCHAR(≈2005) && has varchar -> "too old" * if not MyISAM or Aria or CSV -> silently ignore USE_FRM * rename/truncate/rename * if ALGORITHM=COPY is needed (and not FORCE/USE_FRM), use it * otherwise, per engine: * MyISAM run repair, QUICK or EXTENDED, or QUICK then EXTENDED * Aria, same as MyISAM * S3 -> "Table '%-.192s' is read only" * CSV crashes * Archive, perform OPTIMIZE * all other engines * if needs upgrade: use ALTER ALGORITHM=COPY * otherwise "The storage engine for the table doesn't support REPAIR" * if ALGORITHM=COPY is needed but wasn't used -> "Operation failed" I think it's too complex to explain and not user friendly. Note that MyISAM internally can try one repair method and if that fails fall back to another one. So does Aria. Why cannot the server do the same? So, I'd suggest the following logic instead: * if USE_FRM - do USE_FRM stuff (or an warning) * if marked corrupted or EXTENDED or doesn't need upgrade - run repair * if repair isn't supported = ALTER IGNORE unless needs upgrade * if needs upgrade - run upgrade = ALTER IGNORE COPY * if fails because of corruption - go to step 2 if repair wasn't run yet here USE_FRM is handled as before (except that it adds a warning if USE_FRM isn't supported by the engine). After that the server runs engine's repair if the table is marked corrupted or if EXTENDED was used (so no need for a new FORCE syntax). There is no error "engine does not support REPAIR", the server falls back to ALTER automatically in that case. If the data need to be upgraded and ALTER fails, the server tries repair and then ALTER again - just like MyISAM and Aria try different repair methods. Also, note that ALTER should always use IGNORE to delete duplicates in unique after collation of hash changes - that's what repair in MyISAM/Aria does. Anyway, some detailed comments to your patch are inline, please, see below. On Feb 21, Michael Widenius wrote: > commit ea9a05f0821 > Author: Michael Widenius <monty(a)mariadb.org> > Date: Mon Jan 29 11:52:44 2024 +0200 > > This commit is about improving repair of tables now as there is an MDEV, it should be here in the subject line > > The 3 ways to repair a table are: > 1) ALTER TABLE table_name FORCE" (not other options). > As an alias we allow: "ALTER TABLE table_name ENGINE=original_engine" > 2) "REPAIR TABLE" (without FORCE) > 3) "OPTIMIZE TABLE" > > All of the above commands will optimize row space usage (which means that > space will be needed to hold a temporary copy of the table) and > re-genereate all indexes. They will also try to replicate the original > table definition as exact as possible. > > For ALTER TABLE and "REPAIR TABLE without FORCE", the following will hold: > If the table is from an older MariaDB version and data conversion is > needed (for example for old type HASH columns, MySQL JSON type or new > TIMESTAMP format) "ALTER TABLE table_name FORCE, algorithm=COPY" will be > used. > > The differences between the algoritms are > 1) Will use the fastest algorithm the engine supports to do a full repair > of the table (except if data conversions are is needed). > 2) Will use the storage engine internal REPAIR facility (MyISAM, Aria). > If the engine does not support REPAIR then > "ALTER TABLE FORCE, ALGORITHM=COPY" will be used. > If there was data incompatibilites (which means that FORCE was used) > then there will be a warning after REPAIR that ALTER TABLE FORCE is > still needed. > The reason for this is that REPAIR may be able to go around data > errors (wrong incompatible data, crashed or unreadable sectors) that > ALTER TABLE cannot do. > 3) Will use the storage engine internal OPTIMIZE. If engine does not > support optimize, then "ALTER TABLE FORCE" is used. > > The above will ensure that ALTER TABLE FORCE is able to > correct almost any errors in the row or index data. In case of > corrupted blocks then REPAIR possibile followed by ALTER TABLE is needed. > This is important as mariadb-upgrade executes ALTER TABLE table_name > FORCE for any table that must be re-created. > > Bugs fixed with InnoDB tables: > - No error for INNODB_DEFAULT_ROW_FORMAT=COMPACT even if row length > would be too wide. (Independent of innodb_strict_mode). > - Tables using symlinks will be symlinked after any of the above commands > (Independent of the setting of --symbolic-links) > > If one specifies an algorithm together with ALTER TABLE FORCE, things > will work as before (except if data conversion is required as then > the COPY algoritm is enforced). > > ALTER TABLE .. OPTIMIZE ALL PARTITIONS will work as before. > > Other things: > - FORCE argument added to REPAIR to allow one to first run internal > repair to fix damaged blocks and then follow it with ALTER TABLE. > - REPAIR will not update frm_version if ha_check_for_upgrade() finds > that table is still incompatible with current version. In this case the > REPAIR will end with an error. doesn't look like "the REPAIR will end with an error" > - Default error messages length for %M increased from 128 to 256 to not > cut information from REPAIR. > - Documented HA_ADMIN_XX variables related to repair. > - Added HA_ADMIN_NEEDS_DATA_CONVERSION to signal that we have to > do data conversions when converting the table (and thus ALTER TABLE > copy algorithm is needed). > > diff --git a/include/my_handler_errors.h b/include/my_handler_errors.h > index df414888907..06dde5b3f74 100644 > --- a/include/my_handler_errors.h > +++ b/include/my_handler_errors.h > @@ -101,7 +101,7 @@ static const char *handler_error_messages[]= > "Operation was interrupted by end user (probably kill command?)", > "Disk full", > /* 190 */ > - "Incompatible key or row definition between the MariaDB .frm file and the information in the storage engine. You may have retry or dump and restore the table to fix this", > + "Incompatible key or row definition between the MariaDB .frm file and the information in the storage engine. You can try REPAIR TABLE ... USE_FRM possible followed by ALTER TABLE ... FORCE or dump and restore the table to fix this", 1. typo. "possibly" 2. please, don't recommend USE_FRM, it's a last resort lossy method, designed only for the case when MYI file is lost or corrupted. 3. REPAIR USE_FRM is likey broken for CSV, but I couldn't test it, because REPAIR crashes on CSV tables with or without USE_FRM. > "Too many words in a FTS phrase or proximity search", > "Table encrypted but decryption failed. This could be because correct encryption management plugin is not loaded, used encryption key is not available or encryption method does not match.", > "Foreign key cascade delete/update exceeds max depth", > diff --git a/mysql-test/main/ctype_utf8_def_upgrade.result b/mysql-test/main/ctype_utf8_def_upgrade.result > index d30f8670536..e7c21b76058 100644 > --- a/mysql-test/main/ctype_utf8_def_upgrade.result > +++ b/mysql-test/main/ctype_utf8_def_upgrade.result > @@ -10,24 +10,39 @@ SELECT @@character_set_database; > latin1 > SET @@character_set_database="latin1"; > SELECT COUNT(*) FROM t1; > -ERROR HY000: Got error 190 "Incompatible key or row definition between the MariaDB .frm file and the information in the storage engine. You may have retry " from storage engine MyISAM > +ERROR HY000: Got error 190 "Incompatible key or row definition between the MariaDB .frm file and the information in the storage engine. You can try REPAIR TABLE ... USE_FRM possible followed by ALTER TABLE ... FORCE or dump and restore the table to fix this" from storage engine MyISAM > +ALTER TABLE t1 FORCE; > +ERROR HY000: Got error 190 "Incompatible key or row definition between the MariaDB .frm file and the information in the storage engine. You can try REPAIR TABLE ... USE_FRM possible followed by ALTER TABLE ... FORCE or dump and restore the table to fix this" from storage engine MyISAM > CHECK TABLE t1; > Table Op Msg_type Msg_text > -test.t1 check Error Got error 190 "Incompatible key or row definition between the MariaDB .frm file and the information in the storage engine. You may have retry " from storage engine MyISAM > +test.t1 check Error Got error 190 "Incompatible key or row definition between the MariaDB .frm file and the information in the storage engine. You can try REPAIR TABLE ... USE_FRM possible followed by ALTER TABLE ... FORCE or dump and restore the table to fix this" from storage engine MyISAM > test.t1 check error Corrupt > REPAIR TABLE t1; > Table Op Msg_type Msg_text > -test.t1 repair Error Got error 190 "Incompatible key or row definition between the MariaDB .frm file and the information in the storage engine. You may have retry " from storage engine MyISAM > +test.t1 repair Error Got error 190 "Incompatible key or row definition between the MariaDB .frm file and the information in the storage engine. You can try REPAIR TABLE ... USE_FRM possible followed by ALTER TABLE ... FORCE or dump and restore the table to fix this" from storage engine MyISAM > +test.t1 repair error Corrupt > +REPAIR TABLE t1 FORCE; > +Table Op Msg_type Msg_text > +test.t1 repair Warning Can't open table > test.t1 repair error Corrupt > REPAIR TABLE t1 USE_FRM; > Table Op Msg_type Msg_text > -test.t1 repair status OK > +test.t1 repair note Table data recovered > +t1 repair error Table rebuild required. Please do "ALTER TABLE `test.t1` FORCE" or dump/reload to fix it! better to avoid exclamation marks in the error messages! > +test.t1 repair status Operation failed > SELECT COUNT(*) FROM t1; > COUNT(*) > 0 > CHECK TABLE t1; > Table Op Msg_type Msg_text > +test.t1 check error Table rebuild required. Please do "ALTER TABLE `t1` FORCE" or dump/reload to fix it! > +ALTER TABLE t1 FORCE; > +CHECK TABLE t1; > +Table Op Msg_type Msg_text > test.t1 check status OK > +SELECT COUNT(*) FROM t1; > +COUNT(*) > +0 > SHOW CREATE TABLE t1; > Table Create Table > t1 CREATE TABLE `t1` ( > diff --git a/sql/handler.h b/sql/handler.h > index fe272d841d9..f429cdacc6f 100644 > --- a/sql/handler.h > +++ b/sql/handler.h > @@ -2249,6 +2267,8 @@ struct Table_scope_and_contents_source_pod_st // For trivial members > enum_stats_auto_recalc stats_auto_recalc; > bool varchar; ///< 1 if table has a VARCHAR > bool sequence; // If SEQUENCE=1 was used > + /* True if we are using OPTIMIZE TABLE, REPAIR TABLE or ALTER TABLE FORCE */ it'd be good to clarify here /* this is used to suppress certain warnings that are only used for CREATE/ALTER TABLE */ > + bool recreate_identical_table; > > List<Virtual_column_info> *check_constraint_list; > > diff --git a/sql/sql_admin.cc b/sql/sql_admin.cc > index fcbd8a55c15..d4eabd7c34a 100644 > --- a/sql/sql_admin.cc > +++ b/sql/sql_admin.cc > @@ -48,7 +48,8 @@ const LEX_CSTRING msg_optimize= { STRING_WITH_LEN("optimize") }; > /* Prepare, run and cleanup for mysql_recreate_table() */ > > static bool admin_recreate_table(THD *thd, TABLE_LIST *table_list, > - Recreate_info *recreate_info) > + Recreate_info *recreate_info, > + bool partition_admin, bool table_copy) you don't need to pass partition_admin as an argument, if it's always thd->lex->alter_info.partition_flags & ALTER_PARTITION_ADMIN > { > bool result_code; > DBUG_ENTER("admin_recreate_table"); > @@ -859,20 +867,29 @@ static bool mysql_admin_table(THD* thd, TABLE_LIST* tables, > /* purecov: end */ > } > > - if (operator_func == &handler::ha_repair && > - !(check_opt->sql_flags & TT_USEFRM)) > + if (operator_func == &handler::ha_repair) > { > handler *file= table->table->file; > int check_old_types= file->check_old_types(); > int check_for_upgrade= file->ha_check_for_upgrade(check_opt); why check_old_types is not part of ha_check_for_upgrade? they're always used together, as far as I can see. > + require_data_conversion= > + (check_old_types == HA_ADMIN_NEEDS_DATA_CONVERSION || > + check_for_upgrade == HA_ADMIN_NEEDS_DATA_CONVERSION); > + require_alter_table= > + (check_old_types == HA_ADMIN_NEEDS_ALTER || > + check_for_upgrade == HA_ADMIN_NEEDS_ALTER); > > - if (check_old_types == HA_ADMIN_NEEDS_ALTER || > - check_for_upgrade == HA_ADMIN_NEEDS_ALTER) > + if (!(check_opt->sql_flags & (TT_USEFRM | TT_FORCE))) > + { > + if (require_data_conversion || require_alter_table) > { > /* We use extra_open_options to be able to open crashed tables */ > thd->open_options|= extra_open_options; > - result_code= admin_recreate_table(thd, table, &recreate_info) ? > + result_code= admin_recreate_table(thd, table, &recreate_info, > + (lex->alter_info.partition_flags & > + ALTER_PARTITION_ADMIN), 1) ? > HA_ADMIN_FAILED : HA_ADMIN_OK; > + recreate_used= 1; > thd->open_options&= ~extra_open_options; > goto send_result; > } > if (check_old_types || check_for_upgrade) > { > /* If repair is not implemented for the engine, run ALTER TABLE */ > need_repair_or_alter= 1; > } > } > - > + } > result_code= compl_result_code= HA_ADMIN_OK; > > if (operator_func == &handler::ha_analyze) > @@ -1229,7 +1277,10 @@ static bool mysql_admin_table(THD* thd, TABLE_LIST* tables, > *save_next_global= table->next_global; > table->next_local= table->next_global= 0; > > - result_code= admin_recreate_table(thd, table, &recreate_info); > + result_code= admin_recreate_table(thd, table, &recreate_info, > + (lex->alter_info.partition_flags & > + ALTER_PARTITION_ADMIN), 0); > + recreate_used= 1; that's a bit too late, you check recreate_used earlier > trans_commit_stmt(thd); > trans_commit(thd); > close_thread_tables(thd); > @@ -1641,7 +1696,8 @@ bool Sql_cmd_repair_table::execute(THD *thd) > WSREP_TO_ISOLATION_BEGIN_WRTCHK(NULL, NULL, first_table); > res= mysql_admin_table(thd, first_table, &m_lex->check_opt, &msg_repair, > TL_WRITE, 1, > - MY_TEST(m_lex->check_opt.sql_flags & TT_USEFRM), > + MY_TEST(m_lex->check_opt.sql_flags & > + (TT_USEFRM | TT_FORCE)), better rename repair_table_use_frm argument of mysql_admin_table and open_only_one_table, because it's not only TT_USEFRM anymore. a good name could be no_errors_from_open > HA_OPEN_FOR_REPAIR, &prepare_for_repair, > &handler::ha_repair, &view_repair, true); > > diff --git a/sql/sql_table.cc b/sql/sql_table.cc > index 55319a2242f..7d5a968070a 100644 > --- a/sql/sql_table.cc > +++ b/sql/sql_table.cc > @@ -10868,19 +10869,56 @@ do_continue:; > } > #endif /* WITH_WSREP */ > > + /* > + ALTER TABLE ... ENGINE to the same engine is a common way to > + request table rebuild. Set ALTER_RECREATE flag to force table > + rebuild. > + */ > + if (new_db_type == old_db_type) > + { > + if (create_info->used_fields & HA_CREATE_USED_ENGINE) > + alter_info->flags|= ALTER_RECREATE; > + > + /* > + Check if we are using ALTER TABLE FORCE without any other options > + (except ENGINE == current_engine). > + In this case we will try to recreate an identical to the original. > + This code is also used with REPAIR and OPTIMIZE. > + */ > + if (alter_info->flags == ALTER_RECREATE && > + ((create_info->used_fields & ~HA_CREATE_USED_ENGINE) == 0)) > + create_info->recreate_identical_table= 1; > + } > + > /* > We can use only copy algorithm if one of the following is true: > + - If the table is from an old MariaDB version and requires data > + modifications. In this case we ignore --alter-algorithm as eh, no, you cannot ignore it, if the user specified incompatible set of options, ALTER TABLE has to fail with an error, saying ALGORITHM=INPLACE (for example) is impossible here. The server cannot just silently ignore an explicit user command. > + as we cannot use any other algorithm than COPY (using other > + algorithms could open up the problem that the table .frm version > + is updated without data transformations and the table would be > + corrupted without any way for MariaDB to notice this during > + check/upgrade). > + This logic ensurses that ALTER TABLE ... FORCE (no other > + options) will always be be able to repair a table structure and > + convert data from any old format. > - In-place is impossible for given operation. > - Changes to partitioning which were not handled by fast_alter_part_table() > needs to be handled using table copying algorithm unless the engine > supports auto-partitioning as such engines can do some changes > using in-place API. > */ > + check_opt.init(); > + > + require_copy_algorithm= (table->file->ha_check_for_upgrade(&check_opt) == > + HA_ADMIN_NEEDS_DATA_CONVERSION); > + if (require_copy_algorithm || > - if (is_inplace_alter_impossible(table, create_info, alter_info) > - || IF_PARTITIONING((partition_changed && > + is_inplace_alter_impossible(table, create_info, alter_info) || > + IF_PARTITIONING((partition_changed && > !(old_db_type->partition_flags() & HA_USE_AUTO_PARTITION)), 0)) > { > - if (alter_info->algorithm_is_nocopy(thd)) > + if (alter_info->algorithm_is_nocopy(thd) && > + !(require_copy_algorithm && alter_info->algorithm_not_specified())) > { > my_error(ER_ALTER_OPERATION_NOT_SUPPORTED, MYF(0), > alter_info->algorithm_clause(thd), "ALGORITHM=COPY"); > @@ -12451,10 +12482,14 @@ bool mysql_recreate_table(THD *thd, TABLE_LIST *table_list, > create_info.init(); > create_info.row_type=ROW_TYPE_NOT_USED; > create_info.alter_info= &alter_info; > + create_info.recreate_identical_table= 1; > /* Force alter table to recreate table */ > alter_info.flags= (ALTER_CHANGE_COLUMN | ALTER_RECREATE); > + alter_info.partition_flags= thd->lex->alter_info.partition_flags; > + if (partition_admin) > + alter_info.partition_flags= ALTER_PARTITION_ADMIN; when is this needed? I couldn't find a single code path where alter_info.partition_flags is not ALTER_PARTITION_ADMIN at this point. Neither code analysys nor mtr have found anything. You can safely remove this if() and add DBUG_ASSERT(alter_info.partition_flags & ALTER_PARTITION_ADMIN); > > - if (table_copy) > + if (table_copy && !partition_admin) > alter_info.set_requested_algorithm( > Alter_info::ALTER_TABLE_ALGORITHM_COPY); > > diff --git a/sql/table.cc b/sql/table.cc > index f84eeda297f..b09e9b11f17 100644 > --- a/sql/table.cc > +++ b/sql/table.cc > @@ -1846,6 +1846,8 @@ int TABLE_SHARE::init_from_binary_frm_image(THD *thd, bool write, > */ > if (share->frm_version == FRM_VER_TRUE_VARCHAR -1 && frm_image[33] == 5) > share->frm_version= FRM_VER_TRUE_VARCHAR; > + if (share->frm_version <= FRM_VER_TRUE_VARCHAR) > + share->keep_original_mysql_version= 1; this is wrong. see prepare_frm_header(), frm_version is *always* FRM_VER_TRUE_VARCHAR if there are no vcols. you've basically just disabled update_frm_version() for almost everyone. > > new_field_pack_flag= frm_image[27]; > new_frm_ver= (frm_image[2] - FRM_VER); Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 3

RISC-V CI for MariaDB
by ali.tariq＠10xengineers.ai 04 Mar '24

04 Mar '24

Hi all, I have seen the status of MariaDB in the RISC-V software ecosystem sheet (https://docs.google.com/spreadsheets/d/10rWj7QlzCiXwTZ778vex5_nRLP9gLrNC8Pw…) as "SUPPORTED" but not yet "FULLY SUPPORTED" or "Optimized" for RISC-V architecture, which means optimization for RISC-V is still in progress. I want to bump a CI platform called Cloud-V (https://cloud-v.co/) (created in collaboration with the RISC-V Labs subgroup) which provides CI for RISC-V with emulated and physical RISC-V computes. You can enroll for access at this link (https://cloud-v.co/contactus)

1 0

Storing binlig in InnoDB/engine
by Kristian Nielsen 28 Feb '24

28 Feb '24

Hi Marko, I looked a bit more at the idea that we have discussed a couple times, of storing the binlog in an InnoDB tablespace to avoid the need for two-phase commit between binlog and InnoDB and save one (or potentially both) fsync()s during a commit. I managed to get some InnoDB code written that is able to create a tablespace and write to it, patch below or on github: https://github.com/MariaDB/server/commits/knielsen_binlog_in_engine https://github.com/MariaDB/server/commit/95958c3842ebf0f7e358d6c3f51b887bd9… This is far from a complete patch, just an exercise for me on how InnoDB tablespaces and mini-transactions work in detail. But it turned out to be useful for me to get started and find a few questions to ask. My general approach to writing is to use fsp_page_create() for the first write to a page, and then buf_page_get_gen() for subsequent writes. But maybe this should be refined for the actual implementation. I'm thinking if perhaps fsp_page_create() does too much, you mentioned earlier that some parts of the page header could be simplified/omitted. And maybe there is a way to pin the current page in the buffer pool so buf_page_get_gen() is not needed for every write? I'm currently passing RW_SX_LATCH to buf_page_get_gen() (otherwise I got an assertion when writing). I'm not sure though how these latches work, or if binlog writing would need such latches; maybe it makes more sense to have a simple mutex protecting page access? Another assertion was fixed by doing mtr.set_named_space() before writing. Again, I'm not sure what this does exactly or if it's appropriate? I tried in this patch to reserve 2 "special" tablespace ids for the binlog tablespaces. Idea would be to cycle between them, keeping at most the two last tablespaces active. But do the tablespace IDs appear in the redo log and used for recovery? In that case, I assume that all binlog tablespaces written since the last InnoDB checkpoint will need a unique tablespace ID? So maybe 2 is too few. I was thinking maybe the binlog could allocate new tablespace IDs as necessary, but re-use them after each InnoDB checkpoint. This would avoid wasting ids and eventually hitting the 2**32 limit. - Kristian. commit 95958c3842ebf0f7e358d6c3f51b887bd9948845 (HEAD -> binlog_in_inno, origin/knielsen_binlog_in_engine) Author: Kristian Nielsen <knielsen(a)knielsen-hq.org> Date: Sun Feb 25 17:41:50 2024 +0100 Binlog in Engine: Very first sketch, able to create and write an InnoDB tablespace Signed-off-by: Kristian Nielsen <knielsen(a)knielsen-hq.org> diff --git a/mysql-test/suite/binlog/t/binlog_in_engine.test b/mysql-test/suite/binlog/t/binlog_in_engine.test new file mode 100644 index 00000000000..947139c9bcc --- /dev/null +++ b/mysql-test/suite/binlog/t/binlog_in_engine.test @@ -0,0 +1,11 @@ +--source include/have_innodb.inc +--source include/have_binlog_format_mixed.inc + +CREATE TABLE t1 (a INT PRIMARY KEY) ENGINE=InnoDB; +INSERT INTO t1 VALUES (1); +BEGIN; +INSERT INTO t1 VALUES (2); +INSERT INTO t1 VALUES (3); +COMMIT; +SELECT * FROM t1 ORDER BY a; +DROP TABLE t1; diff --git a/storage/innobase/buf/buf0flu.cc b/storage/innobase/buf/buf0flu.cc index bd43429eb5d..09d8fdaff39 100644 --- a/storage/innobase/buf/buf0flu.cc +++ b/storage/innobase/buf/buf0flu.cc @@ -1626,7 +1626,8 @@ static ulint buf_flush_list(ulint max_n= ULINT_UNDEFINED, bool buf_flush_list_space(fil_space_t *space, ulint *n_flushed) { const auto space_id= space->id; - ut_ad(space_id <= SRV_SPACE_ID_UPPER_BOUND); + ut_ad(space_id <= SRV_SPACE_ID_UPPER_BOUND || + space_id == SRV_SPACE_ID_BINLOG0 || space_id == SRV_SPACE_ID_BINLOG1); bool may_have_skipped= false; ulint max_n_flush= srv_io_capacity; diff --git a/storage/innobase/fil/fil0fil.cc b/storage/innobase/fil/fil0fil.cc index 0ce54df6574..e83225a4883 100644 --- a/storage/innobase/fil/fil0fil.cc +++ b/storage/innobase/fil/fil0fil.cc @@ -184,7 +184,7 @@ it is an absolute path. */ const char* fil_path_to_mysql_datadir; /** Common InnoDB file extensions */ -const char* dot_ext[] = { "", ".ibd", ".isl", ".cfg" }; +const char* dot_ext[] = { "", ".ibd", ".isl", ".cfg", ".ibb" }; /** Number of pending tablespace flushes */ Atomic_counter<ulint> fil_n_pending_tablespace_flushes; @@ -1044,6 +1044,9 @@ fil_space_t *fil_space_t::create(uint32_t id, uint32_t flags, if (UNIV_LIKELY(id <= fil_system.max_assigned_id)) { break; } + if (id == SRV_SPACE_ID_BINLOG0 || id == SRV_SPACE_ID_BINLOG1) { + break; + } if (UNIV_UNLIKELY(srv_operation == SRV_OPERATION_BACKUP)) { break; } @@ -1603,9 +1606,10 @@ inline void mtr_t::log_file_op(mfile_type_t type, uint32_t space_id, ut_ad(!(byte(type) & 15)); /* fil_name_parse() requires that there be at least one path - separator and that the file path end with ".ibd". */ + separator and that the file path end with ".ibd" or "ibb". */ ut_ad(strchr(path, '/')); - ut_ad(!strcmp(&path[strlen(path) - strlen(DOT_IBD)], DOT_IBD)); + ut_ad(!strcmp(&path[strlen(path) - strlen(DOT_IBD)], DOT_IBD) || + !strcmp(&path[strlen(path) - strlen(DOT_IBB)], DOT_IBB)); m_modifications= true; if (!is_logged()) diff --git a/storage/innobase/fsp/fsp0fsp.cc b/storage/innobase/fsp/fsp0fsp.cc index 787bda53895..f1bb42b3a15 100644 --- a/storage/innobase/fsp/fsp0fsp.cc +++ b/storage/innobase/fsp/fsp0fsp.cc @@ -3763,3 +3763,129 @@ void fsp_shrink_temp_space() mtr.commit(); sql_print_information("InnoDB: Temporary tablespace truncated successfully"); } + + + +fil_space_t* binlog_space; +buf_block_t *binlog_cur_block; +uint32_t binlog_cur_page_no; +uint32_t binlog_cur_page_offset; + +/** Create a binlog tablespace file +@param[in] name file name +@return DB_SUCCESS or error code */ +dberr_t fsp_binlog_tablespace_create(const char* name) +{ + pfs_os_file_t fh; + bool ret; + + uint32_t size= (1<<20) >> srv_page_size_shift /* ToDo --max-binlog-size */; + if(srv_read_only_mode) + return DB_ERROR; + + os_file_create_subdirs_if_needed(name); + + /* ToDo: Do we need here an mtr.log_file_op(FILE_CREATE) like in fil_ibd_create(()? */ + fh = os_file_create( + innodb_data_file_key, + name, + OS_FILE_CREATE | OS_FILE_ON_ERROR_NO_EXIT, + OS_FILE_AIO, OS_DATA_FILE, srv_read_only_mode, &ret); + + if (!ret) { + os_file_close(fh); + return DB_ERROR; + } + + /* ToDo: Enryption? */ + fil_encryption_t mode= FIL_ENCRYPTION_OFF; + fil_space_crypt_t* crypt_data= nullptr; + + /* We created the binlog file and now write it full of zeros */ + if (!os_file_set_size(name, fh, + os_offset_t{size} << srv_page_size_shift)) { + ib::error() << "Unable to allocate " << name; + os_file_close(fh); + os_file_delete(innodb_data_file_key, name); + return DB_ERROR; + } + + mysql_mutex_lock(&fil_system.mutex); + uint32_t space_id= SRV_SPACE_ID_BINLOG0; + if (!(binlog_space= fil_space_t::create(space_id, + ( FSP_FLAGS_FCRC32_MASK_MARKER | + FSP_FLAGS_FCRC32_PAGE_SSIZE()), + FIL_TYPE_TABLESPACE, crypt_data, + mode, true))) { + mysql_mutex_unlock(&fil_system.mutex); + return DB_ERROR; + } + + fil_node_t* node = binlog_space->add(name, fh, size, false, true); + IF_WIN(node->find_metadata(), node->find_metadata(fh, true)); + mysql_mutex_unlock(&fil_system.mutex); + + binlog_cur_page_no= 0; + binlog_cur_page_offset= FIL_PAGE_DATA; + return DB_SUCCESS; +} + +void fsp_binlog_write_start(uint32_t page_no, + const uchar *data, uint32_t len, mtr_t *mtr) +{ + buf_block_t *block= fsp_page_create(binlog_space, page_no, mtr); + mtr->memcpy<mtr_t::MAYBE_NOP>(*block, FIL_PAGE_DATA + block->page.frame, + data, len); + binlog_cur_block= block; +} + +void fsp_binlog_write_offset(uint32_t page_no, uint32_t offset, + const uchar *data, uint32_t len, mtr_t *mtr) +{ + dberr_t err; + /* ToDo: Is RW_SX_LATCH appropriate here? */ + buf_block_t *block= buf_page_get_gen(page_id_t{binlog_space->id, page_no}, + 0, RW_SX_LATCH, binlog_cur_block, + BUF_GET, mtr, &err); + ut_a(err == DB_SUCCESS); + mtr->memcpy<mtr_t::MAYBE_NOP>(*block, + offset + block->page.frame, + data, len); +} + +void fsp_binlog_append(const uchar *data, uint32_t len, mtr_t *mtr) +{ + ut_ad(binlog_cur_page_offset <= srv_page_size - FIL_PAGE_DATA_END); + uint32_t remain= ((uint32_t)srv_page_size - FIL_PAGE_DATA_END) - + binlog_cur_page_offset; + // ToDo: Some kind of mutex to protect binlog access. + while (len > 0) { + if (remain < 4) { + binlog_cur_page_offset= FIL_PAGE_DATA; + remain= ((uint32_t)srv_page_size - FIL_PAGE_DATA_END) - + binlog_cur_page_offset; + ++binlog_cur_page_no; + } + uint32_t this_len= std::min<uint32_t>(len, remain); + if (binlog_cur_page_offset == FIL_PAGE_DATA) + fsp_binlog_write_start(binlog_cur_page_no, data, this_len, mtr); + else + fsp_binlog_write_offset(binlog_cur_page_no, binlog_cur_page_offset, + data, this_len, mtr); + len-= this_len; + data+= this_len; + binlog_cur_page_offset+= this_len; + } +} + + +void fsp_binlog_test(const uchar *data, uint32_t len) +{ + mtr_t mtr; + mtr.start(); + if (!binlog_space) + fsp_binlog_tablespace_create("./binlog-000000.ibb"); + mtr.set_named_space(binlog_space); + fsp_binlog_append(data, len, &mtr); + mtr.commit(); +} diff --git a/storage/innobase/handler/ha_innodb.cc b/storage/innobase/handler/ha_innodb.cc index 93127bb1c3a..df2bd07d2dc 100644 --- a/storage/innobase/handler/ha_innodb.cc +++ b/storage/innobase/handler/ha_innodb.cc @@ -4481,6 +4481,10 @@ innobase_commit( if (commit_trx || (!thd_test_options(thd, OPTION_NOT_AUTOCOMMIT | OPTION_BEGIN))) { + /* ToDo: This is just a random very initial test of writing + something into a binlog tablespace. */ + if (!opt_bootstrap) + fsp_binlog_test((const uchar *)"Hulubulu!!?!", 12); /* Run the fast part of commit if we did not already. */ if (!trx->active_commit_ordered) { innobase_commit_ordered_2(trx, thd); diff --git a/storage/innobase/include/fil0fil.h b/storage/innobase/include/fil0fil.h index f3660eff7c6..17b35f2f892 100644 --- a/storage/innobase/include/fil0fil.h +++ b/storage/innobase/include/fil0fil.h @@ -1129,10 +1129,12 @@ enum ib_extention { NO_EXT = 0, IBD = 1, ISL = 2, - CFG = 3 + CFG = 3, + IBB = 4 }; extern const char* dot_ext[]; #define DOT_IBD dot_ext[IBD] +#define DOT_IBB dot_ext[IBB] #define DOT_ISL dot_ext[ISL] #define DOT_CFG dot_ext[CFG] diff --git a/storage/innobase/include/fsp0fsp.h b/storage/innobase/include/fsp0fsp.h index ddc45e53fe6..26a45518ba2 100644 --- a/storage/innobase/include/fsp0fsp.h +++ b/storage/innobase/include/fsp0fsp.h @@ -579,6 +579,8 @@ void fsp_system_tablespace_truncate(); /** Truncate the temporary tablespace */ void fsp_shrink_temp_space(); +extern void fsp_binlog_test(const uchar *data, uint32_t len); + #ifndef UNIV_DEBUG # define fsp_init_file_page(space, block, mtr) fsp_init_file_page(block, mtr) #endif diff --git a/storage/innobase/include/fsp0types.h b/storage/innobase/include/fsp0types.h index 757ead55d03..e3d45796190 100644 --- a/storage/innobase/include/fsp0types.h +++ b/storage/innobase/include/fsp0types.h @@ -27,8 +27,12 @@ Created May 26, 2009 Vasil Dimov #pragma once #include "ut0byte.h" -/** All persistent tablespaces have a smaller fil_space_t::id than this. */ +/** All persistent tablespaces (except binlog tablespaces) have a smaller +fil_space_t::id than this. */ constexpr uint32_t SRV_SPACE_ID_UPPER_BOUND= 0xFFFFFFF0U; +/** Binlog tablespaces. */ +constexpr uint32_t SRV_SPACE_ID_BINLOG0 = SRV_SPACE_ID_UPPER_BOUND + 1; +constexpr uint32_t SRV_SPACE_ID_BINLOG1 = SRV_SPACE_ID_UPPER_BOUND + 2; /** The fil_space_t::id of the innodb_temporary tablespace. */ constexpr uint32_t SRV_TMP_SPACE_ID= 0xFFFFFFFEU; diff --git a/storage/innobase/include/mtr0log.h b/storage/innobase/include/mtr0log.h index e2419309764..86f6e3794f6 100644 --- a/storage/innobase/include/mtr0log.h +++ b/storage/innobase/include/mtr0log.h @@ -25,7 +25,7 @@ Mini-transaction log record encoding and decoding #include "mtr0mtr.h" /** The smallest invalid page identifier for persistent tablespaces */ -constexpr page_id_t end_page_id{SRV_SPACE_ID_UPPER_BOUND, 0}; +constexpr page_id_t end_page_id{SRV_SPACE_ID_BINLOG1, 0}; /** The minimum 2-byte integer (0b10xxxxxx xxxxxxxx) */ constexpr uint32_t MIN_2BYTE= 1 << 7; diff --git a/storage/innobase/log/log0recv.cc b/storage/innobase/log/log0recv.cc index ef31a4d00c1..310acb73071 100644 --- a/storage/innobase/log/log0recv.cc +++ b/storage/innobase/log/log0recv.cc @@ -2973,7 +2973,8 @@ recv_sys_t::parse_mtr_result recv_sys_t::parse(source &l, bool if_exists) if (is_predefined_tablespace(space_id)) goto file_rec_error; - if (fnend - fn < 4 || memcmp(fnend - 4, DOT_IBD, 4)) + if (fnend - fn < 4 || + (memcmp(fnend - 4, DOT_IBD, 4) && memcmp(fnend - 4, DOT_IBB, 4))) goto file_rec_error; if (UNIV_UNLIKELY(!recv_needed_recovery && srv_read_only_mode))

2 5

[PATCH 0/2] Suggestion for smaller fix to XA parallel replication performance, MDEV-31949
by Kristian Nielsen 28 Feb '24

28 Feb '24

As was discussed recently off-list, there is a desire to find a minimal fix to the performance issues in 10.6 with parallel replication of user XA due to non-optimal scheduling (MDEV-31949). How about something like this two-patch series? It _only_ modifies the scheduling of event groups to worker threads, and only for XA transactions, and is quite a bit simpler than the other proposed patches for MDEV-31949. The current 10.6 code for parallel replication of user XA needs to avoid applying in parallel two different event groups that have the same XA XID. Typically the XA PREPARE and corresponding XA COMMIT/ROLLBACK, but could also be another transaction that uses a duplicate XID. The method currently used in 10.6 takes a hash of the XID and assigns the transaction to worker (hash MOD N) amongst the N workers. This is sub-optimal, since there is a high chance that amonst N transactions, several of them will map to the same worker. Thus less than N transactions will be able to replicate in parallel and group commit together. The second patch in the series improves this by keeping track of recently active XIDs and which worker they were scheduled on. A subsequent event group with the same XID is scheduled on the same worker. If the XID is different, the event group is scheduled normally on the next free worker in round-robin fashion. This permits most XA PREPARE to be scheduled without dependency restrictions, same as normal non-XA transactions. Only if the user submits multiple XA transactions close together using the same duplicate XID will there be dependency restrictions. The XA COMMIT will normally be scheduled on the same worker as the XA PREPARE, unless the two events are far apart in the replication stream. This is mostly unavoidable in the current XA replication design, since the XA PREPARE and XA COMMIT of a single transaction cannot group-commit together. The first patch is a refactor/preparation patch. It rewrites the scheduling to use an explicit FIFO for scheduling the threads instead of a simple `i := (i+1) % N` cyclic counter. This allows to combine explicit scheduling of some transactions with round-robin scheduling of the rest in the second patch. Patches also available on github: https://github.com/MariaDB/server/commits/knielsen_xa_sched_minimal_fix https://github.com/MariaDB/server/commit/66d6cce96f831b638812490844d7542317… https://github.com/MariaDB/server/commit/3a32fb546c111e4627cad7c66bcc089bc1… This is just a quick proof-of-concept I did today, but it seems to work, and is small and simple and suitable for targeted fixing and cherry-picking. Brandon and/or Andrei, maybe you could try your benchmarks with this patch and see if it also solves the performance issues you were seeing? Or if there's something else required for XA that I'm missing? - Kristian. Kristian Nielsen (2): Refactor parallel replication round-robin scheduling to use explicit FIFO More precise dependency tracking of XA XID in parallel replication sql/rpl_parallel.cc | 158 ++++++++++++++++++++++++++++++++++++++------ sql/rpl_parallel.h | 48 +++++++++++++- 2 files changed, 181 insertions(+), 25 deletions(-) -- 2.30.2

1 3

Fwd: 2342aaf46eb: MDEV-30973 Implement debug logging for slave_transaction_retries
by Aleksey Midenkov 28 Feb '24

28 Feb '24

Hi Sergei, the commits are in bb-11.4-midenok-MDEV-30973 New changes are in 4 top commits. There is also a small CMake change "Columnstore empty submodule fix" which I would push ASAP. On Tue, Nov 7, 2023 at 8:35 PM Sergei Golubchik <serg(a)mariadb.org> wrote: > > Hi, Aleksey, > > On Nov 07, Aleksey Midenkov wrote: > > revision-id: 2342aaf46eb (mariadb-11.0.1-256-g2342aaf46eb) > > parent(s): 80bc5df1c80 > > author: Aleksey Midenkov > > committer: Aleksey Midenkov > > timestamp: 2023-10-10 13:11:19 +0300 > > message: > > > > MDEV-30973 Implement debug logging for slave_transaction_retries > > > > Configuration options: > > > > log_slave_retries > > log_slave_retries_max > > 1. > a commit comment of >100 lines that's a bit too much, I think. > > 2. > comments in MDEV-30973 say that one should be able to have everything in > one file, namely in the error log. I don't see how it's possible in your > commit. Fixed. > > 3. > one of the reasons for not simply writing into erorr log was > "you get what are relatively large entries in there which can make it > hard/annoying to also look for the normal, one-line warning and error > output". But as far as I can see your slave retry entries are not large, > but normal one-line messages. I'd say they can fit into the error log > just fine. "large entries" here is not fully correct. "large number of entries" is more correct. > > I've still commented below in the assumption that the separate log > implementation will stay. But I honestly don't see a reason for it > anymore. > > > diff --git a/mysql-test/suite/rpl/t/slave_deadlock.test b/mysql-test/suite/rpl/t/slave_deadlock.test > > new file mode 100644 > > index 00000000000..9fcda7c9b71 > > --- /dev/null > > +++ b/mysql-test/suite/rpl/t/slave_deadlock.test > > @@ -0,0 +1,194 @@ > > +--source include/have_debug.inc > > +--source include/have_innodb.inc > > +--source include/have_binlog_format_statement.inc > > + > > +# On --repeat test infinitely increments thread_id and finally fails --sync_with_master at log_pos 1754: > > +# > > +# 231009 20:58:15 server id 1 end_log_pos 1754 CRC32 0x7e5aecd1 Query thread_id=561 exec_time=0 error_code=0 xid=0 > > I'm sorry, I don't understand that. Could you elaborate please? I don't know, I've just shown what I observed. Some replication bug? I tried to retest and couldn't reproduce. Removed. > > > +# > > +# force_restart seems to fix this > > + > > +--source include/force_restart.inc > > + > > +# Note that test requires big enough --debug-sync-timeout unfair scheduling at high load. > > Sorry, cannot parse that. Could you elaborate please? I missed *due to* unfair scheduling at high load. That is: thread scheduling latency may reach up to 1 hour. It is executed on VM and actually VM is not always good for concurrency testing (this is quite an omission in our CI system). > > > +# slave_deadlock.opt sets this to 1 hour > > + > > +--let $rpl_topology=1->2 > > +--source include/rpl_init.inc > > + > > +--echo # > > +--echo # MDEV-30973 Enhance output of innodb deadlocks (SAMU-124, SAMU-131) > > +--echo # > > +--connection server_2 > > +SET @old_dbug= @@global.debug_dbug; > > +set @old_parallel_threads= @@global.slave_parallel_threads; > > +set @old_domain_parallel_threads= @@global.slave_domain_parallel_threads; > > +set @old_parallel_mode= @@global.slave_parallel_mode; > > +set @old_slave_transaction_retries= @@global.slave_transaction_retries; > > +set @old_innodb_lock_wait_timeout= @@global.innodb_lock_wait_timeout; > > +set @old_log_slave_retries= @@global.log_slave_retries; > > +set @old_log_slave_retries_file= @@global.log_slave_retries_file; > > +set @old_log_slave_retries_max= @@global.log_slave_retries_max; > > All this makes little sense, it's simpler to put all settings into the .opt > file > > generally it's better to do your way, to have less > server restarts, but you have an .opt file already, you can as well > use it. > > except if you decide to remove the .opt file - then nevermind, the > above it good. I'm taking it for future. > > > +--source include/stop_slave.inc > > +change master to master_use_gtid= slave_pos; > > +set global slave_parallel_threads= 10; > > +set global slave_domain_parallel_threads= 1; > > +set global slave_parallel_mode= optimistic; > > +set global slave_transaction_retries= 4; > > +set global innodb_lock_wait_timeout= 1; > ... > > diff --git a/mysql-test/suite/versioning/r/rpl_row.result b/mysql-test/suite/versioning/r/rpl_row.result > > index c161f68d0bd..6343fbecdec 100644 > > --- a/mysql-test/suite/versioning/r/rpl_row.result > > +++ b/mysql-test/suite/versioning/r/rpl_row.result > > @@ -37,12 +37,7 @@ select *, unix_timestamp(row_start) as row_start, unix_timestamp(row_end) as row > > x y row_start row_end > > 1 1 12345.000000 2147483647.999999 > > set timestamp= default; > > -### INSERT INTO `test`.`t1` > > -### SET > > -### @1=1 > > -### @2=1 > > -### @3=12345.000000 > > -### @4=2147483647.999999 > > why? That was a bug in grep.inc (fixed). I guess this commit was temporary which I forgot to deal with. > > > +/*!50530 SET @@SESSION.PSEUDO_SLAVE_MODE=1*/; > > connection slave; > > select * from t1; > > x y > > diff --git a/sql/log.cc b/sql/log.cc > > index 877315c60de..78e874fda5b 100644 > > --- a/sql/log.cc > > +++ b/sql/log.cc > > @@ -1188,9 +1188,17 @@ bool Log_to_file_event_handler:: > > log_error(enum loglevel level, const char *format, > > va_list args) > > { > > - return vprint_msg_to_log(level, format, args); > > + return vprint_msg_to_log(stderr, level, format, args); > > } > > > > +#ifdef HAVE_REPLICATION > > +bool Log_to_file_event_handler::log_slave_retry(const char *format, va_list args) > > +{ > > + DBUG_ASSERT(slave_retries_file.get()); > > + return vprint_msg_to_log(slave_retries_file.get(), NO_LEVEL, format, args); > > I don't think you should use vprint_msg_to_log and add a NO_LEVEL hack. > vprint_msg_to_log also prints into the windows event log, into > wsrep json log, and it locks LOCK_error_log. You don't need any of that, > just create your own function, it's just one fprintf() anyway. I still think formatting log message in single function is a good idea. Any other above deficiencies were fixed. > > > +} > > +#endif > > + > > void Log_to_file_event_handler::init_pthread_objects() > > { > > mysql_log.init_pthread_objects(); > > @@ -9752,13 +9787,15 @@ static void print_buffer_to_file(enum loglevel level, const char *buffer, > > } > > } > > > > +#ifndef EMBEDDED_LIBRARY > > mysql_mutex_lock(&LOCK_error_log); > > +#endif > > > > skr= my_time(0); > > localtime_r(&skr, &tm_tmp); > > start=&tm_tmp; > > > > - fprintf(stderr, "%d-%02d-%02d %2d:%02d:%02d %lu [%s] %.*s%.*s\n", > > + fprintf(file, "%d-%02d-%02d %2d:%02d:%02d [T%lu] %s%.*s%.*s\n", > > eh... no. don't change the format of existing log files, please. This change is because of a serious deficiency in logging format: you cannot grep by thread number (with simple expression) and grepping by thread number is important. I understand this may cause incompatibilities but I hope the drawbacks are less than the advantage. What if we keep it? > > > start->tm_year + 1900, > > start->tm_mon+1, > > start->tm_mday, > > @@ -9834,6 +9872,22 @@ void sql_print_error(const char *format, ...) > > } > > > > > > +#ifdef HAVE_REPLICATION > > +void sql_print_error2(const char *format, ...) > > s/sql_print_error2/print_slave_error/ ? Fixed (slave_print_error). > > > +{ > > + va_list args; > > + DBUG_ENTER("sql_print_error"); > > + > > + va_start(args, format); > > + error_log_print(ERROR_LEVEL, format, args); > > + slave_retries_print(format, args); > > + va_end(args); > > + > > + DBUG_VOID_RETURN; > > +} > > +#endif > > + > > + > > void sql_print_warning(const char *format, ...) > > { > > va_list args; > > diff --git a/sql/mysqld.cc b/sql/mysqld.cc > > index 8cc3ad51ee9..4f6b2273597 100644 > > --- a/sql/mysqld.cc > > +++ b/sql/mysqld.cc > > @@ -765,6 +765,90 @@ char *opt_relay_logname = 0, *opt_relaylog_index_name=0; > > char *opt_logname, *opt_slow_logname, *opt_bin_logname; > > char *opt_binlog_index_name=0; > > > > +#ifdef HAVE_REPLICATION > > +char default_slave_retries_path[FN_REFLEN]; > > +bool opt_slave_retries= false; > > +char *opt_slave_retries_path= NULL; > > +uint opt_slave_retries_max_log= 0; > > +#endif > > +Slave_retries_file slave_retries_file; > > + > > +bool Slave_retries_file::open() > > +{ > > +#ifdef HAVE_REPLICATION > > + DBUG_ASSERT(opt_slave_retries); > > + DBUG_ASSERT(opt_slave_retries_path); > > + file= fopen(opt_slave_retries_path, "a"); > > my_fopen or my_open > > and anyway, why a new class, why not to use MYSQL_LOG like general and slow > logs do? I rewrote the code via MYSQL_LOG. > > > + if (!file) > > + { > > + sql_print_error("Open of log_slave_retries '%s' failed: %s (%d)", > > + opt_slave_retries_path, strerror(errno), errno); > > This is inconsistent, the server doesn't print this for other log files. > > I've just tried with general log and got > > 2023-11-06 17:58:19 0 [ERROR] mysqld: File '/tmp' not found (Errcode: 21 "Is a directory") > 2023-11-06 17:58:19 0 [ERROR] Could not use /tmp for logging (error 21). Turning logging off for the whole duration of the MariaDB server process. To turn it on again: fix the cause, shutdown the MariaDB server and restart it. > > Please, do the same Looks like wrong behavior: you supplied wrong arguments and the server was started anyway! What is the justification for this? It is easy to miss: if you don't read the error log, you don't even know something is wrong. And if you got lucky to notice that, the server is already up and running (and loaded with clients). Anyway since my logging now works like general log it now works the same. > > > + return true; > > + } > > + else > > + { > > + slave_retries_print("%s (mariadbd %s) starting as process %lu ...", > > + my_progname, server_version, (ulong) getpid()); > > Any reason, why it shouldn't be the same as in the error log? > > sql_print_information("Starting MariaDB %s source revision %s as process %lu", > server_version, SOURCE_REVISION, (ulong) getpid()); > I think it would be nice to add my_progname into the message. Made it now like in the original code. > > + } > > +#endif > > + return false; > > +} > > + > > +void Slave_retries_file::close() > > +{ > > +#ifdef HAVE_REPLICATION > > + if (file) > > + { > > + acquire(); > > + fclose(file); > > + release(); > > + } > > + file= NULL; > > +#endif > > +} > > + > > +void Slave_retries_file::init() > > +{ > > +#ifdef HAVE_REPLICATION > > + mysql_mutex_init(key_LOCK_global_system_variables, &mutex, MY_MUTEX_INIT_FAST); > > +#endif > > +} > > + > > +void Slave_retries_file::destroy() > > +{ > > +#ifdef HAVE_REPLICATION > > + mysql_mutex_destroy(&mutex); > > +#endif > > +} > > + > > +bool Slave_retries_file::acquire() > > +{ > > +#ifdef HAVE_REPLICATION > > + if (!file) > > + return false; > > + mysql_mutex_lock(&mutex); > > + return true; > > +#else > > + return false; > > +#endif > > +} > > + > > +void Slave_retries_file::release() > > +{ > > +#ifdef HAVE_REPLICATION > > + mysql_mutex_unlock(&mutex); > > +#endif > > +} > > + > > +void Slave_retries_file::flush() > > +{ > > + if (acquire()) > > + { > > + fflush(file); > > + release(); > > + } > > +} > > + > > /* Static variables */ > > > > my_bool opt_stack_trace; > > @@ -5055,6 +5145,35 @@ static int init_server_components() > > "this server. However this will be ignored as the " > > "--log-bin option is not defined."); > > } > > + > > + slave_retries_file.init(); > > + if (!log_error_file_ptr[0]) > > + fn_format(default_slave_retries_path, pidfile_name, mysql_data_home, > > + "-retries.err", MY_REPLACE_EXT); > > + else > > + fn_format(default_slave_retries_path, log_error_file_ptr, mysql_data_home, > > + "-retries.err", MY_REPLACE_EXT); > > that's not how all other log names work, > grep mysqld.cc for opt_slow_logname Now it works like other logs. > > > + if (opt_slave_retries_path && !opt_abort) > > + { > > + if (!opt_slave_retries_path[0]) > > + { > > + if (!default_slave_retries_path[0]) > > + { > > + /* fn_format failed */ > > + opt_slave_retries_path= NULL; > > + opt_slave_retries= false; > > + } > > + else > > + { > > + opt_slave_retries_path= my_strdup(key_memory_Sys_var_charptr_value, > > + default_slave_retries_path, MYF(0)); > > + } > > + } > > + > > + if (opt_slave_retries && > > + slave_retries_file.open()) > > + unireg_abort(1); > > + } > > #endif > > > > if (opt_bin_log) > > diff --git a/sql/mysqld.h b/sql/mysqld.h > > index 69cfc0cad56..5f1561ca10d 100644 > > --- a/sql/mysqld.h > > +++ b/sql/mysqld.h > > @@ -822,6 +848,8 @@ enum options_mysqld > > OPT_KEY_CACHE_CHANGED_BLOCKS_HASH_SIZE, > > OPT_LOG_BASENAME, > > OPT_LOG_ERROR, > > + OPT_SLAVE_RETRIES, > > + OPT_SLAVE_RETRIES_LOG, > > why do you need them? Removed. > > > OPT_LOWER_CASE_TABLE_NAMES, > > OPT_PLUGIN_LOAD, > > OPT_PLUGIN_LOAD_ADD, > > diff --git a/sql/sql_show.cc b/sql/sql_show.cc > > index d742bc6c1c3..c53ca0f0097 100644 > > --- a/sql/sql_show.cc > > +++ b/sql/sql_show.cc > > @@ -10662,6 +10663,15 @@ char *thd_get_error_context_description(THD *thd, char *buffer, > > /* Don't wait if LOCK_thd_data is used as this could cause a deadlock */ > > if (!mysql_mutex_trylock(&thd->LOCK_thd_data)) > > { > > + if (thd->rgi_slave) > > is rgi_slave and its content protected by thd->LOCK_thd_data ? AFAICS they are per-thread and don't require protection. > > > + { > > + rpl_group_info *rgi= thd->rgi_slave; > > + len= my_snprintf(header, sizeof(header), > > + ", GTID %u-%u-%llu", rgi->current_gtid.domain_id, > > + rgi->current_gtid.server_id, rgi->current_gtid.seq_no); > > + str.append(header, len); > > + } > > + > > if (const char *info= thread_state_info(thd)) > > { > > str.append(' '); > > diff --git a/sql/sys_vars.cc b/sql/sys_vars.cc > > index ebb64b340b0..11c321f06dc 100644 > > --- a/sql/sys_vars.cc > > +++ b/sql/sys_vars.cc > > @@ -5862,6 +5862,61 @@ Sys_slave_trans_retry_interval( > > "slave_transaction_retry_errors", > > GLOBAL_VAR(slave_trans_retry_interval), CMD_LINE(REQUIRED_ARG), > > VALID_RANGE(0, 3600), DEFAULT(0), BLOCK_SIZE(1)); > > + > > +static bool update_slave_retries_log(sys_var *self, THD* thd, enum_var_type type) > > +{ > > + if (opt_slave_retries) > > + { > > + if (!slave_retries_file.get() && slave_retries_file.open()) > > + { > > + my_error(ER_CANT_OPEN_FILE, MYF(0), opt_slave_retries_path, errno); > > + opt_slave_retries= false; > > + return true; > > + } > > + } > > + else > > + slave_retries_file.close(); > > + return false; > > +} > > + > > +static Sys_var_mybool Sys_slave_retries_log( > > + "log_slave_retries", "Log transaction retries", > > + GLOBAL_VAR(opt_slave_retries), > > + CMD_LINE(OPT_ARG, OPT_SLAVE_RETRIES), DEFAULT(false), > > + NO_MUTEX_GUARD, NOT_IN_BINLOG, ON_CHECK(0), ON_UPDATE(update_slave_retries_log)); > > + > > + > > +static bool check_slave_retries_file(sys_var *self, THD *thd, set_var *var) > > +{ > > + if (opt_slave_retries) > > + { > > + my_error(ER_NOT_SUPPORTED_YET, MYF(0), "modifying while log_slave_retries is ON"); > > as always, please don't hard-code error messages. > you can use > > my_error(ER_OPTION_PREVENTS_STATEMENT, MYF(0), "log_slave_retries=ON"); Fixed. > > > + return true; > > + } > > + return false; > > +} > > + > > +static bool update_slave_retries_file(sys_var *self, THD* thd, enum_var_type type) > > +{ > > + if (!opt_slave_retries_path) > > + opt_slave_retries_path= my_strdup(key_memory_Sys_var_charptr_value, > > + default_slave_retries_path, MYF(0)); > > sys_var.cc can handle that for you, see PREALLOCATED > in e.g. opt_slow_logname Yes. > > also, you don't reopen the log when its name changes, how comes? It reopens. > > > + return false; > > +} > > + > > +static Sys_var_charptr Sys_slave_retries_log_path( > > + "log_slave_retries_file", "Log file path for transaction retries", > > + GLOBAL_VAR(opt_slave_retries_path), > > + CMD_LINE(REQUIRED_ARG, OPT_SLAVE_RETRIES_LOG), DEFAULT(NULL), > > + NO_MUTEX_GUARD, NOT_IN_BINLOG, ON_CHECK(check_slave_retries_file), > > + ON_UPDATE(update_slave_retries_file)); > > + > > + > > +static Sys_var_uint Sys_slave_retries_log_max( > > + "log_slave_retries_max", "Log max N transaction retries " > > + "(default 0 means infinity)", > > just "(0 means infinity)". Default is printed separately > Fixed. > > + GLOBAL_VAR(opt_slave_retries_max_log), CMD_LINE(REQUIRED_ARG), > > + VALID_RANGE(0, UINT_MAX), DEFAULT(0), BLOCK_SIZE(1)); > > #endif > > > > static bool check_locale(sys_var *self, THD *thd, set_var *var) > > diff --git a/storage/innobase/lock/lock0lock.cc b/storage/innobase/lock/lock0lock.cc > > index 0e9cef669d4..8a8eeaf39f6 100644 > > --- a/storage/innobase/lock/lock0lock.cc > > +++ b/storage/innobase/lock/lock0lock.cc > > @@ -49,6 +49,7 @@ Created 5/7/1996 Heikki Tuuri > > #include <debug_sync.h> > > > > #include <set> > > +#include <stdio.h> > > I presume Marko has reviewed (or will review) InnoDB changes I didn't receive his explicit confirmation. > > > > > #ifdef WITH_WSREP > > #include <mysql/service_wsrep.h> > > diff --git a/mysql-test/main/mysqld--help.result b/mysql-test/main/mysqld--help.result > > index 77d6d10b3e6..274c034d455 100644 > > --- a/mysql-test/main/mysqld--help.result > > +++ b/mysql-test/main/mysqld--help.result > > @@ -1673,6 +1678,9 @@ log-isam myisam.log > > log-output FILE > > log-queries-not-using-indexes FALSE > > log-short-format FALSE > > +log-slave-retries FALSE > > +log-slave-retries-file (No default value) > > this is another sign that log-slave-retries-file doesn't behave like > other log filename variables. > > > +log-slave-retries-max 0 > > log-slave-updates FALSE > > log-slow-admin-statements TRUE > > log-slow-disabled-statements sp > > Regards, > Sergei > Chief Architect, MariaDB Server > and security(a)mariadb.org Thanks! -- @midenok

1 0

Re: f22662c36b1: MDEV-33442 REPAIR TABLE corrupts UUIDs
by Sergei Golubchik 26 Feb '24

26 Feb '24

Hi, Alexander, On Feb 24, Alexander Barkov wrote: > revision-id: f22662c36b1 (mariadb-10.11.7-6-gf22662c36b1) > parent(s): 656f8867720 > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2024-02-15 18:11:03 +0400 > message: > > MDEV-33442 REPAIR TABLE corrupts UUIDs Looks quite good. A question: you removed a hard-coded check for MYSQL_TYPE_NEWDECIMAL, but I don't see that you added Type_handler_olddecimal::type_handler_for_implicit_upgrade() A suggestion, rename Column_definition_implicit_upgrade(). You've changed its semantics, it's good to make sure that the code using old Column_definition_implicit_upgrade() (added e.g. in 10.6 and merged up) won't compile. Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 2

Re: 7ff9e91a7d2: MDEV-32975 Default charset doesn't work with PHP MySQLi extension
by Sergei Golubchik 25 Feb '24

25 Feb '24

Hi, Alexander, the fix is good, thanks. a couple of notes about comments, see below. ok to push after fixing them On Feb 25, Alexander Barkov wrote: > revision-id: 7ff9e91a7d2 (mariadb-10.4.32-179-g7ff9e91a7d2) > parent(s): 1070575a890 > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2024-01-26 13:12:03 +0400 > message: > > MDEV-32975 Default charset doesn't work with PHP MySQLi extension > > When sending the server default collation ID to the client > in the handshake packet, translate a 2-byte collation ID > to the ID of the default collation for the character set. > > diff --git a/sql/sql_acl.cc b/sql/sql_acl.cc > index 1949a0c02b6..5103ff02105 100644 > --- a/sql/sql_acl.cc > +++ b/sql/sql_acl.cc > @@ -12831,7 +12831,32 @@ static bool send_server_handshake_packet(MPVIO_EXT *mpvio, > > int2store(end, thd->client_capabilities); > /* write server characteristics: up to 16 bytes allowed */ may be move this comment to be directly above end[2]=... line, as before? Otherwise they're separated by another comment and an if() block, so the comment looks out of place > - end[2]= (char) default_charset_info->number; > + > + /* > + Send the default server collation ID. > + Note, since MySQL-4.1 it's actually the > + client responsibility to send its character set to the server, > + so this value means nothing and is ignored by most connectors. > + But some rare connectors (e.g. PHP) still read it. where did you get it from? protocol description doesn't say that the charset field in the handshake packet is obsolete and means nothing. better to remove this claim. If you want to obsolete this field, don't do it in an obscure comment in the code, but change the protocol description in KB. > + */ > + CHARSET_INFO *handshake_cs= default_charset_info; > + if (handshake_cs->number > 0xFF) > + { > + /* > + A workaround for a 2-byte collation ID: translate it into > + the ID of the primary collation of this character set. > + */ > + CHARSET_INFO *cs= get_charset_by_csname(handshake_cs->csname, > + MY_CS_PRIMARY, MYF(MY_WME)); > + /* > + cs should not normally be NULL, however it may be possible > + with a dynamic character set incorrectly defined in Index.xml. > + For safety let's fallback to latin1 in case cs is NULL. > + */ > + handshake_cs= cs ? cs : &my_charset_latin1; > + } > + end[2]= (char) handshake_cs->number; > + > int2store(end+3, mpvio->auth_info.thd->server_status); > int2store(end+5, thd->client_capabilities >> 16); > end[7]= data_len; > Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

1 0

Re: 367c87a34f8: Optimize performance of my_bitmap
by Sergei Golubchik 23 Feb '24

23 Feb '24

Hi, Monty, On Feb 23, Michael Widenius wrote: > commit 367c87a34f8 > Author: Michael Widenius <monty(a)mariadb.org> > Date: Sun Feb 18 17:30:01 2024 +0200 > > diff --git a/include/my_bitmap.h b/include/my_bitmap.h > --- a/include/my_bitmap.h > +++ b/include/my_bitmap.h > @@ -81,14 +79,15 @@ extern void bitmap_union(MY_BITMAP *map, const MY_BITMAP *map2); > extern void bitmap_xor(MY_BITMAP *map, const MY_BITMAP *map2); > extern void bitmap_invert(MY_BITMAP *map); > extern void bitmap_copy(MY_BITMAP *map, const MY_BITMAP *map2); > +extern void bitmap_copy_data(MY_BITMAP *map, const uchar *ptr, uint bits); Why? In all cases where you use it, the source is a bitmap too. you can use a function that copies MY_BITMAP to MY_BITMAP. > > extern uint bitmap_lock_set_next(MY_BITMAP *map); > extern void bitmap_lock_clear_bit(MY_BITMAP *map, uint bitmap_bit); > /* Fast, not thread safe, bitmap functions */ > -#define bitmap_buffer_size(bits) (((bits)+31)/32)*4 > +#define bitmap_buffer_size(bits) (((bits)+63)/64)*8 may be MY_ALIGN(bits, sizeof(my_bitmap_map)*8)*sizeof(my_bitmap_map) ? and the same below? less error-prone or, even better #define my_bitmap_map_bytes sizeof(my_bitmap_map) #define my_bitmap_map_bits (my_bitmap_map_bytes*8) #define bitmap_buffer_size(bits) MY_ALIGN(bits,my_bitmap_map_bits)*my_bitmap_map_bytes > #define no_bytes_in_map(map) (((map)->n_bits + 7)/8) > -#define no_words_in_map(map) (((map)->n_bits + 31)/32) > -#define bytes_word_aligned(bytes) (4*((bytes + 3)/4)) > +#define no_words_in_map(map) (((map)->n_bits + 63)/64) > +#define bytes_word_aligned(bytes) (8*((bytes + 7)/8)) > /* The following functions must be compatible with create_last_word_mask()! */ > static inline void > bitmap_set_bit(MY_BITMAP *map,uint bit) > diff --git a/mysys/my_bitmap.c b/mysys/my_bitmap.c > --- a/mysys/my_bitmap.c > +++ b/mysys/my_bitmap.c > @@ -31,95 +42,82 @@ > New version written and test program added and some changes to the interface > was made by Mikael Ronstrom 2005, with assistance of Tomas Ulin and Mats > Kindahl. > + Updated to 64 bits and use my_find_first_bit() to speed up > + bitmap_get_next_set() by Monty in 2024 > */ > > #include "mysys_priv.h" > #include <my_bitmap.h> > #include <m_string.h> > #include <my_bit.h> > +#include <my_byteorder.h> > + > + > +/* Defines to check bitmaps */ > + > +#define DBUG_ASSERT_BITMAP(M) \ > + DBUG_ASSERT((M)->bitmap); \ > + DBUG_ASSERT((M)->n_bits > 0); \ > + DBUG_ASSERT((M)->last_word_ptr == (M)->bitmap + no_words_in_map(M)-1); \ > + DBUG_ASSERT((*(M)->last_word_ptr & (M)->last_word_mask) == 0); > + > +#define DBUG_ASSERT_BITMAP_AND_BIT(M,B) \ > + DBUG_ASSERT_BITMAP(M); \ > + DBUG_ASSERT((B) < (M)->n_bits); > + > +#define DBUG_ASSERT_DIFFERENT_BITMAPS(M,N) \ > + DBUG_ASSERT_BITMAP(M); \ > + DBUG_ASSERT_BITMAP(N); > + > +#define DBUG_ASSERT_IDENTICAL_BITMAPS(M,N) \ > + DBUG_ASSERT_BITMAP(M); \ > + DBUG_ASSERT_BITMAP(N); \ > + DBUG_ASSERT((M)->n_bits == (N)->n_bits); > > /* > - Create a mask with the upper 'unused' bits set and the lower 'used' > - bits clear. The bits within each byte is stored in big-endian order. > + Create a mask for the usable bits on the LAST my_bitmap_map position for > + a bitmap with 'bits' number of bits. > + > + The lowest 'bits' bits are set to zero and the rest bits are set to 1. > + For (bits & 63) == 0 , 0 is returned as in this case all bits in > + the my_bitmap_position are significant > + > + This code assumes the underlying storage is a 64 bit ulonglong. doesn't have to be if you'll use my_bitmap_map_bits > */ > > -static inline uchar invers_last_byte_mask(uint bits) > -{ > - return last_byte_mask(bits) ^ 255; > -} > +static inline my_bitmap_map last_word_mask(uint bits) > +{ > + uint bits_in_last_map= (bits & 63); > + return bits_in_last_map ? ~((1ULL << bits_in_last_map)-1) : 0ULL; you sure a condition is faster than a linear code of few bit operations? > +} > > > +static inline my_bitmap_map first_word_mask(uint bits) > +{ > + uint bits_in_last_map= (bits & 63); > + return ~((1ULL << bits_in_last_map)-1); and no condition here. strange. and shouldn't one of these two functions be inverted? supposedly one masks all bits before the first and the other - all bits after the last. > +} > -void create_last_word_mask(MY_BITMAP *map) > -{ > - unsigned char const mask= invers_last_byte_mask(map->n_bits); > - > - /* > - The first bytes are to be set to zero since they represent real bits > - in the bitvector. The last bytes are set to 0xFF since they represent > - bytes not used by the bitvector. Finally the last byte contains bits > - as set by the mask above. > - */ > - unsigned char *ptr= (unsigned char*)&map->last_word_mask; > - > - map->last_word_ptr= map->bitmap + no_words_in_map(map)-1; > - switch (no_bytes_in_map(map) & 3) { > - case 1: > - map->last_word_mask= ~0U; > - ptr[0]= mask; > - return; > - case 2: > - map->last_word_mask= ~0U; > - ptr[0]= 0; > - ptr[1]= mask; > - return; > - case 3: > - map->last_word_mask= 0U; > - ptr[2]= mask; > - ptr[3]= 0xFFU; > - return; > - case 0: > - map->last_word_mask= 0U; > - ptr[3]= mask; > - return; > - } > -} > > > -static inline my_bitmap_map last_word_mask(uint bit) > -{ > - my_bitmap_map last_word_mask; > - uint n_bits= bit + 1; > - unsigned char const mask= invers_last_byte_mask(n_bits); > - > - /* > - The first bytes are to be set to zero since they represent real bits > - in the bitvector. The last bytes are set to 0xFF since they represent > - bytes not used by the bitvector. Finally the last byte contains bits > - as set by the mask above. > - */ > - unsigned char *ptr= (unsigned char*)&last_word_mask; > - > - switch ((n_bits + 7)/8 & 3) { > - case 1: > - last_word_mask= ~0U; > - ptr[0]= mask; > - break; > - case 2: > - last_word_mask= ~0U; > - ptr[0]= 0; > - ptr[1]= mask; > - break; > - case 3: > - last_word_mask= 0U; > - ptr[2]= mask; > - ptr[3]= 0xFFU; > - break; > - case 0: > - last_word_mask= 0U; > - ptr[3]= mask; > - break; > - } > - return last_word_mask; > -} > +/* > + Update the bitmap's last_word_ptr and last_word_mask > + Also ensure that the last world is all zero to make it > + easy to find the next set bit. > + > + Note that if n_bits is 0, then last_word_ptr will point to > + bitmap (safetly). The bitmap will not be usable for almost any operation. ether "safety" or "safely" > +*/ > + > +void create_last_word_mask(MY_BITMAP *map) > +{ > + my_bitmap_map mask= last_word_mask(map->n_bits); > + map->last_word_mask= mask; > + map->last_word_ptr= map->bitmap + MY_MAX(no_words_in_map(map),1) -1; > + if (map->n_bits > 0) > + { > + *map->last_word_ptr&= ~mask; /* Set not used bits to 0 */ > + DBUG_ASSERT_BITMAP(map); > + } > +} > > > @@ -326,37 +312,45 @@ void bitmap_set_prefix(MY_BITMAP *map, uint prefix_size) > } > if ((d= no_bytes_in_map(map)-prefix_bytes)) > memset(m, 0, d); > + DBUG_ASSERT_BITMAP(map); > } > > > +/** > + Check if bitmap is a bitmap of prefix bits set in the beginning > + > + @param map bitmap > + @param prefix_size number of bits that should be set. 0 is allowed. > + > + @return 1 Yes, prefix bits where set or prefix_size == 0. > + @return 0 No > +*/ > + > my_bool bitmap_is_prefix(const MY_BITMAP *map, uint prefix_size) > { > uint prefix_mask= last_byte_mask(prefix_size); > uchar *m= (uchar*) map->bitmap; > uchar *end_prefix= m+(prefix_size-1)/8; > uchar *end; > - DBUG_ASSERT(m); > - DBUG_ASSERT(prefix_size <= map->n_bits); > > /* Empty prefix is always true */ > if (!prefix_size) > return 1; > > + DBUG_ASSERT_BITMAP_AND_BIT(map, prefix_size-1); > + > while (m < end_prefix) > if (*m++ != 0xff) > return 0; > > - end= ((uchar*) map->bitmap) + no_bytes_in_map(map) - 1; > - if (m == end) > - return ((*m & last_byte_mask(map->n_bits)) == prefix_mask); > - > + end= ((uchar*) map->last_word_ptr) + sizeof(*map->last_word_ptr)-1; good, here you use sizeof(), not just 7. > if (*m != prefix_mask) > return 0; > > - while (++m < end) > + while (++m <= end) > if (*m != 0) > return 0; > - return ((*m & last_byte_mask(map->n_bits)) == 0); > + return 1; > } > > > @@ -447,50 +442,51 @@ void bitmap_intersect(MY_BITMAP *map, const MY_BITMAP *map2) > > /* > Check if there is some bit index between start_bit and end_bit, such that > - this is bit is set for all bitmaps in bitmap_list. > + this is atleast on bit that set for all bitmaps in bitmap_list. "at least" and "one" and "that is set" ? > > SYNOPSIS > bitmap_exists_intersection() > bitmpap_array [in] a set of MY_BITMAPs > - bitmap_count [in] number of elements in bitmpap_array > + bitmap_count [in] number of elements in bitmap_array > start_bit [in] beginning (inclusive) of the range of bits to search > end_bit [in] end (inclusive) of the range of bits to search, must be > no bigger than the bits of the shortest bitmap. > > - NOTES > - This function assumes that for at least one of the bitmaps in bitmap_array all > - bits outside the range [start_bit, end_bit] are 0. As a result is not > - necessary to take care of the bits outside the range [start_bit, end_bit]. > - > RETURN > TRUE if an intersecion exists > FALSE no intersection > */ > > -my_bool bitmap_exists_intersection(const MY_BITMAP **bitmap_array, > +my_bool bitmap_exists_intersection(MY_BITMAP **bitmap_array, > uint bitmap_count, > uint start_bit, uint end_bit) > { > uint i, j, start_idx, end_idx; > - my_bitmap_map cur_res; > + my_bitmap_map cur_res, first_map; > > DBUG_ASSERT(bitmap_count); > DBUG_ASSERT(end_bit >= start_bit); > for (j= 0; j < bitmap_count; j++) > - DBUG_ASSERT(end_bit < bitmap_array[j]->n_bits); > + { > + DBUG_ASSERT_BITMAP_AND_BIT(bitmap_array[j], end_bit); > + } > > start_idx= start_bit/8/sizeof(my_bitmap_map); > end_idx= end_bit/8/sizeof(my_bitmap_map); > > + first_map= first_word_mask(start_bit); > + cur_res= first_map; > for (i= start_idx; i < end_idx; i++) > { > - cur_res= ~0; > for (j= 0; cur_res && j < bitmap_count; j++) > cur_res &= bitmap_array[j]->bitmap[i]; > if (cur_res) > return TRUE; > + cur_res= ~(my_bitmap_map) 0; > } > - cur_res= ~last_word_mask(end_bit); > + cur_res= ~last_word_mask(end_bit+1); > + if (start_idx == end_idx) > + cur_res&= first_map; > for (j= 0; cur_res && j < bitmap_count; j++) > cur_res &= bitmap_array[j]->bitmap[end_idx]; > return cur_res != 0; > @@ -501,21 +497,26 @@ my_bool bitmap_exists_intersection(const MY_BITMAP **bitmap_array, > > my_bool bitmap_union_is_set_all(const MY_BITMAP *map1, const MY_BITMAP *map2) > { > - my_bitmap_map *m1= map1->bitmap, *m2= map2->bitmap, *end; > + my_bitmap_map *m1= map1->bitmap, *m2= map2->bitmap, *end= map1->last_word_ptr; > + DBUG_ASSERT_IDENTICAL_BITMAPS(map1,map2); > > - DBUG_ASSERT(map1->bitmap); > - DBUG_ASSERT(map2->bitmap); > - DBUG_ASSERT(map1->n_bits==map2->n_bits); > - end= map1->last_word_ptr; > while ( m1 < end) > - if ((*m1++ | *m2++) != 0xFFFFFFFF) > + if ((*m1++ | *m2++) != 0xFFFFFFFFFFFFFFFFULL) better ~(my_bitmap_map)0 > return FALSE; > /* here both maps have the same number of bits - see assert above */ > - return ((*m1 | *m2 | map1->last_word_mask) != 0xFFFFFFFF); > + return ((*m1 | *m2 | map1->last_word_mask) != 0xFFFFFFFFFFFFFFFFULL); > } > > > > +#ifdef NOT_USED > + > +/* > + If we re-introduce this one one, we should change the interface to > + use_from_bit and not from_byte. We should also add testing for this > + in bitmap-t.c > +*/ just remove it, it's easier to write it from scratch again when needed then to undertand and fix this old dead code > + > /* > Set/clear all bits above a bit. > > @@ -603,45 +601,51 @@ uint bitmap_bits_set(const MY_BITMAP *map) > my_bitmap_map *data_ptr= map->bitmap; > my_bitmap_map *end= map->last_word_ptr; > uint res= 0; > - DBUG_ASSERT(map->bitmap); > + DBUG_ASSERT_BITMAP(map); > > - for (; data_ptr < end; data_ptr++) > - res+= my_count_bits_uint32(*data_ptr); > + for (; data_ptr <= end; data_ptr++) > + res+= my_count_bits(*data_ptr); > > - /*Reset last bits to zero*/ > - res+= my_count_bits_uint32(*map->last_word_ptr & ~map->last_word_mask); > return res; > } > > void bitmap_copy(MY_BITMAP *map, const MY_BITMAP *map2) > { > - my_bitmap_map *to= map->bitmap, *from= map2->bitmap, *end; > - > - DBUG_ASSERT(map->bitmap); > - DBUG_ASSERT(map2->bitmap); > - DBUG_ASSERT(map->n_bits == map2->n_bits); > - end= map->last_word_ptr; > + my_bitmap_map *to= map->bitmap, *from= map2->bitmap, *end= map->last_word_ptr; > + DBUG_ASSERT_IDENTICAL_BITMAPS(map,map2); > > while (to <= end) > *to++ = *from++; > } > > > +/* > + Copy data into the bitmap from a byte array > +*/ > + > +void bitmap_copy_data(MY_BITMAP *map, const uchar *ptr, uint bits) not needed, as far as I understand > +{ > + my_bitmap_map *to= map->bitmap; > + uint length= (bits + 7)/8; > + uint map_length= no_bytes_in_map(map); > + DBUG_ASSERT_BITMAP_AND_BIT(map, bits-1); > + > + memcpy(to, ptr, length); > + if (map_length != length) > + bzero(to + length, map_length - length); > + *map->last_word_ptr&= ~map->last_word_mask; > +} > + > + > uint bitmap_get_first_set(const MY_BITMAP *map) > { > - uint i; > my_bitmap_map *data_ptr= map->bitmap, *end= map->last_word_ptr; > + DBUG_ASSERT_BITMAP(map); > > - DBUG_ASSERT(map->bitmap); > - > - for (i=0; data_ptr < end; data_ptr++, i++) > + for (uint i=0; data_ptr <= end; data_ptr++, i++) > if (*data_ptr) > - goto found; > - if (!(*data_ptr & ~map->last_word_mask)) > - return MY_BIT_NONE; > - > -found: > - return get_first_set(*data_ptr, i); > + return my_find_first_bit(*data_ptr) + i * sizeof(my_bitmap_map)*8; > + return MY_BIT_NONE; > } > > > @@ -703,35 +696,18 @@ uint bitmap_get_next_set(const MY_BITMAP *map, uint bitmap_bit) > > uint bitmap_get_first(const MY_BITMAP *map) this should be named bitmap_get_first_clear(). because it's always is_bit_set/is_bit_clear, is_set_all/is_clear_all, test_and_set/test_and_clear, set_bit/clear_bit, etc. there is bitmap_get_first_set, so this one must be bitmap_get_first_clear > { > - uchar *byte_ptr; > - uint i,j,k; > - my_bitmap_map *data_ptr, *end= map->last_word_ptr; > - > - DBUG_ASSERT(map->bitmap); > - data_ptr= map->bitmap; > - *map->last_word_ptr|= map->last_word_mask; > + uint i; > + my_bitmap_map *data_ptr= map->bitmap, *end= map->last_word_ptr; > + DBUG_ASSERT_BITMAP(map); > > - for (i=0; data_ptr < end; data_ptr++, i++) > - if (*data_ptr != 0xFFFFFFFF) > + for (i= 0; data_ptr < end; data_ptr++, i++) > + if (*data_ptr != 0xFFFFFFFFFFFFFFFFULL) > goto found; > - if ((*data_ptr | map->last_word_mask) == 0xFFFFFFFF) > + if ((*data_ptr | map->last_word_mask) == 0xFFFFFFFFFFFFFFFFULL) > return MY_BIT_NONE; > - > found: > - byte_ptr= (uchar*)data_ptr; > - for (j=0; ; j++, byte_ptr++) > - { > - if (*byte_ptr != 0xFF) > - { > - for (k=0; ; k++) > - { > - if (!(*byte_ptr & (1 << k))) > - return (i*32) + (j*8) + k; > - } > - } > - } > - DBUG_ASSERT(0); > - return MY_BIT_NONE; /* Impossible */ > + /* find first zero bit by reverting all bits and find first bit */ > + return my_find_first_bit(~*data_ptr) + i * sizeof(my_bitmap_map)*8; > } > > Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

1 0

Re: 264aef71a37: Have ha_partition ignore HA_EXTRA..CHILDREN extra() calls if no myisamrg
by Sergei Golubchik 23 Feb '24

23 Feb '24

Hi, Michael, On Feb 23, Michael Widenius wrote: > revision-id: 264aef71a37 (mariadb-10.6.17-15-g264aef71a37) > parent(s): 042c3fc432b > author: Michael Widenius > committer: Michael Widenius > timestamp: 2024-02-22 09:20:51 +0200 > message: > > Have ha_partition ignore HA_EXTRA..CHILDREN extra() calls if no myisamrg > > diff --git a/sql/ha_partition.cc b/sql/ha_partition.cc > index e174ef8d219..f2b3ab3aff9 100644 > --- a/sql/ha_partition.cc > +++ b/sql/ha_partition.cc > @@ -3030,6 +3031,10 @@ bool ha_partition::create_handlers(MEM_ROOT *mem_root) > DBUG_PRINT("info", ("InnoDB")); > m_innodb= TRUE; > } > + else if (ha_legacy_type(hton0) == DB_TYPE_MRG_MYISAM) > + { > + m_myisammrg= TRUE; > + } > DBUG_RETURN(FALSE); > } > > @@ -9419,9 +9424,14 @@ int ha_partition::extra(enum ha_extra_function operation) > } > /* Category 9) Operations only used by MERGE */ > case HA_EXTRA_ADD_CHILDREN_LIST: > + if (!m_myisammrg) > + DBUG_RETURN(0); > DBUG_RETURN(loop_partitions(extra_cb, &operation)); > case HA_EXTRA_ATTACH_CHILDREN: > { > + if (!m_myisammrg) > + DBUG_RETURN(0); > + > int result; This is, of course, quite horrible. Checking the legacy type, doing something for a specify underlying engine. But I don't see a better way of doing it, short of changing the storage engine API. What if we redefine ::extra() call to return HA_ERR_WRONG_COMMAND for unsupported extra operations? The caller will not abort on it, but it'll allow the caller to know whether the operation was executed or ignored. This will allow partitioning handler to skip all extra() methods that are not implemented in the underlying engine. For all engines and all extra operations. Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

1 0

Re: Second part of review of XA patches in bb-10.6-MDEV-31949
by Kristian Nielsen 17 Feb '24

17 Feb '24

Andrei Elkin <andrei.elkin(a)mariadb.com> writes: > > I think it's better to define a proper API for this. > > It feels an overkill in the current case. "Define a proper API" means that you think through how this should work for different storage engines and TC_LOG implementations. And then you document those thoughts. I don't understand how doing this can be overkill, you are re-defining how durability of external XA transactions are coordinated between the server layer and the storage engines. >> And vice versa, for every xid >> X_COMPLETE found, roll it back if X is found prepared in InnoDB, > > Actually >> else commit it. > safer must be to commit both X_COMPLETE and X. Generally even if > X_COMPLETE and X commits in this order, X can race to be durable before > crash. Right, and "commit X" would mean "XA ROLLBACK X" if the "complete" operation is a rollback. I didn't follow why rolling back the X_COMPLETE would be a problem, but it's not important for now; I can see how committing should work as well. >> One reason I'm asking is I'm wondering how this will affect the scheduling >> of XA in parallel replication? Like, in case of duplicate XID, the X_PREPARE >> or X_COMPLETE could conflict between two transactions, so maybe the >> wakeup_subsequent_commits can only be done after the XA COMMIT of the >> X_PREPARE and X_COMMIT? Which I think would require changes to the current >> patch. > > Hmm, right now in the branch (see Sliding_window etc) for the duplicate > XID case, like > P_1,C_2,P_3,C_4 > P - XA-PREPARE, C - ... commit) P_3 can start only when C_2 wakes it up. Yes. So my point is: In this case, the C_2 cannot do wakeup_subsequent_commits inside the group commit along with run_commit_ordered(), that would be too early. C_2 must first complete the extra internal XA transaction on the mysql.gtid_slave_pos table before it can wakeup P_3. Or is this not a problem? > My bad I left out a mention of the plan, to treat - call it > XAC_0 -> XAP_1 > dependency - by the optimistic retry. That's in order to minimize the Aha. So if we optimistically apply the XA PREPARE, and we get an error ER_XAER_DUPID that the XID already exists as prepared, then we would do a retry like other temporary errors, is that what you mean? Is this implemented in the current patch? Maybe it is automatically handled by convert_kill_to_deadlock_error() ? If this XAC_0 -> XAP_1 conflict can be handled by normal optimistic rollback/retry, then I agree there is no need to handle the dependency explicitly in the sliding window. That was not clear to me from the code. >> This I don't understand at all. >> How can an XA PREPARE/COMMIT/ROLLBACK be marked as "ddl" ? > > That's a binlogging whim. E.g a temp table "DDL" involved into > transaction. I was not sure I could quickly eliminate the flagging, so > decided to live with it. Aha, right, CREATE TEMPORARY TABLE can be part of a transaction and thus DDL (in statement-based binlogging). I missed that. > But I could not get the meaning of > > ... handler.cc to avoid special cases for whether the binlog is enabled > The burden of checking whether the binlog is enabled, e.g for taking a > TC_LOG->log_and_order() path is on the programmer. If binlog is globally enabled (--log-bin option), then tc_log is an object of class MYSQL_BIN_LOG. Else it is class TC_LOG_MMAP (or TC_LOG_DUMMY). So a virtual function can be used instead of a conditional on opt_bin_log. But binlogging can be disabled on a transaction-by-transaction basis by setting SQL_LOG_BIN, is that the case here? Then yes, it seems an explicit condition will be needed. - Kristian.

1 0

[Andrei Elkin] Re: Third/last part of review of XA patches in bb-10.6-MDEV-31949
by andrei.elkin＠pp.inet.fi 12 Feb '24

12 Feb '24

Kristian, Using the chance to forwarding the reply that did not make to the developers list, let me comment on binlog_commit() that I might hastily rate as a big subject. > Hi Andrei, > > Here is the last part of my review of the changes in branch > bb-10.6-MDEV-31949: > >> --- a/sql/log_event_server.cc >> +++ b/sql/log_event_server.cc >> @@ -3324,6 +3324,11 @@ Gtid_log_event::Gtid_log_event(THD *thd_arg, uint64 seq_no_arg, >> flags2|= thd->lex->sql_command == SQLCOM_XA_PREPARE ? >> FL_PREPARED_XA : FL_COMPLETED_XA; >> xid.set(xid_state.get_xid()); >> + >> + // multi-engine external completion is unaware of the prepared engine #. >> + extra_engines= thd->transaction->all.ha_list ? >> + ha_count_rw_2pc(thd_arg, >> + thd_arg->in_multi_stmt_transaction_mode()) - 1 : 0; > > I cannot understand the comment. > I assume it's about that an XA COMMIT of a detached xid (different > session > that the XA PREPARE, or replication slave) - right? Correct. > > But what is the consequences of not knowing extra_engines in this > case? > Is the extra_engines used for anything than the optional truncate of > the > binlog after master crash? In the multi-engine case when the # of prepared in Engine is greater than in the value in GTID event then the trx is partially committed, so is decided for to be completed. > >> @@ -4175,7 +4173,8 @@ int XA_prepare_log_event::do_commit() >> thd->lex->xid= &xid; >> if (!one_phase) >> { >> - if ((res= thd->wait_for_prior_commit())) >> + if (thd->is_current_stmt_binlog_disabled() && >> + (res= thd->wait_for_prior_commit())) > > Why?000 > Is it to allow XA_PREPARE_LOG_EVENT to group commit with other event > groups? > That makes sense, but then shouldn't the wait_for_prior_commit() be > somewhere in handler.cc (or maybe xa.cc)? > Look at Xid_log_event::do_commit(), it doesn't have any > wait_for_prior_commit(), that is in ha_commit_one_phase(). Right. > (The wait-for-prior-commit facility is designed as a server-level > feature, > not a replication-level, it could in principle be used outside of > replication, even though it not currently (IIRC).) I think I understood what you mean which is follow the ha_commit_one_phase() pattern. [todo] > >> diff --git a/sql/sql_array.h b/sql/sql_array.h >> index 8610e971016..94c6d86ac05 100644 >> --- a/sql/sql_array.h >> +++ b/sql/sql_array.h >> @@ -172,6 +172,8 @@ template <class Elem> class Dynamic_array >> >> size_t size() const { return array.elements; } >> >> + size_t max_size() const { return array.max_element; } >> + > > Not that it hurts that much, but I think you can avoid adding this. Left over of a cleanup from the previous version. [todo] > >> --- a/sql/log.cc >> +++ b/sql/log.cc >> @@ -85,6 +85,7 @@ const char *log_bin_index= 0; >> const char *log_bin_basename= 0; >> >> MYSQL_BIN_LOG mysql_bin_log(&sync_binlog_period); >> +XID_cache_element *xid_cache_search(THD *thd, XID *xid); > > This duplicates the same declaration in sql/xa.h > So probably this can be removed now? [todo] > >> @@ -2186,16 +2202,13 @@ int binlog_commit(THD *thd, bool all, bool >> ro_1pc) >> } >> >> if (cache_mngr->trx_cache.empty() && >> - (thd->transaction->xid_state.get_state_code() != XA_PREPARED || >> - !(thd->ha_data[binlog_hton->slot].ha_info[1].is_started() && >> - thd->ha_data[binlog_hton->slot].ha_info[1].is_trx_read_write()))) >> + (!(is_preparing_xa(thd) || is_prepared_xa(thd)) || >> + !is_binlog_read_write(thd))) > > This binlog_commit function started out as the "commit" method for the > binlog handlerton. But that is not how it is used anymore, now it is > called > directly from several different places, including some places specific > to > external user XA. > > A large part of the function has become complex conditions like this > that > are basically impossible for the developer (or reviewer) to understand > exactly why this set of conditions is the correct one. Again, the > function > is trying to guess what the caller wants, depending on the state of > various > thd properties etc. > > Instead, the function should be split into multiple functions. One > that is > called to handle XA PREPARE; one for binlog_commit_by_xid(), etc. Then > each > version should be able to greatly reduce the number of conditions > needed. > > And if any logic is shared between the different versions, this can > then be > put in smaller functions that are called from each version. [my initial response was: Your proposal leads to a significant refactoring, which I add only one hunk. I'd really consider that as a separate exercise I think I've understood. Sure, binlog_prepare() that logs XA-PREPARE can call a common function with binlog_commit() that originally intended to log only normal trx:s. I can look into that. [todo] >> >> @@ -11050,10 +11124,30 @@ bool >> Recovery_context::decide_or_assess(xid_recovery_member *member, int >> round, >> } >> else if (member->in_engine_prepare < last_gtid_engines) >> { >> - DBUG_ASSERT(member->in_engine_prepare > 0); >> + DBUG_ASSERT(member->in_engine_prepare > 0 || >> + /* XA event over xid that was completed in engines */ >> + (member->xid == 0 && >> + member->xa_binlog_state >= xid_recovery_member::XA_PREPARE)); >> + if (member->in_engine_prepare > 0 && >> + member->xa_binlog_state == xid_recovery_member::XA_PREPARE) >> + { > > The function decide_or_assess() is hard to understand. > I can see that it's called in 3 cases: XID_LOG_EVENT, XA PREPARE, XA > COMMIT/ROLLBACK. > So again here, I think this should be split into 3 different > functions, one > for each case (or maybe two, one for internal XID and one for external > XA, > whatever makes the more sense). > > I don't understand the comment "XA event over xid that was completed > in > engines". But I remember seeing something in log_event_server.cc that > sets > extra_engines to 0 in the external user XA case. > > So is that the case here, that last_gtid_engines==0 marks this is user > XA? > That would be much clearer if the logic for internal and external XA > was not > mixed in the same function. > >> + /* >> + the first of the two options below can occur when round 1 turns to 2 >> + so event sequences across binlog files are like the following >> + g_current(B_0, xid), ... g_last(B^*, xid), >> + where `g`:s are xa event groups associated with the same member via >> + common `xid`. As elsewhere B_0 is the 2nd round's first and, >> + B^* the 1st round binlogs. >> + */ >> if (reset_truncate_coord(member->binlog_coord < last_gtid_coord ? >> member->binlog_coord.second : pos)) > > I'm trying to understand what is going on here. I think this is refering to > the way binlog files are scanned? > First we scan the most recent binlog file. At the end of it, we may find > that the latest binlog checkpoint is in an earlier file, then we continue to > scan those older files. So for example we may scan in order: > > master-bin.000004, master-bin.000002, master-bin.000003 > > This means that we may see the XA PREPARE and XA COMMIT out-of-order as we > go from the first scan to the remaining - "round 1 turns to 2". > > But I don't understand that this is "the first of the two options below": > > member->binlog_coord < last_gtid_coord ? member->binlog_coord.second : pos > > If last_gtid_coord is in master-bin.000002, but member->binlog_coord is in > master-bin.000004 ("round 1 turns to 2"), then I would think we would have > > member->binlog_coord > last_gtid_coord > > So it would be the *second* of the options below? What am I missing here? > > Also, I would suggest a simplification here. After we do the initial scan in > round 1, if we find that we need to scan more files, then reset all the > state and hashes. And then do a full scan of all necessary binlog files > in-order. In my example, that would be: > > master-bin.000004 > <reset state> > master-bin.000002 > master-bin.000003 > master-bin.000004 > > This way, the logic becomes simpler, as we don't need logic to handle seeing > events out-of-order. > > This will mean that occasionally we will need to scan one more binlog file, > but I think that is a good trade-off. The recovery code is notoriously > difficult to test and get correct, because it's an unusual occurence to need > recovery in the first place, and very hard to reproduce any problems. And > robust and correct recovery is the whole point of having XA in the first > palce. So making the code as simple as possible is crucial. > > The extra cost is small: > > - For one, recovery is a rare event > - Re-scanning only costs CPU, the file is already read from storage into OS > buffers. > - Normally, the binlog checkpoint completes shortly after rotation, so the > last binlog file that must be rescanned will be short. > > So simply redo the scan from the correct point when a "round 2" is needed, > to avoid any logic to handle seeing events out-of-order during recovery. > > >> @@ -11182,7 +11311,9 @@ void Recovery_context::process_gtid(int round, Gtid_log_event *gev, >> last_gtid_no2pc= false; >> last_gtid_standalone= >> (gev->flags2 & Gtid_log_event::FL_STANDALONE) ? true : false; >> - if (do_truncate && last_gtid_standalone) >> + /* a standalone XA-COMPLETE is truncation-safe */ >> + if (do_truncate && last_gtid_standalone >> + && !(gev->flags2 & Gtid_log_event::FL_COMPLETED_XA)) >> update_binlog_unsafe_coord_if_needed(linfo); > > I don't understand the comment here. > What is "a standalone XA-COMPLETE" ? > What does it mean to be "truncation-safe"? > Please reword the comment to be comprehensible. > >> + /* >> + the user XA events are decided similary to the normal transaction, >> + difference is just in the timing which is Gtid event read now. >> + */ > > I don't understand this comment. Please reformulate to make it > understandable. What does normal transactions do, and what does user > XA do? > How does the timing differ, and why? > >> + sql_print_warning("GTID %u-%u-%s XA transaction state % " >> + "in binlog file:%s pos:%llu is inconsistent " >> + "with the former same state; " >> + "consider filtered binary loggig.", >> + ctx.last_gtid.domain_id, ctx.last_gtid.server_id, >> + buf, is_xac ? "'COMPLETE'" : "'PREPARE'", >> + linfo->log_file_name, ctx.last_gtid_coord.second); > > Typo: s/loggig/logging/ > > Also, the phrase "the former same state" is not understandable (at > least to > me), what does it mean? Please reformulate. > I also have to be short now in replying to the rest, which I'll expand on tomorrow morning. Now just this. >> @@ -909,14 +910,16 @@ ha_commit_checkpoint_request(void *cookie, void (*pre_hook)(void *)) >> there's no need to rollback here as all transactions must >> be rolled back already >> */ >> -void ha_close_connection(THD* thd) >> +void ha_close_connection(THD* thd, bool skip_binlog) > > I tried to understand what the purpose of this change is. > ha_close_connection() is now called from two new places related to > replication of XA PREPARE. And in one place with this mysterious > skip_binlog=true condition. > > My guess is that this is to detach the XA PREPARE from the current > THD? The reason is described in 6cbb63f592 commit name II.1. ``` OTHER CHANGES, to ha_close_connection() are done to keep the binlog hton connection alive for the end of XA-prepare's binlogging on the parallel slave. Other than the binlog hton:s are disconnected from the slave transaction handler in TC_LOG::run_commit_ordered() which makes the prepared transaction engine objects available to binlog-commit-order followers. ``` In other words XAP_k in run_commit_ordered() releases with ha_close_connection(thd, true) all resources that the following XAC_k+i need. The latter does not need XAP_k to release binlog which it still need while being is this stack of TC_LOG::run_commit_ordered MYSQL_BIN_LOG::trx_group_commit_leader MYSQL_BIN_LOG::write_transaction_to_binlog_events MYSQL_BIN_LOG::write_transaction_to_binlog binlog_flush_cache binlog_commit_flush_xa_prepare binlog_commit binlog_prepare prepare_or_error ha_prepare > So > that it becomes visible in XA RECOVER to other threads, and (maybe > more > important) to make the THD available to run another transaction? One > of the > things InnoDB does in close_connection is: trx_disconnect_prepared(). > > But I think the purpose of the close_connection handlerton call is > different And you must have meant 'ha_close_connection', right? > - it is to free things at the end of a client connection? Well, with the ha- prefix, it's to disconnect THD from all transaction branches/htons. > So it can do > more > than detach the current external XA - which is presumably the reason > for the > need to hack it to not call the specific binlog_close_connection()? > > So this shouldn't hijack the close_connection to mean > "detach_xa_trx". It > looks like we need an explicit handlerton call for detaching an xa > transaction, that can then be used instead of this hack in > ha_close_connection(). I need to digest that in the morning. Generally I don't see a problem to conduct hton release flexibly. > >> -static my_bool xacommit_handlerton(THD *unused1, plugin_ref plugin, >> - void *arg) >> +my_bool xacommit_handlerton(THD *unused1, plugin_ref plugin, void *arg) >> { >> handlerton *hton= plugin_hton(plugin); >> - if (hton->recover) >> + if (hton->recover && hton != binlog_hton) >> { >> hton->commit_by_xid(hton, ((struct xahton_st *)arg)->xid); > >> -int ha_commit_or_rollback_by_xid(XID *xid, bool commit) >> +int ha_commit_or_rollback_by_xid(XID *xid, bool commit, THD *thd) >> { > >> + if (!skip_binlog) >> + { >> + // when binlog is ON start from its transaction branch >> if (commit) >> binlog_commit_by_xid(binlog_hton, xid); >> else >> binlog_rollback_by_xid(binlog_hton, xid); > > I think the purpose if this is to ensure that commit/rollback of XA > calls > into the binlog code first (to binlog before doing it in the engines). > > But as I think Serg also remarked already, this should not be > needed. The > binlog is the (internal) transaction coordinator when it is enabled, > we have > a dedicated interface for that, class TC_LOG. This is used elsewhere > in > handler.cc to avoid special cases for whether the binlog is enabled or > not. > > So it looks like there needs to be extra virtual functions in TC_LOG > for > the XA commit/rollback, and these can then be called on the tc_log > object > and do what is required. I need to process that tomorrow. > Then some logic in handler.cc can be omitted/simplified. > > ---- > > I think with this, I got through all of the code changes in branch > bb-10.6-MDEV-31949 as of commit > 1bdc9bc8077c79e55cb4dbfc7cc47a193d17d54f > from January 20, 2024. So this concludes the current review. I have > not > looked at the test cases in this round of review. Thanks again! Cheers, Andrei

1 1

Re2: Starting on review of XA patches in bb-10.6-MDEV-31949
by andrei.elkin＠pp.inet.fi 12 Feb '24

12 Feb '24

Salve Kristian, this is the 2nd follow-up replay to your 1st review, to cover few deferred notes in particular. I mark with [x] any addressed [todo] note. > Kristian Nielsen <knielsen(a)knielsen-hq.org> writes: > >> Hi Andrei, > So you're in this context > > 1 innobase_commit_by_xid( > 2 /*===================*/ > 3 handlerton* hton, > 4 XID* xid) /*!< in: X/Open XA transaction > identification */ > 5 { > 6 DBUG_ASSERT(hton == innodb_hton_ptr); > 7 > 8 DBUG_EXECUTE_IF("innobase_xa_fail", > 9 return XAER_RMFAIL;); > 10 > 11 if (high_level_read_only) { > 12 return(XAER_RMFAIL); > 13 } > 14 > 15 if (trx_t* trx = trx_get_trx_by_xid(xid)) { > 16 THD *thd = current_thd; > 17 if (thd) > 18 thd_binlog_pos(thd, &trx->mysql_log_file_name, > 19 &trx->mysql_log_offset); > 20 if (trx->mysql_log_offset > 0) > 21 trx->flush_log_later = true; > > and actually considering `trx->mysql_log_offset` as the return value? > ... I could not grasp your way of thinking in this place. > Could you please expand on? In another mail yesterday concerning with this part I suggested to replace if (trx->mysql_log_offset > 0) with if (thd_binlog_format(thd) != UNSPEC) > > >> >> It would be much better if this could be done similar to how normal >> commits >> are done. We have the commit_ordered handlerton which does the fast >> part of >> commit-to-memory. And then the slow part which does the rest in commit >> handlerton. > > Notice that the *slow* part of the XA commit is the same as in the > normal > case: > > trx_commit_complete_for_mysql > *innobase_commit* > commit_one_phase_2 > ha_commit_one_phase > > The fast parts of the two - the XA and normal - could be unified - into > > innobase_commit_ordered() > > but I did not think (had time for) towards that. > >> >> So we could have commit_by_xid_ordered() and commit_by_xid(). The >> problem is >> that commit_by_xid_ordered() needs to release the xid, so it becomes >> harder >> to get hold of the transaction (which the commit handlerton just finds >> from >> the thd). Maybe commit_by_xid_ordered() would need to return some trx >> handle. >> >> What part of the commit is skipped by this in InnoDB? Is it _only_ >> flushing >> the log up to the LSN of the commit record, or are there more things? > > > So the xa's fast innobase_commit_by_xid() skips a "slow" > trx_commit_complete_for_mysql() > and the same does the normal's innobase_commit_ordered(). > >> >> How is another storage engine supposed to handle this? This also needs >> to be >> documented somewhere, eg. in comments on the handlerton calls. >> >> This also means that the --innodb-flush-log-at-trx-commit=3 is not >> supported >> for XA COMMIT when it's done from a different session that the XA >> PREPARE. >> For example, with --innodb-flush-log-at-trx-commit=3 but >> --sync-binlog=0, >> normal transactions are still durable in InnoDB. But external XA >> transactions will not be. This could be solved if the commit_by_xid() >> was >> split into a fast and slow part similar to normal commit. > > [todo] Let me process that carefully later. [x] My 1st reply already said, just missed to underscore that, that the XA also the slow and fast commit paths, therefore iflatc=3 is good. The past paths are commit_one_phase_2 and innobase_commit_by_xid functionally equivalent. One would vote to unify them, and I'd accept that. > >> >> Probably you'll say that this works for now, and perhaps that's true. >> But at >> least it really _really_ needs some detailed comments explaining what >> is >> going on, and what should be kept in mind if modifying/improving this >> code >> in the future, otherwise it will be impossible for the next developer >> to >> understand. > > [todo] Agreed. > >> >> >>> diff --git a/sql/log.cc b/sql/log.cc >>> index e54d4087d46..7e3dbeaef29 100644 >>> --- a/sql/log.cc >>> +++ b/sql/log.cc >> >>> @@ -9365,18 +9381,84 @@ TC_LOG::run_commit_ordered(THD *thd, bool >>> all) >>> all ? thd->transaction->all.ha_list : >>> thd->transaction->stmt.ha_list; >>> >>> mysql_mutex_assert_owner(&LOCK_commit_ordered); >>> - for (; ha_info; ha_info= ha_info->next()) >>> + >>> + if (!ha_info) >> >> Your patch puts a lot of XA/binlog logic into run_commit_ordered(). >> But >> that's not appropriate. The run_commit_ordered() function has the >> purpose >> solely to call the commit_ordered() handlerton in each engine, and >> it's also >> called from TC_LOG_MMAP (that's why it's TC_LOG::, not >> MYSQL_BIN_LOG::). >> >> Instead make a new function that handles the part of binlog commit >> that >> needs to be done under LOCK_commit_ordered. Be sure to comment it that >> this >> runs under this highly contended mutex, and needs to do as little work >> as >> possible. Agreed. [todo] >> >> This function can then call run_commit_ordered(). It can also be used >> to >> share a few bits of code that are now duplicated between the >> opt_optimize_thread_scheduling and !opt_optimize_thread_scheduling >> cases, >> like: >> >> ++num_commits; >> run_commit_ordered(...) >> entry->thd->wakeup_subsequent_commits() >> if (entry->queued_by_other) { ... } >> >> (if it makes sense to do so in the actual code, only if makes things >> simpler, that's easier to decide when working with the actual code). Will see. [todo] >> >> There sesems to be 3 new cases: >> >> - XA PREPARE >> - XA COMMIT from same session as XA PREPARE >> - XA COMMIT of detached XA transaction >> >> The logic to decide this was already done earlier, when the event >> group was >> queued for group commit, eg. to decide which end_event to use. For >> example >> there is no need to call xid_cache_search() here, that can be done >> earlier >> (and it's also wrong to call my_error() here, as we're in the wrong >> thread >> context). Indeed. The error's thread context was overlooked. Thanks for catching it. >> >> So the logic about what to do should be placed higher up in the call >> stack, >> ideally where we already know, eg. binlog_commit_by_xid(). And errors >> such >> as ER_XAER_NOTA can be thrown already there, and the decision on what >> to do >> put consisely as a flag into the struct group_commit_entry, so the >> logic >> during binlog group commit becomes as simple as possible. >> >> (It might even make sense to simply store a function pointer to the >> method >> that should be called to handle this; normal, attached XA COMMIT, >> detached >> XA COMMIT, XA PREPARE. But again it's easier when actually working >> with the >> code to decide if this is simpler than a couple flags or case on enum >> value). > > > [todo] Let me process the above suggestion block a bit later. [x] To the need of xid_cache_search() etc, let me combine this part to reply to the 2nd review mail's notes around a Dummy XID though ... (read on) > >> >>> + if (thd->rgi_slave && thd->rgi_slave->is_parallel_exec) >>> + { >>> + DBUG_ASSERT(thd->transaction->xid_state.is_dummy_XA()); >>> + >>> + auto xs= xid_cache_search(thd, xid); >>> + if(!xs) >>> + { >>> + my_error(ER_XAER_NOTA, MYF(0)); >>> + return; >>> + } >>> + else >>> + { >>> + thd->transaction->xid_state.xid_cache_element= xs; >>> + } >>> + DBUG_ASSERT(thd->transaction->xid_state.is_recovered()); >>> + } >>> + >>> plugin_foreach(NULL, commit ? xacommit_handlerton : >>> xarollback_handlerton, >>> MYSQL_STORAGE_ENGINE_PLUGIN, &xaop); >>> xid_cache_delete(thd); >> >> Why this check for thd->rgi_slave->is_parallel_exec here deep in >> binlog >> group commit? That seems very wrong, and again no comment whatsoever >> explaining what is going on, so again I have to try to guess... >> >> I'm guessing that it's something with calling xid_cache_delete() which >> needs >> to find thd->transaction->xid_state.xid_cache_element ? > > Right. > >> >> But this could be done much better in a higher-level part of the code >> that's >> specific to xa and (parallel) replication, not here deep down in >> binlog >> group commit code. >> >> And why should this be needed only for parallel replication, not for >> non-parellel replication or non-replication SQL? I don't understand. > > > It's a kind of optimization for the slave side, in that `xid` of XAP > (Prepare) gets released for a XAC (Commit) the soonest. > ... the reason's said here. >> >> >> A couple remarks about the tests I pushed to >> knielsen_mdev31949_review: >> > >> >> I think the failures in rpl.rpl_recovery_xa I linked are from >> duplicate XID >> in XA PREPARE ; XA COMMIT ; XA PREPARE. There comes an ambiguity >> between >> whether the transaction prepared in the engine at recovery is from the >> first >> XA PREPARE (so it should be committed) or the second (so it should be >> rolled >> back). This happens when the second XA PREPARE is in the engine but >> not >> binlogged. >> >> One way to fix could actually be to use the binlog position stored >> inside >> InnoDB. That's a neat reminder on the Innodb durably committed binlog position! Indeed with the duplicate xid case there exists this trouble of identification and the Engine is able and must come to rescue. >> If the XA COMMIT is before this binlog position, then we know the >> first transaction became durable in InnoDB, so we know we have to roll >> back >> the current prepared transaction in the engine. Otherwise we should >> commit >> it. I see the recovery code already uses binlog positions to decide >> the fate >> of pending XA transactions in the engines, but I did not yet fully >> understand what is going on there. >> >> Another way could be to binlog an extra event before the second XA >> PREPARE >> with the duplicate XID. Well, this does not cover the case of prior to crash XA PREPARE only made to Engine. The above consulting with Innodb resolves the ambiguity generally. >> This event would mean "This xid is now durable >> committed in the engine", so we would know not to commit any prepared >> trx in >> the engine with the same xid. This event would only be needed in case >> of XA >> PREPARE with a duplicate xid. This though requires the prior XA COMMIT >> to >> fsync inside InnoDB first, I think. > > Thanks for this piece of analysis! > [todo] I am processing this a bit later. [x] > >> >> >> Another test failure in rpl.slave_provision_xa I think happens because >> the >> XA PREPARE does not update the binlog position inside innodb. >> Maybe the way >> to solve this problem is to have the slave be idempotent, and silently >> ignore any duplicate XA PREPARE if the xid is already prepared? Indeed, the fact that XAP does not update the innodb's binlog position must cause few concerns. I still need to have a close look. [todo] > > [todo] I am processing this a bit later. [x] > >> >> >> About the "ToDo" of using some Xid_list_log_event. I think this is a >> way >> during recovery to get the state of the XA PREPARE/COMMIT's in earlier >> binlog files before the last binlog checkpoint, if I understood things >> correctly? > > True. > >> >> Another way to do this is to delay the binlog checkpoints until all XA >> PREPARE in a binlog file have been XA COMMIT'ted (or XA ROLLBACK'ed), >> so >> that they will all be scanned during recovery. I actually implemented >> this >> in my branch knielsen_mdev32020. The advantage of this is that we may >> anyway >> need this to provide the XA PREPARE events to a slave set up from a >> logical >> backup (like mysqldump), to avoid purging XA PREPARE events too early. >> And >> then it's a bit simpler to simply scan an extra file rather than both >> scan >> and read Xid_list_log_event. But I think Xid_list_log_event could work >> fine >> as well, just mentioning it for you to consider. > > Xlle can't be generally avoided so we have to take conservatively > first. > To add up to a separate thread discussion over this problematics, I've grown more inclined with the first implementation basing on your idea to defer Binlog-CheckPoint. That might work for a majority of XA users, Xid list to be added later if time permits or will be convincingly :-) requested. Now I am turning to account the rest - 2,3 reviews you sent that only glanced through - to reply within few hours. Cheers, Andrei

2 2

Re: 64-bit time_t support in MariaDB?
by Otto Kekäläinen 12 Feb '24

12 Feb '24

> it's not related to time, it's just an assert that's verified at compile > time, that is, during compilation. Like static_assert in C++. > > I just need to find 32-bit arm somewhere. Unfortunately I don't have any with SSH access to give. Seems the reported of MDEV-33429 has armhf hardware and is helping out. Once you have a patch, I can push it to Launchpad builders (or you can do it yourself, anyone with an Launchpad can push code to their own PPA). Related, Cfarm is a project that gives free accounts to open source developers in case you want to have access to play around with multiple architectures: https://portal.cfarm.net/machines/list/ Unfortunately they don't seem to have armhf/armel.

2 1

Re2: thd_binlog_pos / Re: Starting on review of XA patches in bb-10.6-MDEV-31949
by andrei.elkin＠pp.inet.fi 08 Feb '24

08 Feb '24

>> So you're in this context >> >> 1 innobase_commit_by_xid( > >> 18 thd_binlog_pos(thd, &trx->mysql_log_file_name, >> 19 &trx->mysql_log_offset); >> 20 if (trx->mysql_log_offset > 0) >> 21 trx->flush_log_later = true; >> >> and actually considering `trx->mysql_log_offset` as the return value? >> Could you please expand on? > > My point was just, that what we want to do here is for the (internal) > transaction coordinator to tell the storage engine that the engine does > not > need to fsync() for the commit to be durable, another fsync was already > done > to ensure this. This is what we in the normal commit path do by calling > commit_ordered(), as documented sql/handler.h. > > The thd_binlog_pos() is completely unrelated, it's a way for the > storage... I see your point. My patch's block of if (trx_t* trx = trx_get_trx_by_xid(xid)) { + THD *thd = current_thd; + if (thd) + thd_binlog_pos(thd, &trx->mysql_log_file_name, + &trx->mysql_log_offset); + if (trx->mysql_log_offset > 0) + trx->flush_log_later = true; + /* use cases are: disconnected xa, slave xa, recovery */ innobase_commit_low(trx); + trx->mysql_log_file_name = NULL; should not indeed rely on anything that thd_binlog_pos() computes. I think I wanted to spare --skip-log-bin setups from the flush_log_later policy. I see now that alternatively thd_binlog_format(thd) could be used which will always return BINLOG_FORMAT_UNSPEC when that's the case (or @@skip_log_bin=1). You think it's better to covert the block to use that?

1 0

Starting on review of XA patches in bb-10.6-MDEV-31949
by Kristian Nielsen 07 Feb '24

07 Feb '24

Hi Andrei, It's not an easy review of the XA patches in bb-10.6-MDEV-31949. One thing that would have helped is a clear design document that lists all the different concerns and points that need to be considered around recovery, backup, replication, possible conflicts, etc. This could then be referenced by the reviewer to see if a given concern was considered, and if so what the resolution is. But I'll try as best I can with what we have. As I'm trying to understand the different issues, I found making test cases to check my understanding helped. I have pushed some to the branch knielsen_mdev31949_review . They seem to show some problems in the current patch: mysql-test/suite/rpl/t/rpl_recovery_xa.test - A trx is left in "XA PREPARED" state after crash recovery, even though the binlogged event group is incomplete (it should have been rolled back). - A trx is committed after crash recovery, even though it is not binlogged (it should have been rolled back). - An incomplete XA event group is output by mysqlbinlog with a "ROLLBACK" at the end which gives an error (should probably be XA ROLLBACK). mysql-test/suite/rpl/t/rpl_recovery_xa2.test - An XA PREPAREd transaction is lost after recovery when it was binlogged in an earlier binlog file. (I think this might be the issue that's referenced as a ToDo with Xid_list_log_event in patch?) mysql-test/suite/mariabackup/slave_provision_xa.test - An XA PREPARE does not update the binlog position inside InnoDB. So a slave provisioned with mariabackup from this starting position can fail with duplicate apply of XA PREPARE. Some other initial remarks follow. Since there's a desire to progress with this quickly, I thought it would be useful to send these early, so you can start looking/discussions before I complete building an understanding of the whole design and code picture. > diff --git a/storage/innobase/handler/ha_innodb.cc b/storage/innobase/handler/ha_innodb.cc > index 17e512bf34e..a4cdc72e18c 100644 > --- a/storage/innobase/handler/ha_innodb.cc > +++ b/storage/innobase/handler/ha_innodb.cc > @@ -17144,8 +17144,16 @@ innobase_commit_by_xid( > } > > if (trx_t* trx = trx_get_trx_by_xid(xid)) { > + THD *thd = current_thd; > + if (thd) > + thd_binlog_pos(thd, &trx->mysql_log_file_name, > + &trx->mysql_log_offset); I guess this is necessary to get the correct binlog position stored inside innodb to match the binlogged XA COMMIT event, right? But will it work if called during crash recovery? During crash recovery, we could actually update the position, but only if this is trx is the last event group in the old binlog. Maybe that's better handled elsewhere (there are currently other cases where the binlog position in InnoDB can be wrong immediately after restart until the first new transaction is committed). > + if (trx->mysql_log_offset > 0) > + trx->flush_log_later = true; But I'm guessing this is the motivation. If called from within binlog group commit, you want to disable fsync in InnoDB. But if called from recovery, or when binlog is disabled, this condition will be false, and fsync will be done. If so, that's a _really_ obfuscated way to do it :-/ It surely needs a very clear comment explaining what is going on. And it is very fragile. What if thd_binlog_pos() is changed at some point, to slightly change the conditions under which it returns 0 or not? That could silently break this code. It would be much better if this could be done similar to how normal commits are done. We have the commit_ordered handlerton which does the fast part of commit-to-memory. And then the slow part which does the rest in commit handlerton. So we could have commit_by_xid_ordered() and commit_by_xid(). The problem is that commit_by_xid_ordered() needs to release the xid, so it becomes harder to get hold of the transaction (which the commit handlerton just finds from the thd). Maybe commit_by_xid_ordered() would need to return some trx handle. What part of the commit is skipped by this in InnoDB? Is it _only_ flushing the log up to the LSN of the commit record, or are there more things? How is another storage engine supposed to handle this? This also needs to be documented somewhere, eg. in comments on the handlerton calls. This also means that the --innodb-flush-log-at-trx-commit=3 is not supported for XA COMMIT when it's done from a different session that the XA PREPARE. For example, with --innodb-flush-log-at-trx-commit=3 but --sync-binlog=0, normal transactions are still durable in InnoDB. But external XA transactions will not be. This could be solved if the commit_by_xid() was split into a fast and slow part similar to normal commit. Probably you'll say that this works for now, and perhaps that's true. But at least it really _really_ needs some detailed comments explaining what is going on, and what should be kept in mind if modifying/improving this code in the future, otherwise it will be impossible for the next developer to understand. > diff --git a/sql/log.cc b/sql/log.cc > index e54d4087d46..7e3dbeaef29 100644 > --- a/sql/log.cc > +++ b/sql/log.cc > @@ -9365,18 +9381,84 @@ TC_LOG::run_commit_ordered(THD *thd, bool all) > all ? thd->transaction->all.ha_list : thd->transaction->stmt.ha_list; > > mysql_mutex_assert_owner(&LOCK_commit_ordered); > - for (; ha_info; ha_info= ha_info->next()) > + > + if (!ha_info) Your patch puts a lot of XA/binlog logic into run_commit_ordered(). But that's not appropriate. The run_commit_ordered() function has the purpose solely to call the commit_ordered() handlerton in each engine, and it's also called from TC_LOG_MMAP (that's why it's TC_LOG::, not MYSQL_BIN_LOG::). Instead make a new function that handles the part of binlog commit that needs to be done under LOCK_commit_ordered. Be sure to comment it that this runs under this highly contended mutex, and needs to do as little work as possible. This function can then call run_commit_ordered(). It can also be used to share a few bits of code that are now duplicated between the opt_optimize_thread_scheduling and !opt_optimize_thread_scheduling cases, like: ++num_commits; run_commit_ordered(...) entry->thd->wakeup_subsequent_commits() if (entry->queued_by_other) { ... } (if it makes sense to do so in the actual code, only if makes things simpler, that's easier to decide when working with the actual code). There sesems to be 3 new cases: - XA PREPARE - XA COMMIT from same session as XA PREPARE - XA COMMIT of detached XA transaction The logic to decide this was already done earlier, when the event group was queued for group commit, eg. to decide which end_event to use. For example there is no need to call xid_cache_search() here, that can be done earlier (and it's also wrong to call my_error() here, as we're in the wrong thread context). So the logic about what to do should be placed higher up in the call stack, ideally where we already know, eg. binlog_commit_by_xid(). And errors such as ER_XAER_NOTA can be thrown already there, and the decision on what to do put consisely as a flag into the struct group_commit_entry, so the logic during binlog group commit becomes as simple as possible. (It might even make sense to simply store a function pointer to the method that should be called to handle this; normal, attached XA COMMIT, detached XA COMMIT, XA PREPARE. But again it's easier when actually working with the code to decide if this is simpler than a couple flags or case on enum value). > + if (thd->rgi_slave && thd->rgi_slave->is_parallel_exec) > + { > + DBUG_ASSERT(thd->transaction->xid_state.is_dummy_XA()); > + > + auto xs= xid_cache_search(thd, xid); > + if(!xs) > + { > + my_error(ER_XAER_NOTA, MYF(0)); > + return; > + } > + else > + { > + thd->transaction->xid_state.xid_cache_element= xs; > + } > + DBUG_ASSERT(thd->transaction->xid_state.is_recovered()); > + } > + > plugin_foreach(NULL, commit ? xacommit_handlerton : xarollback_handlerton, > MYSQL_STORAGE_ENGINE_PLUGIN, &xaop); > xid_cache_delete(thd); Why this check for thd->rgi_slave->is_parallel_exec here deep in binlog group commit? That seems very wrong, and again no comment whatsoever explaining what is going on, so again I have to try to guess... I'm guessing that it's something with calling xid_cache_delete() which needs to find thd->transaction->xid_state.xid_cache_element ? But this could be done much better in a higher-level part of the code that's specific to xa and (parallel) replication, not here deep down in binlog group commit code. And why should this be needed only for parallel replication, not for non-parellel replication or non-replication SQL? I don't understand. A couple remarks about the tests I pushed to knielsen_mdev31949_review: I think the failures in rpl.rpl_recovery_xa I linked are from duplicate XID in XA PREPARE ; XA COMMIT ; XA PREPARE. There comes an ambiguity between whether the transaction prepared in the engine at recovery is from the first XA PREPARE (so it should be committed) or the second (so it should be rolled back). This happens when the second XA PREPARE is in the engine but not binlogged. One way to fix could actually be to use the binlog position stored inside InnoDB. If the XA COMMIT is before this binlog position, then we know the first transaction became durable in InnoDB, so we know we have to roll back the current prepared transaction in the engine. Otherwise we should commit it. I see the recovery code already uses binlog positions to decide the fate of pending XA transactions in the engines, but I did not yet fully understand what is going on there. Another way could be to binlog an extra event before the second XA PREPARE with the duplicate XID. This event would mean "This xid is now durable committed in the engine", so we would know not to commit any prepared trx in the engine with the same xid. This event would only be needed in case of XA PREPARE with a duplicate xid. This though requires the prior XA COMMIT to fsync inside InnoDB first, I think. Another test failure in rpl.slave_provision_xa I think happens because the XA PREPARE does not update the binlog position inside innodb. Maybe the way to solve this problem is to have the slave be idempotent, and silently ignore any duplicate XA PREPARE if the xid is already prepared? About the "ToDo" of using some Xid_list_log_event. I think this is a way during recovery to get the state of the XA PREPARE/COMMIT's in earlier binlog files before the last binlog checkpoint, if I understood things correctly? Another way to do this is to delay the binlog checkpoints until all XA PREPARE in a binlog file have been XA COMMIT'ted (or XA ROLLBACK'ed), so that they will all be scanned during recovery. I actually implemented this in my branch knielsen_mdev32020. The advantage of this is that we may anyway need this to provide the XA PREPARE events to a slave set up from a logical backup (like mysqldump), to avoid purging XA PREPARE events too early. And then it's a bit simpler to simply scan an extra file rather than both scan and read Xid_list_log_event. But I think Xid_list_log_event could work fine as well, just mentioning it for you to consider. That's it for now, I will continue trying to understand what is going on and complete the review. - Kristian.

1 2

Re: Xlle question
by andrei.elkin＠pp.inet.fi 06 Feb '24

06 Feb '24

>> Xlle can't be generally avoided so we have to take conservatively >> first. > > Why it cannot be avoided? What is it needed for? The main purpose is to discern at recovery a prepared XA (XAP) that was never binlogged from one that was. Your case >>> Another way to do this is to delay the binlog checkpoints until all >>> XA >>> PREPARE in a binlog file have been XA COMMIT'ted assumes that XAC must always come in "normal" time. But of course, being a part of the user's decision, it can be late significantly up until a purge from binary logs that contain the incl the total RESET MASTER. In the latter case its eventual post RM XA-COMMIT would be bin-logged with the full awareness.

1 0

Fwd/Re: Starting on review of XA patches in bb-10.6-MDEV-31949
by andrei.elkin＠pp.inet.fi 05 Feb '24

05 Feb '24

From: Andrei Elkin <andrei.elkin(a)mariadb.com> Subject: Re: Starting on review of XA patches in bb-10.6-MDEV-31949 To: Kristian Nielsen <knielsen(a)knielsen-hq.org> Cc: developers(a)lists.mariadb.org, andrei elkin <andrei.elkin(a)mariadb.com>, Michael Widenius <michael.widenius(a)gmail.com> Date: Mon, 05 Feb 2024 14:11:01 +0200 (36 minutes, 34 seconds ago) Organization: MariaDB Kristian Nielsen <knielsen(a)knielsen-hq.org> writes: > Hi Andrei, > > It's not an easy review of the XA patches in bb-10.6-MDEV-31949. > > One thing that would have helped is a clear design document that lists > all > the different concerns and points that need to be considered around > recovery, backup, replication, possible conflicts, etc. This could then > be > referenced by the reviewer to see if a given concern was considered, > and if > so what the resolution is. I am sorry for discomfort the lack of the above caused. Unfortunately the IV recovery part has consumed more time than was expected (which is at the final stage of my local testing). > But I'll try as best I can with what we > have. If you'll need any online discussion to compensate for that I'd be glad to do it. > > As I'm trying to understand the different issues, I found making test > cases > to check my understanding helped. I have pushed some to the branch > knielsen_mdev31949_review. Thank you for streamlining it this way! > They seem to show some problems in the current > patch: > > mysql-test/suite/rpl/t/rpl_recovery_xa.test > - A trx is left in "XA PREPARED" state after crash recovery, even > though > the binlogged event group is incomplete (it should have been > rolled > back). > - A trx is committed after crash recovery, even though it is not > binlogged (it should have been rolled back). > - An incomplete XA event group is output by mysqlbinlog with a > "ROLLBACK" > at the end which gives an error (should probably be XA ROLLBACK). > > mysql-test/suite/rpl/t/rpl_recovery_xa2.test > - An XA PREPAREd transaction is lost after recovery when it was > binlogged > in an earlier binlog file. (I think this might be the issue that's > referenced as a ToDo with Xid_list_log_event in patch?) > > mysql-test/suite/mariabackup/slave_provision_xa.test > - An XA PREPARE does not update the binlog position inside InnoDB. > So a > slave provisioned with mariabackup from this starting position can > fail > with duplicate apply of XA PREPARE. > > Some other initial remarks follow. Since there's a desire to progress > with > this quickly, I thought it would be useful to send these early, so you > can > start looking/discussions before I complete building an understanding > of the > whole design and code picture. Super! > > >> diff --git a/storage/innobase/handler/ha_innodb.cc >> b/storage/innobase/handler/ha_innodb.cc >> index 17e512bf34e..a4cdc72e18c 100644 >> --- a/storage/innobase/handler/ha_innodb.cc >> +++ b/storage/innobase/handler/ha_innodb.cc >> @@ -17144,8 +17144,16 @@ innobase_commit_by_xid( >> } >> >> if (trx_t* trx = trx_get_trx_by_xid(xid)) { >> + THD *thd = current_thd; >> + if (thd) >> + thd_binlog_pos(thd, &trx->mysql_log_file_name, >> + &trx->mysql_log_offset); > > I guess this is necessary to get the correct binlog position stored > inside > innodb to match the binlogged XA COMMIT event, right? Correct. > But will it work if called during crash recovery? Right. The unpushed part IV makes sure of that. > > During crash recovery, we could actually update the position, but only > if > this is trx is the last event group in the old binlog. Maybe that's > better > handled elsewhere (there are currently other cases where the binlog > position > in InnoDB can be wrong immediately after restart until the first new > transaction is committed). > >> + if (trx->mysql_log_offset > 0) >> + trx->flush_log_later = true; > > But I'm guessing this is the motivation. If called from within binlog > group > commit, you want to disable fsync in InnoDB. But if called from > recovery, or > when binlog is disabled, this condition will be false, and fsync will > be > done. Indeed. > > If so, that's a _really_ obfuscated way to do it :-/ I remember it was a very quick decision for myself to miss this sort of reflection. [todo] I'll add comments of course. > It surely needs a very clear comment explaining what is going on. > And it is very fragile. What if thd_binlog_pos() is changed at some > point, > to slightly change the conditions under which it returns 0 or not? That > could silently break this code. So you're in this context 1 innobase_commit_by_xid( 2 /*===================*/ 3 handlerton* hton, 4 XID* xid) /*!< in: X/Open XA transaction identification */ 5 { 6 DBUG_ASSERT(hton == innodb_hton_ptr); 7 8 DBUG_EXECUTE_IF("innobase_xa_fail", 9 return XAER_RMFAIL;); 10 11 if (high_level_read_only) { 12 return(XAER_RMFAIL); 13 } 14 15 if (trx_t* trx = trx_get_trx_by_xid(xid)) { 16 THD *thd = current_thd; 17 if (thd) 18 thd_binlog_pos(thd, &trx->mysql_log_file_name, 19 &trx->mysql_log_offset); 20 if (trx->mysql_log_offset > 0) 21 trx->flush_log_later = true; and actually considering `trx->mysql_log_offset` as the return value? ... I could not grasp your way of thinking in this place. Could you please expand on? > > It would be much better if this could be done similar to how normal > commits > are done. We have the commit_ordered handlerton which does the fast > part of > commit-to-memory. And then the slow part which does the rest in commit > handlerton. Notice that the *slow* part of the XA commit is the same as in the normal case: trx_commit_complete_for_mysql *innobase_commit* commit_one_phase_2 ha_commit_one_phase The fast parts of the two - the XA and normal - could be unified - into innobase_commit_ordered() but I did not think (had time for) towards that. > > So we could have commit_by_xid_ordered() and commit_by_xid(). The > problem is > that commit_by_xid_ordered() needs to release the xid, so it becomes > harder > to get hold of the transaction (which the commit handlerton just finds > from > the thd). Maybe commit_by_xid_ordered() would need to return some trx > handle. > > What part of the commit is skipped by this in InnoDB? Is it _only_ > flushing > the log up to the LSN of the commit record, or are there more things? So the xa's fast innobase_commit_by_xid() skips a "slow" trx_commit_complete_for_mysql() and the same does the normal's innobase_commit_ordered(). > > How is another storage engine supposed to handle this? This also needs > to be > documented somewhere, eg. in comments on the handlerton calls. > > This also means that the --innodb-flush-log-at-trx-commit=3 is not > supported > for XA COMMIT when it's done from a different session that the XA > PREPARE. > For example, with --innodb-flush-log-at-trx-commit=3 but > --sync-binlog=0, > normal transactions are still durable in InnoDB. But external XA > transactions will not be. This could be solved if the commit_by_xid() > was > split into a fast and slow part similar to normal commit. [todo] Let me process that carefully later. > > Probably you'll say that this works for now, and perhaps that's true. > But at > least it really _really_ needs some detailed comments explaining what > is > going on, and what should be kept in mind if modifying/improving this > code > in the future, otherwise it will be impossible for the next developer > to > understand. [todo] Agreed. > > >> diff --git a/sql/log.cc b/sql/log.cc >> index e54d4087d46..7e3dbeaef29 100644 >> --- a/sql/log.cc >> +++ b/sql/log.cc > >> @@ -9365,18 +9381,84 @@ TC_LOG::run_commit_ordered(THD *thd, bool all) >> all ? thd->transaction->all.ha_list : >> thd->transaction->stmt.ha_list; >> >> mysql_mutex_assert_owner(&LOCK_commit_ordered); >> - for (; ha_info; ha_info= ha_info->next()) >> + >> + if (!ha_info) > > Your patch puts a lot of XA/binlog logic into run_commit_ordered(). But > that's not appropriate. The run_commit_ordered() function has the > purpose > solely to call the commit_ordered() handlerton in each engine, and it's > also > called from TC_LOG_MMAP (that's why it's TC_LOG::, not > MYSQL_BIN_LOG::). > > Instead make a new function that handles the part of binlog commit that > needs to be done under LOCK_commit_ordered. Be sure to comment it that > this > runs under this highly contended mutex, and needs to do as little work > as > possible. > > This function can then call run_commit_ordered(). It can also be used > to > share a few bits of code that are now duplicated between the > opt_optimize_thread_scheduling and !opt_optimize_thread_scheduling > cases, > like: > > ++num_commits; > run_commit_ordered(...) > entry->thd->wakeup_subsequent_commits() > if (entry->queued_by_other) { ... } > > (if it makes sense to do so in the actual code, only if makes things > simpler, that's easier to decide when working with the actual code). > > There sesems to be 3 new cases: > > - XA PREPARE > - XA COMMIT from same session as XA PREPARE > - XA COMMIT of detached XA transaction > > The logic to decide this was already done earlier, when the event group > was > queued for group commit, eg. to decide which end_event to use. For > example > there is no need to call xid_cache_search() here, that can be done > earlier > (and it's also wrong to call my_error() here, as we're in the wrong > thread > context). > > So the logic about what to do should be placed higher up in the call > stack, > ideally where we already know, eg. binlog_commit_by_xid(). And errors > such > as ER_XAER_NOTA can be thrown already there, and the decision on what > to do > put consisely as a flag into the struct group_commit_entry, so the > logic > during binlog group commit becomes as simple as possible. > > (It might even make sense to simply store a function pointer to the > method > that should be called to handle this; normal, attached XA COMMIT, > detached > XA COMMIT, XA PREPARE. But again it's easier when actually working with > the > code to decide if this is simpler than a couple flags or case on enum > value). [todo] Let me process the above suggestion block a bit later. > >> + if (thd->rgi_slave && thd->rgi_slave->is_parallel_exec) >> + { >> + DBUG_ASSERT(thd->transaction->xid_state.is_dummy_XA()); >> + >> + auto xs= xid_cache_search(thd, xid); >> + if(!xs) >> + { >> + my_error(ER_XAER_NOTA, MYF(0)); >> + return; >> + } >> + else >> + { >> + thd->transaction->xid_state.xid_cache_element= xs; >> + } >> + DBUG_ASSERT(thd->transaction->xid_state.is_recovered()); >> + } >> + >> plugin_foreach(NULL, commit ? xacommit_handlerton : >> xarollback_handlerton, >> MYSQL_STORAGE_ENGINE_PLUGIN, &xaop); >> xid_cache_delete(thd); > > Why this check for thd->rgi_slave->is_parallel_exec here deep in binlog > group commit? That seems very wrong, and again no comment whatsoever > explaining what is going on, so again I have to try to guess... > > I'm guessing that it's something with calling xid_cache_delete() which > needs > to find thd->transaction->xid_state.xid_cache_element ? Right. > > But this could be done much better in a higher-level part of the code > that's > specific to xa and (parallel) replication, not here deep down in binlog > group commit code. > > And why should this be needed only for parallel replication, not for > non-parellel replication or non-replication SQL? I don't understand. It's a kind of optimization for the slave side, in that `xid` of XAP (Prepare) gets released for a XAC (Commit) the soonest. > > > A couple remarks about the tests I pushed to knielsen_mdev31949_review: > > > I think the failures in rpl.rpl_recovery_xa I linked are from duplicate > XID > in XA PREPARE ; XA COMMIT ; XA PREPARE. There comes an ambiguity > between > whether the transaction prepared in the engine at recovery is from the > first > XA PREPARE (so it should be committed) or the second (so it should be > rolled > back). This happens when the second XA PREPARE is in the engine but not > binlogged. > > One way to fix could actually be to use the binlog position stored > inside > InnoDB. If the XA COMMIT is before this binlog position, then we know > the > first transaction became durable in InnoDB, so we know we have to roll > back > the current prepared transaction in the engine. Otherwise we should > commit > it. I see the recovery code already uses binlog positions to decide the > fate > of pending XA transactions in the engines, but I did not yet fully > understand what is going on there. > > Another way could be to binlog an extra event before the second XA > PREPARE > with the duplicate XID. This event would mean "This xid is now durable > committed in the engine", so we would know not to commit any prepared > trx in > the engine with the same xid. This event would only be needed in case > of XA > PREPARE with a duplicate xid. This though requires the prior XA COMMIT > to > fsync inside InnoDB first, I think. Thanks for this piece of analysis! [todo] I am processing this a bit later. > > > Another test failure in rpl.slave_provision_xa I think happens because > the > XA PREPARE does not update the binlog position inside innodb. Maybe the > way > to solve this problem is to have the slave be idempotent, and silently > ignore any duplicate XA PREPARE if the xid is already prepared? [todo] I am processing this a bit later. > > > About the "ToDo" of using some Xid_list_log_event. I think this is a > way > during recovery to get the state of the XA PREPARE/COMMIT's in earlier > binlog files before the last binlog checkpoint, if I understood things > correctly? True. > > Another way to do this is to delay the binlog checkpoints until all XA > PREPARE in a binlog file have been XA COMMIT'ted (or XA ROLLBACK'ed), > so > that they will all be scanned during recovery. I actually implemented > this > in my branch knielsen_mdev32020. The advantage of this is that we may > anyway > need this to provide the XA PREPARE events to a slave set up from a > logical > backup (like mysqldump), to avoid purging XA PREPARE events too early. > And > then it's a bit simpler to simply scan an extra file rather than both > scan > and read Xid_list_log_event. But I think Xid_list_log_event could work > fine > as well, just mentioning it for you to consider. Xlle can't be generally avoided so we have to take conservatively first. > > > That's it for now, I will continue trying to understand what is going > on and > complete the review. Thank you for so prompt piece of feedback! I'll return with a reply within a work day to [todo] tagged. Cheers, Andrei ----------

2 1

Review of MDEV-7850: Extend GTID Binlog Events with Thread Id
by Kristian Nielsen 31 Jan '24

31 Jan '24

> commit c37b2087b4abe576f1b0391c8d379dba6299dcb5 (origin/bb-11.4-MDEV-7850) > Author: Brandon Nesterenko <brandon.nesterenko(a)mariadb.com> > Date: Mon Jul 10 18:53:19 2023 +0300 > > MDEV-7850: Extend GTID Binlog Events with Thread Id > > This patch augments Gtid_log_event with the user thread-id. > In particular that compensates for the loss of this info in > Rows_log_events. Here's my review of this patch (which was already pushed, but there are some misunderstandings that should be fixed; Monty asked me about it as he had been told that I had reviewed the patch, but that is not the case (until now)): > diff --git a/sql/log_event.cc b/sql/log_event.cc > index 41ddc038594..f167284ad1d 100644 > --- a/sql/log_event.cc > +++ b/sql/log_event.cc > @@ -1901,11 +1904,12 @@ Query_log_event::begin_event(String *packet, ulong ev_offset, > /* > Currently we only need to replace GTID event. > The length of GTID differs depending on whether it contains commit id. > + And/or thread id. > */ > - DBUG_ASSERT(data_len == LOG_EVENT_HEADER_LEN + GTID_HEADER_LEN || > - data_len == LOG_EVENT_HEADER_LEN + GTID_HEADER_LEN + 2); > - if (data_len != LOG_EVENT_HEADER_LEN + GTID_HEADER_LEN && > - data_len != LOG_EVENT_HEADER_LEN + GTID_HEADER_LEN + 2) > + DBUG_ASSERT(data_len >= LOG_EVENT_HEADER_LEN + GTID_HEADER_LEN && > + data_len <= LOG_EVENT_HEADER_LEN + GTID_HEADER_LEN + 2 + 9); > + if (data_len < LOG_EVENT_HEADER_LEN + GTID_HEADER_LEN || > + data_len > LOG_EVENT_HEADER_LEN + GTID_HEADER_LEN + 2 + 9) This change of the asserts/checks doesn't make any sense. They were there originally, because we are hand-crafting a BEGIN query event, and there were only two cases of the length, so two fixed-sized events were crafted by two different code paths. And the asserts are there to safe-guard if the length would change. But now this check was already wrong, as more optional info has been added to the GTID event without updating the code for Query_log_event::begin_event() (XID, flags_extra, extra_engines). But the new code in this patch is generic and can handle any length GTID event, so these checks no longer make any sense, and should just be removed (rather than attempt to correct them). > @@ -1926,9 +1930,19 @@ Query_log_event::begin_event(String *packet, ulong ev_offset, > } > else > { > - DBUG_ASSERT(data_len == LOG_EVENT_HEADER_LEN + GTID_HEADER_LEN + 2); > - /* Put in an empty time_zone_str to take up the extra 2 bytes. */ > - int2store(q + Q_STATUS_VARS_LEN_OFFSET, 2); > + DBUG_ASSERT(data_len <= LOG_EVENT_HEADER_LEN + GTID_HEADER_LEN + 11); This assert should not be there. The length can be larger than 11, and the code below works fine for arbitrary length. > + DBUG_ASSERT(data_len >= LOG_EVENT_HEADER_LEN + GTID_HEADER_LEN + 2); > + > + /* Put in an empty time_zone_str to take up the extra 2 plus the number of > + dummy_bytes. */ > + size_t dummy_bytes= > + data_len - (LOG_EVENT_HEADER_LEN + GTID_HEADER_LEN + 2); > + > + int2store(q + Q_STATUS_VARS_LEN_OFFSET, dummy_bytes + 2); > + for (size_t i= 0; i < dummy_bytes; i++) > + q[Q_DATA_OFFSET + i]= Q_DUMMY; > + q+= dummy_bytes; > + > q[Q_DATA_OFFSET]= Q_TIME_ZONE_CODE; > q[Q_DATA_OFFSET+1]= 0; /* Zero length for empty time_zone_str */ > q[Q_DATA_OFFSET+2]= 0; /* Zero terminator for empty db */ This new code with Q_DUMMY is generic and independent of the length of the GTID event. That's good! But it needs a comment explaining (that the Q_DUMMY is unknown to the old slave, which will make the old slave just skip the rest of the user vars, which are just there to fill up the extra bytes). And now the code that adds the dummy empty time_zone_str is redundant, and should just be removed, as should the code for the case of data_len == LOG_EVENT_HEADER_LEN + GTID_HEADER_LEN. Since now any extra bytes in the GTID can just be replaced by Q_DUMMY. This way, the code will be fixed to work for all the possible GTID lengths, and be shorter and simpler (and a lot less confusing). > @@ -1584,6 +1584,9 @@ Query_log_event::Query_log_event(const uchar *buf, uint event_len, > sa_seq_no = uint8korr(pos); > pos+= 8; > } > + } > + case Q_DUMMY: > + { > break; > } > default: > /* That's why you must write status vars in growing order of code */ > DBUG_PRINT("info",("Query_log_event has unknown status vars (first has code: %u), skipping the rest of them", (uint) *(pos-1))); > pos= (const uchar*) end; // Break loop There's no point in having this code. Q_DUMMY is supposed to be higher than any known status var type, so it will be ignored anyway in the `default` case. And besides, server that knows about Q_DUMMY can never receive it in the first place. (But if this is really wanted, then don't delete the `break` statement in the prior switch case, that's likely to introduce a bug the next time a case is added. And assign Q_DUMMY the next free value, not the value 255). > diff --git a/sql/log_event.h b/sql/log_event.h > index 473506fafed..d324642ad5e 100644 > --- a/sql/log_event.h > +++ b/sql/log_event.h > @@ -324,6 +324,7 @@ class String; > > #define Q_HRNOW 128 > #define Q_XID 129 > +#define Q_DUMMY 255 This needs a good comment, explaining how Q_DUMMY is a code that will be unknown to any server version, and can be used to make the slave skip this (and any following) user vars. diff --git a/mysql-test/suite/rpl/t/rpl_gtid_thread_id.test b/mysql-test/suite/rpl/t/rpl_gtid_thread_id.test new file mode 100644 index 00000000000..b8ed1865da9 --- /dev/null +++ b/mysql-test/suite/rpl/t/rpl_gtid_thread_id.test @@ -0,0 +1,125 @@ +# +# Verify that GTID log events are written into the binary log along with the +# id of the thread which executed the transaction. On the primary, this is the <snip> Disclaimer: I did not review this testcase. - Kristian.

2 3

Slow or hanging parallel replication in 10.6
by Kristian Nielsen 30 Jan '24

30 Jan '24

As we are discussing possible problems with XA parallel replication being slow or hanging in 10.6, I just want to remind of these recently fixed bugs: MDEV-31482: Lock wait timeout with INSERT-SELECT, autoinc, and statement-based replication This can cause slow or hanging parallel replication, as it hangs for --innodb-lock-wait-timeout MDEV-32096 Parallel replication lags because innobase_kill_query() may fail to interrupt a lock wait This can cause show or hanging parallel replication MDEV-31655: Parallel replication deadlock victim preference code errorneously removed This can cause parallel replication to be slow or fail with "too many retries" These bugs were fixed in 10.6.16. These bugs are not related to XA. But the symptoms of these bugs are that optimistic parallel replication occasionally is slow, hangs, or breaks under high load. So it is important not to mistakenly assume that these symptoms are necessarily caused by XA, even though the workload that has problems might be using external XA transactions. In versions prior to 10.6.16, these kinds of problems are likely to be caused by the above known bugs. (I got this list from a quick scan through the 10.6 commit history, there may be one or two more like them I missed. But the conclusion is the same, using optimistic parallel replication under high load in 10.6 requires using at least 10.6.16 to work). - Kristian.

1 0

PR#2779: Fix merge commit 5ea5291: No test file or result files should be executable
by Otto Kekäläinen 28 Jan '24

28 Jan '24

Hi Sanja! Did you notice my PR https://github.com/MariaDB/server/pull/2779? I tagged you about a minor bug in you previous merge. I was hoping you would notice it and review that PR, but Daniel Black jumped on it. I am asking now because it seems the exact same thing happened for mysql-test/main/lowercase_table2.result in your commit fecd78b8378 from November 8th, so I thought I'd send an extra email now to ensure you are aware of this happening. - Otto

2 3

Suggestions for the problems around replication of XA in 10.5
by Kristian Nielsen 27 Jan '24

27 Jan '24

Hi Monty, As promised, here my thoughts around the issues with the implementation of replication of external XA as implemented since MariaDB 10.5. I see at least two architectural issues with the current implementation. One is that it splits transactions in two separate GTIDs in the binlog (XA PREPARE and XA COMMIT). This breaks with the fundamental principle that replication applies transactions one after the other in strict sequence, and the replication position/GTID is the single transaction last replicated. This for example means that a mysqldump backup can no longer be used to provision a slave, since any XA PREPAREd event at the time of the dump will be missing; a testcase rpl_xa_provision.test in MDEV-32020 demonstrates this. Another architectural issue is that each XA PREPARE keeps row locks around on every slave until commit (or rollback). This means replication will break if _any_ transaction replicated after the XA PREPARE gets blocked on a lock. This can easily happen; surely in many ways in statement-based replication, and even in row-based replication without primary key as demonstrated by testcase rpl_mdev32020.test in MDEV-32020. There are other problems; for example the update of mysql.gtid_slave_pos cannot be committed crash-safe together with the transaction for XA PREPARE (since the transaction is not committed). I believe the root of the problem is architectural: external XA should be replicated only after they commit on the master. Trying to fix individual problems one by one will not address the root problem and will lead to ever increasing complexity without ever being fully successful. The current implementation appears to only address a very specific and rare use-case, where enhanced semi-synchronous replication is used with row-based binlogging to try to fail-over to a slave and preserve any external XA that was in PREPAREd state on the master before the failover. Mixed-mode replication, provisioning slaves with mysqldump, slaves not intended for failover, etc., seem to be not considered and basically broken since 10.5. Here is my idea for a design that solves most of these problems. At XA PREPARE, we can still write the events to the binlog, but without a GTID, and we do not replicate it to slaves by default. Then at XA COMMIT we binlog a commit transaction that can be replicated normally to the slaves without problems. If necessary, the events for XA COMMIT can be read from the PREPARE earlier in the binlog, eg. after server crash/restart. We already have the binlog checkpoint mechanism to ensure that required binlog files are preserved until no longer needed for transaction recovery. This way we make external XA preserved across server restart, and all normal replication features continue to work - mysqldump, mixed-mode, etc. Nice and simple. Then optionally we can support the specific usecase of being able to recover external XA PREPAREd transactions on a slave after failover. When enabled, the slave can receive the XA PREPARE events and binlog them itself, without applying. Then as part of failover, those XA PREPARE in the binlog that are still pending can be applied, leaving them in PREPAREd state on the new master. This way, _only_ the few transactions that need to be failed-over need special handling, the majority can still just replicate normally. There are different refinements and optimizations that can be added on top of this. But the point is that this is a simple implementation that is robust, correct, and crash-safe from the start, without needing to add complexity and fixes on top. I've done some initial proof-of-concept code for this, and continue to work on it on the branch knielsen_mdev32020 on github. - Kristian.

3 6

Presentation and Interest in "SIMD for SQL Articulations and Capabilities" Undertaking
by Rayan Chowdhury 25 Jan '24

25 Jan '24

Dear MariaDB Developers, I trust this email thinks that you are well. I am Rayan Chowdhury, and I'm an understudy from India as of now seeking Computer Science in NIIT University. I'm excited about the chance to contribute to MariaDB after discovering the "SIMD for SQL Expressions and Functions" project recently. I have a strong fascination with data set frameworks and streamlining, and the objectives of this undertaking adjust impeccably with my goals. Adding vectorized assessment for SQL capabilities and articulations in the MariaDB ColumnStore motor is both captivating and testing, and I'm anxious to be a piece of the work to further develop execution around here. I've started my journey by signing up for the developers' mailing list, and I'm excited to get involved in the MariaDB community. Prior to jumping into commitments, I needed to connect with express my advantage in the undertaking and look for direction on how I can get everything rolling. I'm anticipating adding to MariaDB and gaining from the accomplished designers locally. Much thanks to you for your time, and I'm amped up for the chance of cooperating on this venture. Best respects, Rayan Chowdhury CSE Student, NIIT University

2 1

Re: 1a3974f534f: MDEV-33299 Assertion `(tm->tv_usec % (int) log_10_int[6 - dec]) == 0' failed in void my_timestamp_to_binary(const timeval*, uchar*, uint)
by Sergei Golubchik 24 Jan '24

24 Jan '24

Hi, Alexander, On Jan 24, Alexander Barkov wrote: > revision-id: 1a3974f534f (mariadb-11.3.1-12-g1a3974f534f) > parent(s): 83a79ba33b0 > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2024-01-23 18:04:31 +0400 > message: > > MDEV-33299 Assertion `(tm->tv_usec % (int) log_10_int[6 - dec]) == 0' failed in void my_timestamp_to_binary(const timeval*, uchar*, uint) > > diff --git a/sql/opt_rewrite_date_cmp.cc b/sql/opt_rewrite_date_cmp.cc > index 5a6410414da..8ff0ab3847e 100644 > --- a/sql/opt_rewrite_date_cmp.cc > +++ b/sql/opt_rewrite_date_cmp.cc > @@ -288,10 +288,13 @@ Item *Date_cmp_func_rewriter::create_end_bound() > const_arg_ts.hour= 23; > const_arg_ts.minute= TIME_MAX_MINUTE; > const_arg_ts.second= TIME_MAX_SECOND; > - const_arg_ts.second_part= TIME_MAX_SECOND_PART; > + const_arg_ts.second_part= TIME_MAX_SECOND_PART - > + my_time_fraction_remainder(TIME_MAX_SECOND_PART, > + field_ref->decimals); > if (check_datetime_range(&const_arg_ts)) > return nullptr; > res= new (thd->mem_root) Item_datetime(thd); > + res->decimals= field_ref->decimals; would it work you if always set res->decimals=6 here (and below)? looks like a simpler way to make the item consistent. > res->set(&const_arg_ts); > break; Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 2

Re: af51be1106b: MDEV-32837 long unique does not work like unique key when using replace
by Sergei Golubchik 24 Jan '24

24 Jan '24

Hi, Alexander, Generally, ok to push. One request for a change and one question below: On Jan 23, Alexander Barkov wrote: > revision-id: af51be1106b (mariadb-10.5.23-57-gaf51be1106b) > parent(s): c9902a20b3a > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2024-01-17 15:51:54 +0400 > message: > > MDEV-32837 long unique does not work like unique key when using replace > > write_record() when performing REPLACE has an optimization: > - if the unique violation happened in the last unique key, then do UPDATE > - otherwise, do DELETE+INSERT > > This patch skips this optimization in case if a non-versioned table > has hash unique (i.e. UNIQUE..USING HASH) constraints. > > Rationale: > > hash unique constraints are checked before regular unique indexes, > so even if a hash unique is the last unique constraint in the > table->key_info[] array - it guarantees nothing - there still > can be regular (non-hash) contraints that violate as well. > So just go the non-optmized path with DELETE followed > by INSERT on the next loop iteration. Don't forget to update the commit comment to match the final fix. > diff --git a/mysql-test/main/long_unique_bugs_no_protocol.test b/mysql-test/main/long_unique_bugs_no_protocol.test > new file mode 100644 > index 00000000000..a6f1f8e57e8 > --- /dev/null > +++ b/mysql-test/main/long_unique_bugs_no_protocol.test > @@ -0,0 +1,61 @@ > +--source include/no_protocol.inc why no protocol? Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 1

Re: 0c0721cb2f7: MDEV-31616 Problems with a stored function EMPTY() on upgrade to 10.6.
by Sergei Golubchik 22 Jan '24

22 Jan '24

Hi, Alexey, On Dec 27, Alexey Botchkov wrote: > revision-id: 0c0721cb2f7 (mariadb-10.6.16-45-g0c0721cb2f7) > parent(s): aff5ed3988a > author: Alexey Botchkov > committer: Alexey Botchkov > timestamp: 2023-12-19 00:00:48 +0400 > message: > > MDEV-31616 Problems with a stored function EMPTY() on upgrade to 10.6. > > The IDENT_sys doesn't include keywords, so the fucntion with the typo: fucntion > keyword name can be created, but cannot be called. > Moving some keywords to keyword_funcname set so the functions with these > names are allowed. The current logic in the code is: * built-in functions use IDENT_sys (no keywords allowed) in function_call_generic * functions with keyword names are explicitly listed in function_call_conflict. * stored function invocations use IDENT_sys via function_call_generic but * stored function declaration uses sp_name (that is keyword_ident) rule so, it'd make sense to introduce IDENT_funcname and use it instead of IDENT_sys in function_call_generic. That was a good idea. But also, I expect it to make function_call_conflict redundant, because all names from there should be in IDENT_funcname. And the patch doesn't solve the issue that many keywords from IDENT_sys are not in IDENT_funcname. So it's still possible to create functions that cannot be invoked. Note, that simply changing sp_name to use IDENT_funcname is not correct, because stored procedure names have different limitations. Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 2

Re: 02b85f1cf9b: MDEV-31340 Remove MY_COLLATION_HANDLER::strcasecmp()
by Sergei Golubchik 19 Jan '24

19 Jan '24

Hi, Alexander, Phew. It was a big patch. Mostly mechanical changes, admittedly, but still a lot to read. Generally I quite liked it, great work! Shorter way to write an identifier comparison, type safety that enforces namespaces and correct collations. Nice. But there are, of course, comments and questions, please, see below. On Dec 10, Alexander Barkov wrote: > commit 02b85f1cf9b > Author: Alexander Barkov <bar(a)mariadb.com> > Date: Wed Apr 26 15:27:01 2023 +0400 > > MDEV-31340 Remove MY_COLLATION_HANDLER::strcasecmp() > > - Removing the virtual function strnncoll() from MY_COLLATION_HANDLER > > - Adding a wrapper function CHARSET_INFO::streq(), to compare > two strings for equality. For now it calls strnncoll() internally. > In the future it will turn into a virtual function. why would you want it as a virtual function? What is collation-dependent there? > > - Adding new accent sensitive case insensitive collations: > - utf8mb4_general1400_as_ci > - utf8mb3_general1400_as_ci > They implements accent sensitive case insensitive comparison. > The weight of a character is equal to the code point of its > upper case variant. These collation use Unicode-14.0.0 casefolding data. > > The result of > my_charset_utf8mb3_general1400_as_ci.strcoll() > is very close to the former > my_charset_utf8mb3_general_ci.strcasecmp() > > There is only a difference in a couple dozen rare characters, because: > - the switch from "tolower" to "toupper" comparison, to make > utf8mb3_general1400_as_ci closer to utf8mb3_general_ci > - the switch from Unicode-3.0.0 to Unicode-14.0.0 > This difference should be tolarable. See the list of affected > characters in the MDEV description. > > Note, utf8mb4_general1400_as_ci correctly handles non-BMP characters! > Unlike utf8mb4_general_ci, it does not treat all BMP characters > as equal. > > - Adding classes representing names of the file based database objects: > > Lex_ident_db > Lex_ident_table > Lex_ident_trigger > > Their comparison collation depends on the underlying > file system case sensitivity and on --lower-case-table-names > and can be either my_charset_bin or my_charset_utf8mb3_general1400_as_ci. > > - Adding classes representing names of other database objects, > whose names have case insensitive comparison style, > using my_charset_utf8mb3_general1400_as_ci: > > Lex_ident_column > Lex_ident_sys_var > Lex_ident_user_var > Lex_ident_sp_var I don't think Lex_ident_sp_var is a separate domain, it's the same as Lex_ident_column. Both logically: select a from t1 where b > c (here you don't know what's a column and what's an sp var) and also in the code, in fill_spvar_definition() you yourself cast from Lex_ident_sp_var to Lex_ident_column, which kind of defeats the purpose of having two different types and emphasizes that it's actually just one and the same. > Lex_ident_ps > Lex_ident_i_s_table > Lex_ident_window > Lex_ident_func > Lex_ident_partition > Lex_ident_with_element > Lex_ident_rpl_filter > Lex_ident_master_info > Lex_ident_host > Lex_ident_locale > Lex_ident_plugin > Lex_ident_engine > Lex_ident_server > Lex_ident_savepoint > Lex_ident_table_option_name > Lex_ident_table_option_value option value is a string, not an ident > Lex_ident_charset > > - All the mentioned Lex_ident_xxx classes implement a method streq(): > > if (ident1.streq(ident2)) > do_equal(); > > This method works as a wrapper for CHARSET_INFO::streq(). > > - Changing a lot of "LEX_CSTRING name" to "Lex_ident_xxx name" > in class members and in function/method parameters. > > - Replacing all calls like > system_charset_info->coll->strcasecmp(ident1, ident2) > to > ident1.streq(ident2) > > - Taking advantage of the c++11 user defined literal operator > for Lex_ident_xxx data type. See lex_ident.h. > Use example: > > const Lex_ident_column primary_key_name= "PRIMARY"_Lex_ident_column; > > is now a shorter version of: > > const Lex_ident_column primary_key_name= > Lex_ident_column({STRING_WITH_LEN("PRIMARY")}); Of all the things in your patch this will be the first thing that Monty will remove when he'll stumble upon it. Could you instead implement a constructor that understands const Lex_ident_column primary_key_name= {STRING_WITH_LEN("PRIMARY")}; > diff --git a/include/m_ctype.h b/include/m_ctype.h > index 9d13989d1fe..e5c5145b256 100644 > --- a/include/m_ctype.h > +++ b/include/m_ctype.h > @@ -997,6 +1006,18 @@ struct charset_info_st > return state & MY_CS_COMPILED; > } > > + /* > + There will be a separate virtual function streq() in > + MY_COLLATION_HANDLER eventually. > + */ > + my_bool streq(const LEX_CSTRING a, const LEX_CSTRING b) const > + { > + DBUG_ASSERT(is_valid_comparison_operand(a)); > + DBUG_ASSERT(is_valid_comparison_operand(b)); > + return 0 == (coll->strnncoll)(this, (const uchar *) a.str, a.length, > + (const uchar *) b.str, b.length, FALSE); as you have also added strnncoll helper, you can write the above as return 0 == strnncoll(a, b); also, perhaps you should move these asserts into strnncoll helper? > + } > + > int strnncoll(const uchar *a, size_t alen, > const uchar *b, size_t blen, my_bool b_is_prefix= FALSE) const > { > diff --git a/mysql-test/main/ctype_utf8mb4_general1400_as_ci_casefold.result b/mysql-test/main/ctype_utf8mb4_general1400_as_ci_casefold.result > new file mode 100644 > index 00000000000..c6ccf662d16 > --- /dev/null > +++ b/mysql-test/main/ctype_utf8mb4_general1400_as_ci_casefold.result > @@ -0,0 +1,1446 @@ > +# > +# Start of 11.4 tests > +# > +# > +# MDEV-31340 Remove MY_COLLATION_HANDLER::strcasecmp() > +# > +SET NAMES utf8mb4 COLLATE utf8mb4_general1400_as_ci; > +EXECUTE IMMEDIATE SFORMAT(' > +CREATE VIEW v_bmp AS > +SELECT > + seq AS codepoint, > + LPAD(HEX(seq),4,''0'') AS codepoint_hex4, > + CONVERT(CHAR(seq USING utf32) USING {}) COLLATE {} AS c > +FROM > + seq_0_to_65535', @@character_set_connection, @@collation_connection); > +SELECT COLLATION(c) FROM v_bmp LIMIT 1; > +COLLATION(c) > +utf8mb4_general1400_as_ci > +SELECT > +codepoint_hex4, > +HEX(CAST(LOWER(c) AS CHAR CHARACTER SET ucs2)), > +HEX(CAST(UPPER(c) AS CHAR CHARACTER SET ucs2)) > +FROM v_bmp > +WHERE BINARY(c)<>BINARY(LOWER(c)) OR BINARY(c)<>BINARY(UPPER(c)); > +codepoint_hex4 HEX(CAST(LOWER(c) AS CHAR CHARACTER SET ucs2)) HEX(CAST(UPPER(c) AS CHAR CHARACTER SET ucs2)) what is the point of this table? detect changes when going to the next unicode version? > +0041 0061 0041 ... > +# > +# End of 11.4 tests > +# > diff --git a/mysql-test/main/ctype_utf8mb4_general1400_as_ci_ws.result b/mysql-test/main/ctype_utf8mb4_general1400_as_ci_ws.result > new file mode 100644 > index 00000000000..49f43777feb > --- /dev/null > +++ b/mysql-test/main/ctype_utf8mb4_general1400_as_ci_ws.result > diff --git a/mysql-test/main/ctype_utf8mb4_general1400_as_ci_ws.test b/mysql-test/main/ctype_utf8mb4_general1400_as_ci_ws.test > new file mode 100644 > index 00000000000..b5eb369ae9a > --- /dev/null > +++ b/mysql-test/main/ctype_utf8mb4_general1400_as_ci_ws.test > @@ -0,0 +1,39 @@ > +--source include/have_utf32.inc > +--source include/have_sequence.inc > + > +--echo # > +--echo # Start of 11.1 tests > +--echo # > + > +--echo # > +--echo # Collation utfmb4_general1400_as_ci > +--echo # > + > +SET NAMES utf8mb4 COLLATE utf8mb4_general1400_as_ci; > + > +EXECUTE IMMEDIATE SFORMAT(' > +CREATE VIEW v_bmp AS > +SELECT > + seq AS codepoint, > + LPAD(HEX(seq),6,''0'') AS codepoint_hex6, > + CONVERT(CHAR(seq USING utf32) USING {}) COLLATE {} AS c > +FROM > + seq_0_to_1114111', @@character_set_connection, @@collation_connection); > + > +SELECT COLLATION(c) FROM v_bmp LIMIT 1; > + > +SELECT > + SUM(codepoint_hex6=HEX(LOWER(c))) AS count_bmp_weight_is_lower, > + SUM(codepoint_hex6<>HEX(LOWER(c))) AS count_bmp_weight_is_not_lower > +FROM v_bmp; > + > +SELECT codepoint_hex6,HEX(WEIGHT_STRING(c)) > +FROM v_bmp > +WHERE codepoint_hex6<>HEX(WEIGHT_STRING(c)); > + > +DROP VIEW v_bmp; > + > + > +--echo # > +--echo # End of 11.1 tests 11.4 ? > +--echo # > diff --git a/mysql-test/suite/compat/oracle/r/lower_case_table_names.result b/mysql-test/suite/compat/oracle/r/lower_case_table_names.result > new file mode 100644 > index 00000000000..2518be31402 > --- /dev/null > +++ b/mysql-test/suite/compat/oracle/r/lower_case_table_names.result > @@ -0,0 +1,9 @@ > +SELECT @@lower_case_table_names; > +@@lower_case_table_names > +1 > +# > +# TODO: Report this problem as MDEV what is the problem? isn't the comparison accent-sensitive? > +# > +SET NAMES utf8; > +CREATE TABLE t1 (a öracle_schema.date); > +ERROR HY000: Unknown data type: 'öracle_schema.date' > diff --git a/mysql-test/suite/unit/disabled.def b/mysql-test/suite/unit/disabled.def > new file mode 100644 > index 00000000000..3a68bea32d0 > --- /dev/null > +++ b/mysql-test/suite/unit/disabled.def > @@ -0,0 +1,12 @@ > +############################################################################## > +# > +# List the test cases that are to be disabled temporarily. > +# > +# Separate the test case name and the comment with ':'. > +# > +# <testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> > +# > +# Do not use any TAB characters for whitespace. > +# > +############################################################################## > +conc_charset : TODO: client side ID for utf8mb4_tolower_ci 1. There is no utf8mb4_tolower_ci 2. what do you mean "client side ID"? > diff --git a/sql/ddl_log.cc b/sql/ddl_log.cc > index 61c8f50d1e3..fdfb605a57f 100644 > --- a/sql/ddl_log.cc > +++ b/sql/ddl_log.cc > @@ -1146,22 +1146,23 @@ static void execute_rename_table(DDL_LOG_ENTRY *ddl_log_entry, handler *file, > static void rename_triggers(THD *thd, DDL_LOG_ENTRY *ddl_log_entry, > bool swap_tables) > { > - LEX_CSTRING to_table, from_table, to_db, from_db, from_converted_name; > + Lex_ident_table to_table, from_table, from_converted_name; > + Lex_ident_db to_db, from_db; > char to_path[FN_REFLEN+1], from_path[FN_REFLEN+1], conv_path[FN_REFLEN+1]; > > if (!swap_tables) > { > - from_db= ddl_log_entry->db; > - from_table= ddl_log_entry->name; > - to_db= ddl_log_entry->from_db; > - to_table= ddl_log_entry->from_name; > + from_db= Lex_ident_db(ddl_log_entry->db); this would look better with implicit casting, that is, if you'd have an assignment operator from LEX_CSTRING > + from_table= Lex_ident_table(ddl_log_entry->name); > + to_db= Lex_ident_db(ddl_log_entry->from_db); > + to_table= Lex_ident_table(ddl_log_entry->from_name); > } > else > { > - from_db= ddl_log_entry->from_db; > - from_table= ddl_log_entry->from_name; > - to_db= ddl_log_entry->db; > - to_table= ddl_log_entry->extra_name; > + from_db= Lex_ident_db(ddl_log_entry->from_db); > + from_table= Lex_ident_table(ddl_log_entry->from_name); > + to_db= Lex_ident_db(ddl_log_entry->db); > + to_table= Lex_ident_table(ddl_log_entry->extra_name); > } > > build_filename_and_delete_tmp_file(from_path, sizeof(from_path), > @@ -1211,11 +1212,11 @@ static void rename_triggers(THD *thd, DDL_LOG_ENTRY *ddl_log_entry, > > (void) Table_triggers_list::prepare_for_rename(thd, > &trigger_param, > - &from_db, > - &from_table, > - &from_converted_name, > - &to_db, > - &to_table); > + from_db, > + from_table, > + from_converted_name, > + to_db, > + to_table); You rather inconsistently replace pointers with const references in some methods, like above, but not in others, like below or in the next hunk. > (void) Table_triggers_list::change_table_name(thd, > &trigger_param, > &from_db, > @@ -1707,9 +1708,8 @@ static int ddl_log_execute_action(THD *thd, MEM_ROOT *mem_root, > > switch (ddl_log_entry->phase) { > case DDL_DROP_DB_PHASE_INIT: > - drop_database_objects(thd, &path, &db, > - !my_strcasecmp(system_charset_info, > - MYSQL_SCHEMA_NAME.str, db.str)); > + // QQ: was a bug here? This comment makes little sense in the code, as one cannot see what *was* here before. Just drop it. > + drop_database_objects(thd, &path, &db, MYSQL_SCHEMA_NAME.streq(db)); > > strxnmov(to_path, sizeof(to_path)-1, path.str, MY_DB_OPT_FILE, NullS); > mysql_file_delete_with_symlink(key_file_misc, to_path, "", MYF(0)); > diff --git a/sql/debug_sync.cc b/sql/debug_sync.cc > index 362463a7f22..4704691615a 100644 > --- a/sql/debug_sync.cc > +++ b/sql/debug_sync.cc > @@ -858,6 +858,13 @@ static st_debug_sync_action *debug_sync_get_action(THD *thd, > } > > > +class Debug_token: public Lex_ident_ci > +{ > +public: > + using Lex_ident_ci::Lex_ident_ci; > +}; I wonder if you can create those types with less boilerplate will typedef work? will using work? if not, may be, like template<> Lex_ident_templ: Lex_ident_ci { ... } Lex_ident_templ<> Debug_token; > + > + > /** > Set a debug sync action. > > @@ -909,7 +916,8 @@ static bool debug_sync_set_action(THD *thd, st_debug_sync_action *action) > } > else > { > - const char *dsp_name= action->sync_point.c_ptr(); > + const Debug_token dsp_name(action->sync_point.c_ptr(), > + action->sync_point.length()); or dsp_name(action->sync_point->to_lex_cstring()) ? > #ifdef DBUG_TRACE > DBUG_EXECUTE("debug_sync", { > /* Functions as DBUG_PRINT args can change keyword and line nr. */ > diff --git a/sql/grant.cc b/sql/grant.cc > index 1ba197bc1d9..0ebf47431bc 100644 > --- a/sql/grant.cc > +++ b/sql/grant.cc > @@ -31,8 +31,8 @@ bool Grant_privilege::add_column_privilege(THD *thd, > class LEX_COLUMN *point; > while ((point=iter++)) > { > - if (!my_strcasecmp(system_charset_info, > - point->column.c_ptr(), new_str->c_ptr())) > + const Lex_ident_column tmp(point->column.c_ptr(), point->column.length()); That feels like an overkill, creating Lex_ident_column just to compare two strings. What is it compiled to at -O3? > + if (tmp.streq(new_str->to_lex_cstring())) > break; > } > m_column_privilege_total|= which_grant; > diff --git a/sql/ha_partition.cc b/sql/ha_partition.cc > index a615d26b9a9..fc840c21191 100644 > --- a/sql/ha_partition.cc > +++ b/sql/ha_partition.cc > @@ -118,7 +118,7 @@ int ha_partition::notify_tabledef_changed(LEX_CSTRING *db, > LEX_CSTRING table_name; > const char *table_name_ptr; > if (create_partition_name(from_buff, sizeof(from_buff), > - from_path, name_buffer_ptr, > + from_path, Lex_cstring_strlen(name_buffer_ptr), Don't, please. You _always_ use Lex_cstring_strlen, in _all_ invocations of create_partition_name(). Just pass a char* down as before and do strlen inside, if you need it. > NORMAL_PART_NAME, FALSE)) > res=1; > table_name_ptr= from_buff + dirname_length(from_buff); > diff --git a/sql/handler.cc b/sql/handler.cc > index a977054d0da..7b9d11d00a2 100644 > --- a/sql/handler.cc > +++ b/sql/handler.cc > @@ -7531,7 +7528,8 @@ int handler::check_duplicate_long_entry_key(const uchar *new_rec, uint key_no) > else > { > Item_func_left *fnc= static_cast<Item_func_left *>(arguments[j]); > - DBUG_ASSERT(!my_strcasecmp(system_charset_info, "left", fnc->func_name())); > + DBUG_ASSERT(Lex_ident_func(fnc->func_name_cstring()). eh, why func_name_cstring() doesn't return Lex_ident_func? > + streq("left"_Lex_cstring)); > DBUG_ASSERT(fnc->arguments()[0]->type() == Item::FIELD_ITEM); > t_field= static_cast<Item_field *>(fnc->arguments()[0])->field; > uint length= (uint)fnc->arguments()[1]->val_int(); > diff --git a/sql/item.cc b/sql/item.cc > index 5c4f1704ebe..5689756eca9 100644 > --- a/sql/item.cc > +++ b/sql/item.cc > @@ -3463,13 +3463,11 @@ bool Item_field::eq(const Item *item, bool binary_cmp) const > (In cases where we would choose wrong we would have to generate a > ER_NON_UNIQ_ERROR). > */ > - return (!lex_string_cmp(system_charset_info, &item_field->name, > - &field_name) && > - (!item_field->table_name.str || !table_name.str || > - (!my_strcasecmp(table_alias_charset, item_field->table_name.str, > - table_name.str) && > - (!item_field->db_name.str || !db_name.str || > - (item_field->db_name.str && !strcmp(item_field->db_name.str, > + return (item_field->name.streq(field_name) && > + (!item_field->table_name.str || !table_name.str || > + (item_field->table_name.streq(table_name) && > + (!item_field->db_name.str || !db_name.str || > + (item_field->db_name.str && !strcmp(item_field->db_name.str, > db_name.str)))))); I presume, # db_name should also be compared with streq > } > > diff --git a/sql/item_create.cc b/sql/item_create.cc > index 4f0df8f732d..501db19c1b8 100644 > --- a/sql/item_create.cc > +++ b/sql/item_create.cc > @@ -2638,7 +2639,8 @@ Create_qfunc::create_func(THD *thd, const LEX_CSTRING *name, > if (thd->lex->copy_db_to(&db)) > return NULL; > > - return create_with_db(thd, &db, name, false, item_list); Again, can LEX_CSTRING be implicitly cast to the correct Lex_ident variant? > + return create_with_db(thd, Lex_ident_db(db), Lex_ident_func(*name), > + false, item_list); > } > > > diff --git a/sql/item_windowfunc.cc b/sql/item_windowfunc.cc > index 5438cade27a..f0a5737bdf7 100644 > --- a/sql/item_windowfunc.cc > +++ b/sql/item_windowfunc.cc > @@ -28,7 +28,7 @@ Item_window_func::resolve_window_name(THD *thd) > return false; > } > DBUG_ASSERT(window_name != NULL && window_spec == NULL); > - const char *ref_name= window_name->str; > + const LEX_CSTRING &ref_name= *window_name; Why LEX_CSTRING ? > > /* !TODO: Add the code to resolve ref_name in outer queries */ > /* > diff --git a/sql/opt_trace.cc b/sql/opt_trace.cc > index 1008690ccaa..dbb2708f487 100644 > --- a/sql/opt_trace.cc > +++ b/sql/opt_trace.cc > @@ -43,7 +43,7 @@ bool list_has_optimizer_trace_table(const TABLE_LIST *tbl) > for (; tbl; tbl= tbl->next_global) > { > if (tbl->schema_table && > - 0 == strcmp(tbl->schema_table->table_name, I_S_table_name)) > + 0 == strcmp(tbl->schema_table->table_name.str, I_S_table_name)) Why not streq? > return true; > } > return false; > @@ -190,9 +190,8 @@ void opt_trace_disable_if_no_security_context_access(THD *thd) > if (!(thd->main_security_ctx.check_access(GLOBAL_ACLS & ~GRANT_ACL)) && > (0 != strcmp(thd->main_security_ctx.priv_user, > thd->security_context()->priv_user) || > - 0 != my_strcasecmp(system_charset_info, > - thd->main_security_ctx.priv_host, > - thd->security_context()->priv_host))) > + !Lex_ident_host(Lex_cstring_strlen(thd->main_security_ctx.priv_host)). > + streq(Lex_cstring_strlen(thd->security_context()->priv_host)))) Doesn't really look like an improvement :) May be you can get rid of intermediate Lex_cstring_strlen()'s ? > trace->missing_privilege(); > } > > diff --git a/sql/partition_info.cc b/sql/partition_info.cc > index 1b65de6e3c0..7b76c6d65b1 100644 > --- a/sql/partition_info.cc > +++ b/sql/partition_info.cc > @@ -1403,8 +1397,8 @@ bool partition_info::check_partition_info(THD *thd, handlerton **eng_type, > num_parts_not_set++; > part_elem->engine_type= default_engine_type; > } > - if (check_table_name(part_elem->partition_name, > - strlen(part_elem->partition_name), FALSE)) > + if (check_table_name(part_elem->partition_name.str, > + part_elem->partition_name.length, FALSE)) better change check_table_name to accept a LEX_CSTRING or Lex_ident, it'll simplify almost all (except one) invocations of it > { > my_error(ER_WRONG_PARTITION_NAME, MYF(0)); > goto end; > @@ -2604,8 +2596,8 @@ bool partition_info::has_same_partitioning(partition_info *new_part_info) > part_state must be PART_NORMAL! > */ > if (!part_elem || !new_part_elem || > - strcmp(part_elem->partition_name, > - new_part_elem->partition_name) || > + strcmp(part_elem->partition_name.str, > + new_part_elem->partition_name.str) || why strcmp? > part_elem->part_state != PART_NORMAL || > new_part_elem->part_state != PART_NORMAL || > part_elem->max_value != new_part_elem->max_value || > @@ -2666,8 +2658,8 @@ bool partition_info::has_same_partitioning(partition_info *new_part_info) > sub_part_elem->engine_type != new_sub_part_elem->engine_type) > DBUG_RETURN(false); > > - if (strcmp(sub_part_elem->partition_name, > - new_sub_part_elem->partition_name) || > + if (strcmp(sub_part_elem->partition_name.str, > + new_sub_part_elem->partition_name.str) || same here > sub_part_elem->part_state != PART_NORMAL || > new_sub_part_elem->part_state != PART_NORMAL || > sub_part_elem->part_min_rows != > diff --git a/sql/partition_info.h b/sql/partition_info.h > index 3a8c3a37524..92e8906274f 100644 > --- a/sql/partition_info.h > +++ b/sql/partition_info.h > @@ -526,11 +527,13 @@ void partition_info::vers_update_el_ids() > } > > > -inline > -bool make_partition_name(char *move_ptr, uint i) > +static inline > +Lex_ident_partition make_partition_name_ls(char *move_ptr, uint i) it's not _ls really, is it? why not to keep the old name? > { > int res= snprintf(move_ptr, MAX_PART_NAME_SIZE + 1, "p%u", i); > - return res < 0 || res > MAX_PART_NAME_SIZE; > + return res < 0 || res > MAX_PART_NAME_SIZE ? > + Lex_ident_partition() : > + Lex_ident_partition(move_ptr, (size_t) res); > } > > > @@ -549,15 +552,16 @@ uint partition_info::next_part_no(uint new_parts) const > for (uint cur_part= 0; cur_part < new_parts; ++cur_part, ++suffix) > { > uint32 cur_suffix= suffix; > - if (make_partition_name(part_name, suffix)) > + Lex_ident_partition part_name_ls(make_partition_name_ls(part_name, suffix)); > + if (!part_name_ls.str) > return 0; > partition_element *el; > it.rewind(); > while ((el= it++)) > { > - if (0 == my_strcasecmp(&my_charset_latin1, el->partition_name, part_name)) > + if (el->partition_name.streq(part_name_ls)) > { > - if (make_partition_name(part_name, ++suffix)) > + if (!(part_name_ls= make_partition_name_ls(part_name, ++suffix)).str) you don't seem to use part_name_ls below > return 0; > it.rewind(); > } > diff --git a/sql/set_var.h b/sql/set_var.h > index aed4955ef62..8f7e3609f5f 100644 > --- a/sql/set_var.h > +++ b/sql/set_var.h > @@ -383,7 +383,7 @@ class set_var_default_role: public set_var_base > { > LEX_USER *user, *real_user; > LEX_CSTRING role; > - const char *real_role; > + LEX_CSTRING real_role; Not Lex_ident_user ? > public: > set_var_default_role(LEX_USER *user_arg, LEX_CSTRING role_arg) : > user(user_arg), role(role_arg) {} > diff --git a/sql/slave.cc b/sql/slave.cc > index 0a15cc55146..daeb05f423e 100644 > --- a/sql/slave.cc > +++ b/sql/slave.cc > @@ -3403,8 +3403,9 @@ static bool send_show_master_info_data(THD *thd, Master_info *mi, bool full, > static int cmp_mi_by_name(const Master_info **arg1, > const Master_info **arg2) > { > - return my_strcasecmp(system_charset_info, (*arg1)->connection_name.str, > - (*arg2)->connection_name.str); > + return Lex_ident_master_info::charset_info()->strnncoll( > + (*arg1)->connection_name, > + (*arg2)->connection_name); Why not (*arg1)->connection_name.streq() ? > } > > > diff --git a/sql/sp_pcontext.cc b/sql/sp_pcontext.cc > index 848d1f0c655..761f535832d 100644 > --- a/sql/sp_pcontext.cc > +++ b/sql/sp_pcontext.cc > @@ -139,8 +139,7 @@ sp_pcontext *sp_pcontext::push_context(THD *thd, sp_pcontext::enum_scope scope) > > bool cmp_labels(sp_label *a, sp_label *b) > { > - return (lex_string_cmp(system_charset_info, &a->name, &b->name) == 0 && > - a->type == b->type); > + return a->name.streq(b->name) && a->type == b->type; put type comparison first, please > } > > sp_pcontext *sp_pcontext::pop_context() > diff --git a/sql/sql_acl.cc b/sql/sql_acl.cc > index 09af5523c76..dd36504ed56 100644 > --- a/sql/sql_acl.cc > +++ b/sql/sql_acl.cc > @@ -567,14 +560,17 @@ static uchar* acl_role_get_key(ACL_ROLE *entry, size_t *length, > > struct ROLE_GRANT_PAIR : public Sql_alloc > { > - char *u_uname; > - char *u_hname; > - char *r_uname; > + LEX_CSTRING u_uname; > + LEX_CSTRING u_hname; > + LEX_CSTRING r_uname; Lex_ident_user ? (and in many places below) > LEX_STRING hashkey; > bool with_admin; > > - bool init(MEM_ROOT *mem, const char *username, const char *hostname, > - const char *rolename, bool with_admin_option); > + bool init(MEM_ROOT *mem, > + const LEX_CSTRING &username, > + const LEX_CSTRING &hostname, > + const LEX_CSTRING &rolename, > + bool with_admin_option); > }; > > static uchar* acl_role_map_get_key(ROLE_GRANT_PAIR *entry, size_t *length, > @@ -2731,7 +2733,7 @@ static bool acl_load(THD *thd, const Grant_tables& tables) > char *db_name; > db.user=safe_str(get_field(&acl_memroot, db_table.user())); > const char *hostname= get_field(&acl_memroot, db_table.host()); > - if (!hostname && find_acl_role(db.user, true)) > + if (!hostname && find_acl_role(Lex_cstring_strlen(db.user), true)) Perhaps it'd be better to change user and db to be Lex_ident_* in ACL_DB? > hostname= ""; > update_hostname(&db.host, hostname); > db.db= db_name= get_field(&acl_memroot, db_table.db()); > @@ -2822,14 +2824,29 @@ static bool acl_load(THD *thd, const Grant_tables& tables) > init_alloc_root(key_memory_acl_mem, &temp_root, ACL_ALLOC_BLOCK_SIZE, 0, MYF(0)); > while (!(read_record_info.read_record())) > { > - char *hostname= safe_str(get_field(&temp_root, roles_mapping_table.host())); > - char *username= safe_str(get_field(&temp_root, roles_mapping_table.user())); > - char *rolename= safe_str(get_field(&temp_root, roles_mapping_table.role())); > + StringBuffer<MAX_FIELD_WIDTH> host_buff, user_buff, role_buff; > + /* > + find_user_by_name() called recursively later in this code block There's no find_user_by_name() and acl_find_user_by_name does not seem to be called recursively. What did you mean here? also, why you didn't change acl_find_user_by_name to take a Lex_ident? > + needs a 0-terminated string. So does sql_print_error(). > + As val_lex_cstring() does not guarantee a 0-terminated string, > + let's alloc 0-terminated copies of val_lex_cstring() results on > + the memory root. you've already added Field::val_lex_string_strmake() in August, why not to use it here? > + */ > + const Lex_cstring hostname= safe_lexcstrdup_root(&temp_root, > + roles_mapping_table.host()-> > + val_lex_cstring(&host_buff)); > + const Lex_cstring username= safe_lexcstrdup_root(&temp_root, > + roles_mapping_table.user()-> > + val_lex_cstring(&user_buff)); > + const Lex_cstring rolename= safe_lexcstrdup_root(&temp_root, > + roles_mapping_table.role()-> > + val_lex_cstring(&role_buff)); > + > bool with_grant_option= get_YN_as_bool(roles_mapping_table.admin_option()); > > if (add_role_user_mapping(username, hostname, rolename)) { > sql_print_error("Invalid roles_mapping table entry user:'%s@%s', rolename:'%s'", > - username, hostname, rolename); > + username.str, hostname.str, rolename.str); > continue; > } > > @@ -4443,28 +4484,31 @@ static ACL_USER * find_user_wild(const char *host, const char *user, const char > /* > Find a role with the specified name > */ > -static ACL_ROLE *find_acl_role(const char *role, bool allow_public) > +static ACL_ROLE *find_acl_role(const LEX_CSTRING &role, bool allow_public) > { > - size_t length= strlen(role); > DBUG_ENTER("find_acl_role"); > - DBUG_PRINT("enter",("role: '%s'", role)); > + DBUG_PRINT("enter",("role: '%s'", role.str)); > DBUG_PRINT("info", ("Hash elements: %ld", acl_roles.records)); > > mysql_mutex_assert_owner(&acl_cache->lock); > > - if (!length || (!allow_public && strcasecmp(role, public_name.str) == 0)) > + if (!role.length || > + (!allow_public && > + my_charset_utf8mb3_general1400_as_ci.streq(role, public_name))) Lex_ident_user would've been nice here too > DBUG_RETURN(NULL); > > - ACL_ROLE *r= (ACL_ROLE *)my_hash_search(&acl_roles, (uchar *)role, length); > + ACL_ROLE *r= (ACL_ROLE *)my_hash_search(&acl_roles, (uchar *)role.str, > + role.length); > DBUG_RETURN(r); > } > > /* > Finds a grantee - something that privileges or roles can be granted to. > */ > -static ACL_USER_BASE *find_acl_user_base(const char *user, const char *host) > +static ACL_USER_BASE *find_acl_user_base(const LEX_CSTRING &user, > + const LEX_CSTRING &host) > { > - if (*host) > + if (*host.str) > return find_user_exact(host, user); > > return find_acl_role(user, true); > @@ -5657,8 +5696,8 @@ static GRANT_NAME *name_hash_search(HASH *name_hash, > { > if (!grant_name->host.hostname || > (host && > - !my_strcasecmp(system_charset_info, host, > - grant_name->host.hostname)) || > + Lex_ident_host(Lex_cstring_strlen(host)). Still don't like these Lex_ident_something(Lex_cstring_strlen(xxx)) Let's get rid of them > + streq(Lex_cstring_strlen(grant_name->host.hostname))) || > (ip && !strcmp(ip, grant_name->host.hostname))) > return grant_name; > } > @@ -7373,8 +7414,9 @@ int mysql_table_grant(THD *thd, TABLE_LIST *table_list, > } > } > if (Str->is_role()) > - propagate_role_grants(find_acl_role(Str->user.str, true), > - PRIVS_TO_MERGE::TABLE_COLUMN, db_name, table_name); > + propagate_role_grants(find_acl_role(Str->user, true), > + PRIVS_TO_MERGE::TABLE_COLUMN, > + db_name.str, table_name.str); may be pass Lex_ident_* down into propagate_role_grants too? > } > > thd->mem_root= old_root; > diff --git a/sql/sql_base.cc b/sql/sql_base.cc > index 9d2415d3a4d..d7d7ebee024 100644 > --- a/sql/sql_base.cc > +++ b/sql/sql_base.cc > @@ -233,33 +233,41 @@ uint get_table_def_key(const TABLE_LIST *table_list, const char **key) > # Pointer to list of names of open tables. > */ > > -struct list_open_tables_arg > +class list_open_tables_arg > { > +public: > THD *thd; > - const char *db; > + const Lex_ident_db db; > const char *wild; > TABLE_LIST table_list; > OPEN_TABLE_LIST **start_list, *open_list; > + > + list_open_tables_arg(THD *thd_arg, const LEX_CSTRING &db_arg, > + const char *wild_arg) > + :thd(thd_arg), db(db_arg), wild(wild_arg), > + start_list(&open_list), open_list(0) > + { > + bzero((char*) &table_list, sizeof(table_list)); > + } > }; > > > static my_bool list_open_tables_callback(TDC_element *element, > list_open_tables_arg *arg) > { > - const char *db= (char*) element->m_key; > - size_t db_length= strlen(db); > - const char *table_name= db + db_length + 1; > + const Lex_ident_db > + db= Lex_ident_db(Lex_cstring_strlen((const char*) element->m_key)); > + const char *table_name= db.str + db.length + 1; > > - if (arg->db && my_strcasecmp(system_charset_info, arg->db, db)) > + // QQ: a bug was here? same as above, "was" > + if (arg->db.str && !arg->db.streq(db)) > return FALSE; > if (arg->wild && wild_compare(table_name, arg->wild, 0)) > return FALSE; > > /* Check if user has SELECT privilege for any column in the table */ > - arg->table_list.db.str= db; > - arg->table_list.db.length= db_length; > - arg->table_list.table_name.str= table_name; > - arg->table_list.table_name.length= strlen(table_name); > + arg->table_list.db= db; > + arg->table_list.table_name= Lex_cstring_strlen(table_name); > arg->table_list.grant.privilege= NO_ACL; > > if (check_table_access(arg->thd, SELECT_ACL, &arg->table_list, TRUE, 1, TRUE)) > @@ -1906,7 +1904,7 @@ bool open_table(THD *thd, TABLE_LIST *table_list, Open_table_context *ot_ctx) > DBUG_ASSERT(!table_list->table); > > /* an open table operation needs a lot of the stack space */ > - if (check_stack_overrun(thd, STACK_MIN_SIZE_FOR_OPEN, (uchar *)&alias)) > + if (check_stack_overrun(thd, STACK_MIN_SIZE_FOR_OPEN, (uchar *)&alias.str)) better pass `key` above. doesn't matter much but looks a bit less weird > DBUG_RETURN(TRUE); > > if (!(flags & MYSQL_OPEN_IGNORE_KILLED) && thd->killed) > @@ -1954,7 +1952,10 @@ bool open_table(THD *thd, TABLE_LIST *table_list, Open_table_context *ot_ctx) > if (table->s->table_cache_key.length == key_length && > !memcmp(table->s->table_cache_key.str, key, key_length)) > { > - if (!my_strcasecmp(system_charset_info, table->alias.c_ptr(), alias) && > + // QQ: Why table_alias_charset is not used for comparison? Did you try to correct this? > + if (my_charset_utf8mb3_general1400_as_ci.streq( > + table->alias.to_lex_cstring(), > + alias) && > table->query_id != thd->query_id && /* skip tables already used */ > (thd->locked_tables_mode == LTM_LOCK_TABLES || > table->query_id == 0)) > @@ -6353,26 +6355,25 @@ find_field_in_natural_join(THD *thd, TABLE_LIST *table_ref, const char *name, si > */ > > Field * > -find_field_in_table(THD *thd, TABLE *table, const char *name, size_t length, > +find_field_in_table(THD *thd, TABLE *table, const Lex_ident_column &name, > bool allow_rowid, field_index_t *cached_field_index_ptr) > { > Field *field; > field_index_t cached_field_index= *cached_field_index_ptr; > DBUG_ENTER("find_field_in_table"); > DBUG_PRINT("enter", ("table: '%s', field name: '%s'", table->alias.c_ptr(), > - name)); > + name.str)); > > /* We assume here that table->field < NO_CACHED_FIELD_INDEX = UINT_MAX */ > if (cached_field_index < table->s->fields && > - !my_strcasecmp(system_charset_info, > - table->field[cached_field_index]->field_name.str, name)) > + table->field[cached_field_index]->field_name.streq(name)) > { > field= table->field[cached_field_index]; > DEBUG_SYNC(thd, "table_field_cached"); > } > else > { > - LEX_CSTRING fname= {name, length}; > + LEX_CSTRING fname= name; Do you need that? You can use `name` directly below > field= table->find_field_by_name(&fname); > } > > @@ -6999,9 +7000,9 @@ find_item_in_list(Item *find, List<Item> &items, uint *counter, > List_iterator<Item> li(items); > uint n_items= limit == 0 ? items.elements : limit; > Item **found=0, **found_unaliased= 0, *item; > - const char *db_name=0; > + const LEX_CSTRING *db_name=0; > const LEX_CSTRING *field_name= 0; > - const char *table_name=0; > + const LEX_CSTRING *table_name=0; Why not Lex_ident's ? > bool found_unaliased_non_uniq= 0; > /* > true if the item that we search for is a valid name reference > @@ -7059,12 +7060,11 @@ find_item_in_list(Item *find, List<Item> &items, uint *counter, > item is not fix_field()'ed yet. > */ > if (item_field->field_name.str && item_field->table_name.str && > - !lex_string_cmp(system_charset_info, &item_field->field_name, > - field_name) && > - !my_strcasecmp(table_alias_charset, item_field->table_name.str, > - table_name) && > - (!db_name || (item_field->db_name.str && > - !strcmp(item_field->db_name.str, db_name)))) > + item_field->field_name.streq(*field_name) && > + item_field->table_name.streq(*table_name) && > + (!db_name || !db_name->str || > + (item_field->db_name.str && > + !strcmp(item_field->db_name.str, db_name->str)))) why strcmp? > { > if (found_unaliased) > { > @@ -7135,11 +7132,10 @@ find_item_in_list(Item *find, List<Item> &items, uint *counter, > } > } > } > - else if (!table_name) > + else if (!table_name || !table_name->str) > { > if (is_ref_by_name && find->name.str && item->name.str && > - find->name.length == item->name.length && > - !lex_string_cmp(system_charset_info, &item->name, &find->name)) > + item->name.streq(find->name)) Do you have tests that will fail if streq will compare lengths? > { > found= li.ref(); > *counter= i; > diff --git a/sql/sql_class.cc b/sql/sql_class.cc > index a2414de645b..1b36cd9416f 100644 > --- a/sql/sql_class.cc > +++ b/sql/sql_class.cc > @@ -855,10 +853,13 @@ THD::THD(my_thread_id id, bool is_wsrep_applier) > profiling.set_thd(this); > #endif > user_connect=(USER_CONN *)0; > - my_hash_init(key_memory_user_var_entry, &user_vars, system_charset_info, > + my_hash_init(key_memory_user_var_entry, &user_vars, > + Lex_ident_user_var::charset_info(), > USER_VARS_HASH_SIZE, 0, 0, (my_hash_get_key) get_var_key, > (my_hash_free_key) free_user_var, HASH_THREAD_SPECIFIC); > - my_hash_init(PSI_INSTRUMENT_ME, &sequences, system_charset_info, > + my_hash_init(PSI_INSTRUMENT_ME, &sequences, > + // QQ: why case insensitive? looks like a bug, can you do a test case for it, please? > + &my_charset_utf8mb3_general1400_as_ci, > SEQUENCES_HASH_SIZE, 0, 0, (my_hash_get_key) > get_sequence_last_key, (my_hash_free_key) free_sequence_last, > HASH_THREAD_SPECIFIC); > @@ -4011,7 +4015,7 @@ Table_ident::to_ident_db_internal_with_error(Query_arena *arena) const > if (is_derived_table()) > { > DBUG_ASSERT(db.str == empty_c_string && db.length == 0); > - return Lex_ident_db(empty_c_string, 0); > + return Lex_ident_db(empty_c_string, (size_t) 0); why? > } > // Normal table or JSON table > return arena->to_ident_db_internal_with_error(db); > diff --git a/sql/sql_class.h b/sql/sql_class.h > index 214118d8431..5685240ceaf 100644 > --- a/sql/sql_class.h > +++ b/sql/sql_class.h > @@ -433,13 +435,14 @@ class Alter_rename_key : public Sql_alloc > class Alter_index_ignorability: public Sql_alloc > { > public: > - Alter_index_ignorability(const char *name, bool is_ignored, bool if_exists) : > + Alter_index_ignorability(const LEX_CSTRING &name, > + bool is_ignored, bool if_exists) : > m_name(name), m_is_ignored(is_ignored), m_if_exists(if_exists) > { > - assert(name != NULL); > + assert(name.str != NULL); DBUG_ASSERT > } > > - const char *name() const { return m_name; } > + const Lex_ident_column &name() const { return m_name; } > bool if_exists() const { return m_if_exists; } > > /* The ignorability after the operation is performed. */ > @@ -7989,42 +7995,24 @@ class Identifier_chain2 > class Database_qualified_name > { > public: > - LEX_CSTRING m_db; > - LEX_CSTRING m_name; > - Database_qualified_name(const LEX_CSTRING *db, const LEX_CSTRING *name) > - :m_db(*db), m_name(*name) > + Lex_ident_db m_db; > + Lex_cstring m_name; // no comparison semantics > + Database_qualified_name() > { } > - Database_qualified_name(const LEX_CSTRING &db, const LEX_CSTRING &name) > + Database_qualified_name(const Lex_ident_db &db, const LEX_CSTRING &name) > :m_db(db), m_name(name) > { } > - Database_qualified_name(const char *db, size_t db_length, > - const char *name, size_t name_length) > - { > - m_db.str= db; > - m_db.length= db_length; > - m_name.str= name; > - m_name.length= name_length; > - } > - > - bool eq(const Database_qualified_name *other) const > + bool eq_routine_name(const Database_qualified_name *other) const eq_func_name, because the comparison semantics comes from Lex_ident_func > { > - CHARSET_INFO *cs= lower_case_table_names ? > - &my_charset_utf8mb3_general_ci : > - &my_charset_utf8mb3_bin; > - return > - m_db.length == other->m_db.length && > - m_name.length == other->m_name.length && > - !cs->strnncoll(m_db.str, m_db.length, > - other->m_db.str, other->m_db.length) && > - !cs->strnncoll(m_name.str, m_name.length, > - other->m_name.str, other->m_name.length); > + return m_db.streq(other->m_db) && no need to take lower_case_table_names into account when comparing db names here? > + Lex_ident_func(m_name).streq(other->m_name); > } > /* > Make copies of "db" and "name" on the memory root in internal format: > - Lower-case "db" if lower-case-table-names==1. > - Preserve "name" as is. > */ > - bool copy_sp_name_internal(MEM_ROOT *mem_root, const LEX_CSTRING &db, > + bool copy_sp_name_internal(MEM_ROOT *mem_root, const Lex_ident_db &db, > const LEX_CSTRING &name); > > // Export db and name as a qualified name string: 'db.name' > diff --git a/sql/sql_connect.cc b/sql/sql_connect.cc > index c5619b28f69..4e10f498e3c 100644 > --- a/sql/sql_connect.cc > +++ b/sql/sql_connect.cc > @@ -509,7 +514,9 @@ extern "C" void free_table_stats(TABLE_STATS* table_stats) > > void init_global_table_stats(void) > { > - my_hash_init(PSI_INSTRUMENT_ME, &global_table_stats, system_charset_info, > + my_hash_init(PSI_INSTRUMENT_ME, &global_table_stats, > + // QQ: why not Lex_ident_table::charset_info() ? just change it > + &my_charset_utf8mb3_general1400_as_ci, > max_connections, 0, 0, (my_hash_get_key) get_key_table_stats, > (my_hash_free_key) free_table_stats, 0); > } > diff --git a/sql/sql_db.cc b/sql/sql_db.cc > index 08441d9f067..51635bdf0ce 100644 > --- a/sql/sql_db.cc > +++ b/sql/sql_db.cc > @@ -98,12 +98,10 @@ typedef struct my_dbopt_st > */ > > static inline bool > -cmp_db_names(LEX_CSTRING *db1_name, const LEX_CSTRING *db2_name) > +cmp_db_names(const Lex_ident_db &db1_name, const Lex_ident_db &db2_name) > { > - return (db1_name->length == db2_name->length && > - (db1_name->length == 0 || can it be used with empty names, really? > - my_strcasecmp(table_alias_charset, > - db1_name->str, db2_name->str) == 0)); > + return (db1_name.length == 0 && db2_name.length == 0) || > + db1_name.streq(db2_name); > } > > #ifdef HAVE_PSI_INTERFACE > @@ -1087,8 +1085,7 @@ mysql_rm_db_internal(THD *thd, const Lex_ident_db &db, bool if_exists, > Disable drop of enabled log tables, must be done before name locking. > This check is only needed if we are dropping the "mysql" database. > */ > - if ((rm_mysql_schema= > - (my_strcasecmp(system_charset_info, MYSQL_SCHEMA_NAME.str, db.str) == 0))) > + if ((rm_mysql_schema= MYSQL_SCHEMA_NAME.streq(db))) // QQ: was a bug here? again, "was" > { > for (table= tables; table; table= table->next_local) > if (check_if_log_table(table, TRUE, "DROP")) > diff --git a/sql/sql_handler.cc b/sql/sql_handler.cc > index 406f67ff955..def1fc44392 100644 > --- a/sql/sql_handler.cc > +++ b/sql/sql_handler.cc > @@ -1080,11 +1079,14 @@ static SQL_HANDLER *mysql_ha_find_match(THD *thd, TABLE_LIST *tables) > { > if (tables->is_anonymous_derived_table()) > continue; > + /* > + TODO: for some reasons the code user case-insensitive comparison. > + Shouldn't it use Lex_ident_db/Lex_ident_table instead, > + to take into account the filesystem case sensitivity? > + */ must be a bug. try to create a test case for it. I think it could be something like (untested) create table t1 ...; create table T1; handler t1 open; handler t1 read first; handler t1 read next; drop table T1; handler t1 read next; > if ((! tables->db.str[0] || > - ! my_strcasecmp(&my_charset_latin1, hash_tables->db.str, > - tables->get_db_name())) && > - ! my_strcasecmp(&my_charset_latin1, hash_tables->table_name.str, > - tables->get_table_name())) > + Lex_ident_ci(hash_tables->db).streq(tables->get_db_name())) && > + Lex_ident_ci(hash_tables->table_name).streq(tables->get_table_name())) > { > /* Link into hash_tables list */ > hash_tables->next= head; > diff --git a/sql/sql_i_s.h b/sql/sql_i_s.h > index 263031ae2c9..127b35f6ab2 100644 > --- a/sql/sql_i_s.h > +++ b/sql/sql_i_s.h > @@ -329,7 +325,7 @@ typedef class Item COND; > > typedef struct st_schema_table > { > - const char *table_name; > + LEX_CSTRING table_name; why LEX_CSTRING? > ST_FIELD_INFO *fields_info; > /* for FLUSH table_name */ > int (*reset_table) (); > diff --git a/sql/sql_lex.cc b/sql/sql_lex.cc > index a84fd29affc..b74ecb51a69 100644 > --- a/sql/sql_lex.cc > +++ b/sql/sql_lex.cc > @@ -9370,8 +9359,8 @@ bool LEX::create_package_finalize(THD *thd, > { > if (name2 && > (name2->m_explicit_name != name->m_explicit_name || > - strcmp(name2->m_db.str, name->m_db.str) || > - !Sp_handler::eq_routine_name(name2->m_name, name->m_name))) > + // QQ: was a bug here? "was" may be it was intentional, looking for an exact match only? > + !name2->eq_routine_name(name))) > { > bool exp= name2->m_explicit_name || name->m_explicit_name; > my_error(ER_END_IDENTIFIER_DOES_NOT_MATCH, MYF(0), > diff --git a/sql/sql_parse.cc b/sql/sql_parse.cc > index 665d2143872..1430aa473af 100644 > --- a/sql/sql_parse.cc > +++ b/sql/sql_parse.cc > @@ -1131,8 +1131,8 @@ static bool wsrep_tables_accessible_when_detached(const TABLE_LIST *tables) > { > for (const TABLE_LIST *table= tables; table; table= table->next_global) I'd personally use const TABLE_LIST *t= tables; ... it's a 3-line loop > { > - LEX_CSTRING db= table->db, tn= table->table_name; > - if (get_table_category(&db, &tn) < TABLE_CATEGORY_INFORMATION) > + if (get_table_category(table->db, > + table->table_name) < TABLE_CATEGORY_INFORMATION) > return false; > } > return tables != NULL; > diff --git a/sql/sql_partition.cc b/sql/sql_partition.cc > index 958c52fd95c..c8d64ea37ff 100644 > --- a/sql/sql_partition.cc > +++ b/sql/sql_partition.cc > @@ -8055,13 +8050,9 @@ void make_used_partitions_str(MEM_ROOT *alloc, > if (parts_str->length()) > parts_str->append(','); > uint index= parts_str->length(); > - parts_str->append(head_pe->partition_name, > - strlen(head_pe->partition_name), > - system_charset_info); > + parts_str->append(head_pe->partition_name, system_charset_info); > parts_str->append('_'); > - parts_str->append(pe->partition_name, > - strlen(pe->partition_name), > - system_charset_info); > + parts_str->append(pe->partition_name, system_charset_info); better pe->partition_name->charset_info() (here and above and below) > used_partitions_list.append_str(alloc, parts_str->ptr() + index); > } > partition_id++; > diff --git a/sql/sql_plugin.cc b/sql/sql_plugin.cc > index 8bab244c709..f335226b9a2 100644 > --- a/sql/sql_plugin.cc > +++ b/sql/sql_plugin.cc > @@ -374,16 +374,16 @@ bool check_valid_path(const char *path, size_t len) > > static void fix_dl_name(MEM_ROOT *root, LEX_CSTRING *dl) > { > - const size_t so_ext_len= sizeof(SO_EXT) - 1; > - if (dl->length < so_ext_len || > - my_strcasecmp(&my_charset_latin1, dl->str + dl->length - so_ext_len, > - SO_EXT)) > + const Lex_ident_plugin so_ext(SO_EXT, sizeof(SO_EXT) - 1); STRING_WITH_LEN > + if (dl->length < so_ext.length || > + !so_ext.streq(Lex_cstring(dl->str + dl->length - so_ext.length, > + so_ext.length))) > { > - char *s= (char*)alloc_root(root, dl->length + so_ext_len + 1); > + char *s= (char*)alloc_root(root, dl->length + so_ext.length + 1); > memcpy(s, dl->str, dl->length); > strcpy(s + dl->length, SO_EXT); > dl->str= s; > - dl->length+= so_ext_len; > + dl->length+= so_ext.length; > } > } > > diff --git a/sql/sql_show.cc b/sql/sql_show.cc > index 1fcafd6ec08..7548ff1ec57 100644 > --- a/sql/sql_show.cc > +++ b/sql/sql_show.cc > @@ -7182,10 +7170,12 @@ static int get_schema_views_record(THD *thd, TABLE_LIST *tables, > Security_context *sctx= thd->security_ctx; > if (!tables->allowed_show) > { > - if (!my_strcasecmp(system_charset_info, tables->definer.user.str, > - sctx->priv_user) && > - !my_strcasecmp(system_charset_info, tables->definer.host.str, > - sctx->priv_host)) > + // QQ: why case insensitive comparision? Same as above? Try a test case and fix if a bug? it's likely create user foo; create user FOO; create view ... definer=foo v1 ... connect(as FOO); select * from information schema.views; > + if (my_charset_utf8mb3_general1400_as_ci.streq( > + tables->definer.user, > + Lex_cstring_strlen(sctx->priv_user)) && > + Lex_ident_host(tables->definer.host). > + streq(Lex_cstring_strlen(sctx->priv_host))) > tables->allowed_show= TRUE; > #ifndef NO_EMBEDDED_ACCESS_CHECKS > else > @@ -7893,8 +7883,7 @@ static int get_schema_partitions_record(THD *thd, TABLE_LIST *tables, > > while ((part_elem= part_it++)) > { > - table->field[3]->store(part_elem->partition_name, > - strlen(part_elem->partition_name), cs); > + table->field[3]->store(part_elem->partition_name, cs); may be part_elem->partition_name->charset_info() ? > table->field[3]->set_notnull(); > /* PARTITION_ORDINAL_POSITION */ > table->field[5]->store((longlong) ++part_pos, TRUE); > diff --git a/sql/sql_trigger.cc b/sql/sql_trigger.cc > index 5304a535ad0..dcba31d06c9 100644 > --- a/sql/sql_trigger.cc > +++ b/sql/sql_trigger.cc > @@ -470,7 +470,7 @@ bool mysql_create_or_drop_trigger(THD *thd, TABLE_LIST *tables, bool create) > /* > We don't allow creating triggers on tables in the 'mysql' schema > */ > - if (create && lex_string_eq(&tables->db, STRING_WITH_LEN("mysql"))) > + if (create && tables->db.streq(MYSQL_SCHEMA_NAME)) // QQ: was a bug here? "was" > { > my_error(ER_NO_TRIGGERS_ON_SYSTEM_SCHEMA, MYF(0)); > DBUG_RETURN(TRUE); > diff --git a/sql/sql_trigger.h b/sql/sql_trigger.h > index 493349afb9d..5648f8f15ff 100644 > --- a/sql/sql_trigger.h > +++ b/sql/sql_trigger.h > @@ -74,8 +74,10 @@ struct st_trg_execution_order > /** > Trigger name referenced in the FOLLOWS/PRECEDES clause of the > CREATE TRIGGER statement. > + Cannot be Lex_ident_trigger, > + as this structure is used in %union in sql_yacc.yy Thanks. I already wrote above "why it's not simply a Lex_ident_trigger. That explains it, deleted my previous comment :) > */ > - LEX_CSTRING anchor_trigger_name; > + LEX_CSTRING anchor_trigger_name; // Used in sql_yacc %union > }; > > > diff --git a/sql/sql_truncate.cc b/sql/sql_truncate.cc > index beeee9da72f..e3efc3122df 100644 > --- a/sql/sql_truncate.cc > +++ b/sql/sql_truncate.cc > @@ -147,14 +147,12 @@ fk_truncate_illegal_if_parent(THD *thd, TABLE *table) > /* Loop over the set of foreign keys for which this table is a parent. */ > while ((fk_info= it++)) > { > - if (lex_string_cmp(system_charset_info, fk_info->referenced_db, > - &table->s->db) || > - lex_string_cmp(system_charset_info, fk_info->referenced_table, > - &table->s->table_name) || > - lex_string_cmp(system_charset_info, fk_info->foreign_db, > - &table->s->db) || > - lex_string_cmp(system_charset_info, fk_info->foreign_table, > - &table->s->table_name)) > + // QQ: why db and table don't use table_alias_charset for comparison? Same as above? Try a test case and fix if a bug? test looks fairly straightforward here too > + if (!Lex_ident_ci(*fk_info->referenced_db).streq(table->s->db) || > + !Lex_ident_ci(*fk_info->referenced_table). > + streq(table->s->table_name) || > + !Lex_ident_ci(*fk_info->foreign_db).streq(table->s->db) || > + !Lex_ident_ci(*fk_info->foreign_table).streq(table->s->table_name)) > break; > } > > diff --git a/sql/sql_view.cc b/sql/sql_view.cc > index 10588b54fa5..751373b4571 100644 > --- a/sql/sql_view.cc > +++ b/sql/sql_view.cc > @@ -1347,12 +1347,9 @@ bool mysql_make_view(THD *thd, TABLE_SHARE *share, TABLE_LIST *table, > precedent; > precedent= precedent->referencing_view) > { > - if (precedent->view_name.length == table->table_name.length && > - precedent->view_db.length == table->db.length && > - my_strcasecmp(system_charset_info, > - precedent->view_name.str, table->table_name.str) == 0 && > - my_strcasecmp(system_charset_info, > - precedent->view_db.str, table->db.str) == 0) > + // QQ: was a bug here? "was" > + if (precedent->view_name.streq(table->table_name) && > + precedent->view_db.streq(table->db)) > { > my_error(ER_VIEW_RECURSIVE, MYF(0), > top_view->view_db.str, top_view->view_name.str); > diff --git a/sql/sql_yacc.yy b/sql/sql_yacc.yy > index fcfe490c035..d01c195cb8a 100644 > --- a/sql/sql_yacc.yy > +++ b/sql/sql_yacc.yy > @@ -15676,7 +15680,8 @@ user_maybe_role: > It's OK to use in-place lowercase as long as > the character set is utf8. > */ > - my_casedn_str(system_charset_info, (char*) $$->host.str); > + $$->host.length= my_casedn_str(system_charset_info, > + (char*) $$->host.str); I thought my_casedn_str() changes were in the different commit? > } > else > { > diff --git a/sql/structs.h b/sql/structs.h > index 75bdc9e276b..d349ab72970 100644 > --- a/sql/structs.h > +++ b/sql/structs.h > @@ -321,11 +322,25 @@ typedef struct user_conn { > uint conn_per_hour, updates, questions; > /* Maximum amount of resources which account is allowed to consume. */ > USER_RESOURCES user_resources; > + > + /* > + The CHARSET_INFO used for hashes to compare the entire 'user\0hash' key. > + QQ: perhaps the user should be compared case sensitively, > + while host host should be comparad case insensitively. correct > + */ > + static CHARSET_INFO *user_host_key_charset_info_for_hash() > + { > + return &my_charset_utf8mb3_general1400_as_ci; > + } > } USER_CONN; > > typedef struct st_user_stats > { > char user[MY_MAX(USERNAME_LENGTH, LIST_PROCESS_HOST_LEN) + 1]; > + static CHARSET_INFO *user_key_charset_info_for_hash() > + { > + return &my_charset_utf8mb3_general1400_as_ci; > + } > // Account name the user is mapped to when this is a user from mapped_user. > // Otherwise, the same value as user. > char priv_user[MY_MAX(USERNAME_LENGTH, LIST_PROCESS_HOST_LEN) + 1]; > diff --git a/sql/table.cc b/sql/table.cc > index 8bc16753e68..d9bc5fabb4f 100644 > --- a/sql/table.cc > +++ b/sql/table.cc > @@ -88,23 +88,34 @@ struct extra2_fields > static Virtual_column_info * unpack_vcol_info_from_frm(THD *, > TABLE *, String *, Virtual_column_info **, bool *); > > +/* > + Lex_ident_db does not have operator""_Lex_ident_db, > + because its constructor contains DBUG_SLOW_ASSERT > + and therefore it's not a constexpr constructor. > + So let's initialize a number of Lex_ident_db constants > + using operator""_Lex_cstring. > +*/ > + > /* INFORMATION_SCHEMA name */ > -LEX_CSTRING INFORMATION_SCHEMA_NAME= {STRING_WITH_LEN("information_schema")}; > +Lex_ident_db > + INFORMATION_SCHEMA_NAME= Lex_ident_db("information_schema"_Lex_cstring); don't think so. information_schema is case insensitive, even if fs is case sensitivity > > /* PERFORMANCE_SCHEMA name */ > -LEX_CSTRING PERFORMANCE_SCHEMA_DB_NAME= {STRING_WITH_LEN("performance_schema")}; > +Lex_ident_db > + PERFORMANCE_SCHEMA_DB_NAME= Lex_ident_db("performance_schema"_Lex_cstring); same > > /* MYSQL_SCHEMA name */ > -LEX_CSTRING MYSQL_SCHEMA_NAME= {STRING_WITH_LEN("mysql")}; > +Lex_ident_db > + MYSQL_SCHEMA_NAME= Lex_ident_db("mysql"_Lex_cstring); this uses fs case sensitivity, correct > > /* GENERAL_LOG name */ > -LEX_CSTRING GENERAL_LOG_NAME= {STRING_WITH_LEN("general_log")}; > +Lex_ident_table GENERAL_LOG_NAME= "general_log"_Lex_ident_table; > > /* SLOW_LOG name */ > -LEX_CSTRING SLOW_LOG_NAME= {STRING_WITH_LEN("slow_log")}; > +Lex_ident_table SLOW_LOG_NAME= "slow_log"_Lex_ident_table; > > -LEX_CSTRING TRANSACTION_REG_NAME= {STRING_WITH_LEN("transaction_registry")}; > -LEX_CSTRING MYSQL_PROC_NAME= {STRING_WITH_LEN("proc")}; > +Lex_ident_table TRANSACTION_REG_NAME= "transaction_registry"_Lex_ident_table; > +Lex_ident_table MYSQL_PROC_NAME= "proc"_Lex_ident_table; > > /* > Keyword added as a prefix when parsing the defining expression for a > @@ -280,42 +291,38 @@ const char *fn_frm_ext(const char *name) > } > > > -TABLE_CATEGORY get_table_category(const LEX_CSTRING *db, > - const LEX_CSTRING *name) > +TABLE_CATEGORY get_table_category(const Lex_ident_db &db, > + const Lex_ident_table &name) > { > - DBUG_ASSERT(db != NULL); > - DBUG_ASSERT(name != NULL); > - > #ifdef WITH_WSREP > - if (db->str && > - my_strcasecmp(system_charset_info, db->str, WSREP_SCHEMA) == 0) > + if (db.str && db.streq(MYSQL_SCHEMA_NAME)) //QQ: was a bug here? "was" > { > - if ((my_strcasecmp(system_charset_info, name->str, WSREP_STREAMING_TABLE) == 0 || > - my_strcasecmp(system_charset_info, name->str, WSREP_CLUSTER_TABLE) == 0 || > - my_strcasecmp(system_charset_info, name->str, WSREP_MEMBERS_TABLE) == 0)) > + if (name.streq(Lex_ident_table{STRING_WITH_LEN(WSREP_STREAMING_TABLE)}) || > + name.streq(Lex_ident_table{STRING_WITH_LEN(WSREP_CLUSTER_TABLE)}) || > + name.streq(Lex_ident_table{STRING_WITH_LEN(WSREP_MEMBERS_TABLE)})) > { > return TABLE_CATEGORY_INFORMATION; > } > } > #endif /* WITH_WSREP */ > - if (is_infoschema_db(db)) > + if (is_infoschema_db(&db)) > return TABLE_CATEGORY_INFORMATION; > > - if (is_perfschema_db(db)) > + if (is_perfschema_db(&db)) > return TABLE_CATEGORY_PERFORMANCE; > > - if (lex_string_eq(&MYSQL_SCHEMA_NAME, db)) > + if (db.streq(MYSQL_SCHEMA_NAME)) // QQ: was a bug here? "was" > { > - if (is_system_table_name(name->str, name->length)) > + if (is_system_table_name(name.str, name.length)) > return TABLE_CATEGORY_SYSTEM; > > - if (lex_string_eq(&GENERAL_LOG_NAME, name)) > + if (name.streq(GENERAL_LOG_NAME)) // QQ: was bug here? "was" > return TABLE_CATEGORY_LOG; > > - if (lex_string_eq(&SLOW_LOG_NAME, name)) > + if (name.streq(SLOW_LOG_NAME)) // QQ: was a bug here? "was" > return TABLE_CATEGORY_LOG; > > - if (lex_string_eq(&TRANSACTION_REG_NAME, name)) > + if (name.streq(TRANSACTION_REG_NAME)) // QQ: was a bug here? "was" > return TABLE_CATEGORY_LOG; > } > > diff --git a/sql/table.h b/sql/table.h > index 35be3bd4b2d..fe65025478d 100644 > --- a/sql/table.h > +++ b/sql/table.h > @@ -3402,27 +3409,27 @@ static inline int set_zone(int nr,int min_zone,int max_zone) > } > > /* performance schema */ > -extern LEX_CSTRING PERFORMANCE_SCHEMA_DB_NAME; > +extern Lex_ident_db PERFORMANCE_SCHEMA_DB_NAME; > > -extern LEX_CSTRING GENERAL_LOG_NAME; > -extern LEX_CSTRING SLOW_LOG_NAME; > -extern LEX_CSTRING TRANSACTION_REG_NAME; > +extern Lex_ident_table GENERAL_LOG_NAME; > +extern Lex_ident_table SLOW_LOG_NAME; > +extern Lex_ident_table TRANSACTION_REG_NAME; > > /* information schema */ > -extern LEX_CSTRING INFORMATION_SCHEMA_NAME; > -extern LEX_CSTRING MYSQL_SCHEMA_NAME; > +extern Lex_ident_db INFORMATION_SCHEMA_NAME; > +extern Lex_ident_db MYSQL_SCHEMA_NAME; > > /* table names */ > -extern LEX_CSTRING MYSQL_PROC_NAME; > +extern Lex_ident_table MYSQL_PROC_NAME; > > inline bool is_infoschema_db(const LEX_CSTRING *name) > { > - return lex_string_eq(&INFORMATION_SCHEMA_NAME, name); > + return lex_string_eq(&INFORMATION_SCHEMA_NAME, name); // QQ: use Lex_ident_ci ? yes > } > > inline bool is_perfschema_db(const LEX_CSTRING *name) > { > - return lex_string_eq(&PERFORMANCE_SCHEMA_DB_NAME, name); > + return lex_string_eq(&PERFORMANCE_SCHEMA_DB_NAME, name); // QQ: use Lex_ident_ci? yes > } > > inline void mark_as_null_row(TABLE *table) > diff --git a/storage/blackhole/ha_blackhole.cc b/storage/blackhole/ha_blackhole.cc > index 343f3c70286..ce48c95f837 100644 > --- a/storage/blackhole/ha_blackhole.cc > +++ b/storage/blackhole/ha_blackhole.cc > @@ -416,7 +416,9 @@ static int blackhole_init(void *p) > mysql_mutex_init(bh_key_mutex_blackhole, > &blackhole_mutex, MY_MUTEX_INIT_FAST); > (void) my_hash_init(PSI_INSTRUMENT_ME, &blackhole_open_tables, > - system_charset_info, 32, 0, 0, > + // QQ: was a bug here? "was" > + Lex_ident_table::charset_info(), > + 32, 0, 0, > (my_hash_get_key) blackhole_get_key, > (my_hash_free_key) blackhole_free_key, 0); > > diff --git a/storage/csv/ha_tina.cc b/storage/csv/ha_tina.cc > index e705ff0e7c0..9fc0bcf604b 100644 > --- a/storage/csv/ha_tina.cc > +++ b/storage/csv/ha_tina.cc > @@ -184,7 +184,9 @@ static int tina_init_func(void *p) > tina_hton= (handlerton *)p; > mysql_mutex_init(csv_key_mutex_tina, &tina_mutex, MY_MUTEX_INIT_FAST); > (void) my_hash_init(csv_key_memory_tina_share, &tina_open_tables, > - system_charset_info, 32, 0, 0, (my_hash_get_key) > + // QQ: was a bug here? "was" > + table_alias_charset, > + 32, 0, 0, (my_hash_get_key) > tina_get_key, 0, 0); > tina_hton->db_type= DB_TYPE_CSV_DB; > tina_hton->create= tina_create_handler; > diff --git a/storage/innobase/dict/dict0dict.cc b/storage/innobase/dict/dict0dict.cc > index 5545727b015..231bcc1a132 100644 > --- a/storage/innobase/dict/dict0dict.cc > +++ b/storage/innobase/dict/dict0dict.cc > @@ -284,27 +284,26 @@ otherwise table->n_def */ > ulint > dict_table_has_column( > const dict_table_t* table, > - const char* col_name, > + const LEX_CSTRING &col_name, looks ok to me, but to avoid surprises Marko should review it too > ulint col_nr) > { > ulint col_max = table->n_def; > > ut_ad(table); > - ut_ad(col_name); > + ut_ad(col_name.str); > ut_ad(table->magic_n == DICT_TABLE_MAGIC_N); > > if (col_nr < col_max > - && innobase_strcasecmp( > - col_name, dict_table_get_col_name(table, col_nr)) == 0) { > + && Lex_ident_column(col_name). > + streq(dict_table_get_col_name_ls(table, col_nr))) { > return(col_nr); > } > > /** The order of column may changed, check it with other columns */ > for (ulint i = 0; i < col_max; i++) { > if (i != col_nr > - && innobase_strcasecmp( > - col_name, dict_table_get_col_name(table, i)) == 0) { > - > + && Lex_ident_column(col_name). > + streq(dict_table_get_col_name_ls(table, i))) { > return(i); > } > } > diff --git a/storage/mroonga/ha_mroonga.cpp b/storage/mroonga/ha_mroonga.cpp > index 44fd09e74ed..71784dad475 100644 > --- a/storage/mroonga/ha_mroonga.cpp > +++ b/storage/mroonga/ha_mroonga.cpp > @@ -1936,7 +1936,10 @@ static int mrn_init(void *p) > MY_MUTEX_INIT_FAST) != 0)) { > goto err_allocated_thds_mutex_init; > } > - if (mrn_my_hash_init(&mrn_allocated_thds, system_charset_info, 32, 0, 0, > + if (mrn_my_hash_init(&mrn_allocated_thds, > + // QQ: was a bug here? "was" > + &my_charset_bin, > + 32, 0, 0, > mrn_allocated_thds_get_key, 0, 0)) { > goto error_allocated_thds_hash_init; > } > @@ -1945,7 +1948,10 @@ static int mrn_init(void *p) > MY_MUTEX_INIT_FAST) != 0)) { > goto err_allocated_open_tables_mutex_init; > } > - if (mrn_my_hash_init(&mrn_open_tables, system_charset_info, 32, 0, 0, > + if (mrn_my_hash_init(&mrn_open_tables, > + // QQ: was a bug here? "was" > + Lex_ident_table::charset_info(), > + 32, 0, 0, > mrn_open_tables_get_key, 0, 0)) { > goto error_allocated_open_tables_hash_init; > } > @@ -1954,7 +1960,10 @@ static int mrn_init(void *p) > MY_MUTEX_INIT_FAST) != 0)) { > goto error_allocated_long_term_share_mutex_init; > } > - if (mrn_my_hash_init(&mrn_long_term_share, system_charset_info, 32, 0, 0, > + if (mrn_my_hash_init(&mrn_long_term_share, > + // QQ: was a bug here? "was" > + Lex_ident_table::charset_info(), > + 32, 0, 0, > mrn_long_term_share_get_key, 0, 0)) { > goto error_allocated_long_term_share_hash_init; > } > diff --git a/storage/perfschema/pfs_engine_table.h b/storage/perfschema/pfs_engine_table.h > index 1ef1e0282de..b4f4c9f0acc 100644 > --- a/storage/perfschema/pfs_engine_table.h > +++ b/storage/perfschema/pfs_engine_table.h > @@ -25,6 +25,22 @@ > > #include "table.h" > #include "sql_acl.h" > + > +/* > + The performance schema is implemented as a storage engine, in memory. > + The current storage engine interface exposed by the server, > + and in particular handlerton::discover, uses 'FRM' files to describe a > + table structure, which are later stored on disk, by the server, > + in ha_create_table_from_engine(). > + Because the table metadata is stored on disk, the table naming rules > + used by the performance schema then have to comply with the constraints > + imposed by the disk storage, and in particular with lower_case_table_names. > +*/ > + > +using PFS_ident_db= Lex_ident_db; > +using PFS_ident_table = Lex_ident_table; No, table metadata for performance schema are never stored on disk. let's use case insensitive comparison here, if possible. > + > + > /** > @file storage/perfschema/pfs_engine_table.h > Performance schema tables (declarations). Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 5

Re: 9d6de9212d5: MDEV-29095 REGEXP_REPLACE treats empty strings different than REPLACE in ORACLE mode
by Sergei Golubchik 16 Jan '24

16 Jan '24

Hi, Alexander, On Jan 10, Alexander Barkov wrote: > revision-id: 9d6de9212d5 (mariadb-10.4.32-45-g9d6de9212d5) > parent(s): bdfd93d30c1 > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2023-11-24 13:23:56 +0400 > message: > > MDEV-29095 REGEXP_REPLACE treats empty strings different than REPLACE in ORACLE mode > > Turning REGEXP_REPLACE into two schema-qualified functions: > - mariadb_schema.regexp_replace() > - oracle_schema.regexp_replace() > > Fixing oracle_schema.regexp_replace(subj,pattern,replacement) to treat > NULL in "replacement" as an empty string. > > Adding new classes implementing oracle_schema.regexp_replace(): > - Item_func_regexp_replace_oracle > - Create_func_regexp_replace_oracle > > diff --git a/sql/item_strfunc.cc b/sql/item_strfunc.cc > index 92d5e196da4..b688ba97d3e 100644 > --- a/sql/item_strfunc.cc > +++ b/sql/item_strfunc.cc > @@ -1385,20 +1385,22 @@ bool Item_func_regexp_replace::append_replacement(String *str, > } > > > -String *Item_func_regexp_replace::val_str(String *str) > +String *Item_func_regexp_replace::val_str_internal(String *str, > + bool null_to_empty) > { > DBUG_ASSERT(fixed == 1); > char buff0[MAX_FIELD_WIDTH]; > char buff2[MAX_FIELD_WIDTH]; > String tmp0(buff0,sizeof(buff0),&my_charset_bin); > String tmp2(buff2,sizeof(buff2),&my_charset_bin); > - String *source= args[0]->val_str(&tmp0); > - String *replace= args[2]->val_str(&tmp2); > + String *source, *replace; > LEX_CSTRING src, rpl; > int startoffset= 0; > > - if ((null_value= (args[0]->null_value || args[2]->null_value || > - re.recompile(args[1])))) > + if ((null_value= > + (!(source= args[0]->val_str(&tmp0)) || > + !(replace= args[2]->val_str_null_to_empty(&tmp2, null_to_empty)) || why are you fixing it differently from Item_func_replace? If you think this approach is better then, please, change Item_func_replace to use it too, in a followup commit. > + re.recompile(args[1])))) > return (String *) 0; > > if (!(source= re.convert_if_needed(source, &re.subject_converter)) || > diff --git a/sql/item_strfunc.h b/sql/item_strfunc.h > index 826c7f784af..e014cc5fcbc 100644 > --- a/sql/item_strfunc.h > +++ b/sql/item_strfunc.h > @@ -412,11 +420,34 @@ class Item_func_regexp_replace :public Item_str_func > re.cleanup(); > DBUG_VOID_RETURN; > } > - String *val_str(String *str); > + String *val_str(String *str) > + { > + return val_str_internal(str, false); > + } > bool fix_fields(THD *thd, Item **ref); > bool fix_length_and_dec(); > const char *func_name() const { return "regexp_replace"; } > - Item *get_copy(THD *thd) { return 0;} > + Item *get_copy(THD *thd) { return 0;} // QQ: why? you could've checked the history in git. it was commit 343bcb152f8 Author: Igor Babaev <igor(a)askmonty.org> Date: Sun Nov 5 22:52:41 2017 -0800 Fixed mdev-14237 Server crash on query with regexp_substr It's better to prohibit pushdown of conditions that involve regexp_substr() and regexp_replace() into materialized derived tables / views until proper implementations of the get_copy() virtual method are provided for those functions. and I suspect the reason is Regexp_processor_pcre inside regexp Items. > +}; > + > + > +class Item_func_regexp_replace_oracle: public Item_func_regexp_replace > +{ > +public: > + Item_func_regexp_replace_oracle(THD *thd, Item *a, Item *b, Item *c) > + :Item_func_regexp_replace(thd, a, b, c) > + {} > + const Schema *schema() const { return &oracle_schema_ref; } > + bool fix_length_and_dec() > + { > + bool rc= Item_func_regexp_replace::fix_length_and_dec(); > + maybe_null= true; // Empty result is converted to NULL > + return rc; > + } > + String *val_str(String *str) > + { > + return val_str_internal(str, true); > + } > }; Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 1

Re: 22e2f082400: MDEV-25370 Update for portion changes autoincrement key in bi-temp table
by Sergei Golubchik 12 Jan '24

12 Jan '24

Hi, Nikita, On Jan 06, Nikita Malyavin wrote: > revision-id: 22e2f082400 (mariadb-10.5.23-39-g22e2f082400) > parent(s): e472b682e06 > author: Nikita Malyavin > committer: Nikita Malyavin > timestamp: 2023-12-20 02:46:09 +0100 > message: > > MDEV-25370 Update for portion changes autoincrement key in bi-temp table > > According to the standard, the autoincrement column (i.e. *identity > column*) should be advanced each insert implicitly made by > UPDATE/DELETE ... FOR PORTION. When you refer to the standard, specify the exact reference so that someone could verify it. E.g. as I've commented in MDEV, "SQL:2016, Part 2, 15.13 Effect of replacing rows in base tables, paragraph 10) b) i)" Also, the standard doesn't have autoincrement, it'd be better to say "According to the standard, the identity column (which autoincrement is a functional analog of) should be advanced...." > This is very unconvenient use in several notable cases. Concider a > WITHOUT OVERLAPS key with an autoinc column: > id int auto_increment, unique(id, p without overlaps) > > An update or delete with FOR PORTION creates a sense that id will > remain unchanged in such case. > > This commit makes an exception the autoinc columns that are part of > WITHOUT OVERLAPS keys and infers the following rule for the newly > inserted rows: Better to say here not that we're making an exception and violating the standard (because we are not), but that in the standard IDENTITY uses its own independent sequence generator, which knows nothing about periods. While autoincrement always requires an index (that is, our "identity" column is a pair {autoincrement, index}, not autoincrement alone), and the index can be declared WITHOUT OVERLAPS, so our "identity" column is period-aware. This is an extension of the standard, and we can freely define how it behaves in the cases not covered by the standard, that is in the WITHOUT OVERLAPS case. > ===== > Let V_j a set of values for currently updating/deleting row. > Let VI_j a set of values to be inserted. > * Case: > (a) if V_i is an identity column that is also a part of a unique constraint > that includes <without overlap specification>, then let VI_i be V_i > (b) otherwise, let VI_i be the value deduced as described by the rules of > ISO/IEC 9075-2, clauses 15.7 Effect of deleting rows from base tables > and 15.13 Effect of replacing rows in base tables. > ===== That's confusing, I'm sure that anyone who doesn't remember the corresponding part of the sql standard will think it's a direct quote from it. You've imitated the style rather convincingly :) It's also rather unnecessary, we don't have identity columns, so nobody cares how they should behave in this case. Perhaps it'd be better if you simply write that if the key, that defines autoincrement values, is declared WITHOUT OVERLAPS, then new rows created by UPDATE/DELETE FOR PORTION OF will use old autoincrement values, as they belong to non-overlapping ranges. But INSERT will generate a new autoincrement value, it will not try to find the smallest value in the overlapping ranges. > Note that this also infers that if an identity column is not a part of > any unique constraint that includes <without overlap specification>, then > its value will be DEFAULT and, that is, auto-incremented. > > Apart from WITHOUT OVERLAPS there is also another notable case, referred > by the reporter - a unique key that has an autoincrement column and a field > from the period specification: > id int auto_increment, unique(id, s), period for p(s, e) > > for this case, no exception is made, and the autoincrementing rules will be > proceeded accordung to the standard (i.e. the value will be advanced on > implicit inserts). > > diff --git a/mysql-test/suite/period/r/overlaps.result b/mysql-test/suite/period/r/overlaps.result > --- a/mysql-test/suite/period/r/overlaps.result > +++ b/mysql-test/suite/period/r/overlaps.result > @@ -449,3 +449,86 @@ VALUES > ('2000-01-01 00:00:00.000000', '2001-01-01 00:00:00.000000', 'abc'); > ERROR 23000: Duplicate entry 'abc-2001-01-01 00:00:00.000000-2000-01-01 00:00:00.000000' for key 'index_name' > DROP TABLE t1; > +create or replace table cars(id int auto_increment, > +price int, s date, e date, > +period for p(s,e), > +primary key(id, p without overlaps)); > +insert into cars(price, s, e) values (1000, '2018-01-01', '2020-01-01'); > +select * from cars; > +id price s e > +1 1000 2018-01-01 2020-01-01 > +update cars for portion of p from '2019-01-01' to '2019-12-01' set price= 1100; > +select * from cars; > +id price s e > +1 1000 2018-01-01 2019-01-01 > +1 1000 2019-12-01 2020-01-01 > +1 1100 2019-01-01 2019-12-01 > +delete from cars for portion of p from '2019-12-10' to '2019-12-20'; > +select * from cars; > +id price s e > +1 1000 2018-01-01 2019-01-01 > +1 1000 2019-12-01 2019-12-10 > +1 1000 2019-12-20 2020-01-01 > +1 1100 2019-01-01 2019-12-01 ok > +# AUTO_INCREMENT field is separate from WITHOUT OVERLAPS > +create or replace table cars(id int primary key auto_increment, > +car_id int, > +price int, s date, e date, > +period for p(s,e), > +unique(car_id, p without overlaps)); > +insert cars(car_id, price, s, e) values (1, 1000, '2018-01-01', '2020-01-01'); > +select * from cars; > +id car_id price s e > +1 1 1000 2018-01-01 2020-01-01 > +update cars for portion of p from '2019-01-01' to '2019-12-01' set price= 1100; > +select * from cars; > +id car_id price s e > +1 1 1100 2019-01-01 2019-12-01 > +2 1 1000 2018-01-01 2019-01-01 > +3 1 1000 2019-12-01 2020-01-01 > +delete from cars for portion of p from '2019-12-10' to '2019-12-20'; > +select * from cars; > +id car_id price s e > +1 1 1100 2019-01-01 2019-12-01 > +2 1 1000 2018-01-01 2019-01-01 > +4 1 1000 2019-12-01 2019-12-10 > +5 1 1000 2019-12-20 2020-01-01 right > +# AUTO_INCREMENT field is both standalone and in WITHOUT OVERLAPS > +create or replace table cars(id int primary key auto_increment, > +price int, s date, e date, > +period for p(s,e), > +unique(id, p without overlaps)); > +insert cars(price, s, e) values (1000, '2018-01-01', '2020-01-01'); > +select * from cars; > +id price s e > +1 1000 2018-01-01 2020-01-01 > +update cars for portion of p from '2019-01-01' to '2019-12-01' set price= 1100; > +ERROR 23000: Duplicate entry '1' for key 'PRIMARY' uhm, I don't know. I'd expect that here PRIMARY would define autoincrement values. > +truncate cars; > +insert cars(price, s, e) values (1000, '2018-01-01', '2020-01-01'); > +delete from cars for portion of p from '2019-12-10' to '2019-12-20'; > +ERROR 23000: Duplicate entry '1' for key 'PRIMARY' > +# AUTO_INCREMENT field is non-unique not sure here. I think here unique should define autoincrement values. That is, in both cases, the most restrictive index should define. > +create or replace table cars(id int auto_increment, key(id), > +car_id int, > +price int, s date, e date, > +period for p(s,e), > +unique(car_id, p without overlaps)); > +insert cars(car_id, price, s, e) values (1, 1000, '2018-01-01', '2020-01-01'); > +select * from cars; > +id car_id price s e > +1 1 1000 2018-01-01 2020-01-01 > +update cars for portion of p from '2019-01-01' to '2019-12-01' set price= 1100; > +select * from cars; > +id car_id price s e > +1 1 1100 2019-01-01 2019-12-01 > +2 1 1000 2018-01-01 2019-01-01 > +3 1 1000 2019-12-01 2020-01-01 > +delete from cars for portion of p from '2019-12-10' to '2019-12-20'; > +select * from cars; > +id car_id price s e > +1 1 1100 2019-01-01 2019-12-01 > +2 1 1000 2018-01-01 2019-01-01 > +4 1 1000 2019-12-01 2019-12-10 > +5 1 1000 2019-12-20 2020-01-01 please also add test for INSERT. Just add a second insert like insert cars(price, s, e) values (1000, '2021-01-01', '2022-01-01'); to show that id will *not* be 1 here, even though 1 would not conflict with anything in the specified date range. Simply to have the expected behavior recorded as a test result. > +drop table cars; > diff --git a/sql/table.cc b/sql/table.cc > --- a/sql/table.cc > +++ b/sql/table.cc > @@ -8917,6 +8916,21 @@ int TABLE::update_generated_fields() > return res; > } > > +void TABLE::period_prepare_autoinc() > +{ > + if (!found_next_number_field) > + return; > + > + key_map::Iterator it(found_next_number_field->part_of_key_not_clustered); > + for (uint i; (i= it++) != key_map::Iterator::BITMAP_END;) > + { > + KEY &key= key_info[i]; > + if (key.without_overlaps) > + return; > + } No, I don't think you need to check *all* indexes that the field is part of. I believe you should only check the "main" index, the one that defines autoincrement field behavior. That is only if (key_info[s->next_number_index].without_overlaps) return; > + next_number_field= found_next_number_field; > +} > + > int TABLE::period_make_insert(Item *src, Field *dst) > { > THD *thd= in_use; Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 10

Re: 6835d2ce268: MDEV-28649 json_array(false) in the definition of view return 0 instead of "true".
by Sergei Golubchik 12 Jan '24

12 Jan '24

Hi, Alexey, On Jan 12, Alexey Botchkov wrote: > revision-id: 6835d2ce268 (mariadb-10.4.32-116-g6835d2ce268) > parent(s): fbe604d8839 > author: Alexey Botchkov > committer: Alexey Botchkov > timestamp: 2023-12-15 00:56:20 +0400 > message: > > MDEV-28649 json_array(false) in the definition of view return 0 instead of "true". > > The 'temporary-only Field_bool introduced'; Why did you need a new Field and to change Type_handler_bool from long to tiny? On the first glance it seems that a dedicated Item_bool::print() should've been enough. Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

1 0

Re: 424ee6eac78: MDEV-32837 long unique does not work like unique key when using replace
by Sergei Golubchik 11 Jan '24

11 Jan '24

Hi, Alexander, On Jan 11, Alexander Barkov wrote: > revision-id: 424ee6eac78 (mariadb-10.5.23-57-g424ee6eac78) > parent(s): c9902a20b3a > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2024-01-10 15:35:25 +0400 > message: > > MDEV-32837 long unique does not work like unique key when using replace > > diff --git a/sql/sql_insert.cc b/sql/sql_insert.cc > index ec9e379768f..edc2be207f3 100644 > --- a/sql/sql_insert.cc > +++ b/sql/sql_insert.cc > @@ -2065,7 +2065,8 @@ int write_record(THD *thd, TABLE *table, COPY_INFO *info, select_result *sink) > we just should not expose this fact to users by invoking > ON UPDATE triggers. > */ > - if (last_uniq_key(table,key_nr) && > + if ((!table->s->long_unique_table || table->versioned()) && > + last_uniq_key(table,key_nr) && > !table->file->referenced_by_foreign_key() && > (!table->triggers || !table->triggers->has_delete_triggers())) > { 1. why table->versioned() ? 2. use show status in your test for handler_update/delete to see what REPLACE actually did 3. technically, you can keep the optimization if the long unique is the last *and there are no in-engine uniques*, this can be checked with && !(table->key_info->flags & HA_NOSAME) 4. but here's the problem that last_uniq_key() doesn't see long uniques, so it'll treat every one of them as the last. So it should be something like bool is_long_unique= table->s->key_info[key_nr].flags & HA_UNIQUE_HASH; if ((is_long_unique ? last_uniq_key(table, table->s->key_info, key_nr) && !(table->key_info->flags & HA_NOSAME) : last_uniq_key(table, table->key_info, key_nr)) && !table->file->referenced_by_foreign_key() && (!table->triggers || !table->triggers->has_delete_triggers())) I suspect this will correctly use the UPDATE optimization in all cases (only long unique / mix of uniques)x(last / not last key conflicting) Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

1 0