- developers - lists.mariadb.org

Re: [PATCH] MDEV-32014 Rename binlog cache temporary file to binlog file for large transaction
by Kristian Nielsen 03 Sep '24

03 Sep '24

Andrei Elkin <andrei.elkin(a)mariadb.com> writes: >> So please do a small refactoring of trx_group_commit_leader() and keep >> only ... > > and no need for refactoring really arises. Well, the refactoring is good, even without the MDEV-32014 patch. The function trx_group_commit_leader() is already too long and complicated, and does too much in a single function. And the first part (obtaining the queue) is cleanly separated from the second part (committing each entry in the queue in-order in binlog and participating engines). It's just nice that this improvement of the existing code also makes the MDEV-32014 patch integrate cleaner. > I noted earlier on this part on the PR page and LiBing seems to have > practically proven (by the most recent commit that you must've missed) that > conducting the rename inside the flush stage of > trx_group_commit_leader() is viable: > > https://github.com/MariaDB/server/pull/3402/commits/287511a703f7dff47939633… > > By that measure the flushing is expanded onto possibly multiple "implicitly" > rotating log files Hm, I don't like the change 287511a703f7dff47939633475edd436a542cf6f. In fact, that is one of the things I like the most about Libing Song's patch, that it so cleanly separates the commit-by-rotate logic from the complex group commit logic in queue_for_group_commit() and trx_group_commit_leader(), which is already too complex. The original patch without 287511a703f7dff47939633475edd436a542cf6f seems much cleaner. It is better to do as in the original patch, and treat commit-by-rotate like a normal binlog rotate, not try to participate that transaction in binlog group commit. We cannot share the fsync anyway, as the files are different. And it introduces a lot of complexities to suddenly have binlog rotations in the middle of group commit - what about the complex accounting around xid_count and binlog checkpoints? There will be a lot of corner cases eg. if _two_ commit-by-rotate happens in the same group commit, with lots of potential for subtle bugs that will occur only very rarely in production and be almost impossible to track down. I strongly prefer the original patch I reviewed (with my suggested changes to write_transaction_to_binlog_events() and trx_group_commit_leader()), where the commit-by-rotate logic is cleanly separated from the group commit logic and implemented mostly using the existing binlog rotate logic. - Kristian.

2 2

Re: [PATCH] MDEV-34857: Implement --slave-abort-blocking-timeout
by Michael Widenius 03 Sep '24

03 Sep '24

Review of MDEV-34857: Implement --slave-abort-blocking-timeout If a slave replicating an event has waited for more than @@slave_abort_blocking_timeout for a conflicting metadata lock held by a non-replication thread, the blocking query is killed to allow replication to proceed and not be blocked indefinitely by a user query. <cut> diff --git a/sql/mdl.cc b/sql/mdl.cc index faccd1c9476..9845718e165 100644 --- a/sql/mdl.cc +++ b/sql/mdl.cc <cut> @@ -2397,14 +2398,39 @@ MDL_context::acquire_lock(MDL_request *mdl_request, double lock_wait_timeout) find_deadlock(); - struct timespec abs_timeout, abs_shortwait; + struct timespec abs_timeout, abs_shortwait, abs_abort_blocking_timeout; + bool abort_blocking_enabled= false; + double abort_blocking_timeout= slave_abort_blocking_timeout; Add a comment here. Something like /* This code implements aborting of queries on the slaves that stops replication from continuing. */ + if (abort_blocking_timeout < lock_wait_timeout && + m_owner->get_thd()->rgi_slave) + { + set_timespec_nsec(abs_abort_blocking_timeout, + (ulonglong)(abort_blocking_timeout * 1000000000ULL)); + abort_blocking_enabled= true; + } set_timespec_nsec(abs_timeout, (ulonglong)(lock_wait_timeout * 1000000000ULL)); - set_timespec(abs_shortwait, 1); wait_status= MDL_wait::EMPTY; - while (cmp_timespec(abs_shortwait, abs_timeout) <= 0) + for (;;) { + bool abort_blocking= false; + set_timespec(abs_shortwait, 1); + if (abort_blocking_enabled && + cmp_timespec(abs_shortwait, abs_abort_blocking_timeout) >= 0) + { + /* + If a slave DDL has waited for --slave-abort-select-timeout, then notify + any blocking SELECT once before continuing to wait until the full + timeout. + */ + abs_shortwait= abs_abort_blocking_timeout; + abort_blocking= true; + abort_blocking_enabled= false; + } + if (cmp_timespec(abs_shortwait, abs_timeout) > 0) + break; Is the above right last compare right? Assuming that cmp_timespec(abs_shortwait, abs_abort_blocking_timeout) > 0 then the above will not break and we will not signal other threads to break even if the previous code expectrs us to abort blocking. (I think the cmp_timespec() return test should be the same in both cases) Shouldn't the code be: for (;;) { bool abort_blocking= false; set_timespec(abs_shortwait, 1); if (abort_blocking_enabled && cmp_timespec(abs_shortwait, abs_abort_blocking_timeout) >= 0) { /* If a slave DDL has waited for --slave-abort-select-timeout, then notify any blocking SELECT once before continuing to wait until the full timeout. */ abort_blocking= true; abort_blocking_enabled= false; } else if (cmp_timespec(abs_shortwait, abs_timeout) > 0) break; Or have I missed something? /* abs_timeout is far away. Wait a short while and notify locks. */ wait_status= m_wait.timed_wait(m_owner, &abs_shortwait, FALSE, mdl_request->key.get_wait_state_name()); @@ -2425,9 +2451,8 @@ MDL_context::acquire_lock(MDL_request *mdl_request, double lock_wait_timeout) mysql_prlock_wrlock(&lock->m_rwlock); if (lock->needs_notification(ticket)) - lock->notify_conflicting_locks(this); + lock->notify_conflicting_locks(this, abort_blocking); mysql_prlock_unlock(&lock->m_rwlock); - set_timespec(abs_shortwait, 1); I like the new code in that we are not setting abs_shortwait two times anymore. (Using a while loop in the is case was sub optimal. Ok to push after the above change or if you have a good argument why the current code is correct

1 0

Re: [PATCH] MDEV-32014 Rename binlog cache temporary file to binlog file for large transaction
by andrei.elkin＠pp.inet.fi 02 Sep '24

02 Sep '24

(Re-sending to the list) Howdy Kristian. > > The patch calls the trx_group_commit_leader() function. That is ackward, as > commit-by-rotate does not actually participate in group commit. But the > problem is with the existing function trx_group_commit_leader(), which does > too much in a single function. I noted earlier on this part on the PR page and LiBing seems to have practically proven (by the most recent commit that you must've missed) that conducting the rename inside the flush stage of trx_group_commit_leader() is viable: https://github.com/MariaDB/server/pull/3402/commits/287511a703f7dff47939633… By that measure the flushing is expanded onto possibly multiple "implicitly" rotating log files > > So please do a small refactoring of trx_group_commit_leader() and keep > only ... and no need for refactoring really arises. Could you please assess that approach, cheers, Andrei

1 0

Next LTS version in February 2025?
by Otto Kekäläinen 29 Aug '24

29 Aug '24

Hi! Traditionally there has been a new major MariaDB version in February of the year Debian has a new major version. What is the plan for February 2025? I was reading https://mariadb.org/about/#maintenance-policy and old mariadb.orb blog posts but none was explicit about this.

2 2

Re: fec3696fef6: MDEV-15751 CURRENT_TIMESTAMP should return a TIMESTAMP [WITH TIME ZONE?]
by Sergei Golubchik 28 Aug '24

28 Aug '24

Hi, Alexander, This has waited far too long :( > commit fec3696fef6 > Author: Alexander Barkov <bar(a)mariadb.com> > Date: Tue Feb 14 13:27:46 2023 +0400 > > diff --git a/sql/sql_class.h b/sql/sql_class.h > index f0bf695a1c1..7378225eaa2 100644 > --- a/sql/sql_class.h > +++ b/sql/sql_class.h > @@ -3900,7 +3900,8 @@ class THD: public THD_count, /* this must be first */ > } > const Type_handler *type_handler_for_datetime() const; > bool timestamp_to_TIME(MYSQL_TIME *ltime, my_time_t ts, > - ulong sec_part, date_mode_t fuzzydate); > + ulong sec_part, date_mode_t fuzzydate, > + bool zero_timesteamp_means_zero_datetime); I don't understand it. NOW() cannot return a zero timestamp, how this feature can possibly need changes to zero timestamp treatment? > inline my_time_t query_start() { return start_time; } > inline ulong query_start_sec_part() > { used|= QUERY_START_SEC_PART_USED; return start_time_sec_part; } > diff --git a/sql/item_timefunc.h b/sql/item_timefunc.h > index a5f6d9307c6..1c463c8a332 100644 > --- a/sql/item_timefunc.h > +++ b/sql/item_timefunc.h > @@ -707,6 +707,49 @@ class Item_datetimefunc :public Item_func > }; > > > +class Item_timestampfunc: public Item_func > +{ > +protected: > + Datetime to_datetime(THD *thd) > + { > + return Timestamp_or_zero_datetime_native_null(thd, this).to_datetime(thd); how does it differ from Datetime(this) ? > + } > +public: > + Item_timestampfunc(THD *thd): Item_func(thd) {} > + Item_timestampfunc(THD *thd, Item *a): Item_func(thd, a) {} > + const Type_handler *type_handler() const override > + { return &type_handler_timestamp2; } > + bool get_date(THD *thd, MYSQL_TIME *ltime, date_mode_t fuzzydate) override > + { > + return null_value= to_datetime(thd).copy_to_mysql_time(ltime); > + } > + double val_real() override > + { > + Datetime dt= to_datetime(current_thd); > + null_value= !dt.is_valid_datetime(); > + return dt.to_double(); may be `return null_value ? 0 : dt.to_double()` ? why to call to_double if it isn't a valid datetime? (same everywhere below) > + } > + longlong val_int() override > + { > + Datetime dt= to_datetime(current_thd); > + null_value= !dt.is_valid_datetime(); > + return dt.to_longlong(); > + } > + my_decimal *val_decimal(my_decimal *to) override > + { > + Datetime dt= to_datetime(current_thd); > + null_value= !dt.is_valid_datetime(); > + return dt.to_decimal(to); > + } > + String *val_str(String *to) override > + { > + Datetime dt= to_datetime(current_thd); > + null_value= !dt.is_valid_datetime(); > + return dt.to_string(to, decimals); > + } > +}; > + > + > /* Abstract CURTIME function. Children should define what time zone is used */ > > class Item_func_curtime :public Item_timefunc > @@ -837,20 +880,48 @@ class Item_func_now :public Item_datetimefunc > }; > > > -class Item_func_now_local :public Item_func_now > +/* > + This is a replacement for Item_func_now_local, > + returning TIMESTAMP instead of DATETIME. why did you rename it? why didn't you change Item_func_now_utc? > +*/ > +class Item_func_current_timestamp: public Item_timestampfunc > { > public: > - Item_func_now_local(THD *thd, uint dec): Item_func_now(thd, dec) {} > + Item_func_current_timestamp(THD *thd, uint dec) > + :Item_timestampfunc(thd) > + { decimals= dec; } why not Item_timestampfunc(thd, dec) ? > LEX_CSTRING func_name_cstring() const override > { > static LEX_CSTRING name= {STRING_WITH_LEN("current_timestamp") }; > return name; > } > - int save_in_field(Field *field, bool no_conversions) override; > - void store_now_in_TIME(THD *thd, MYSQL_TIME *now_time) override; > + void print(String *str, enum_query_type query_type) override > + { > + str->append(func_name_cstring()); > + str->append('('); > + if (decimals) > + str->append_ulonglong(decimals); > + str->append(')'); > + } > + bool fix_length_and_dec(THD *thd) override > + { > + if (check_decimal_scale_or_error(TIME_SECOND_PART_DIGITS)) > + return true; > + fix_attributes_datetime(decimals); > + return false; > + } > + bool val_native(THD *thd, Native *to) override; > + bool check_vcol_func_processor(void *arg) override > + { > + /* > + NOW is safe for replication as slaves will run with same time as > + master > + */ what does this replication comment do in check_vcol_func_processor? (I know it was in the old code too, but "old code did it" never stopped your cleanups before :) > + return mark_unsupported_function(func_name(), "()", arg, VCOL_TIME_FUNC); > + } > enum Functype functype() const override { return NOW_FUNC; } > Item *get_copy(THD *thd) override > - { return get_item_copy<Item_func_now_local>(thd, this); } > + { return get_item_copy<Item_func_current_timestamp>(thd, this); } > }; > > > diff --git a/sql/item_timefunc.cc b/sql/item_timefunc.cc > index 52b9ae7a682..3e224e3e6f7 100644 > --- a/sql/item_timefunc.cc > +++ b/sql/item_timefunc.cc > @@ -1642,32 +1633,10 @@ void Item_func_now::print(String *str, enum_query_type query_type) > } > > > -int Item_func_now_local::save_in_field(Field *field, bool no_conversions) > +bool Item_func_current_timestamp::val_native(THD *thd, Native *to) > { > - if (field->type() == MYSQL_TYPE_TIMESTAMP) > - { > - THD *thd= field->get_thd(); > - my_time_t ts= thd->query_start(); > - ulong sec_part= decimals ? thd->query_start_sec_part() : 0; > - sec_part-= my_time_fraction_remainder(sec_part, decimals); > - field->set_notnull(); > - field->store_timestamp(ts, sec_part); > - return 0; > - } > - else > - return Item_datetimefunc::save_in_field(field, no_conversions); > -} > - > - > -/** > - Converts current time in my_time_t to MYSQL_TIME representation for local > - time zone. Defines time zone (local) used for whole NOW function. > -*/ > -void Item_func_now_local::store_now_in_TIME(THD *thd, MYSQL_TIME *now_time) > -{ > - thd->variables.time_zone->gmt_sec_to_TIME(now_time, thd->query_start()); > - set_sec_part(thd->query_start_sec_part(), now_time, this); > - thd->used|= THD::TIME_ZONE_USED; > + Timestamp ts(Timeval(thd->query_start(), thd->query_start_sec_part())); > + return null_value= ts.trunc(decimals).to_native(to, decimals); can this possibly fail? > } > > > @@ -2732,7 +2693,6 @@ String *Item_func_tochar::val_str(String* str) > > bool Item_func_from_unixtime::fix_length_and_dec(THD *thd) > { > - thd->used|= THD::TIME_ZONE_USED; would be nice to have a test case for this. Like direct insert into a timestamp column no longer uses timezone - e.g. timestamp within dst switch time is inserted correctly. and vice versa, when from_unixtime is converted to a string, like `insert ... values(concat(from_unixtime(...)))` then TIME_ZONE_USED is still set and show binlog events (or mysqlbinlog) should show that the timezone was correctly binlogged, as before. > tz= thd->variables.time_zone; > Type_std_attributes::set( > Type_temporal_attributes_not_fixed_dec(MAX_DATETIME_WIDTH, Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 1

MariaDB patches in Debian
by Otto Kekäläinen 26 Aug '24

26 Aug '24

Hi! Could perhaps Tuukka or Daniel review the patches currently carried in Debian if some of them would be applicable to be imported or re-implemented directly upstream so they can be dropped in Debian? https://udd.debian.org/patches.cgi?src=mariadb&version=1%3A11.4.3-1 Thanks

1 0

Re: [MariaDB commits] [PATCH 1/6] MDEV-34696: do_gco_wait() completes too early on InnoDB dict stats updates
by andrei.elkin＠pp.inet.fi 23 Aug '24

23 Aug '24

Howdy Kristian. > Before doing mark_start_commit(), check that there is no pending deadlock > kill. If there is a pending kill, we won't commit (we will abort, roll back, > and retry). Then we should not mark the commit as started, since that could > potentially make the following GCO start too early, before we completed the > commit after the retry. > > This condition could trigger in some corner cases, where InnoDB would take > temporarily table/row locks that are released again immediately, not held > until the transaction commits. This happens with dict_stats updates and > possibly auto-increment locks. > > Such locks can be passed to thd_rpl_deadlock_check() and cause a deadlock > kill to be scheduled in the background. But since the blocking locks are > held only temporarily, they can be released before the background kill > happens. This way, the kill can be delayed until after mark_start_commit() > has been called. Thus we need to check the synchronous indication > rgi->killed_for_retry, not just the asynchroneous thd->killed. I think I understood what's going on, also thanks to a verbose Jira ticket description. > > Signed-off-by: Kristian Nielsen <knielsen(a)knielsen-hq.org> > --- > sql/rpl_parallel.cc | 20 ++++++++++++++++---- > sql/rpl_rli.cc | 17 +++++++++++++++++ > 2 files changed, 33 insertions(+), 4 deletions(-) > > diff --git a/sql/rpl_parallel.cc b/sql/rpl_parallel.cc > index 1cfdf96ee3b..9c4222d7817 100644 > --- a/sql/rpl_parallel.cc > +++ b/sql/rpl_parallel.cc > @@ -1450,11 +1450,23 @@ handle_rpl_parallel_thread(void *arg) > after mark_start_commit(), we have to unmark, which has at least a > theoretical possibility of leaving a window where it looks like all > transactions in a GCO have started committing, while in fact one > - will need to rollback and retry. This is not supposed to be possible > - (since there is a deadlock, at least one transaction should be > - blocked from reaching commit), but this seems a fragile ensurance, > - and there were historically a number of subtle bugs in this area. > + will need to rollback and retry. > + > + Normally this will not happen, since the kill is there to resolve a > + deadlock that is preventing at least one transaction from proceeding. > + One case it can happen is with InnoDB dict stats update, which can > + temporarily cause transactions to block each other, but locks are > + released immediately, they don't linger until commit. There could be > + other similar cases, there were historically a number of subtle bugs > + in this area. > + > + But once we start the commit, we can expect that no new lock > + conflicts will be introduced. So by handling any lingering deadlock > + kill at this point just before mark_start_commit(), we should be > + robust even towards spurious deadlock kills. > */ > + if (rgi->killed_for_retry != rpl_group_info::RETRY_KILL_NONE) > + wait_for_pending_deadlock_kill(thd, rgi); Assuming my understanding, please correct me if anything, I left a question on this block in https://github.com/MariaDB/server/commit/df0c36a354ffde2766ebed2615642c74b7… quoted further. ... this guard would not let a victim of an optimistic conflict proceed until it clears out itself it is such a victim. For instance in the binlog sequence like T_1, T_2, D_3 where T_2 is the victim, D_3 is a DDL, the victim T_2 won't anymore release the next gco member D_3 into its execution. But what if at this point T_2 is only going to be marked in rgi->killed_for_retry by T_1? That is it is apparently able, with this patch as well, having ...NONE killed status go through. T_1 would reach this very point later and when that is done before T_2 finds itself killed, T_1 would release D_3, and afterward T_2 would finally see itself KILLed into retry. Cheers, Andrei

2 4

Open PRs pending second approver (has already one)
by Otto Kekäläinen 22 Aug '24

22 Aug '24

Hi! There are couple PRs that have 1 approval, and are pending the second one. Would somebody have time to review them? In order of importance: https://github.com/MariaDB/server/pull/3401 "MDEV-34680 Asynchronous and Buffered Logging in MySQL Audit Plugin" https://github.com/MariaDB/server/pull/3406 (related but zero approvals: https://github.com/MariaDB/server/pull/3324 "MDEV-12182 Add client tcp port to audit plugin logs") https://github.com/MariaDB/server/pull/3208 "Refine OpenSSL test names and allow more complete OpenSSL version coverage" https://github.com/MariaDB/server/pull/3225 "MDEV-29955: add ${PCRE_LIBRARY_PATH} to link directories" Thanks, Otto

1 0

Re: 186aa192d1f: MDEV-34704 Quick mode produces the bug for mariadb client
by Sergei Golubchik 22 Aug '24

22 Aug '24

Hi, Sanja, ok to push, but see one comment below On Aug 22, Oleksandr Byelkin wrote: > revision-id: 186aa192d1f (mariadb-11.5.1-34-g186aa192d1f) > parent(s): 9e1923ca238 > author: Oleksandr Byelkin > committer: Oleksandr Byelkin > timestamp: 2024-08-05 16:16:09 +0200 > message: > > MDEV-34704 Quick mode produces the bug for mariadb client > > --quick-max-column-width parameter added to limit field > width in --quick mode. > > diff --git a/mysql-test/main/client.test b/mysql-test/main/client.test > new file mode 100644 > index 00000000000..59e687ff8b9 > --- /dev/null > +++ b/mysql-test/main/client.test > @@ -0,0 +1,33 @@ > +--echo # > +--echo # MDEV-34704: Quick mode produces the bug for mariadb client > +--echo # > + > + > +create table t1 (aaaaaaaaa char (5), aaaaa char (10), a char (127), b char(1)); > +insert into t1 values ("X", "X", "X", "X"); > + to make the result file more readable, I'd suggest to add here, like --echo mysql --table --quick ... --echo mysql --table --quick --quick-max-column-width=0 ... etc before every corresponding --exec > +--exec echo "select * from test.t1;" | $MYSQL --table --quick 2>&1 > + > +--exec echo "select * from test.t1;" | $MYSQL --table --quick --quick-max-column-width=0 2>&1 > + > +--exec echo "select * from test.t1;" | $MYSQL --table --quick --quick-max-column-width=10 2>&1 > + > +--exec echo "select * from test.t1;" | $MYSQL --table --quick --quick-max-column-width=20 2>&1 > + > +insert into t1 values ("01234", "0123456789", "01234567890123456789", "1"); > + > +--exec echo "select * from test.t1;" | $MYSQL --table --quick 2>&1 > + > + > +--exec echo "select * from test.t1;" | $MYSQL --table --quick --quick-max-column-width=0 2>&1 > + > +--exec echo "select * from test.t1;" | $MYSQL --table --quick --quick-max-column-width=10 2>&1 > + > +--exec echo "select * from test.t1;" | $MYSQL --table --quick --quick-max-column-width=20 2>&1 > + > +drop table t1; > + > +--echo # > +--echo # End of 10.7 tests > +--echo # As Marko said, use here only one line --echo # End of 10.7 tests otherwise merge mixes it up with --echo # --echo # MDEV-34704: Quick mode produces the bug for mariadb client --echo # Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

1 0

Suggestion: use 'main' as the development branch instead of an ever changing 11.x
by Otto Kekäläinen 22 Aug '24

22 Aug '24

Hi! Has the core developers had discussions about using branch name 'main' for development instead of switching to a new branch every 3 months? If all new features and other additions would always target the branch 'main' it would lower the barrier of entry for new contributors, and decrease the number of unnecessary rebases done by contributors. If the permanent development target was 'main', you would need to create new 11.x branches only at "freeze" when a new major version branch is cut for release. MariaDB is currently the only open source project I know that does *not* have a main/master/trunk branch and instead does these constant new branch names. I imagine this is not only annoying for contributors, but also the core developers who waste time asking contributors to rebase PRs every 3 months (see https://github.com/MariaDB/server/pull/2556 and https://github.com/MariaDB/server/pull/2671 as examples). - Otto

7 11