- developers - lists.mariadb.org

review for MDEV-32839 LONG UNIQUE gives error when used with REPLACE
by Alexander Barkov 12 Dec '23

12 Dec '23

Hello Sergei, - This patch: https://github.com/MariaDB/server/commit/4f2d0541b501a7a27bf44aa16821d09395… cleanup: remove innodb-specific code around update_auto_increment() is OK to push. Please write in the commit comments that it was a dead code and that InnoDB team does know why this code was there. - This patch https://github.com/MariaDB/server/commit/c8d0f3370b7d77ecb07f8f1357da6bb1f0… MDEV-32839 LONG UNIQUE gives error when used with REPLACE is OK. I have only a proposal about tests. You added tests above the "# End of 10.5 tests" line, but the patch is for 10.6. Also please add tests for at lease these engines: - MyISAM - InnoDB - Spider as well as partitioned tables for the same engines, because your patch for ha_partition.cc changes the code related to need_info_for_auto_inc(), which has a special implementation in ha_partition and ha_spider. - This patch: https://github.com/MariaDB/server/commit/94f26086663b7c8dbcd6b26930b64de36a… cleanup: remove partition-specific code around update_auto_increment() > diff --git a/sql/ha_partition.cc b/sql/ha_partition.cc > index 25bbcfdd75ce0..e174ef8d21940 100644 > --- a/sql/ha_partition.cc > +++ b/sql/ha_partition.cc > @@ -4308,6 +4308,7 @@ int ha_partition::external_lock(THD *thd, int lock_type) > m_part_info->part_expr->walk(&Item::register_field_in_read_map, 1, 0); > if ((error= m_part_info->vers_set_hist_part(thd))) > goto err_handler; > + need_info_for_auto_inc(); auto-increment code is often done under a condition like this: if (table->next_number_field && buf == table->record[0]) { some_auto_increment code; } This includes the code that you are removing from ha_partition::write_row() Is it ok to call need_info_for_auto_inc() unconditionally here? > } > DBUG_RETURN(0); > > @@ -4623,33 +4624,8 @@ int ha_partition::write_row(const uchar * buf) > */ > if (have_auto_increment) > { > - if (!table_share->next_number_keypart) > - if (unlikely(error= update_next_auto_inc_val())) > - goto exit; Here there was an extra condition !table_share->next_number_keypart before calling update_next_auto_inc_val(). Why was it needed? Why it's not needed in the new version of external_lock() before the call for need_info_for_auto_inc()? Also, the old code checked the return value. The new code int external_code() does not check the result of need_info_for_auto_inc(). Can't the behavior change because of this change? > - > - /* > - If we have failed to set the auto-increment value for this row, > - it is highly likely that we will not be able to insert it into > - the correct partition. We must check and fail if necessary. > - */ > if (unlikely(error= update_auto_increment())) > goto exit; > - > - /* > - Don't allow generation of auto_increment value the partitions handler. > - If a partitions handler would change the value, then it might not > - match the partition any longer. > - This can occur if 'SET INSERT_ID = 0; INSERT (NULL)', > - So allow this by adding 'MODE_NO_AUTO_VALUE_ON_ZERO' to sql_mode. > - The partitions handler::next_insert_id must always be 0. Otherwise > - we need to forward release_auto_increment, or reset it for all > - partitions. > - */ > - if (table->next_number_field->val_int() == 0) > - { > - table->auto_increment_field_not_null= TRUE; > - thd->variables.sql_mode|= MODE_NO_AUTO_VALUE_ON_ZERO; > - } Why is it safe to remove this code? > } > old_map= dbug_tmp_use_all_columns(table, &table->read_set); > error= m_part_info->get_partition_id(m_part_info, &part_id, &func_value); > @@ -11012,10 +10988,7 @@ void ha_partition::get_auto_increment(ulonglong offset, ulonglong increment, > else > { > THD *thd= ha_thd(); > - /* > - This is initialized in the beginning of the first write_row call. > - */ > - DBUG_ASSERT(part_share->auto_inc_initialized); > + update_next_auto_inc_val(); > /* > Get a lock for handling the auto_increment in part_share > for avoiding two concurrent statements getting the same number. > diff --git a/sql/ha_partition.h b/sql/ha_partition.h > index d75e38ec272a5..0ac8f696fcf55 100644 > --- a/sql/ha_partition.h > +++ b/sql/ha_partition.h > @@ -1408,9 +1408,8 @@ class ha_partition final :public handler > { > ulonglong nr= (((Field_num*) field)->unsigned_flag || > field->val_int() > 0) ? field->val_int() : 0; > + update_next_auto_inc_val(); > lock_auto_increment(); > - DBUG_ASSERT(part_share->auto_inc_initialized || > - !can_use_for_auto_inc_init()); > /* must check when the mutex is taken */ > if (nr >= part_share->next_auto_inc_val) > part_share->next_auto_inc_val= nr + 1;

2 2

MDEV-31404 Review
by Kristian Nielsen 11 Dec '23

11 Dec '23

Hi Monty, here is my review of the patch for MDEV-31404: > commit d51086a9f1b1b6a66c7ba8eb6a6876b433fa118f (origin/bb-11.4-MDEV-31404) > Author: Monty <monty(a)mariadb.org> > Date: Sun Dec 3 21:42:44 2023 +0200 > > MDEV-31404 Implement binlog_space_limit > > binlog_space_limit is a variable in Percona server used to limit the total > size of all binary logs. > > This implementation is based on code from Percona server 5.7. General comment: The --slave-connections-needed-for-purge defaults to 1. I think this means that --log-expire-days will have no effect on systems that have binlog enabled but no slaves configured. This is something to be aware of for backwards compatibility. What I do not like about this is the consequence it will have for all systems that have binlog enabled with some --expire-log-days set but no slaves configured. After upgrade, these systems will silently start accumulating binlogs without limit, possibly filling up disk eventually. On the other hand, it seems good to have --slave-connections-needed-for-purge enabled for the new --max-binlog-total-size option; as you pointed out, otherwise a single large transaction could immediately cause deletion of all binlogs when no slaves are connected. And having --slave-connections-needed-for-purge affect purge from --max-binlog-total-size but not for --expire-log-days seems inconsistent. In the end, maybe it isn't too serious. Many small and default-configured systems will probably be able to run for ages without binlogs getting excessive due to low traffic; and it will be easy for DBAs to just delete the binlogs if needed, or google the --slave-connections-needed-for-purge if they care enough. Two suggestions to make it easier for the DBAs to know how to handle this: 1. How about emitting a note or warning in the error log, if purge from --log-expire-days=N gets delayed for more than 2*N because no slaves are connected and --slave-connections-needed-for-purge=1 ? Note: Purge of binlogs prevented for <seconds> because no slaves are connected. To allow purge of binlogs with no slaves connected, set --slave-connections-needed-for-purge=0 2. Mention in the changelogs for 11.4 that binlogs will by default no longer be purged when no slaves are configured, and point to some documentation about how to handle this for the DBA. I think with those two suggestions it should be fine. Detailed comments on the patch below. - Kristian. > diff --git a/mysql-test/suite/binlog_encryption/binlog_index-master.opt b/mysql-test/suite/binlog_encryption/binlog_index-master.opt > new file mode 100644 > index 00000000000..f2673d08020 > --- /dev/null > +++ b/mysql-test/suite/binlog_encryption/binlog_index-master.opt > @@ -0,0 +1 @@ > +--slave_connections_needed_for_purge=0 Is this needed? If so, would it be better to put this in a general my.cnf for the entire binlog_encryption suite, like you did for mysql-test/suite/binlog/my.cnf ? So that a similar .opt will not need to be added manually for each new binlog_encryption test that will need it. > --- a/mysql-test/suite/perfschema/t/show_sanity.test > +++ b/mysql-test/suite/perfschema/t/show_sanity.test > @@ -533,6 +533,7 @@ insert into test.sanity values > ("JUNK: GLOBAL-ONLY", "I_S.SESSION_VARIABLES", "MAX_BINLOG_CACHE_SIZE"), > ("JUNK: GLOBAL-ONLY", "I_S.SESSION_VARIABLES", "MAX_BINLOG_SIZE"), > ("JUNK: GLOBAL-ONLY", "I_S.SESSION_VARIABLES", "MAX_BINLOG_STMT_CACHE_SIZE"), > + ("JUNK: GLOBAL-ONLY", "I_S.SESSION_VARIABLES", "MAX_BINLOGS_SIZE"), I think this is a left-over from previous naming, should probably be "MAX_BINLOG_TOTAL_SIZE ? > +FLUSH LOGS; > +FLUSH LOGS; > +FLUSH LOGS; > +source include/show_binary_logs.inc; > +show status like "binlog_disk_use"; I think you will need --source include/wait_for_binlog_checkpoint.inc after each of these FLUSH LOGS. Otherwise random delay of binlog checkpoint could cause the binlog_disk_use value to fluctuate and get rare random test failures. (Or alternatively consider if you could omit the select of binlog_disk_use completely, since it will probably differ between versions as event formats change.) Adding a small sleep in binlog_background_thread() just before calling mysql_bin_log.mark_xid_done() can usually reproduce the random failures from delayed binlog checkpoint. > +INSERT INTO t1 VALUES (0,repeat("a",3000)); > +show status like "binlog_disk_use"; > +Variable_name Value > +Binlog_disk_use 3848 > +# First binary should be binary.000004 > +show binary logs; > +Log_name File_size > +binary.000004 # > +INSERT INTO t1 VALUES (2,repeat("b",10)); > +# First binary should be binary.000004 > +show binary logs; > +Log_name File_size > +binary.000004 # > +binary.000005 # Here, I don't understand why binary.000004 is not purged. The binary.000004 contains the large transaction 'repeat("a", 3000)', so the binlog total size is too large. And the transaction 'repeat("n",10)' caused a binlog rotation. So why does that not cause binary.000004 to be purged due to --max-binlog-total-size=1500 ? > +show status like "binlog_disk_use"; > +Variable_name Value > +Binlog_disk_use 3647 > +INSERT INTO t1 VALUES (7,repeat("g",3000)); > +# binary.000001 should be deleted now > +show binary logs; > +Log_name File_size > +binary.000002 # > +binary.000003 # Similarly here, why binary.000002 is not purged? > +INSERT INTO t1 VALUES (4,repeat("d",3000)); > +--echo # First binary should be binary.000011 > +source include/show_binary_logs.inc; > + > +RESET MASTER; Idea: Here you could (or ask one of the other devs) add a test that the SHOW STATUS LIKE "binlog_disk_use" is equal to the size of the files on disk, to check that the running calculation of disk use doesn't miss the odd byte or two. > + if ((longlong) (binlog_space_total + binlog_pos - found_space <= > + binlog_space_limit)) > + break; // Found enough The (longlong) cast here looks misplaced. It's casting the result of the (X <= Y) comparison. I think the cast can just be removed, all the variables seem to be ulonglong type. > +static bool waiting_for_slave_to_change_binlog= 0; > +static ulonglong purge_sending_new_binlog_file= 0; I think a short comment here on these variables would be good, explaining their purpose. And same on the declaration of Atomic_counter<ulonglong> sending_new_binlog_file. > + DBUG_ASSERT(!is_relay_log || binlog_xid_count_list.is_empty()); > + if (!is_relay_log) > { > - I_List_iterator<xid_count_per_binlog> it(binlog_xid_count_list); > - while ((b= it++) && > - 0 != strncmp(log_file_name_arg+dirname_length(log_file_name_arg), > - b->binlog_name, b->binlog_name_len)) > - ; > + xid_count_per_binlog *b; > + mysql_mutex_lock(&LOCK_xid_list); Good catch that we were uselessly doing xid_count operations on the relay log, thanks! > +int MYSQL_BIN_LOG::count_binlog_space() > +{ > + int error; > + LOG_INFO log_info; > + DBUG_ENTER("count_binlog_space"); > + if (is_relay_log || !binlog_space_limit) > + DBUG_RETURN(0); Maybe this if (is_relay_log || !binlog_space_limit) could be an assert? We probably should not be calling this function at all in that case. It looks like that binlog_space_limit is already checked in all callers, not sure about relay log. (A small point, just something to consider as you like). > @@ -7608,7 +7807,8 @@ int MYSQL_BIN_LOG::rotate(bool force_rotate, bool* check_purge) > @retval > nonzero - error in rotating routine. > */ > -void MYSQL_BIN_LOG::purge() > + > +void MYSQL_BIN_LOG::purge(bool all) Would be good with a small comment here explaining the meaning of the new parameter `all`. Minor stuff: Commit message: > Binlog file file sizes are checked on startup Typo (s/file file/file/). > when updating the binary log on on binary log rotation Typo (s/on on/and on/ ?) > Without this option max-binlog-total-size would potentially deleted > binlogs needed by slaves on server startup or when a slave disconnects Typo s/deleted/delete/. > +static Sys_var_ulonglong Sys_max_binlog_total_size( > + "max_binlog_total_size", > + "Maximum space to use for all binary logs. Extra logs are deleted on " > + "server start, log rotation, FLUSH LOGGS or when writing to binlog. " > + "Default is 0, which means no size restrictions." > + "See also slave_connections_needed_for_purge", Missing space at end of line (s/restrictions."/restrictions. "/). > +static Sys_var_ulong Sys_slave_connections_needed_for_purge( > + "slave_connections_needed_for_purge", > + "Number of minimum connected slaves needed for automatic binary " > + "log purge with max_binlog_total_size, binlog_expire_logs_seconds " > + "or binlog_expire_logs_days.", Suggestion for better wording: "Minimum number of connected slaves required for automatic binary " "log purge with max_binlog_total_size, binlog_expire_logs_seconds " "or binlog_expire_logs_days."

2 2

Re: d25ab9fdfa1: MDEV-31608 - Connector/NET fails to connect since 10.10
by Sergei Golubchik 30 Nov '23

30 Nov '23

Hi, Vladislav, On Nov 29, Vladislav Vaintroub wrote: > revision-id: d25ab9fdfa1 (mariadb-10.11.2-62-gd25ab9fdfa1) > parent(s): 034848c6c27 > author: Vladislav Vaintroub > committer: Vladislav Vaintroub > timestamp: 2023-10-08 19:37:20 +0200 > message: > > MDEV-31608 - Connector/NET fails to connect since 10.10 > > Connector/NET does not expect collation IDs returned by "show collations" > to be NULL, runs into an exception. > > The fix is to determine connector/net using its connection attributes, > exclude null-id collations from "show collations".The patch introduces > new old_mode NO_NULL_COLLATION_IDs, that is automatically set, > once MySQL Connector/NET connection is determined. I don't like this hack at all, but I don't have any better ideas, so just one question below > A test was added, that uses MySql.Data from powershell - only works > if MySql.Data is installed into GAC (i.e with C/NET MSI package) please, make sure that MySql.Data is installed on at least one buildbot builder. > diff --git a/sql/sql_show.cc b/sql/sql_show.cc > index bff55f7a431..bea993305be 100644 > --- a/sql/sql_show.cc > +++ b/sql/sql_show.cc > @@ -6386,7 +6386,8 @@ int fill_schema_collation(THD *thd, TABLE_LIST *tables, COND *cond) > character set prefix). > */ > if (is_context && > - cmp(tmp_cl->cs_name, Lex_cstring(STRING_WITH_LEN("utf8mb4")))) > + ((thd->variables.old_behavior & OLD_MODE_NO_NULL_COLLATION_IDS) || > + cmp(tmp_cl->cs_name, Lex_cstring(STRING_WITH_LEN("utf8mb4"))))) wouldn't it be better to do instead - bool is_context= cmp(context_collation_name, full_collation_name); + bool is_context= cmp(context_collation_name, full_collation_name) + && !(thd->variables.old_behavior & OLD_MODE_NO_NULL_COLLATION_IDS); ? > continue; > restore_record(table, s->default_values); > table->field[0]->store(context_collation_name, scs); Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 1

Re: [PATCH] MDEV-32551: "Read semi-sync reply magic number error" warnings on master
by Kristian Nielsen 23 Nov '23

23 Nov '23

Hi Monty, > From: Michael Widenius <monty(a)mariadb.com> > > commit 7abf83f86829ed94de618af4749eb96d5379bd90 (origin/bb-10.6-monty) > Author: Michael Widenius <monty(a)mariadb.com> > Date: Wed Nov 8 16:57:58 2023 -0700 > > MDEV-32551: "Read semi-sync reply magic number error" warnings on master I read through the patch, looks good. A few comments below, mostly minor things and typos that you can consider as you like. - Kristian. > - We should in 11.3 changed the default value for > rpl-semi-sync-master-wait-no-slave from TRUE to FALSE as the TRUE > Sys_semisync_master_wait_no_slave( > "rpl_semi_sync_master_wait_no_slave", > - "Wait until timeout when no semi-synchronous replication slave " > - "available (enabled by default).", > + "Wait until timeout when no semi-synchronous replication slave is " > + "available.", > GLOBAL_VAR(rpl_semi_sync_master_wait_no_slave), > CMD_LINE(OPT_ARG), DEFAULT(TRUE), > - NO_MUTEX_GUARD, NOT_IN_BINLOG, ON_CHECK(0), > - ON_UPDATE(fix_rpl_semi_sync_master_wait_no_slave)); > + NO_MUTEX_GUARD, NOT_IN_BINLOG, ON_CHECK(0)); > -my_bool rpl_semi_sync_master_wait_no_slave = 1; > +my_bool rpl_semi_sync_master_wait_no_slave = 0; There seems to be som inconsistency here: - The commit is based on 10.6, but the commit comment says 11.3 - The option help text is changed in the commit - The "DEFAULT(TRUE)" in the sysvar definition is _not_ changed in the commit - The C initialization values of the my_bool rpl_semi_sync_master_wait_no_slave _is_ changed in the commit. Should it be DEFAULT(FALSE) also in this patch, and the commit message say "10.6"? Or should these changes be removed from the 10.6 patch and put in 11.3 instead? At least this looks inconsistent. > diff --git a/mysql-test/include/wait_for_pattern_in_file.inc b/mysql-test/include/wait_for_pattern_in_file.inc > new file mode 100644 > index 00000000000..d088cb125c2 > --- /dev/null > +++ b/mysql-test/include/wait_for_pattern_in_file.inc > @@ -0,0 +1,54 @@ > +# ==== Purpose ==== > +# > +# Waits until pattern comes into log file or until a timeout is reached. Nice addition, this will be useful for other things! > +--let $keep_include_silent= 1 Why do you set this in the .inc? (Normally this is set in the .test files that want to skip the printout). +let SEARCH_SILENT=1; Maybe the .inc should restore the old value at the end (in case a .test will at some point set it globally for its own calls to include/search_pattern_in_file.inc) > +--let $keep_include_silent= 0 Here, better restore the original value (in case some .test has set it to 1 for itself). > +# Test is binlog format indepdendent, so save resources (Typo s/indepdendent/independent/) > +# The followowing command is likely to cause the slave master is not yet setup > +# for semi-sync (Typo s/followowing/following/) > - strcpy(m_reply_file_name, ""); > - strcpy(m_wait_file_name, ""); > + m_reply_file_name[0]= m_wait_file_name[0]= 0; Interestingly, GCC actually creates identical code for these (it knows to optimize strcpy()). (I'm just mentioning this as I know we both are interested in low-level code details, not asking you to revert the change). Example code: ----------------------------------------------------------------------- #include <string.h> struct X { char m_a[100]; char m_b[10]; void init1(); void init2(); }; void X::init1() { strcpy(m_a, ""); strcpy(m_b, ""); } void X::init2() { m_a[0]= m_b[0]= 0; } ----------------------------------------------------------------------- Result from gcc -O3: ----------------------------------------------------------------------- .type _ZN1X5init1Ev, @function _ZN1X5init1Ev: movb $0, (%rdi) movb $0, 100(%rdi) ret .type _ZN1X5init2Ev, @function _ZN1X5init2Ev: movb $0, 100(%rdi) movb $0, (%rdi) ret ----------------------------------------------------------------------- Identical, except that we now initialize in the opposite order ;-) > +#ifndef DBUG_OFF > +void dbug_verify_no_duplicate_slaves(Slave_ilist *m_slaves, THD *thd) > +{ You could make this (debug-only) function "static". > + Master is not responding (gone away?) or it has turned semi sync > + off. Turning off semi-sync repsonces as there is no point in sending (Typo s/repsonces/responses/)

2 1

Re: dadcf5ed7b1: MDEV-24486 Add `table_privileges` view to the `sys` schema
by Sergei Golubchik 21 Nov '23

21 Nov '23

Hi, Oleg, ok to push. Just one comment below. On Nov 20, Oleg Smirnov wrote: > revision-id: dadcf5ed7b1 (mariadb-11.0.1-259-gdadcf5ed7b1) > parent(s): b8f9f796ff8 > author: Oleg Smirnov > committer: Oleg Smirnov > timestamp: 2023-11-20 19:35:36 +0700 > message: > > MDEV-24486 Add `table_privileges` view to the `sys` schema > > The existing INFORMATION_SCHEMA.TABLE_PRIVILEGES displays only those > privileges that were specifically granted on the table level, > whereas it may be useful to see privileges granted at the database > and global level. > > diff --git a/scripts/sys_schema/NEWS.md b/scripts/sys_schema/NEWS.md > index 7aeefb7edf9..12ce1382b75 100644 > --- a/scripts/sys_schema/NEWS.md > +++ b/scripts/sys_schema/NEWS.md > @@ -1,5 +1,12 @@ > # Change history for the MySQL sys schema > > +## 1.5.2 (11/20/23) > + > +### Improvements > + > +* A new `table_privileges` view was added, which displays privileges to tables granted at all levels (user, schema, table) is it "privileges to tables"? I feel like "privileges on tables" would be more correct. It's GRANT privilege ON object TO user same it two other places. > + > + > ## 1.5.1 (07/07/16) Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

3 3

Question about STACK_DIRETION fix removal in merge commit 6c05edf
by Otto Kekäläinen 20 Nov '23

20 Nov '23

Hi Marko! I noticed you removed from debian/rules a section that flipped -DSTACK_DIRECTION=-1/1 based on architecture in the merge commit https://github.com/MariaDB/server/commit/6c05edfdcd335b9587ba140ebef5d9b082… Was this intentional? I am not sure, but it looks like a mistake to me, so I wanted to confirm. The code originates from cross-compilation fix in Debian in https://salsa.debian.org/mariadb-team/mariadb-server/-/commit/f0ba31e1565fe…, which was later upstreamed to 10.5 (but not on 10.4) in https://github.com/MariaDB/server/commit/ea56841997995283f04e85e957c8d7f315…. - Otto

2 1

Re: 73097e18b77: MDEV-32771 Server crash upon online alter with concurrent XA
by Sergei Golubchik 14 Nov '23

14 Nov '23

Hi, Nikita, ok to push. spelling and renaming suggestions below On Nov 13, Nikita Malyavin wrote: > revision-id: 73097e18b77 (mariadb-11.2.1-20-g73097e18b77) > parent(s): f7646d890b9 > author: Nikita Malyavin > committer: Nikita Malyavin > timestamp: 2023-11-13 15:28:19 +0100 > message: > > MDEV-32771 Server crash upon online alter with concurrent XA > > In case of a non-recovery XA rollback/commit in the same connection, > thon->rollback is called instead of rollback_by_xid, Though reviously, "previously" > thd_ha_data was moved to thd->transaction->xid_state.xid in hton->prepare. > > Like it wasn't enough, XA PREPARE can be skipped upon user and thus we > can end up in hton->commit/rollback with and unprepared XA, so checking > xid_state.is_explicit_XA is not enough -- we should check > xid_state.get_state_code() == XA_PREPARED, which will also guarantee > is_explicit_XA() == true. > --- a/sql/online_alter.cc > +++ b/sql/online_alter.cc > @@ -315,17 +315,35 @@ int online_alter_savepoint_rollback(handlerton *hton, THD *thd, sv_id_t sv_id) > > static int online_alter_commit(handlerton *hton, THD *thd, bool all) > { > - int res= online_alter_end_trans(get_cache_list(hton, thd), thd, > - ending_trans(thd, all), true); > - cleanup_tables(thd); > + int res; > + bool commit= ending_trans(thd, all); > + if (thd->transaction->xid_state.get_state_code() == XA_PREPARED && commit) > + { > + res= hton->commit_by_xid(hton, thd->transaction->xid_state.get_xid()); > + // cleanup is done by prepare() this is conusing, there's no prepare after commit_by_xid. may be you wanted to say "cleanup was already done by prepare()" ? > + } > + else > + { > + res= online_alter_end_trans(get_cache_list(hton, thd), thd, commit, true); > + cleanup_tables(thd); > + } > return res; > }; > > static int online_alter_rollback(handlerton *hton, THD *thd, bool all) > { > - int res= online_alter_end_trans(get_cache_list(hton, thd), thd, > - ending_trans(thd, all), false); > - cleanup_tables(thd); > + int res; > + bool commit= ending_trans(thd, all); 'commit' is weird here, on the rollback path. may be you rename it to, like "is_ending_tran" or "end_tran" or "is_persistent" or something along these lines. And then 'commit' in online_alter_commit too, for consistency. > + if (thd->transaction->xid_state.get_state_code() == XA_PREPARED && commit) > + { > + res= hton->rollback_by_xid(hton, thd->transaction->xid_state.get_xid()); > + // cleanup is done by prepare() same as in online_alter_commit > + } > + else > + { > + res= online_alter_end_trans(get_cache_list(hton, thd), thd, commit, false); > + cleanup_tables(thd); > + } > return res; > }; Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

1 0

Re: MDEV-27576 Use reverse index for max/min optimization
by Sergey Petrunia 13 Nov '23

13 Nov '23

Hi Yuchen, > commit c4c4bd8e47e5bfd4d0a650e286ea9daa3e6276f0 > Author: Yuchen Pei <ycp(a)mariadb.com> > Date: Thu Nov 2 11:55:42 2023 +1100 > > MDEV-27576 Use reverse index for max/min optimization > > We add a field in st_table_ref to indicate that the key used is > a descending index, which will flip the functions and flags used in > get_index_max_value() and get_index_min_value(), that allows correct > optimization for max/min for descending index. Generally the patch looks good. I have only a few cosmetic comments. Please rename the test file from mdev_27576.test to something more descriptive like desc_index_min_max.test (following desc_index_range.test) Also please rename the table in the test to t1. > diff --git a/sql/sql_select.h b/sql/sql_select.h > index e0bbadadc69..45d7527280a 100644 > --- a/sql/sql_select.h > +++ b/sql/sql_select.h > @@ -126,6 +126,7 @@ typedef struct st_table_ref > uint key_parts; ///< num of ... > uint key_length; ///< length of key_buff > int key; ///< key no > + bool reverse; ///< key is reverse > uchar *key_buff; ///< value to look for with key > uchar *key_buff2; ///< key_buff+key_length > store_key **key_copy; // It's actually not "key is reverse" but "last used key part is reverse"... also this member is never set for the most common scenario of TABLE_REF use, which is ref access. The member is set in a rather trivial logic in find_key_for_maxmin(), and then it's used in get_index_min_value() and get_index_max_value(). I suggest that we do without this member. get_index_min_value() and get_index_max_value() can check: bool reverse= table->key_info[ref->key].key_part[ref->key_part]->key_part_flag & HA_REVERSE_SORT > diff --git a/sql/opt_sum.cc b/sql/opt_sum.cc > index 794ec40fc66..1f94a4f4dc7 100644 > --- a/sql/opt_sum.cc > +++ b/sql/opt_sum.cc > @@ -114,7 +114,8 @@ static int get_index_min_value(TABLE *table, TABLE_REF *ref, > int error; > > if (!ref->key_length) > - error= table->file->ha_index_first(table->record[0]); > + error= ref->reverse ? table->file->ha_index_last(table->record[0]) : > + table->file->ha_index_first(table->record[0]); Please add { ... } as the if-branch is a multi-line statement now. > else > { > /* > @@ -206,8 +210,12 @@ static int get_index_max_value(TABLE *table, TABLE_REF *ref, uint range_fl) > table->file->ha_index_read_map(table->record[0], ref->key_buff, > make_prev_keypart_map(ref->key_parts), > range_fl & NEAR_MAX ? > - HA_READ_BEFORE_KEY : > - HA_READ_PREFIX_LAST_OR_PREV) : > + (ref->reverse ? HA_READ_AFTER_KEY : > + HA_READ_BEFORE_KEY) : > + (ref->reverse ? HA_READ_KEY_OR_NEXT : > + HA_READ_PREFIX_LAST_OR_PREV)) : > + ref->reverse ? > + table->file->ha_index_first(table->record[0]) : > table->file->ha_index_last(table->record[0])); Multiple levels of x?y:z statements are hard to read. Please change to if (ref->key_length) { ... } else { ... } BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Blog: http://petrunia.net

2 2

Re: 28deba0a611: MDEV-32101 CREATE PACKAGE [BODY] for sql_mode=DEFAULT
by Sergei Golubchik 10 Nov '23

10 Nov '23

Hi, Alexander, Looks great! Just a couple of questions below On Nov 07, Alexander Barkov wrote: > revision-id: 28deba0a611 (mariadb-11.0.1-239-g28deba0a611) > parent(s): e9573c05965 > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2023-09-22 14:38:52 +0400 > message: > > MDEV-32101 CREATE PACKAGE [BODY] for sql_mode=DEFAULT > > This patch adds PACKAGE support with SQL/PSM dialect for sql_mode=DEFAULT: > > - CREATE PACKAGE > - DROP PACKAGE > - CREATE PACKAGE BODY > - DROP PACKAGE BODY > - Package function and procedure invocation from outside of the package: > -- using two step identifiers > SELECT pkg.f1(); > CALL pkg.p1() > > -- using three step identifiers > SELECT db.pkg.f1(); > CALL db.pkg.p1(); > > This is a non-standard MariaDB extension. > > However, later this code can be used to implement > the SQL Standard and DB2 dialects of CREATE MODULE. > > diff --git a/sql/sql_yacc.yy b/sql/sql_yacc.yy > index af2ec678daa..fca39b63312 100644 > --- a/sql/sql_yacc.yy > +++ b/sql/sql_yacc.yy > @@ -1285,7 +1285,7 @@ bool my_yyoverflow(short **a, YYSTYPE **b, size_t *yystacksize); > %left TRANSACTION_SYM TIMESTAMP PERIOD_SYM SYSTEM USER COMMENT_SYM > > %left PREC_BELOW_SP_OBJECT_TYPE > -%left FUNCTION_SYM > +%left PACKAGE_MARIADB_SYM FUNCTION_SYM Why PACKAGE_ORACLE_SYM and PACKAGE_ALLMODES_SYM don't need it? (same for BODY_ORACLE_SYM and BODY_ALLMODES_SYM) > > > /* > @@ -3049,6 +3059,8 @@ sp_handler: > | PROCEDURE_SYM { $$= &sp_handler_procedure; } > | PACKAGE_ORACLE_SYM { $$= &sp_handler_package_spec; } > | PACKAGE_ORACLE_SYM BODY_ORACLE_SYM { $$= &sp_handler_package_body; } > + | PACKAGE_MARIADB_SYM { $$= &sp_handler_package_spec; } > + | PACKAGE_MARIADB_SYM BODY_MARIADB_SYM { $$= &sp_handler_package_body; } Why not to use here PACKAGE_ALLMODES_SYM and BODY_ALLMODES_SYM? (also, I'd personally call them simply PACKAGE_SYM and BODY_SYM) > ; > > > @@ -19428,35 +19489,33 @@ create_routine: > (Item_result) $8, $10)) > MYSQL_YYABORT; > } > - | create_or_replace definer_opt PACKAGE_ORACLE_SYM > + | create_or_replace definer_opt PACKAGE_ALLMODES_SYM > opt_if_not_exists sp_name opt_create_package_chistics_init > { > sp_package *pkg; > if (unlikely(!(pkg= Lex-> > - create_package_start(thd, > - SQLCOM_CREATE_PACKAGE, > - &sp_handler_package_spec, > - $5, $1 | $4)))) > + create_package_start(thd, &sp_handler_package_spec, > + $5, $1 | $4, > + Lex->sp_chistics)))) > MYSQL_YYABORT; > pkg->set_c_chistics(Lex->sp_chistics); Looks redundant > Lex->sphead->set_body_start(thd, YYLIP->get_cpp_tok_start()); > } > sp_tail_is > opt_package_specification_element_list END > - remember_end_opt opt_sp_name > + remember_end_opt opt_trailing_sp_name > { > if (unlikely(Lex->create_package_finalize(thd, $5, $12, $11))) > MYSQL_YYABORT; > } > - | create_or_replace definer_opt PACKAGE_ORACLE_SYM BODY_ORACLE_SYM > + | create_or_replace definer_opt PACKAGE_ALLMODES_SYM BODY_ALLMODES_SYM > opt_if_not_exists sp_name opt_create_package_chistics_init > { > sp_package *pkg; > if (unlikely(!(pkg= Lex-> > - create_package_start(thd, > - SQLCOM_CREATE_PACKAGE_BODY, > - &sp_handler_package_body, > - $6, $1 | $5)))) > + create_package_start(thd, &sp_handler_package_body, > + $6, $1 | $5, > + Lex->sp_chistics)))) > MYSQL_YYABORT; > pkg->set_c_chistics(Lex->sp_chistics); same > Lex->sphead->set_body_start(thd, YYLIP->get_cpp_tok_start()); Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 2

Re: 2342aaf46eb: MDEV-30973 Implement debug logging for slave_transaction_retries
by Sergei Golubchik 07 Nov '23

07 Nov '23

Hi, Aleksey, On Nov 07, Aleksey Midenkov wrote: > revision-id: 2342aaf46eb (mariadb-11.0.1-256-g2342aaf46eb) > parent(s): 80bc5df1c80 > author: Aleksey Midenkov > committer: Aleksey Midenkov > timestamp: 2023-10-10 13:11:19 +0300 > message: > > MDEV-30973 Implement debug logging for slave_transaction_retries > > Configuration options: > > log_slave_retries > log_slave_retries_max 1. a commit comment of >100 lines that's a bit too much, I think. 2. comments in MDEV-30973 say that one should be able to have everything in one file, namely in the error log. I don't see how it's possible in your commit. 3. one of the reasons for not simply writing into erorr log was "you get what are relatively large entries in there which can make it hard/annoying to also look for the normal, one-line warning and error output". But as far as I can see your slave retry entries are not large, but normal one-line messages. I'd say they can fit into the error log just fine. I've still commented below in the assumption that the separate log implementation will stay. But I honestly don't see a reason for it anymore. > diff --git a/mysql-test/suite/rpl/t/slave_deadlock.test b/mysql-test/suite/rpl/t/slave_deadlock.test > new file mode 100644 > index 00000000000..9fcda7c9b71 > --- /dev/null > +++ b/mysql-test/suite/rpl/t/slave_deadlock.test > @@ -0,0 +1,194 @@ > +--source include/have_debug.inc > +--source include/have_innodb.inc > +--source include/have_binlog_format_statement.inc > + > +# On --repeat test infinitely increments thread_id and finally fails --sync_with_master at log_pos 1754: > +# > +# 231009 20:58:15 server id 1 end_log_pos 1754 CRC32 0x7e5aecd1 Query thread_id=561 exec_time=0 error_code=0 xid=0 I'm sorry, I don't understand that. Could you elaborate please? > +# > +# force_restart seems to fix this > + > +--source include/force_restart.inc > + > +# Note that test requires big enough --debug-sync-timeout unfair scheduling at high load. Sorry, cannot parse that. Could you elaborate please? > +# slave_deadlock.opt sets this to 1 hour > + > +--let $rpl_topology=1->2 > +--source include/rpl_init.inc > + > +--echo # > +--echo # MDEV-30973 Enhance output of innodb deadlocks (SAMU-124, SAMU-131) > +--echo # > +--connection server_2 > +SET @old_dbug= @@global.debug_dbug; > +set @old_parallel_threads= @@global.slave_parallel_threads; > +set @old_domain_parallel_threads= @@global.slave_domain_parallel_threads; > +set @old_parallel_mode= @@global.slave_parallel_mode; > +set @old_slave_transaction_retries= @@global.slave_transaction_retries; > +set @old_innodb_lock_wait_timeout= @@global.innodb_lock_wait_timeout; > +set @old_log_slave_retries= @@global.log_slave_retries; > +set @old_log_slave_retries_file= @@global.log_slave_retries_file; > +set @old_log_slave_retries_max= @@global.log_slave_retries_max; All this makes little sense, it's simpler to put all settings into the .opt file generally it's better to do your way, to have less server restarts, but you have an .opt file already, you can as well use it. except if you decide to remove the .opt file - then nevermind, the above it good. > +--source include/stop_slave.inc > +change master to master_use_gtid= slave_pos; > +set global slave_parallel_threads= 10; > +set global slave_domain_parallel_threads= 1; > +set global slave_parallel_mode= optimistic; > +set global slave_transaction_retries= 4; > +set global innodb_lock_wait_timeout= 1; ... > diff --git a/mysql-test/suite/versioning/r/rpl_row.result b/mysql-test/suite/versioning/r/rpl_row.result > index c161f68d0bd..6343fbecdec 100644 > --- a/mysql-test/suite/versioning/r/rpl_row.result > +++ b/mysql-test/suite/versioning/r/rpl_row.result > @@ -37,12 +37,7 @@ select *, unix_timestamp(row_start) as row_start, unix_timestamp(row_end) as row > x y row_start row_end > 1 1 12345.000000 2147483647.999999 > set timestamp= default; > -### INSERT INTO `test`.`t1` > -### SET > -### @1=1 > -### @2=1 > -### @3=12345.000000 > -### @4=2147483647.999999 why? > +/*!50530 SET @@SESSION.PSEUDO_SLAVE_MODE=1*/; > connection slave; > select * from t1; > x y > diff --git a/sql/log.cc b/sql/log.cc > index 877315c60de..78e874fda5b 100644 > --- a/sql/log.cc > +++ b/sql/log.cc > @@ -1188,9 +1188,17 @@ bool Log_to_file_event_handler:: > log_error(enum loglevel level, const char *format, > va_list args) > { > - return vprint_msg_to_log(level, format, args); > + return vprint_msg_to_log(stderr, level, format, args); > } > > +#ifdef HAVE_REPLICATION > +bool Log_to_file_event_handler::log_slave_retry(const char *format, va_list args) > +{ > + DBUG_ASSERT(slave_retries_file.get()); > + return vprint_msg_to_log(slave_retries_file.get(), NO_LEVEL, format, args); I don't think you should use vprint_msg_to_log and add a NO_LEVEL hack. vprint_msg_to_log also prints into the windows event log, into wsrep json log, and it locks LOCK_error_log. You don't need any of that, just create your own function, it's just one fprintf() anyway. > +} > +#endif > + > void Log_to_file_event_handler::init_pthread_objects() > { > mysql_log.init_pthread_objects(); > @@ -9752,13 +9787,15 @@ static void print_buffer_to_file(enum loglevel level, const char *buffer, > } > } > > +#ifndef EMBEDDED_LIBRARY > mysql_mutex_lock(&LOCK_error_log); > +#endif > > skr= my_time(0); > localtime_r(&skr, &tm_tmp); > start=&tm_tmp; > > - fprintf(stderr, "%d-%02d-%02d %2d:%02d:%02d %lu [%s] %.*s%.*s\n", > + fprintf(file, "%d-%02d-%02d %2d:%02d:%02d [T%lu] %s%.*s%.*s\n", eh... no. don't change the format of existing log files, please. > start->tm_year + 1900, > start->tm_mon+1, > start->tm_mday, > @@ -9834,6 +9872,22 @@ void sql_print_error(const char *format, ...) > } > > > +#ifdef HAVE_REPLICATION > +void sql_print_error2(const char *format, ...) s/sql_print_error2/print_slave_error/ ? > +{ > + va_list args; > + DBUG_ENTER("sql_print_error"); > + > + va_start(args, format); > + error_log_print(ERROR_LEVEL, format, args); > + slave_retries_print(format, args); > + va_end(args); > + > + DBUG_VOID_RETURN; > +} > +#endif > + > + > void sql_print_warning(const char *format, ...) > { > va_list args; > diff --git a/sql/mysqld.cc b/sql/mysqld.cc > index 8cc3ad51ee9..4f6b2273597 100644 > --- a/sql/mysqld.cc > +++ b/sql/mysqld.cc > @@ -765,6 +765,90 @@ char *opt_relay_logname = 0, *opt_relaylog_index_name=0; > char *opt_logname, *opt_slow_logname, *opt_bin_logname; > char *opt_binlog_index_name=0; > > +#ifdef HAVE_REPLICATION > +char default_slave_retries_path[FN_REFLEN]; > +bool opt_slave_retries= false; > +char *opt_slave_retries_path= NULL; > +uint opt_slave_retries_max_log= 0; > +#endif > +Slave_retries_file slave_retries_file; > + > +bool Slave_retries_file::open() > +{ > +#ifdef HAVE_REPLICATION > + DBUG_ASSERT(opt_slave_retries); > + DBUG_ASSERT(opt_slave_retries_path); > + file= fopen(opt_slave_retries_path, "a"); my_fopen or my_open and anyway, why a new class, why not to use MYSQL_LOG like general and slow logs do? > + if (!file) > + { > + sql_print_error("Open of log_slave_retries '%s' failed: %s (%d)", > + opt_slave_retries_path, strerror(errno), errno); This is inconsistent, the server doesn't print this for other log files. I've just tried with general log and got 2023-11-06 17:58:19 0 [ERROR] mysqld: File '/tmp' not found (Errcode: 21 "Is a directory") 2023-11-06 17:58:19 0 [ERROR] Could not use /tmp for logging (error 21). Turning logging off for the whole duration of the MariaDB server process. To turn it on again: fix the cause, shutdown the MariaDB server and restart it. Please, do the same > + return true; > + } > + else > + { > + slave_retries_print("%s (mariadbd %s) starting as process %lu ...", > + my_progname, server_version, (ulong) getpid()); Any reason, why it shouldn't be the same as in the error log? sql_print_information("Starting MariaDB %s source revision %s as process %lu", server_version, SOURCE_REVISION, (ulong) getpid()); > + } > +#endif > + return false; > +} > + > +void Slave_retries_file::close() > +{ > +#ifdef HAVE_REPLICATION > + if (file) > + { > + acquire(); > + fclose(file); > + release(); > + } > + file= NULL; > +#endif > +} > + > +void Slave_retries_file::init() > +{ > +#ifdef HAVE_REPLICATION > + mysql_mutex_init(key_LOCK_global_system_variables, &mutex, MY_MUTEX_INIT_FAST); > +#endif > +} > + > +void Slave_retries_file::destroy() > +{ > +#ifdef HAVE_REPLICATION > + mysql_mutex_destroy(&mutex); > +#endif > +} > + > +bool Slave_retries_file::acquire() > +{ > +#ifdef HAVE_REPLICATION > + if (!file) > + return false; > + mysql_mutex_lock(&mutex); > + return true; > +#else > + return false; > +#endif > +} > + > +void Slave_retries_file::release() > +{ > +#ifdef HAVE_REPLICATION > + mysql_mutex_unlock(&mutex); > +#endif > +} > + > +void Slave_retries_file::flush() > +{ > + if (acquire()) > + { > + fflush(file); > + release(); > + } > +} > + > /* Static variables */ > > my_bool opt_stack_trace; > @@ -5055,6 +5145,35 @@ static int init_server_components() > "this server. However this will be ignored as the " > "--log-bin option is not defined."); > } > + > + slave_retries_file.init(); > + if (!log_error_file_ptr[0]) > + fn_format(default_slave_retries_path, pidfile_name, mysql_data_home, > + "-retries.err", MY_REPLACE_EXT); > + else > + fn_format(default_slave_retries_path, log_error_file_ptr, mysql_data_home, > + "-retries.err", MY_REPLACE_EXT); that's not how all other log names work, grep mysqld.cc for opt_slow_logname > + if (opt_slave_retries_path && !opt_abort) > + { > + if (!opt_slave_retries_path[0]) > + { > + if (!default_slave_retries_path[0]) > + { > + /* fn_format failed */ > + opt_slave_retries_path= NULL; > + opt_slave_retries= false; > + } > + else > + { > + opt_slave_retries_path= my_strdup(key_memory_Sys_var_charptr_value, > + default_slave_retries_path, MYF(0)); > + } > + } > + > + if (opt_slave_retries && > + slave_retries_file.open()) > + unireg_abort(1); > + } > #endif > > if (opt_bin_log) > diff --git a/sql/mysqld.h b/sql/mysqld.h > index 69cfc0cad56..5f1561ca10d 100644 > --- a/sql/mysqld.h > +++ b/sql/mysqld.h > @@ -822,6 +848,8 @@ enum options_mysqld > OPT_KEY_CACHE_CHANGED_BLOCKS_HASH_SIZE, > OPT_LOG_BASENAME, > OPT_LOG_ERROR, > + OPT_SLAVE_RETRIES, > + OPT_SLAVE_RETRIES_LOG, why do you need them? > OPT_LOWER_CASE_TABLE_NAMES, > OPT_PLUGIN_LOAD, > OPT_PLUGIN_LOAD_ADD, > diff --git a/sql/sql_show.cc b/sql/sql_show.cc > index d742bc6c1c3..c53ca0f0097 100644 > --- a/sql/sql_show.cc > +++ b/sql/sql_show.cc > @@ -10662,6 +10663,15 @@ char *thd_get_error_context_description(THD *thd, char *buffer, > /* Don't wait if LOCK_thd_data is used as this could cause a deadlock */ > if (!mysql_mutex_trylock(&thd->LOCK_thd_data)) > { > + if (thd->rgi_slave) is rgi_slave and its content protected by thd->LOCK_thd_data ? > + { > + rpl_group_info *rgi= thd->rgi_slave; > + len= my_snprintf(header, sizeof(header), > + ", GTID %u-%u-%llu", rgi->current_gtid.domain_id, > + rgi->current_gtid.server_id, rgi->current_gtid.seq_no); > + str.append(header, len); > + } > + > if (const char *info= thread_state_info(thd)) > { > str.append(' '); > diff --git a/sql/sys_vars.cc b/sql/sys_vars.cc > index ebb64b340b0..11c321f06dc 100644 > --- a/sql/sys_vars.cc > +++ b/sql/sys_vars.cc > @@ -5862,6 +5862,61 @@ Sys_slave_trans_retry_interval( > "slave_transaction_retry_errors", > GLOBAL_VAR(slave_trans_retry_interval), CMD_LINE(REQUIRED_ARG), > VALID_RANGE(0, 3600), DEFAULT(0), BLOCK_SIZE(1)); > + > +static bool update_slave_retries_log(sys_var *self, THD* thd, enum_var_type type) > +{ > + if (opt_slave_retries) > + { > + if (!slave_retries_file.get() && slave_retries_file.open()) > + { > + my_error(ER_CANT_OPEN_FILE, MYF(0), opt_slave_retries_path, errno); > + opt_slave_retries= false; > + return true; > + } > + } > + else > + slave_retries_file.close(); > + return false; > +} > + > +static Sys_var_mybool Sys_slave_retries_log( > + "log_slave_retries", "Log transaction retries", > + GLOBAL_VAR(opt_slave_retries), > + CMD_LINE(OPT_ARG, OPT_SLAVE_RETRIES), DEFAULT(false), > + NO_MUTEX_GUARD, NOT_IN_BINLOG, ON_CHECK(0), ON_UPDATE(update_slave_retries_log)); > + > + > +static bool check_slave_retries_file(sys_var *self, THD *thd, set_var *var) > +{ > + if (opt_slave_retries) > + { > + my_error(ER_NOT_SUPPORTED_YET, MYF(0), "modifying while log_slave_retries is ON"); as always, please don't hard-code error messages. you can use my_error(ER_OPTION_PREVENTS_STATEMENT, MYF(0), "log_slave_retries=ON"); > + return true; > + } > + return false; > +} > + > +static bool update_slave_retries_file(sys_var *self, THD* thd, enum_var_type type) > +{ > + if (!opt_slave_retries_path) > + opt_slave_retries_path= my_strdup(key_memory_Sys_var_charptr_value, > + default_slave_retries_path, MYF(0)); sys_var.cc can handle that for you, see PREALLOCATED in e.g. opt_slow_logname also, you don't reopen the log when its name changes, how comes? > + return false; > +} > + > +static Sys_var_charptr Sys_slave_retries_log_path( > + "log_slave_retries_file", "Log file path for transaction retries", > + GLOBAL_VAR(opt_slave_retries_path), > + CMD_LINE(REQUIRED_ARG, OPT_SLAVE_RETRIES_LOG), DEFAULT(NULL), > + NO_MUTEX_GUARD, NOT_IN_BINLOG, ON_CHECK(check_slave_retries_file), > + ON_UPDATE(update_slave_retries_file)); > + > + > +static Sys_var_uint Sys_slave_retries_log_max( > + "log_slave_retries_max", "Log max N transaction retries " > + "(default 0 means infinity)", just "(0 means infinity)". Default is printed separately > + GLOBAL_VAR(opt_slave_retries_max_log), CMD_LINE(REQUIRED_ARG), > + VALID_RANGE(0, UINT_MAX), DEFAULT(0), BLOCK_SIZE(1)); > #endif > > static bool check_locale(sys_var *self, THD *thd, set_var *var) > diff --git a/storage/innobase/lock/lock0lock.cc b/storage/innobase/lock/lock0lock.cc > index 0e9cef669d4..8a8eeaf39f6 100644 > --- a/storage/innobase/lock/lock0lock.cc > +++ b/storage/innobase/lock/lock0lock.cc > @@ -49,6 +49,7 @@ Created 5/7/1996 Heikki Tuuri > #include <debug_sync.h> > > #include <set> > +#include <stdio.h> I presume Marko has reviewed (or will review) InnoDB changes > > #ifdef WITH_WSREP > #include <mysql/service_wsrep.h> > diff --git a/mysql-test/main/mysqld--help.result b/mysql-test/main/mysqld--help.result > index 77d6d10b3e6..274c034d455 100644 > --- a/mysql-test/main/mysqld--help.result > +++ b/mysql-test/main/mysqld--help.result > @@ -1673,6 +1678,9 @@ log-isam myisam.log > log-output FILE > log-queries-not-using-indexes FALSE > log-short-format FALSE > +log-slave-retries FALSE > +log-slave-retries-file (No default value) this is another sign that log-slave-retries-file doesn't behave like other log filename variables. > +log-slave-retries-max 0 > log-slave-updates FALSE > log-slow-admin-statements TRUE > log-slow-disabled-statements sp Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

1 0

Re: [PATCH] MDEV-32673 Handle InnoDB binlog position recovery for RESET MASTER or binlog name change
by Kristian Nielsen 07 Nov '23

07 Nov '23

Hi Marko, Thanks for the patch. It looks good, just a few small comments inline. Approved after fixing the check for !is_relay_log. I have pushed an extra commit on top of bb-10.4-MDEV-32673 with a testcase (feel free to squash it onto your commit). The tests pass with your patch (but fails without it). - Kristian. > diff --git a/sql/handler.h b/sql/handler.h > index 6085111bc25..134212a0ad7 100644 > --- a/sql/handler.h > +++ b/sql/handler.h > @@ -1438,6 +1438,7 @@ struct handlerton > enum_binlog_command binlog_command, > const char *query, uint query_length, > const char *db, const char *table_name); > + void (*reset_binlog_pos)(const char *name, uint64_t pos); I think there should be a comment describing what this new call does. Suggestion: /* Called at server startup or RESET MASTER when the new binlog is opened. Informs the storage engine of the change in binlog file name and position. */ > diff --git a/sql/log.cc b/sql/log.cc > index 59730c3205a..e46f3796b63 100644 > --- a/sql/log.cc > +++ b/sql/log.cc > @@ -4327,6 +4327,8 @@ bool MYSQL_BIN_LOG::reset_logs(THD *thd, bool create_new_log, > err: > if (error == 1) > name= const_cast<char*>(save_name); > + else > + ha_reset_binlog_pos(get_log_fname(), BIN_LOG_HEADER_SIZE); Sorry, I forgot something when we discussed this. This function is also used for resetting the relay logs. So it needs a check for !is_relay_log. For example: --- a/sql/log.cc +++ b/sql/log.cc @@ -4331,6 +4331,10 @@ bool MYSQL_BIN_LOG::reset_logs(THD *thd, bool create_new_log, if (!is_relay_log) { xid_count_per_binlog *b; + + if (!error) + ha_reset_binlog_pos(get_log_fname(), BIN_LOG_HEADER_SIZE); + /* Remove all entries in the xid_count list except the last. Normally we will just be deleting all the entries that we waited for to > diff --git a/storage/innobase/trx/trx0rseg.cc b/storage/innobase/trx/trx0rseg.cc > index be52373d03b..3ffa003588d 100644 > --- a/storage/innobase/trx/trx0rseg.cc > +++ b/storage/innobase/trx/trx0rseg.cc > @@ -826,3 +826,52 @@ trx_rseg_update_binlog_offset(byte* rseg_header, const trx_t* trx, mtr_t* mtr) > mlog_write_string(p, binlog_name, len, mtr); > } > } > + > +void trx_rseg_reset_binlog_pos(const char *name, uint64_t pos) > +{ > + mtr_t mtr; > + /* Invalidate the information on the TRX_SYS page. */ > + mtr.start(); > + if (buf_block_t *sys= trx_sysf_get(&mtr, true)) > + { > + byte *magic= TRX_SYS + TRX_SYS_MYSQL_LOG_INFO + > + TRX_SYS_MYSQL_LOG_MAGIC_N_FLD + sys->frame; > + if (mach_read_from_4(magic)) > + mlog_memset(magic, > + TRX_SYS_MYSQL_LOG_NAME + TRX_SYS_MYSQL_LOG_NAME_LEN, 0, > + &mtr); > + > + const uint32_t page_no= trx_sysf_rseg_get_page_no(sys, 0); > + if (page_no != FIL_NULL) > + { > + const uint32_t space_id= trx_sysf_rseg_get_space(sys, 0); > + ut_ad(space_id == TRX_SYS_SPACE); > + buf_block_t *block= buf_page_get(page_id_t(space_id, page_no), > + 0, RW_X_LATCH, &mtr); > + mlog_write_ull(TRX_RSEG + TRX_RSEG_BINLOG_OFFSET + block->frame, > + pos, &mtr); > + byte *p= TRX_RSEG + TRX_RSEG_BINLOG_NAME + block->frame; > + const size_t len= strlen(name) + 1; > + if (memcmp(name, p, len)) > + mlog_write_string(p, reinterpret_cast<const byte*>(name), len, &mtr); > + } > + > + /* Invalidate the information in the other rollback segment headers. */ > + for (ulint rseg_id= 1; rseg_id < TRX_SYS_N_RSEGS; rseg_id++) > + { > + const uint32_t page_no= trx_sysf_rseg_get_page_no(sys, rseg_id); > + if (page_no != FIL_NULL) > + { > + const uint32_t space_id= trx_sysf_rseg_get_space(sys, rseg_id); > + buf_block_t *block= buf_page_get(page_id_t(space_id, page_no), > + 0, RW_X_LATCH, &mtr); > + byte *name= TRX_RSEG + TRX_RSEG_BINLOG_NAME + block->frame; > + if (*name) > + mlog_write_ulint(name, 0, MLOG_1BYTE, &mtr); > + } > + } > + } > + mtr.commit(); > + /* Durably write the change. */ > + log_write_up_to(mtr.commit_lsn(), true); > +}

1 0

Re: 3aea3bd7422: MDEV-23294 Segfault or assertion upon MyISAM repair
by Sergei Golubchik 06 Nov '23

06 Nov '23

Hi, Aleksey, On Oct 23, Aleksey Midenkov wrote: > revision-id: 3aea3bd7422 (mariadb-10.4.30-176-g3aea3bd7422) > parent(s): 4fa6383c937 > author: Aleksey Midenkov > committer: Aleksey Midenkov > timestamp: 2023-09-26 20:20:46 +0300 > message: > > MDEV-23294 Segfault or assertion upon MyISAM repair > > When computing vcol expression some items use current_thd and that was > not set in MyISAM repair thread. Since all the repair threads belong > to one connection and items should not write into THD we can utilize > table THD for that. > > diff --git a/storage/myisam/ha_myisam.cc b/storage/myisam/ha_myisam.cc > index ddab8bcf741..27151548704 100644 > --- a/storage/myisam/ha_myisam.cc > +++ b/storage/myisam/ha_myisam.cc > @@ -714,6 +714,8 @@ static int compute_vcols(MI_INFO *info, uchar *record, int keynum) > /* This mutex is needed for parallel repair */ > mysql_mutex_lock(&info->s->intern_lock); > TABLE *table= (TABLE*)(info->external_ref); > + if (!current_thd) > + set_current_thd(table->in_use); why not to do it in thr_find_all_keys(), unconditionally? you'd avoid an if() executed per row. Also, I'd suggest a comment, like /* To evaluate vcols we must have current_thd set. This will set current_thd in all threads to the same THD, but it's safe, because vcols are always evaluated under info->s->intern_lock. */ set_current_thd((TABLE*)(info->external_ref)->in_use); > table->move_fields(table->field, record, table->field[0]->record_ptr()); > if (keynum == -1) // update all vcols > { Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 3

Re: 0f0ef93c4d4: MDEV-32444 Data from orphaned XA transaction is lost after online alter
by Sergei Golubchik 02 Nov '23

02 Nov '23

Hi, Nikita, See below. A small suggestion for the test. And a rather larger comment about extending xid_t, as you, likely, expected :) On Oct 19, Nikita Malyavin wrote: > revision-id: 0f0ef93c4d4 (mariadb-11.2.1-15-g0f0ef93c4d4) > parent(s): 7f00853f2f1 > author: Nikita Malyavin > committer: Nikita Malyavin > timestamp: 2023-10-13 13:39:36 +0400 > diff --git a/mysql-test/main/alter_table_online_debug.result b/mysql-test/main/alter_table_online_debug.result > index 1ebb19965eb..154aec003ca 100644 > --- a/mysql-test/main/alter_table_online_debug.result > +++ b/mysql-test/main/alter_table_online_debug.result > @@ -1563,6 +1563,103 @@ connection default; > drop table t1, t2; > set @@binlog_format=default; > set debug_sync= reset; > +# MDEV-32444 Data from orphaned XA transaction is lost after online alter > +create table t (a int primary key) engine=innodb; > +insert into t values (1); > +# XA commit > +set debug_sync= 'alter_table_online_downgraded signal downgraded wait_for go'; > +alter table t force, algorithm=copy, lock=none; > +connection con1; > +set debug_sync= 'now wait_for downgraded'; > +xa begin 'x1'; > +update t set a = 2 where a = 1; > +xa end 'x1'; > +xa prepare 'x1'; > +disconnect con1; > +connection con2; > +xa commit 'x1'; > +set debug_sync= 'now signal go'; > +connection default; > +select * from t; > +a > +2 > +# XA rollback > +set debug_sync= 'alter_table_online_downgraded signal downgraded wait_for go'; > +alter table t force, algorithm=copy, lock=none; > +connect con1, localhost, root,,; > +xa begin 'x2'; > +insert into t values (53); > +xa end 'x2'; > +xa prepare 'x2'; > +disconnect con1; > +connection con2; > +xa rollback 'x2'; > +set debug_sync= 'now signal go'; > +connection default; > +select * from t; > +a > +2 > +# XA transaction is left uncommitted > +# end then is rollbacked after alter fails > +set debug_sync= 'alter_table_online_downgraded signal downgraded wait_for go'; > +set statement innodb_lock_wait_timeout=0, lock_wait_timeout= 0 > +for alter table t force, algorithm=copy, lock=none; > +connect con1, localhost, root,,; > +xa begin 'xuncommitted'; > +insert into t values (3); > +xa end 'xuncommitted'; > +xa prepare 'xuncommitted'; > +set debug_sync= 'now signal go'; > +disconnect con1; > +connection default; > +ERROR HY000: Lock wait timeout exceeded; try restarting transaction > +xa rollback 'xuncommitted'; > +select * from t; > +a > +2 > +# Same, but commit > +set debug_sync= 'alter_table_online_downgraded signal downgraded wait_for go'; > +set statement innodb_lock_wait_timeout=0, lock_wait_timeout= 0 > +for alter table t force, algorithm=copy, lock=none; instead of `alter table t force`, better do something that changes the table, so that you could see whether alter was executed or aborted. For example, you can change the column type int->bigint and back. not only here, but everywhere. Or add/drop column, this has an added benefit that you won't need a separate `show create table`, your already existing `select * from t` will show whether the extra column exists > +connect con1, localhost, root,,; > +xa begin 'committed_later'; > +insert into t values (3); > +xa end 'committed_later'; > +xa prepare 'committed_later'; > +set debug_sync= 'now signal go'; > +disconnect con1; > +connection default; > +ERROR HY000: Lock wait timeout exceeded; try restarting transaction > +xa commit 'committed_later'; > +select * from t; > +a > +2 > +3 > +# Commit, but error in statement, and there is some stmt data to rollback > +set debug_sync= 'alter_table_online_downgraded signal downgraded wait_for go'; > +alter table t force, algorithm=copy, lock=none; > +connect con1, localhost, root,,; > +set debug_sync= 'now wait_for downgraded'; > +xa begin 'x1'; > +insert into t values (4), (3); > +ERROR 23000: Duplicate entry '3' for key 'PRIMARY' > +insert into t values (5); > +xa end 'x1'; > +xa prepare 'x1'; > +disconnect con1; > +connection con2; > +xa commit 'x1'; > +set debug_sync= 'now signal go'; > +connection default; > +select * from t; > +a > +2 > +3 > +5 > +connect con1, localhost, root,,; > +connection default; > +drop table t; > +set debug_sync= reset; > disconnect con1; > disconnect con2; > # > diff --git a/sql/handler.h b/sql/handler.h > index 07b392c010a..4622e49f883 100644 > --- a/sql/handler.h > +++ b/sql/handler.h > @@ -903,6 +905,7 @@ struct xid_t { > long gtrid_length; > long bqual_length; > char data[XIDDATASIZE]; // not \0-terminated ! > + Online_alter_cache_list *online_alter_cache; well, no, we've talked about it. Even the comment above the structure says that you cannot change it. make a separate structure, xid_and_online_alter_cache_t or xid_t_internal or whatever. but xid_t should be defined identically everywhere. For example: https://github.com/berkeleydb/libdb/blob/master/src/dbinc/xa.h === okay, now I've seen how you used it and I'd say it's _such a rare use case_ that you can store your online_alter_cache in the XID_cache_element and look it up there. Or even in a separate data structure, that will be empty most of the time anyway. > > xid_t() = default; /* Remove gcc warning */ > bool eq(struct xid_t *xid) const > diff --git a/sql/online_alter.cc b/sql/online_alter.cc > index 8b8c81319d1..19abe57914d 100644 > --- a/sql/online_alter.cc > +++ b/sql/online_alter.cc > @@ -334,14 +336,57 @@ static int online_alter_log_init(void *p) > online_alter_hton->savepoint_rollback_can_release_mdl= > [](handlerton *hton, THD *thd){ return true; }; > > - online_alter_hton->commit= [](handlerton *hton, THD *thd, bool all) > - { return online_alter_end_trans(hton, thd, all, true); }; > - online_alter_hton->rollback= [](handlerton *hton, THD *thd, bool all) > - { return online_alter_end_trans(hton, thd, all, false); }; > - online_alter_hton->commit_by_xid= [](handlerton *hton, XID *xid) > - { return online_alter_end_trans(hton, current_thd, true, true); }; > - online_alter_hton->rollback_by_xid= [](handlerton *hton, XID *xid) > - { return online_alter_end_trans(hton, current_thd, true, false); }; > + online_alter_hton->commit= [](handlerton *hton, THD *thd, bool all) -> int > + { > + int res= online_alter_end_trans(get_cache_list(hton, thd), thd, > + ending_trans(thd, all), true); > + cleanup_tables(thd); > + return res; > + }; > + online_alter_hton->rollback= [](handlerton *hton, THD *thd, bool all) -> int > + { > + int res= online_alter_end_trans(get_cache_list(hton, thd), thd, > + ending_trans(thd, all), false); > + cleanup_tables(thd); > + return res; > + }; > + > + > + online_alter_hton->recover= [](handlerton*, XID*, uint){ return 0; }; > + online_alter_hton->prepare= [](handlerton *hton, THD *thd, bool all) -> int > + { > + auto &cache_list= get_cache_list(hton, thd); > + int res= 0; > + if (ending_trans(thd, all)) > + { > + thd->transaction->xid_state.set_online_alter_cache(&cache_list); > + thd_set_ha_data(thd, hton, NULL); > + } > + else > + { > + res= online_alter_end_trans(cache_list, thd, false, true); is that possible? > + } > + > + cleanup_tables(thd); > + return res; > + }; > + online_alter_hton->commit_by_xid= [](handlerton *hton, XID *xid) -> int > + { > + int res= online_alter_end_trans(*xid->online_alter_cache, current_thd, > + true, true); > + delete xid->online_alter_cache; > + xid->online_alter_cache= NULL; > + return res; > + }; > + online_alter_hton->rollback_by_xid= [](handlerton *hton, XID *xid) -> int > + { > + int res= online_alter_end_trans(*xid->online_alter_cache, current_thd, > + true, false); > + delete xid->online_alter_cache; > + xid->online_alter_cache= NULL; > + return res; > + }; > + > > online_alter_hton->drop_table= [](handlerton *, const char*) { return -1; }; > online_alter_hton->flags= HTON_NOT_USER_SELECTABLE | HTON_HIDDEN Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 6

Re: 2d937b62c33: MDEV-27744 InnoDB: Failing assertion: !cursor->index->is_committed() in row0ins.cc (from row_ins_sec_index_entry_by_modify) | Assertion `0' failed in row_upd_sec_index_entry (debug) | Corruption
by Sergei Golubchik 02 Nov '23

02 Nov '23

Hi, Alexander, Few questions and suggestions below. Nothing major. On Oct 27, Alexander Barkov wrote: > commit 2d937b62c33 > Author: Alexander Barkov <bar(a)mariadb.com> > Date: Mon Apr 4 14:50:21 2022 +0400 > > MDEV-27744 InnoDB: Failing assertion: !cursor->index->is_committed() in row0ins.cc (from row_ins_sec_index_entry_by_modify) | Assertion `0' failed in row_upd_sec_index_entry (debug) | Corruption > would it make sense to change the MDEV title in Jira to better describe the problem? I sometimes do it with my bugs. Like "LPAD in vcol created in ORACLE mode makes table corrupted in non-ORACLE" (I tried to make it short, appropriate for a title) and, of course, change it in the comment and in the test file to match. > The crash happened with an indexed virtual column whose > value is evaluated using a function that has a different meaning > in sql_mode='' vs sql_mode=ORACLE: > > diff --git a/mysql-test/suite/compat/oracle/r/func_concat.result b/mysql-test/suite/compat/oracle/r/func_concat.result > index 392d579707a..17ca4be078a 100644 > --- a/mysql-test/suite/compat/oracle/r/func_concat.result > +++ b/mysql-test/suite/compat/oracle/r/func_concat.result > @@ -211,14 +211,14 @@ SET sql_mode=ORACLE; > CREATE VIEW v1 AS SELECT 'foo'||NULL||'bar' AS test; > SHOW CREATE VIEW v1; > View Create View character_set_client collation_connection > -v1 CREATE VIEW "v1" AS select concat_operator_oracle(concat_operator_oracle('foo',NULL),'bar') AS "test" latin1 latin1_swedish_ci > +v1 CREATE VIEW "v1" AS select concat(concat('foo',NULL),'bar') AS "test" latin1 latin1_swedish_ci > SELECT * FROM v1; > test > foobar > SET sql_mode=DEFAULT; > SHOW CREATE VIEW v1; > View Create View character_set_client collation_connection > -v1 CREATE ALGORITHM=UNDEFINED DEFINER=`root`@`localhost` SQL SECURITY DEFINER VIEW `v1` AS select concat_operator_oracle(concat_operator_oracle('foo',NULL),'bar') AS `test` latin1 latin1_swedish_ci > +v1 CREATE ALGORITHM=UNDEFINED DEFINER=`root`@`localhost` SQL SECURITY DEFINER VIEW `v1` AS select oracle_schema.concat(oracle_schema.concat('foo',NULL),'bar') AS `test` latin1 latin1_swedish_ci please, add a mysqldump test. Like, create a table with virtual columns, check, defaults in the default sql mode. and create a table in oracle mode. Also, a view in the default mode and a view in oracle mode. and then mysqldump, to see that it dumps and restores everything correctly may be stored routines/triggers/etc, if you'd like, but they aren't directly relevant to this MDEV, as far as I understand. > SELECT * FROM v1; > test > foobar > diff --git a/mysql-test/suite/compat/oracle/r/func_decode.result b/mysql-test/suite/compat/oracle/r/func_decode.result > index 2809e971be3..1870a1ec0d5 100644 > --- a/mysql-test/suite/compat/oracle/r/func_decode.result > +++ b/mysql-test/suite/compat/oracle/r/func_decode.result > @@ -1,8 +1,8 @@ > SET sql_mode=ORACLE; > SELECT DECODE(10); > -ERROR 42000: Incorrect parameter count in the call to native function 'DECODE' > +ERROR 42000: Incorrect parameter count in the call to native function 'oracle_schema.DECODE' Hmm, may be this should say DECODE as before? you know, falls under the case "if you don't change sql_mode back and forth, you won't see schema-qualified names" > SELECT DECODE(10,10); > -ERROR 42000: Incorrect parameter count in the call to native function 'DECODE' > +ERROR 42000: Incorrect parameter count in the call to native function 'oracle_schema.DECODE' > SELECT DECODE(10,10,'x10'); > DECODE(10,10,'x10') > x10 > diff --git a/mysql-test/suite/compat/oracle/r/vcol_innodb.result b/mysql-test/suite/compat/oracle/r/vcol_innodb.result > new file mode 100644 > index 00000000000..9fa97c75c10 > --- /dev/null > +++ b/mysql-test/suite/compat/oracle/r/vcol_innodb.result > @@ -0,0 +1,51 @@ > +SET @table_open_cache=@@GLOBAL.table_open_cache; why do you need to manipulate table_open_cache? to trigger a reopen? Just do flush tables, it's explicit, more readable and more... controllable. > +SET sql_mode=''; > +CREATE TABLE t (d INT,b VARCHAR(1),c CHAR(1),g CHAR(1) GENERATED ALWAYS AS (SUBSTR(b,0,0)) VIRTUAL,PRIMARY KEY(b),KEY g(g)) ENGINE=InnoDB; > +INSERT INTO t VALUES (0); > +ERROR 21S01: Column count doesn't match value count at row 1 > +SET sql_mode='ORACLE'; > +INSERT INTO t SET c=REPEAT (1,0); > +Warnings: > +Warning 1364 Field 'b' doesn't have a default value > +ALTER TABLE t CHANGE COLUMN a b INT; > diff --git a/sql/item_func.h b/sql/item_func.h > index 76a997c33fb..cdbefb82541 100644 > --- a/sql/item_func.h > +++ b/sql/item_func.h > @@ -56,8 +56,40 @@ class Item_func :public Item_func_or_sum, > bool check_argument_types_can_return_date(uint start, uint end) const; > bool check_argument_types_can_return_time(uint start, uint end) const; > void print_cast_temporal(String *str, enum_query_type query_type); > + > + void print_schema_qualified_name(String *to, > + const LEX_CSTRING &schema_name, > + const char *function_name) const I don't see why you'd need this helper. is it something that was used in earlier versions of the patch? > + { > + // e.g. oracle_schema.func() > + to->append(schema_name); > + to->append('.'); > + to->append(function_name); > + } > + > + void print_sql_mode_qualified_name(String *to, > + enum_query_type query_type, > + const char *function_name) const > + { > + const Schema *func_schema= schema(); > + if (!func_schema || func_schema == Schema::find_implied(current_thd)) > + to->append(function_name); > + else > + print_schema_qualified_name(to, func_schema->name(), function_name); > + } > + > + void print_sql_mode_qualified_name(String *to, enum_query_type query_type) > + const > + { > + return print_sql_mode_qualified_name(to, query_type, func_name()); > + } I don't see why you need this helper either, you never use print_sql_mode_qualified_name with the last argument being not func_name(). So you can remove this helper and the third argument of print_sql_mode_qualified_name. > + > public: > > + // Print an error message for a builtin-schema qualified function call > + static void wrong_param_count_error(const LEX_CSTRING &schema_name, > + const LEX_CSTRING &func_name); > + > table_map not_null_tables_cache; > > enum Functype { UNKNOWN_FUNC,EQ_FUNC,EQUAL_FUNC,NE_FUNC,LT_FUNC,LE_FUNC, > diff --git a/sql/item_strfunc.cc b/sql/item_strfunc.cc > index ae078dbb22f..92d5e196da4 100644 > --- a/sql/item_strfunc.cc > +++ b/sql/item_strfunc.cc > @@ -2170,13 +2170,31 @@ bool Item_func_trim::fix_length_and_dec() > > void Item_func_trim::print(String *str, enum_query_type query_type) > { > + LEX_CSTRING suffix= {STRING_WITH_LEN("_oracle")}; > if (arg_count == 1) > { > - Item_func::print(str, query_type); > + if (query_type & QT_FOR_FRM) > + { > + // 10.3 downgrade compatibility for FRM > + str->append(func_name()); > + if (schema() == &oracle_schema_ref) > + str->append(suffix); > + } > + else > + print_sql_mode_qualified_name(str, query_type, func_name()); > + print_args_parenthesized(str, query_type); > return; > } > - str->append(Item_func_trim::func_name()); > - str->append(func_name_ext()); > + > + if (query_type & QT_FOR_FRM) > + { > + // 10.3 downgrade compatibility for FRM > + str->append(Item_func_trim::func_name()); > + if (schema() == &oracle_schema_ref) > + str->append(suffix); > + } > + else > + print_sql_mode_qualified_name(str, query_type, Item_func_trim::func_name()); it'd be simpler if you move the above block that prints the function name before if (arg_count == 1) also you won't need suffix, but can do like in all other functions str->append(STRING_WITH_LEN("trim_oracle"); > str->append('('); > str->append(mode_name()); > str->append(' '); > diff --git a/sql/sql_lex.cc b/sql/sql_lex.cc > index 71f592a3852..bb53d1a510a 100644 > --- a/sql/sql_lex.cc > +++ b/sql/sql_lex.cc > @@ -2084,7 +2100,64 @@ bool Lex_input_stream::get_7bit_or_8bit_ident(THD *thd, uchar *last_char) > } > > > -int Lex_input_stream::scan_ident_sysvar(THD *thd, Lex_ident_cli_st *str) > +/* > + Resolve special SQL functions that have a qualified syntax in sql_yacc.yy. > + These functions are not listed in the native function registry > + because of a special syntax, or a reserved keyword: > + > + mariadb_schema.SUBSTRING('a' FROM 1 FOR 2) -- Special syntax I didn't find it in Oracle's manual, by the way > + mariadb_schema.TRIM(BOTH ' ' FROM 'a') -- Special syntax > + mariadb_schema.REPLACE('a','b','c') -- Verb keyword > +*/ > + > +int Lex_input_stream::find_keyword_qualified_special_func(Lex_ident_cli_st *str, > + uint length) const > +{ > + /* > + There are many other special functions, see the following grammar rules: > + function_call_keyword > + function_call_nonkeyword > + Here we resolve only those that have a qualified syntax to handle > + different behavior in different @@sql_mode settings. > + > + Other special functions do not work in qualified context: > + SELECT mariadb_schema.year(now()); -- Function year is not defined > + SELECT mariadb_schema.now(); -- Function now is not defined > + > + We don't resolve TRIM_ORACLE here, because it does not have > + a qualified syntax yet. Search for "trim_operands" in sql_yacc.yy > + to find more comments. > + */ > diff --git a/sql/sql_schema.h b/sql/sql_schema.h > index 1174bc7a83f..2c52646f2ea 100644 > --- a/sql/sql_schema.h > +++ b/sql/sql_schema.h > @@ -77,5 +98,6 @@ class Schema > > > extern Schema mariadb_schema; > +extern const Schema &oracle_schema_ref; What's the difference between these two definitions. Do you expect someone will need to change mariadb_schema? > > #endif // SQL_SCHEMA_H_INCLUDED Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 2

Eliminating sprintf deprecation warnings (maybe others as well)
by Stein Vidar Hagfors Haugan 27 Oct '23

27 Oct '23

Hi! First time here, so my apologies if I'm violating some spoken or unspoken rules :) I'm developing/maintaining two specialized storage engines(*) and I have one issue: There are numerous uses of sprintf() in the general mariadb codebase, and each one generates a warning during compilation when doing a plain cmake/make build (on MacOS & RHEL7 at least). I am a bit fanatical about hunting down and eliminating warnings if at all possible, no matter how small and insignificant - if you become accustomed to routinely getting warnings during compilation, you can easily overlook an important one. So I am considering to contribute with a patch that will eliminate all sprintf() instances by converting them to snprintf(destination, N, ...). Of course, hunting down the correct N for each instance can be a *lot* of work (I presume this is why this hasn't been done already). So I'm proposing to make a macro UNSAFE_SNPRINTF(destination, ...) which would expand to snprintf(destination, 10000, ...) and replace all current sprintf() calls with that. Yes, 10000 is a ridiculously high number that makes it look like we're introducing a huge security risk, but but the current sprintf() calls have an implicit N = infinity. In some instances, the compiler sniffs out the actual size of the buffer and complains - in that case I would change the code to an appropriate regular snprintf() call. Would doing this be worthwile? I.e., is there a good chance that such a patch would actually be accepted into the codebase? If so, I might also do the same with other warnings (I believe there are some type-punning warnings at least). Sincerely, Stein Haugan *) The engines are for internal use so far, but might make them public at some point

2 2

Re: [PATCH 4/4] MDEV-31273: Precompute binlog checksums
by Michael Widenius 26 Oct '23

26 Oct '23

Hi! Review of MDEV-31273: Precompute binlog checksums On Fri, Aug 25, 2023 at 10:16 AM Kristian Nielsen <knielsen(a)knielsen-hq.org> wrote: > > Compute binlog checksums (when enabled) already when writing events > into the statement or transaction caches, where before it was done > when the caches are copied to the real binlog file. This moves the > checksum computation outside of holding LOCK_log, improving > scalabitily. DBUG_RETURN(my_b_copy_to_cache(from_cache, to_cache, SIZE_T_MAX)); You could use from_cache->end_of_file instead of SIZE_T_MAX <cut> uchar checksum_opt; Wouldn't it be better to have this as an "enum_binlog_checksum_alg" to avoid some casts ? > } > else if (data) > return (enum enum_binlog_checksum_alg)data->checksum_opt; > else > return BINLOG_CHECKSUM_ALG_OFF; > } You can remove the else's <cut> ulong *param_ptr_binlog_cache_disk_use, bool precompute_checksums) : stmt_cache(precompute_checksums), trx_cache(precompute_checksums), last_commit_pos_offset(0), using_xa(FALSE), xa_xid(0) { stmt_cache.set_binlog_cache_info(param_max_binlog_stmt_cache_size, param_ptr_binlog_stmt_cache_use, This code was a bit confusing as we first initialize stmt_cache and trx_cache with checksum and then we initialize them again in the next two lines. I understand why and this does not have to be changed, but it does look a bit strange. The other option would be to have set_binlog_cache_info() take checksum as an argument(). <cut> This one confused me a bit: Log_event_writer writer(file, 0, checksum_alg, &crypto); Log_event_writer writer(file, cache_data, checksum_alg, &crypto); Why this change? It may at least affect calls to: void Log_event_writer::add_status(enum_logged_status status) { if (likely(cache_data)) Fix (after discussions on slack): Remove the following lines in MYSQL_BIN_LOG::write_event(Log_event *ev, binlog_cache_data *cache_data, if (cache_data) cache_data->add_status(ev->logged_status()); <cut> > Don't attempt to precompute checksums if: > - Disabled by user request, --binlog-legacy-event-pos > - Binlog is encrypted, cannot use precomputed checksums > - WSREP/Galera. Why Galera? Would be good to explain this in the commit comment. <cut> > /* > If possible, just copy the cache over byte-by-byte with pre-computed > checksums. > */ > if (likely(binlog_checksum_options == cache_data->checksum_opt) && > likely(!crypto.scheme) && > likely(!opt_binlog_legacy_event_pos)) > { > int res= my_b_copy_all_to_cache(cache, &log_file); > status_var_add(thd->status_var.binlog_bytes_written, my_b_tell(cache)); > DBUG_RETURN(res ? ER_ERROR_ON_WRITE : 0); > } I was just wondering if this could sometimes be optimized to write directly to the real binlog file without another cache in between. <cut> bool precomputed_checksums= (cache_data->checksum_opt != BINLOG_CHECKSUM_ALG_OFF); uint old_checksum_len= precomputed_checksums ? BINLOG_CHECKSUM_LEN : 0; Why two variables when one can as easily do: uint old_checksum_len= ((cache_data->checksum_opt != BINLOG_CHECKSUM_ALG_OFF) ? BINLOG_CHECSUM_LEN : 0); In current write_cache() code, group is a bad name. Please rename 'group' to 'log_file_pos'. By the way, great that you removed the 'end_log_file_pos' variable, which also was a very confusing name related how it was used! > /* > Any old precomputed checksum must _not_ be written here. Instead, it > must be discarded; the new checksum, if needed, is written by > writer.write_footer(). > */ > if (ev_len > old_checksum_len) > { > uint bytes_to_skip= > old_checksum_len - std::min(old_checksum_len, ev_len - chunk); > if (writer.write_data(cache->read_pos, chunk - bytes_to_skip)) > goto error_in_write; > } The above is likely wrong as for long events we would execute the inner part multiple times, while there is only on checksum. As checksum is last in the event, why not just do 'even_len-= old_checksum' Before the loop to copy the event and then disregard 'old_checksum_len' bytes from the cache at the end of the loop? Anyway, to find bugs like this, we need to have a test case with events that are bigger than the IO_CACHE size for cache. Setting binlog_file_cache_size to 8192 (min value) should make it easy to test this. <cut> > binlog_checksum_options= value; > my_atomic_storeul_explicit(&binlog_checksum_options, value, MY_MEMORY_ORDER_RELAXED); Atom is not needed as we have a lock on binlog. ...... Something totally different. I noticed in MYSQL_BIN_LOG::write_cache(): group= (size_t)my_b_tell(&log_file); val= uint4korr(header + LOG_POS_OFFSET) + group + end_log_pos_inc; int4store(header + LOG_POS_OFFSET, val); The first log entry is probably ok, as we have done a rotate() event before. However the cache can have a lot of log_events (as part of a transaction). The end_log_pos after the first event can be wrong as it may not fit into 4 bytes. My understanding is that this is not a problem anymore as we are not using end_log_pos anymore. However I still wonder if rotate() should not only consider the current log file size but also the size of all events we plan to write to the log and do a rotate if the total new log file size > 4G. Regards, Monty

2 1

Re: [PATCH 3/4] MDEV-31273: Refactor MYSQL_BIN_LOG::write_cache()
by Michael Widenius 26 Oct '23

26 Oct '23

Hi! On Fri, Aug 25, 2023 at 10:16 AM Kristian Nielsen <knielsen(a)knielsen-hq.org> wrote: > > Preparatory patch for pre-computing binlog checksums outside of holding > LOCK_log. > > The existing code for MYSQL_BIN_LOG::write_cache() was needlessly complex > and very hard to understand and modify for handling the new case where > pre-computed checksums are already present in the IO_CACHE. MDEV-31273: Refactor MYSQL_BIN_LOG::write_cache() > if (likely(length > LOG_EVENT_HEADER_LEN)) > { > header= cache->read_pos; > cache->read_pos+= LOG_EVENT_HEADER_LEN; > length-= LOG_EVENT_HEADER_LEN; > } > else > { > size_t sofar= length; > size_t remain= LOG_EVENT_HEADER_LEN - sofar; > header= &header_buf[0]; > memcpy(header, cache->read_pos, sofar); > cache->read_pos+= sofar; > > while (hdr_offs < length) > length= my_b_fill(cache); > if (!length) > goto error_in_read; > size_t chunk= std::min(length, remain); > memcpy(header + sofar, cache->read_pos, chunk); > sofar+= chunk; > remain-= chunk; > cache->read_pos+= chunk; > length-= chunk; > } while (unlikely(remain > 0)); > } > The above can be replaced with: if (my_b_read(cache, header, LOG_EVENT_HEAD_LENGTH)) goto error_in_read; It is almost exactly as efficent as the above (one extra if) and avoids using cache internals. Note that you do not have to do call my_fill() if you use my_b_read(). my_b_read() will return 0 if it was able to read all data. In case of end of file, my_b_read() will return 1 and info->error will be 0. If needed, you can als find out how much data left to read from IO_CACHE: left_data_to_read= (cache->end_of_file - my_b_tell(cache)) (I can make an inline function of that if needed). /* Write the rest of the event. */ > while (ev_len > 0) > { > if (length == 0) > length= my_b_fill(cache); > if (!length) > goto error_in_read; > -> while (ev_len > 0) { if (length == 0) { if (!(length= my_b_fill(cache))); goto error_in_read; } <cut> uint chunk= std::min(ev_len, (uint)length); I would have prefer to have MY_MIN() used (like the rest of the code). (not critical) Regards, Monty

2 1

Re: ce9ce585e72: MDEV-31184 Remove parser tokens DECODE_MARIADB_SYM and DECODE_ORACLE_SYM
by Sergei Golubchik 23 Oct '23

23 Oct '23

Hi, Alexander, On Oct 21, Alexander Barkov wrote: > revision-id: ce9ce585e72 (mariadb-10.4.30-103-gce9ce585e72) > parent(s): 1fde785315e > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2023-08-31 13:49:19 +0400 > message: > > MDEV-31184 Remove parser tokens DECODE_MARIADB_SYM and DECODE_ORACLE_SYM > > Changing the code handling sql_mode-dependent function DECODE(): > > - removing parser tokens DECODE_MARIADB_SYM and DECODE_ORACLE_SYM > - removing the DECODE() related code from sql_yacc.yy/sql_yacc_ora.yy > - adding handling of DECODE() with help of a new Create_func_func_decode please, add a test for DECODE_MARIADB_SYM that changes from ER_PARSE_ERROR to ER_WRONG_PARAMCOUNT_TO_NATIVE_FCT. Why sql_yacc.yy had rules for DECODE_ORACLE_SYM and vice versa? Was it possible to get DECODE_ORACLE_SYM in sql_yacc.yy? > diff --git a/mysql-test/suite/compat/oracle/r/func_decode.result b/mysql-test/suite/compat/oracle/r/func_decode.result > index b49bad93627..2809e971be3 100644 > --- a/mysql-test/suite/compat/oracle/r/func_decode.result > +++ b/mysql-test/suite/compat/oracle/r/func_decode.result > @@ -1,8 +1,8 @@ > SET sql_mode=ORACLE; > SELECT DECODE(10); > -ERROR 42000: You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near ')' at line 1 > +ERROR 42000: Incorrect parameter count in the call to native function 'DECODE' > SELECT DECODE(10,10); > diff --git a/sql/sql_yacc.yy b/sql/sql_yacc.yy > index cdc04d93708..048741b6ca1 100644 > --- a/sql/sql_yacc.yy > +++ b/sql/sql_yacc.yy > @@ -11027,18 +11024,6 @@ function_call_nonkeyword: > if (unlikely($$ == NULL)) > MYSQL_YYABORT; > } > - | DECODE_MARIADB_SYM '(' expr ',' expr ')' > - { > - $$= new (thd->mem_root) Item_func_decode(thd, $3, $5); > - if (unlikely($$ == NULL)) > - MYSQL_YYABORT; > - } > - | DECODE_ORACLE_SYM '(' expr ',' decode_when_list_oracle ')' > - { > - $5->push_front($3, thd->mem_root); > - if (unlikely(!($$= new (thd->mem_root) Item_func_decode_oracle(thd, *$5)))) > - MYSQL_YYABORT; > - } > | EXTRACT_SYM '(' interval FROM expr ')' > { > $$=new (thd->mem_root) Item_extract(thd, $3, $5); > @@ -12209,25 +12194,6 @@ when_list_opt_else: > } > ; > > -decode_when_list_oracle: > - expr ',' expr > - { > - $$= new (thd->mem_root) List<Item>; > - if (unlikely($$ == NULL) || > - unlikely($$->push_back($1, thd->mem_root)) || > - unlikely($$->push_back($3, thd->mem_root))) > - MYSQL_YYABORT; > - > - } > - | decode_when_list_oracle ',' expr > - { > - $$= $1; > - if (unlikely($$->push_back($3, thd->mem_root))) > - MYSQL_YYABORT; > - } > - ; > - > - > /* Equivalent to <table reference> in the SQL:2003 standard. */ > /* Warning - may return NULL in case of incomplete SELECT */ Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 1

Re: 56fa1da9c67: MDEV-30048 Prefix keys for CHAR work differently for MyISAM vs InnoDB
by Sergei Golubchik 23 Oct '23

23 Oct '23

Hi, Alexander, The actual code change (replacing strnncollsp with strnncollsp_nchars) is ok. But I didn't like the functions you've created. The *_ft_* family is quite confusing, I would not expect to see fulltext-specific functions my_compare.h. I would expect to see there functions which are generic and named by what they're doing, not by where they should be used. ha_compare_char_fixed and ha_compare_char_varying are better. Names are clearer and it's kind of understandable what they do. I suggest you remove *_ft_* functions. fulltext code should, I guess, just use charset_info->coll->strnncoll() everywhere. Not strnncollsp(), because words normally don't have trailing spaces, and if some custom pluggable parser returns words with trailing spaces, they're likely a part of the word and should be compared too. Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org On Oct 17, Alexander Barkov wrote: > revision-id: 56fa1da9c67 (mariadb-10.4.28-90-g56fa1da9c67) > parent(s): ed2adc8c6f9 > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2023-04-07 14:54:17 +0400 > message: > > MDEV-30048 Prefix keys for CHAR work differently for MyISAM vs InnoDB > > Also fixes: MDEV-30050 Inconsistent results of DISTINCT with NOPAD > > Problem: > > Key segments for CHAR columns where compared using strnncollsp() > for engines MyISAM and Aria. > > This did not work correct in case if the engine applyied trailing > space compression. > > Fix: > > Replacing ha_compare_text() with a number of new functions: > > - ha_compare_ft_text_full() > - ha_compare_ft_text_prefix() > - ha_compare_ft_text() > - ha_compare_char_varying() > - ha_compare_char_fixed() > > The code branch corresponding to comparison of CHAR column keys > (HA_KEYTYPE_TEXT segment type) now uses ha_compare_char_fixed() > which calls strnncollsp_nchars(). > > For the rest of the code: > - comparison of VARCHAR/TEXT column keys > (HA_KEYTYPE_VARTEXT1, HA_KEYTYPE_VARTEXT2 segments types) > - comparison in the fulltext code > this patch does not change the behaviour.

2 3

Re: [PATCH] MDEV-31949 parallel slave xa Round-Robin distribution
by Kristian Nielsen 22 Oct '23

22 Oct '23

Kristian Nielsen <knielsen(a)knielsen-hq.org> writes: Hi Andrei, > commit 96bd9e6b780f3738b5008b89aae6b0b086f15943 > Author: Andrei <andrei.elkin(a)mariadb.com> > Date: Sat Aug 19 19:49:25 2023 +0300 > XA-Prepare group of events > > XA START xid > ... > XA END xid > XA PREPARE xid > > and its XA-"complete" terminator > > XA COMMIT or > XA ROLLBACK > > are made distributed Round-Robin across slave parallel workers. > The former hash-based policy was proven to attribute to execution > latency through creating a big - many times larger than the size > of the worker pool - queue of binlog-ordered transactions > to commit. > > Acronyms and notations used below: > > XAP := XA-Prepare event or the whole prepared XA group of events > XAC := XA-"complete", which is a solitary group of events > |W| := the size of the slave worker pool > Subscripts like `_k' denote order in a corresponding sequence > (e.g binlog file). Here is my review of this patch. > The list is arranged as a sliding window with the size of 2*|W| to account > a possibility of XAP_k -> XAP_k+2|W|-1 the largest (in the group-of-events > count sense) dependency. I spent a lot of effort trying to understand why the factor 2 in the size of 2*|W| occurs. Since each transaction must wait_for_prior_commit for the prior transaction, there should never be more than |W| transactions active. In your example with |W|=4, you say that > Worker #4 can take on its T_8 when T_1 is yet at the beginning of its processing But that does not seem possible. T_8 cannot start until T_1 (and 2..4) have done wakeup_subsequent_commits(), which surely shouldn't happen "at the beginning of its processing", but only at the end. But I think I now understand. The problem is that the XA code calls wakeup_subsequent_commits _before_ updating the XID hash (in XA PREPARE) / deleting from the XID hash (in XA COMMIT/ROLLBACK). Right? This I think is the fundamental issue to address. The wait_for_prior_commit mechanism is to ensure that the prior transaction has completed its commit, which means the state of the transaction is "committed" in memory. Thus, the XID hash, which records information about the status of the (XA) transactions, must also be updated before wakeup_prior_commit may be done. I can see that the XA PREPARE/COMMIT/ROLLBACK already uses the trx_group_commit_leader() code. So what you should do is to arrange for the update of / delete from the XID hash happens inside there, just before the wakeup_subsequent_commits() gets called: if (current->cache_mngr->using_xa && likely(!current->error) && DBUG_EVALUATE_IF("skip_commit_ordered", 0, 1)) { mysql_mutex_lock(&current->thd->LOCK_thd_data); run_commit_ordered(current->thd, current->all); mysql_mutex_unlock(&current->thd->LOCK_thd_data); } current->thd->wakeup_subsequent_commits(current->error); I checked the code, the XID hash update/delete currently happens shortly after that in the code, so there should be no problems moving it in there. [Incidentally, I also noticed that the XA PREPARE/COMMIT does not use the commit_ordered mechanism. While this seems (I think) unrelated to this patch, I think it is something you need to look into. The commit_ordered mechansim is central to ensuring that InnoDB and binlog commit in the same order, and I don't see how this is guaranteed for the current XA code (and I suspect this might also mean that binlog recovery will be broken in some cases).] Once the XID hash update/delete is moved as described, we have the very nice property that after an XA PREPARE/COMMIT/ROLLBACK T_i has done wakeup_subsequent_commits(), it is safe for a following T_j that refers to the same XID to replicate. This means that you can now use the existing wait_for_prior_commit mechanism to handle the dependencies between XA event groups with the same XID. Thus, there is no need to introduce a separate (and very complicated) wait mechanism of xid_cache_insert_maybe_wait() and xid_cache_search_maybe_wait(), solely for parallel replication of user XA. Thus, in the SQL driver thread, using your sliding window (which need only be of size |W| I believe), you can mark a "T1: XA COMMIT <xid> " followed by "T2: XA PREPARE <xid>" of a duplicate XID, that T2 must do a wait_for_prior_commit(T1) before running. This is simple, it can use the existing mechanims for that, using entry->last_committed_sub_id. Just like rgi->wait_commit_sub_id and rgi->wait_commit_group_info, we can introduce eg.: rgi->pre_dependency_sub_id rgi->pre_dependency_group_info and then do a wait_for_prior_commit(&rgi->pre_dependency_group_info->commit_orderer) if the rgi->pre_dependency_sub_id > entry->last_committed_sub_id) Similarly, the sliding window can record for "T3: XA PREPARE" and "T4: XA COMMIT", that T4 should do a wait_for_prior_commit(T3) before running. Then T4 will be sure that the XA transaction is ready. This will also be a good preparation for later introducing more of this kind of pre-calculated dependencies to the parallel replication scheduling, along the lines of the "balanced applier" that you have written about previously. These ideas I believe have tremendous potential, and handling this in a generic way is a very big improvement. I'm not suggesting to implement more than what is necessary now for XA, but keep it in mind that this dependency calculation in the SQL driver thread can be used for other than user XA in the future. You can also name the new introduced dependency fields appropriately (ie. with generic names not specific to XA when applicable). This will greatly simplify the patch, I believe; and more importantly it will integrate the XA-required scheduling in a clean and generic way in the parallel replication code. Following is more detailed comments on the patch: > - DBUG_ASSERT( > - !(thd->rgi_slave && > - !thd->rgi_slave->worker_error && > - thd->rgi_slave->did_mark_start_commit) || > - (thd->transaction->xid_state.is_explicit_XA() || > - (thd->rgi_slave->gtid_ev_flags2 & Gtid_log_event::FL_PREPARED_XA))); > - > + DBUG_ASSERT(!(thd->rgi_slave && > + !thd->rgi_slave->worker_error && > + thd->rgi_slave->did_mark_start_commit) || > + (thd->transaction->xid_state.is_explicit_XA() || > + (thd->rgi_slave->gtid_ev_flags2 & > + (Gtid_log_event::FL_PREPARED_XA | > + Gtid_log_event::FL_COMPLETED_XA)))); > if (thd->rgi_slave && > !thd->rgi_slave->worker_error && > thd->rgi_slave->did_mark_start_commit) > thd->rgi_slave->unmark_start_commit(); Why is it allowed to rollback here while did_mark_start_commit is true for XA PREPARE or XA COMMIT? The comment above should be extended to explain this. And why does the following if () statement then still do an emergency unmark_start_commit(), when the condition is not caught by the assertion? It looks like something is wrong in the earlier code, the intention here is that it should never be necessary to unmark_start_commit() here, and if it is, then it is a bug flagged by the assertion. > @@ -1751,7 +1753,8 @@ binlog_flush_cache(THD *thd, binlog_cache_mngr *cache_mngr, > > if ((using_stmt && !cache_mngr->stmt_cache.empty()) || > (using_trx && !cache_mngr->trx_cache.empty()) || > - thd->transaction->xid_state.is_explicit_XA()) > + (thd->transaction->xid_state.is_explicit_XA() || > + (thd->rgi_slave && thd->rgi_slave->is_async_xac))) Why is this extra condition thd->rgi_slave->is_async_xac necessary? There are many of these conditions spread around the code in the patch. I think perhaps it is because the transaction is somehow not "connected" to the THD? Because it is in the XID cache? But the situation must be similar on the master, if the XA COMMIT happens in a different transaction from XA PREPARE. So this should be done the same way on the slave, so that the XA transaction gets connected to the THD of the worker thread processing it, and so that these extra rgi_slave->is_async_xac conditions are not needed. It is very fragile to have such conditions spread around the code, it will be impossible to avoid bugs due to forgetting such extra condition on one place or another. > + /* > + While xid_state.get_xid() is a robust method to access `xid` > + it can't be used on slave by the asynchronously running XA-"complete". > + In the latter case thd->lex->xid is safely accessible. > + */ > + buflen= serialize_with_xid(is_async_xac? thd->lex->xid : > + thd->transaction->xid_state.get_xid(), Same here, it should be possible to use the same xid_state in the slave thread as on the master, so we avoid special cases all over the code for the slave threads. > +static bool acquire_xid(THD *thd) > +{ > + bool rc= false; > + > + if (thd->rgi_slave && thd->rgi_slave->is_async_xac && > + thd->rgi_slave->gtid_ev_flags2 & Gtid_log_event::FL_COMPLETED_XA) > + { > + XID_STATE &xid_state= thd->transaction->xid_state; This function actually seems to do part of "XA transaction gets connected to the THD". But it's called from binlog_commit_by_xid() and binlog_rollback_by_xid(), which means it has to have yet another of these special-condition checks for thd->rgi_slave->is_async_xac etc. Instead, call this from the code that applies the "XA COMMIT" event (eg. in log_event_server.cc), and make sure it fully connects the XA transaction to the the THD. Then you get rid of the checks for is_async_xac in a lot of places, and also avoid polluting with replication details the code than runs the original transaction on the master. > + rc= binlog_commit(thd, TRUE, FALSE); > + thd->ha_data[binlog_hton->slot].ha_info[1].reset(); > + } > + if (!rc) > + { > + rc= acquire_xid(thd); > + } > + if (thd->is_current_stmt_binlog_disabled()) > + { > + thd->wakeup_subsequent_commits(rc); > + } So IIUC, here we first binlog the XA COMMIT and then wakeup_subsequent_commits(). But the engine commit only happens afterwards, in ha_commit_or_rollback_by_xid. Why wakeup_subsequent_commits() here? It seems too early, the transaction is not yet committed in the engine, how do you avoid that the commits in the engine will happen in the wrong order? And then a mariabackup might take a backup with T2 committed but T1 not and not have a valid replication position to provision a slave. And also, is this skipping the binlog transaction coordinator and two-phase commit with the engines? And in particular, commit_ordered()? Then again, it seems we have the problem that the engine can commit in the opposite order from the binlog. I wonder if this will also affect crash recovery if we crash with different commit order in binlog and engine? So I think wakeup_subsequent_commits() here is wrong, should be removed. > @@ -2185,7 +2293,9 @@ int binlog_commit(THD *thd, bool all, bool ro_1pc) > } > > if (cache_mngr->trx_cache.empty() && > - (thd->transaction->xid_state.get_state_code() != XA_PREPARED || > + ((thd->transaction->xid_state.get_state_code() != XA_PREPARED && > + !(thd->rgi_slave && thd->rgi_slave->is_parallel_exec && > + thd->lex->sql_command == SQLCOM_XA_COMMIT)) || As explained above, this condition (and the similar in binlog_rollback() should be avoided. > @@ -10510,13 +10624,20 @@ int TC_LOG_BINLOG::unlog_xa_prepare(THD *thd, bool all) > > binlog_cache_mngr *cache_mngr= thd->binlog_setup_trx_data(); > int cookie= 0; > + int rc= 0; > + > + if (thd->rgi_slave && thd->is_current_stmt_binlog_disabled()) > + { > + rc= thd->wait_for_prior_commit(); > + if (rc == 0) > + thd->wakeup_subsequent_commits(rc); > + return rc; > + } Why is this necessary? If it is really necessary, then put it in the code that applies the XA PREPARE event, not as a random special case in generic code. > - bool rc= false; > - > > rc= write_empty_xa_prepare(thd, cache_mngr); // normally gains need_unlog > > static bool write_empty_xa_prepare(THD *thd, binlog_cache_mngr *cache_mngr) > { > return binlog_commit_flush_xa_prepare(thd, true, cache_mngr); > } Smaller point: since you're fixing the type of `rc` to be int (which is correct), also fix the return type of write_empty_xa_prepare() to be int and not bool - as binlog_commit_flush_xa_prepare() as well as unlog_xa_prepare() return int, not bool. > @@ -3314,16 +3314,22 @@ Gtid_log_event::Gtid_log_event(THD *thd_arg, uint64 seq_no_arg, > XID_STATE &xid_state= thd->transaction->xid_state; > if (is_transactional) > { > - if (xid_state.is_explicit_XA() && > - (thd->lex->sql_command == SQLCOM_XA_PREPARE || > - xid_state.get_state_code() == XA_PREPARED)) > + bool is_async_xac= false; > + if ((xid_state.is_explicit_XA() && > + (thd->lex->sql_command == SQLCOM_XA_PREPARE || > + xid_state.get_state_code() == XA_PREPARED)) || > + (is_async_xac= (thd->rgi_slave && thd->rgi_slave->is_async_xac))) Again, this is the generic GTID event contructor, it shouldn't need this kind of logic. It's very strange that a simple class constructor returns something different depending on which thread it runs in! Hopefully this is no longer necessary after XA transaction gets properly connected to the THD. But else, it should be handled by passing in suitable flag from the caller, or by the caller setting the required modifications after construction. > +using std::max; Please don't. Using std::max() explicit is not long and makes it explicit what `max` implementation is used. > @@ -760,7 +762,8 @@ convert_kill_to_deadlock_error(rpl_group_info *rgi) > return; > err_code= thd->get_stmt_da()->sql_errno(); > if ((rgi->speculation == rpl_group_info::SPECULATE_OPTIMISTIC && > - err_code != ER_PRIOR_COMMIT_FAILED) || > + (err_code != ER_PRIOR_COMMIT_FAILED && > + err_code != ER_XAER_NOTA)) || Why? What is special about the ER_XAER_NOTA error? Is it really safe to speculatively run the XA COMMIT in parallel, and then retry it for any other error than ER_XAER_NOTA ? > @@ -2467,6 +2463,7 @@ free_rpl_parallel_entry(void *element) > } > mysql_cond_destroy(&e->COND_parallel_entry); > mysql_mutex_destroy(&e->LOCK_parallel_entry); > + e->concurrent_xaps_window.~Dynamic_array(); > my_free(e); No, let's not do this, calling a destructor explicitly in a POD managed by my_malloc() / my_free(). Why not just make concurrent_xaps_window a pointer to the Dynamic_array and manage with new / delete? Then you can also allocate it lazily, so it will not need to be allocated at all for the 99.9xx% of users who are not replicating user XA transactions. The size of the sliding window is fixed anyway (until slave_parallel_threads is changed), so why use Dynamic_array at all, why not just allocate a plain array of the right size? > +template <> > +struct std::hash<XID> > +{ > + std::size_t operator()(const XID& xid) const > + { > + return my_hash_sort(&my_charset_bin, xid.key(), xid.key_length()); I don't understand, what is the purpose of introducing this function object? Why not just call my_hash_sort() directly (seems it's only used in one place) and avoid this code-obfuscation? > + Dynamic_array<std::pair<std::size_t, uint32>> concurrent_xaps_window; This is dangerous as Dynamic_array has a destructor but rpl_parallel_entry is a POD where we don't call any constructor/destructor. Should instead use a pointer to an object constructed with new (or not use Dynamic_array at all as suggested above). > + /* > + When true indicates that the user xa transaction is going to > + complete (with COMMIT or ROLLBACK) by the worker thread, > + *while* another worker is still preparing it. Once the latter is done > + the xid will be acquired and the flag gets reset. > + */ > + bool is_async_xac; > + I don't understand this comment. Surely it is not possible to *complete*, with XA COMMIT, a transaction before the corresponding XA PREPARE has completed (possibly in another worker thread)? Do you mean that it is possible for a worker thread W2 to *start* applying the XA COMMIT speculatively, before another worker W1 has completed the XA PREPARE? But I think this is not necessary, see below the discussion that the XA COMMIT worker can just do a wait_for_prior_commit() on the event group of its XA PREPARE. Then this should not be necessary. > @@ -137,7 +137,7 @@ template <class Elem> class Dynamic_array > */ > Elem& at(size_t idx) > { > - DBUG_ASSERT(idx < array.elements); > + DBUG_ASSERT(idx < max_size()); > return *(((Elem*)array.buffer) + idx); No, this cannot possibly be right. This is a patch about XA replication, why would it suddenly allow accessing deleted elements of any Dynamic_array used in the server?!? Surely you can allocate the elements you need for your sliding window. In fact, since the window is fixed size anyway, why use Dynamic_array at all? @@ -2367,6 +2367,15 @@ struct wait_for_commit event group is fully done. */ bool wakeup_blocked; + /* + The condition variable servers as a part of facilities to handle various + commit time additional dependency between groups of replication events, e.g + XA-Prepare -> XA-Commit, or XA-Prepare -> XA-Prepare all with the same xid. + */ + mysql_cond_t COND_wait_commit_dep; No, this doesn't belong in struct wait_for_commit. wait_for_commit is a low-level mechanism for ordering commits, it should not need any knowledge even of replication, certainly not of user-XA replication. The COND_wait_commit_dep isn't even used in any function related to wait_for_commit or even sql_class.cc, it's only used in sql/xa.cc. [But as discussed at the start, we can use the existing wait_for_prior_commit instead of introducing this new mechanism just for replicating user XA.] > -static XID_cache_element *xid_cache_search(THD *thd, XID *xid) > +XID_cache_element *xid_cache_search(THD *thd, XID *xid) Why remove the `static`? The function is not used outside of sql/xa.cc > @@ -254,16 +259,221 @@ static XID_cache_element *xid_cache_search(THD *thd, XID *xid) > + if (thd->rgi_slave && thd->rgi_slave->is_parallel_exec) > + { > + DBUG_ASSERT(thd->lex->sql_command == SQLCOM_XA_COMMIT || > + thd->lex->sql_command == SQLCOM_XA_ROLLBACK); > + thd->rgi_slave->is_async_xac= true; Here again we have some logic that relates to replication being inserted somewhat randomly in low-level code, requiring this thd->rgi_slave->is_parallel_exec condition. Instead of doing this, the caller higher up should handle this depending on the return of xid_cache_search(). This way, code paths that are not related to replication don't get affected by the replication-specific logic. > +bool xid_cache_insert_maybe_wait(THD* thd) > +{ This new wait mechanism just for parallel replication of user XA seems very complex. Spin loop, and extra locking and condition variables and locking on a lock-free hash. As explained at the start, let's instead avoid introducing a new mechanism and use the existing wait_for_prior_commit, which is central to parallel replication and highly optimized for this. IIUC, the real problem here is that after wakeup_subsequent_commits() of XA COMMIT, the XID is still in the xid_hash. This is because of this code in trans_xa_commit(): ha_commit_or_rollback_by_xid(thd->lex->xid, !res); if (!res && thd->is_error()) { // hton completion error retains xs/xid in the cache, // unless there had been already one as reflected by `res`. res= true; goto _end_external_xid; } xid_cache_delete(thd, xs); xid_deleted= true; ha_commit_or_rollback_by_xid() ends up in queue_for_group_commit() which will do the wakeup_subsequent_commits. But the xid_cache_delete() happens just after that. This is too late. The wakeup_subsequent_commits() must not occur until after the transaction is fully committed in memory. So instead, move the xid_cache_delete() call so it happens inside the group commit code. Then when wakeup_subsequent_commits() is done after XA COMMIT, the XID entry will be gone. It would go here in MYSQL_BIN_LOG::trx_group_commit_leader(): if (current->cache_mngr->using_xa && likely(!current->error) && DBUG_EVALUATE_IF("skip_commit_ordered", 0, 1)) { mysql_mutex_lock(&current->thd->LOCK_thd_data); run_commit_ordered(current->thd, current->all); mysql_mutex_unlock(&current->thd->LOCK_thd_data); } current->thd->wakeup_subsequent_commits(current->error); I noticed that commit_ordered is not done for user XA transactions in replication. The direct reason for that seems to be that they do not use log_and_order(), which sets cache_mngr->using_xa. Can you explain why the XA PREPARE and XA COMMIT is not following the commit_ordered protocol, and what mechanism exists instead to ensure that everything still works correctly? For example that commit order in binlog and InnoDB will be the same, and that binlog recovery works? And finally, just for the record, I still do not agree with the way replication of user XA has been changed (aka MDEV-32020), and this patch does not fix the underlying problem. But changing it to use the wait_for_prior_commit() mechanism as explained at the start will improve the integration with the parallel replication code and be a step towards solving also some of the other underlying issues. - Kristian.

1 0

Locking thd->LOCK_thd_data during group commit
by Kristian Nielsen 21 Oct '23

21 Oct '23

Hi Serg, I came upon this commit by you: > commit 6b685ea7b0776430d45b095cb4be3ef0739a3c04 > Author: Sergei Golubchik <serg(a)mariadb.org> > Date: Wed Sep 28 18:55:15 2022 +0200 > > correctness assert > > thd_get_ha_data() can be used without a lock, but only from the > current thd thread, when calling from anoher thread it *must* > be protected by thd->LOCK_thd_data > > * fix group commit code to take thd->LOCK_thd_data > * remove innobase_close_connection() from the innodb background thread, > it's not needed after 87775402cd0c and was failing the assert with > current_thd==0 > @@ -8512,7 +8512,11 @@ MYSQL_BIN_LOG::trx_group_commit_leader(group_commit_entry *leader) > ++num_commits; > if (current->cache_mngr->using_xa && likely(!current->error) && > DBUG_EVALUATE_IF("skip_commit_ordered", 0, 1)) > + { > + mysql_mutex_lock(&current->thd->LOCK_thd_data); > run_commit_ordered(current->thd, current->all); > + mysql_mutex_unlock(&current->thd->LOCK_thd_data); > + } This seems _really_ expensive :-( This code runs for every transaction commit in replication, and it runs under a global LOCK_commit_ordered which needs to be held for as short as possible. The commit message doesn't mention anything about what goes wrong during the group commit code. And the patch doesn't have any test case that shows the problem, it just adds an assertion that will trigger during the group commit. So what is the actual reason for this change? Note that at this point, the group commit leader thread _is_ effectively acting as if it was each of the participating threads, the other threads are blocked from running during commit_ordered. If it is just so that we can have this assertion, then that needs to be rolled back. We _really_ don't want to take/release N mutexes in release builds during LOCK_commit_ordered, just to have an assertion in debug builds. If there is some reason, then the correct fix could be to set current_thd for the duration of run_commit_ordered, which will satisfy the assertion and presumably the actual bug? Well, it depends _what_ the actual reason/bug is. - Kristian.

3 5

Re: Review of MDEV-31273: Eliminate Log_event::checksum_alg
by Kristian Nielsen 17 Oct '23

17 Oct '23

Hi Monty, Lots of good comments, thanks! Replies inline: Michael Widenius <michael.widenius(a)gmail.com> writes: > enum enum_binlog_checksum_alg > > In c++ we can drop the first 'enum' > You can fix that by doing: Ack, will fix. > event.select_checksum_alg() > > This is called a lot. Would it not be easier to store the checksum_alg > in the event when it is created and access the variable directly? > On the other hand, the number of calls to event.select_checksum_alg() may > not change as we will only create one Yes, this is called only once per event, when writing the event into the binlog or the event cache. So it seems unnecessary to use extra memory to store it. And we don't necessarily even have binlog_cache_data when constructing the event to pass to select_checksum_alg(), so I think it is correct to select the checksum algorithm to use only when we write the event. > @@ -1258,11 +1258,10 @@ Log_event* Log_event::read_log_event(const > uchar *buf, uint event_len, > > if (ev) > { > - ev->checksum_alg= alg; > #ifdef MYSQL_CLIENT > if (ev->checksum_alg != BINLOG_CHECKSUM_ALG_OFF && > ev->checksum_alg != BINLOG_CHECKSUM_ALG_UNDEF) > ev->crc= uint4korr(buf + (event_len)); > + ev->read_checksum_alg= alg; > > Why move the setting of read_checksum_alg inside MYQL_CLIENT? It is because with this patch, ev->checksum_alg is no longer needed in the server (the class member is removed). It is needed in the client (mysqlbinlog), so I added a new member read_checksum_alg, which is the algorithm that was used to read the event. This is needed so that mysqlbinlog can output the checksum (if any) in textual form. But it is not needed in the server > To ensure that this is not wrongly used, please add > #ifdef MYSQL_CLIENT > > here: > > class Log_event: > > #ifdef MYSQL_CLIENT > + enum enum_binlog_checksum_alg read_checksum_alg; > #endif Yes, agree, that is the intention - the read_checksum_alg is only present in the class in the client, not in the server (and we want to avoid using memory for it in the server). This code is already in the #else branch of #ifdef MYSQL_SERVER: #ifdef MYSQL_SERVER ... #else /* The checksum algorithm used (if any) when the event was read. */ enum enum_binlog_checksum_alg read_checksum_alg; I thought this would have the same effect as your suggestion with #ifdef MYSQL_CLIENT. It probably does, but it seems somewhat complex when MYSQL_CLIENT and MYSQL_SERVER are defined. Maybe they can even both be defined, though probably not where log_event.h is included? It's a bit confusing that we have both MYSQL_CLIENT and MYSQL_SERVER used for conditional compilation in log_event.h. But probably not something that should be cleaned up in this patch. Do you think I should still add the #ifdef MYSQL_CLIENT as you suggested (even though it should be redundant in the #else branch of #ifdef MYSQL_SERVER)? It can't hurt after all, even if we always have MYSQL_CLIENT == !MYSQL_SERVER. > in Format_description_log_event(const uchar > > We have: > > else > { > checksum_alg= BINLOG_CHECKSUM_ALG_UNDEF; > used_checksum_alg= BINLOG_CHECKSUM_ALG_OFF; > } > > The used_checksum_alg is not needed as you set it at function start. Sorry, I don't understand? At the start we set it to "undef": + used_checksum_alg= BINLOG_CHECKSUM_ALG_UNDEF; This value will remain if there's an error return; else the else branch will change it to "off". Did you miss that it's set differently at the start of the function and in this else { } branch? Or did I miss something? > sql/log_event.h > > enum enum_binlog_checksum_alg set_checksum_alg(enum > enum_binlog_checksum_alg alg) > As we are just copying and restoring checksum_length, why not just do > > Format_description_log_event::write(Log_event_writer *writer) > > org_checksum_length= checksum_length... > .... > checksum_length= org_checksum_length; > > instead of calling set_checksum_alg() again ? > (The last set we can even do undonditionally which saves us an if) Agree, that code looks silly, I don't know why I did it that way. I'll fix it, remove the function set_checksum_alg() and just do: org_checksum_length= writer->checksum_length; writer->checksum_length= BINLOG_CHECKSUM_LEN; ... writer->checksum_length= org_checksum_length; > log_event_server.cc > > enum enum_binlog_checksum_alg Log_event::select_checksum_alg() > > Much simpler, but we do not check if cache_type is correct > > DBUG_ASSERT(((get_type_code() != ROTATE_EVENT && > get_type_code() != STOP_EVENT) || > get_type_code() != FORMAT_DESCRIPTION_EVENT) || > cache_type == Log_event::EVENT_NO_CACHE); > > Do you think this is not needed? > (This could be added just before: > > return BINLOG_CHECKSUM_ALG_OFF; Agree, I was happy to remove a lot of complexity and the need for a number of asserts, but this assertion could be kept. I'll add it back. > Anyway, please remove the else after: > > if (cache_type == Log_event::EVENT_NO_CACHE) > return (enum_binlog_checksum_alg)binlog_checksum_options; Ack will do. > Format_description_log_event::write(Log_event_writer *writer) > > > uint8 checksum_byte= (uint8) (used_checksum_alg != BINLOG_CHECKSUM_ALG_UNDEF ? > used_checksum_alg : BINLOG_CHECKSUM_ALG_OFF); > > Is it possible the alg can be UNDEF? > You have several asserts in the code that this should not happen. Yeah, honestly, I'm not sure. It's one of the annoying things in the code that I did not get to fix, that we have both the UNDEF and the OFF state for the checksum algorithm. Format_description code is not performance critical (only once per binlog file), so I suggest I leave the extra check for UNDEF as defensive code, but add an assertion that it is not UNDEF and see if it passes the testsuite. Does that sound ok? > slave.cc > > @@ -1870,10 +1873,8 @@ static int get_master_version_and_clock(MYSQL* > mysql, Master_info* mi) > until it has received a new FD_m. > */ > mi->rli.relay_log.description_event_for_queue->checksum_alg= > mi->rli.relay_log.relay_log_checksum_alg; > > Not sure why this is removed. > > Are these guaranteed to be the same here? > If yes, why not add an assert for this? This patch removes Log_event::checksum_alg and instead adds a field Format_description_log_event::used_checksum_alg which gets set in the Format_description_log_event constructor. So this assignment - mi->rli.relay_log.description_event_for_queue->checksum_alg= - mi->rli.relay_log.relay_log_checksum_alg; is replaced by this code just above which serves the same purpose: - Format_description_log_event(4, mysql->server_version); + Format_description_log_event(4, mysql->server_version, + mi->rli.relay_log.relay_log_checksum_alg); So I think adding an assert DBUG_ASSERT(mi->rli.relay_log.description_event_for_queue->checksum_alg == mi->rli.relay_log.relay_log_checksum_alg); is somewhat redundant, but let me know if you still think it will be useful and I'll add it. - Kristian.

2 1

Re: [MariaDB commits] [PATCH 1/4] MDEV-31273: Replace Log_event::writer with function parameter
by Kristian Nielsen 16 Oct '23

16 Oct '23

Hi Monty, [Following up on review on IRC] Kristian Nielsen via commits <commits(a)lists.mariadb.org> writes: > Replace Log_event::writer with just passing the writer object as a > function parameter to Log_event::write(). We wondered how the compiled code will look when passing the writer object as a parameter to Log_event::write() vs. temporarily storing it in the Log_event object. Interestingly, the code is actually "better" after my patch despite an extra parameter being passed along. This is because the compiler can cache the function parameter in a register across nested calls. While with a class member, the compiler cannot determine that the member will not be changed by the nested calls, so the value needs to be repeatedly re-loaded from memory. Some more details on this finding below. My reason for the change was probably more general, based on this code: > - ev->writer= this; > - int res= ev->write(); > - IF_DBUG(ev->writer= 0,); // writer must be set before every Log_event::write > + int res= ev->write(this); It's clear that the intention with this code is to pass the Log_event_writer object `this` to the ev->write() method, and it's a strange way to do it to temporarily set it as a member in the object rather than using a function parameter. Needing an IF_DBUG() to enforce/check that this is done correctly is a warning sign, and I was worried that as I did further changes I would introduce subtle bugs this way. It is interesting that this makes the code clearer not only for the developer (me), but also for the compiler in this case. Function parameter passing is quite well optimized on amd64, but it was still good to see this in detail in this case. And as you predicted we learned something from it! As we discussed, I will extend the commit message to explain these reasons better. Thanks for the suggestion; it's always good in the reviews to question the reason for changes like this. To check the compiled code in detail, I did a release build (BUILD/compile-pentium64-max) with and without this patch, and disassembled the User_var_log_event::write() function, which has a number of calls now passing the writer object as a function parameter: > - return write_header(event_length) || > - write_data(buf, sizeof(buf)) || > - write_data(name, name_len) || > - write_data(buf1, buf1_length) || > - write_data(pos, val_len) || > - write_data(&flags, unsigned_len) || > - write_footer(); > + return write_header(writer, event_length) || > + write_data(writer, buf, sizeof(buf)) || > + write_data(writer, name, name_len) || > + write_data(writer, buf1, buf1_length) || > + write_data(writer, pos, val_len) || > + write_data(writer, &flags, unsigned_len) || > + write_footer(writer); It looks like this would need to pass the `writer` argument many times to called functions. However, write_data() and write_footer() are actually inline functions: bool write_data(const uchar *buf, size_t data_length) { return writer->write_data(buf, data_length); } bool write_footer() { return writer->write_footer(); } So it's actually only write_header() where we now pass an extra parameter. For write_data() and write_footer() it was already being passed as the (implicit) `this` parameter. With the patch, the compiler saves the `writer` parameter in register r12 and uses it for each call: movq %rsi, %r12 ... movq %r12, %rsi # `writer` pointer movq %rbx, %rdi # `this` pointer call _ZN9Log_event12write_headerEP16Log_event_writerm@PLT ... movl $4, %edx # `data_length` parameter movq %r12, %rdi # `writer` pointer call _ZN16Log_event_writer10write_dataEPKhm@PLT ... leaq 137(%rbx), %rsi # `buf` parameter movq %r14, %rdx # `data_length` parameter movq %r12, %rdi # `writer` pointer call _ZN16Log_event_writer10write_dataEPKhm@PLT ... movq %r12, %rdi # `writer` pointer call _ZN16Log_event_writer12write_footerEv@PLT Before the patch, the code for User_var_log_event::write() loads this->writer for each call except write_header(): movq %rbx, %rdi # Here we don't pass `writer`, write_header() loads it from Log_event::writer call _ZN9Log_event12write_headerEm@PLT ... movq 88(%rbx), %rdi # this->writer leaq -126(%rbp), %rsi # `buf` parameter movl $4, %edx # `data_length` parameter call _ZN16Log_event_writer10write_dataEPKhm@PLT ... movq 88(%rbx), %rdi # this->writer leaq 153(%rbx), %rsi # `buf` parameter movq %r13, %rdx # `data_length` parameter call _ZN16Log_event_writer10write_dataEPKhm@PLT ... movq 88(%rbx), %rdi # this->writer call _ZN16Log_event_writer12write_footerEv@PLT I think the reason for the repeated loads of this->writer from 88(%rbx) is because the compiler cannot be sure that this->writer will not be modified by calls to write_data(). So it needs to reload the pointer each time. With the patch the pointer is a function parameter which cannot be modified by nested calls, so it can be saved in a register. Thus, we save some memory loads with my patch for each call of write_data() / write_footer(), which should be a bit faster (though perhaps not noticeably so). In general, passing parameters to functions is cheap (on amd64), as they are passed in registers, and register-to-register moves are very fast on high-performance CPUs, while memory loads have a relative high latency, even when they hit the cache. And like we see here, using local variables helps the compiler do alias analysis and cache values in registers, while this is a lot harder for the compiler to do for class members. - Kristian.

1 0

Re: d4c173ae6df: MDEV-27595 Backport SQL service, introduced by MDEV-19275.
by Sergei Golubchik 16 Oct '23

16 Oct '23

Hi, Alexey, On Oct 16, Alexey Botchkov wrote: > revision-id: d4c173ae6df (mariadb-10.4.30-117-gd4c173ae6df) > parent(s): 837b777bfd7 > author: Alexey Botchkov > committer: Alexey Botchkov > timestamp: 2023-09-05 13:18:12 +0400 > message: > > MDEV-27595 Backport SQL service, introduced by MDEV-19275. > > Fixes to make SQL SERVICE working. > > Conflicts: > storage/spider/spd_table.cc ^^^ better remove these two lines. normally git removes them automatically, and there's no spd_table.cc in this commit anyway. Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

1 0

Re: 837b777bfd7: MDEV-27595 Backport SQL service, introduced by MDEV-19275.
by Sergei Golubchik 16 Oct '23

16 Oct '23

Hi, Alexey, On Oct 16, Alexey Botchkov wrote: > revision-id: 837b777bfd7 (mariadb-10.4.30-116-g837b777bfd7) > parent(s): a49b9314c16 > author: Alexey Botchkov > committer: Alexey Botchkov > timestamp: 2023-09-05 13:18:12 +0400 > message: > > MDEV-27595 Backport SQL service, introduced by MDEV-19275. > > The SQL SERVICE backported into the 10.4. > > diff --git a/plugin/test_sql_service/CMakeLists.txt b/plugin/test_sql_service/CMakeLists.txt > new file mode 100644 > index 00000000000..615508bdc4e > --- /dev/null > +++ b/plugin/test_sql_service/CMakeLists.txt > @@ -0,0 +1,18 @@ > +# Copyright (C) 2013 Alexey Botchkov and SkySQL Ab > +# > +# This program is free software; you can redistribute it and/or modify > +# it under the terms of the GNU General Public License as published by > +# the Free Software Foundation; version 2 of the License. > +# > +# This program is distributed in the hope that it will be useful, > +# but WITHOUT ANY WARRANTY; without even the implied warranty of > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > +# GNU General Public License for more details. > +# > +# You should have received a copy of the GNU General Public License > +# along with this program; if not, write to the Free Software > +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1335 USA > + > +SET(SOURCES test_sql_service.c) > + > +MYSQL_ADD_PLUGIN(test_sql_service ${SOURCES} MODULE_ONLY) COMPONENT Test > diff --git a/plugin/test_sql_service/COPYING b/plugin/test_sql_service/COPYING > new file mode 100644 > index 00000000000..6e475df5526 > --- /dev/null > +++ b/plugin/test_sql_service/COPYING > @@ -0,0 +1,339 @@ > + GNU GENERAL PUBLIC LICENSE No need to, we already have one in the CMAKE_SOURCE_DIR, why do you want to copy it just for one plugin? Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

1 0

Re: About Release Model
by Kristian Nielsen 16 Oct '23

16 Oct '23

Hi Sergei, Sergei Golubchik <serg(a)mariadb.org> writes: > You've said that it's unclear how the release model works, when we > release what and where a feature can go. > > I've documented the current release model in the knowledge base, > https://mariadb.com/kb/en/mariadb-release-model/ > would you like to take a look and see if it answers your questions? This is a fine overview page, and if this info was not previously available, then it's surely good to have it documented. However, it doesn't help explain how to meaningfully participate as a core developer in the releases. Where does the discussion happen about the progress and planning of the release, and addressing problems? How to partipate in those discussions? How to push a patch into a preview release, or a release, how to make a release? Etc. My assumption is that this is all restricted to employees? But even that isn't explained on the page. - Kristian.

2 3

Question about lacking git commit message in dc9498b
by Otto Kekäläinen 12 Oct '23

12 Oct '23

Hi Sergei and others! While reviewing changes in MariaDB 10.11.5 I was wondering about this commit: https://github.com/mariadb/server/commit/dc9498beb6f79f08edd02fb7f7153e5150… It has only the default message from issuing 'git revert' and a human written explanation *why* the revert was done is missing. AFAIK the reverted commit https://github.com/MariaDB/server/commit/1d0e3d80d84c2b375a444cdbb41c28afd7… was correct and good - so why was it reverted?

2 1

Re: 6cb526bd363: MDEV-32123: require_secure_transport doesn't allow TCP connections
by Sergei Golubchik 10 Oct '23

10 Oct '23

Hi, Dmitry, Looks pretty good! Basically no comments, some minor nitpicking below, feel free to ignore. On Oct 10, Dmitry Shulga wrote: > revision-id: 6cb526bd363 (mariadb-11.2.1-9-g6cb526bd363) > parent(s): 872ed5342d8 > author: Dmitry Shulga > committer: Dmitry Shulga > timestamp: 2023-10-03 18:07:35 +0700 > message: > > MDEV-32123: require_secure_transport doesn't allow TCP connections > > In case the option require_secure_transport is on the user can't > establish a secure ssl connection over TCP protocol. Inability to set up > a ssl session over TCP was caused by the fact that a type of client's > connection was checked before ssl handshake performed (ssl handshake > happens at the function acl_authenticate()). At that moment vio type has > the value VIO_TYPE_TCPIP for client connection that uses TCP transport. > In result, checking for allowable vio type for fails despite the fact > that SSL session being established. To fix the issue move checking of > vio type for allowable values inside the function > parse_client_handshake_packet() > right after client's capabilities discovered that SSL is not requested > by the client. > > diff --git a/mysql-test/main/require_secure_transport.result b/mysql-test/main/require_secure_transport.result > index 94474d0fa0e..8fa273cbae4 100644 > --- a/mysql-test/main/require_secure_transport.result > +++ b/mysql-test/main/require_secure_transport.result > @@ -1,8 +1,13 @@ > -CREATE TABLE t1 (t int(1)); > SET GLOBAL require_secure_transport=ON; > -ERROR HY000: Connections using insecure transport are prohibited while --require_secure_transport=ON. > +ERROR 08004: Connections using insecure transport are prohibited while --require_secure_transport=ON. I wonder why sqlstate has changed now, as far as I remember it was the previous patch that changed it. Was the test skipped until now? > +SELECT (VARIABLE_VALUE <> '') AS have_ssl FROM INFORMATION_SCHEMA.SESSION_STATUS WHERE VARIABLE_NAME='Ssl_cipher'; > +have_ssl > +1 > +disconnect with_ssl; > connection default; > SET GLOBAL require_secure_transport=OFF; > +SELECT (VARIABLE_VALUE <> '') AS have_ssl FROM INFORMATION_SCHEMA.SESSION_STATUS WHERE VARIABLE_NAME='Ssl_cipher'; > +have_ssl > +0 > disconnect without_ssl; > connection default; > -DROP TABLE t1; > diff --git a/mysql-test/main/require_secure_transport.test b/mysql-test/main/require_secure_transport.test > index e238e732423..f3da73a5167 100644 > --- a/mysql-test/main/require_secure_transport.test > +++ b/mysql-test/main/require_secure_transport.test > @@ -1,15 +1,19 @@ > -- source include/have_ssl_communication.inc > -CREATE TABLE t1 (t int(1)); > SET GLOBAL require_secure_transport=ON; > --disable_query_log > --error ER_SECURE_TRANSPORT_REQUIRED > connect without_ssl,localhost,root,,,,,TCP NOSSL; > + > +connect with_ssl,localhost,root,,,,,TCP SSL; > --enable_query_log I'd suggest to remove disable/enable_query_log and let connect commmand show in the result file. It'll make them easier to read. You'll need replace_result before the failing connect to replace the port away. > +SELECT (VARIABLE_VALUE <> '') AS have_ssl FROM INFORMATION_SCHEMA.SESSION_STATUS WHERE VARIABLE_NAME='Ssl_cipher'; > +disconnect with_ssl; > + > connection default; > SET GLOBAL require_secure_transport=OFF; > --disable_query_log > connect without_ssl,localhost,root,,,,,TCP NOSSL; > --enable_query_log > +SELECT (VARIABLE_VALUE <> '') AS have_ssl FROM INFORMATION_SCHEMA.SESSION_STATUS WHERE VARIABLE_NAME='Ssl_cipher'; > disconnect without_ssl; > connection default; > -DROP TABLE t1; Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

1 0

Re: 217a7a7ceff: online alter: rework savepoints
by Sergei Golubchik 09 Oct '23

09 Oct '23

Hi, Nikita, On Sep 29, Nikita Malyavin wrote: > revision-id: 217a7a7ceff (mariadb-11.2.1-10-g217a7a7ceff) > parent(s): ad6a305d84d > author: Nikita Malyavin > committer: Nikita Malyavin > timestamp: 2023-09-23 11:46:36 +0400 > message: > > online alter: rework savepoints > > Use standard handlerton functions for savepoint add/rollback. > > To identify the savepoint, the pointer passed is used. > > Every table that has online alter in progress maintains a list of > savepoints independently. > > Also this removes setting a value to a global variable savepoint_alloc_size > without any protection, which was a race condition bug. > > --- a/sql/online_alter.cc > +++ b/sql/online_alter.cc > @@ -6,7 +6,19 @@ > > static handlerton *online_alter_hton; > > -using Online_alter_cache_list= ilist<online_alter_cache_data>; > +using sv_id_t= void *; typedef, please > + > +struct Online_alter_cache_list: ilist<online_alter_cache_data> > +{ > + sv_id_t savepoint_id= 0; > +}; > @@ -281,13 +308,11 @@ static int online_alter_log_init(void *p) > { > online_alter_hton= (handlerton *)p; > online_alter_hton->db_type= DB_TYPE_ONLINE_ALTER; > - online_alter_hton->savepoint_offset= sizeof(my_off_t); > + online_alter_hton->savepoint_offset= 1; you can even use 0, I guess. Otherwise it'll still be 8, as the server will need to align allocation, I suppose. But I suggest to try to swap the implementation. Make online_alter_hton->savepoint_offset=sizeof(void*); and in savepoint_set create a list or an array of all offsets and store it in the savepoint. Something like this: struct { uint savepoints; my_off_t log_prev_pos[1]; } oasv; int online_alter_savepoint_set(handlerton *hton, THD *thd, oasv **sv) { uint count= 0, i=0; for (auto &cache: cache_list) count++; DBUG_ASSERT(count); *sv= malloc(sizeof(oasv) + sizeof(my_off_t)*(count - 1)); (*sv)->savepoints= count; for (auto &cache: cache_list) (*sv)->log_prev_pos[i]= cache.get_byte_position(); } and so on. You just need to make sure you always use push_back and not push_front for adding caches to the cache_list. instead of a structure with a counter you can use an array with MY_OFF_T_UNDEF (not 0!) at the end. > online_alter_hton->close_connection= online_alter_close_connection; > > - online_alter_hton->savepoint_set= // Done by online_alter_savepoint_set > - [](handlerton *, THD *, void *){ return 0; }; > - online_alter_hton->savepoint_rollback= // Done by online_alter_savepoint_rollback > - [](handlerton *, THD *, void *){ return 0; }; > + online_alter_hton->savepoint_set= online_alter_savepoint_set; > + online_alter_hton->savepoint_rollback= online_alter_savepoint_rollback; > online_alter_hton->savepoint_rollback_can_release_mdl= > [](handlerton *hton, THD *thd){ return true; }; Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 2

Re: ad6a305d84d: online alter: use thd->ha_data to store cache_list
by Sergei Golubchik 09 Oct '23

09 Oct '23

Hi, Nikita, On Sep 29, Nikita Malyavin wrote: > revision-id: ad6a305d84d (mariadb-11.2.1-9-gad6a305d84d) > parent(s): 132c56f1325 > author: Nikita Malyavin > committer: Nikita Malyavin > timestamp: 2023-09-23 11:46:36 +0400 > message: > > online alter: use thd->ha_data to store cache_list nice > diff --git a/sql/online_alter.cc b/sql/online_alter.cc > index 3255fc06c3e..5ad2156cd21 100644 > --- a/sql/online_alter.cc > +++ b/sql/online_alter.cc > @@ -6,6 +6,7 @@ > > static handlerton *online_alter_hton; > > +using Online_alter_cache_list= ilist<online_alter_cache_data>; please, use typedef for that > class online_alter_cache_data: public Sql_alloc, public ilist_node<>, > public binlog_cache_data > @@ -246,7 +268,11 @@ int online_alter_savepoint_rollback(THD *thd, LEX_CSTRING name) > > static int online_alter_close_connection(handlerton *hton, THD *thd) > { > - DBUG_ASSERT(thd->online_alter_cache_list.empty()); > + Ha_data &ha= thd->ha_data[online_alter_hton->slot]; > + auto *cache_list= (Online_alter_cache_list*)ha.ha_ptr; why not thd_get_ha_data()? and don't forget thd_set_ha_data(thd, online_alter_hton, 0); THD can be reused, I suppose, so let's play safe. > + > + DBUG_ASSERT(!cache_list || cache_list->empty()); > + delete cache_list; > return 0; > } Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 3

Re: MariaDB 10.11.5 mtr --big results from amd64/arm64/armhf/ppc64el/riscv64/s390x
by Otto Kekäläinen 09 Oct '23

09 Oct '23

Hi! > > I just wanted to share the summary of mariadb-test-run with --big from > > all available architectures on Launchpad > > (https://launchpad.net/~otto/+archive/ubuntu/mariadb/+builds?build_text=&bui…) > > I have tried to fix type_test_double and maria.maria-purge in 10.6. > With 'tried', I mean that I have not been able to repeat the test, but > I have a clue what is going on. > I will push later today. > Any chance you can try to run bb-10.6-monty on launchpad to test my fixes? I cherry-picked these two commits: https://github.com/MariaDB/server/commit/8e093d7597664290b0b5e0eb401b3a180f… https://github.com/MariaDB/server/commit/630cb99823f8e83545ffa16d60c9ac75cd… .. and pushed to https://launchpad.net/~otto/+archive/ubuntu/mariadb/+builds?build_text=&bui… and indeed both tests are now fixed: maria.maria-purge w1 [ pass ] 2134 type_test.type_test_double w1 [ pass ] 59 (from https://launchpadlibrarian.net/691041067/buildlog_ubuntu-mantic-amd64.maria…) Thanks Monty! Last failing that remains after Monty's and Daniel's fixes is: - unit.conc_connection: CONC-640/MDEV-30885

1 0

MariaDB 10.11.5 mtr --big results from amd64/arm64/armhf/ppc64el/riscv64/s390x
by Otto Kekäläinen 07 Oct '23

07 Oct '23

Hi! I just wanted to share the summary of mariadb-test-run with --big from all available architectures on Launchpad (https://launchpad.net/~otto/+archive/ubuntu/mariadb/+builds?build_text=3D&= build_state=3Dall): s390x Completed: Failed 13/5644 tests, 99.77% were successful. Failing test(s): innodb.row_size_error_log_warnings_3 disks.disks_notembedded type_test.type_test_double maria.maria-purge unit.conc_connection main.func_json_notembedded riscv64 Completed: Failed 6/6289 tests, 99.90% were successful. Failing test(s): disks.disks_notembedded type_test.type_test_double unit.conc_connection ppc64el Completed: Failed 9/6291 tests, 99.86% were successful. Failing test(s): disks.disks_notembedded type_test.type_test_double maria.maria-purge unit.conc_connection armhf Too many failed: Failed 19/1767 tests, 98.92% were successful. Failing test(s): federated.federatedx_versioning spider/bg.basic_sql spider/bg.spider_fixes_part spider.basic_sql spider.spider_fixes_part versioning.online versioning.partition period.versioning arm64 completed: Failed 9/6291 tests, 99.86% were successful. Failing test(s): disks.disks_notembedded type_test.type_test_double maria.maria-purge unit.conc_connection amd64 Completed: Failed 9/6291 tests, 99.86% were successful. Failing test(s): disks.disks_notembedded type_test.type_test_double maria.maria-purge unit.conc_connection As you can see, these seem to affect all architectures: - maria.maria-purge: MDEV-31349, easily repeatable failure on all architectures on Debian CI, Salsa CI and Launchpad - type_test.type_test_double: MDEV-22243 wasn't actually fixed yet, and failure visible on all architectures and platforms tested - unit.conc_connection: CONC-640/MDEV-30885 visible on all architectures tested - disk.disk{_notembedded}: MDEV-21587 Of these, Daniel Black is currently fixing the item 4 in the list (disk.disk) at https://github.com/MariaDB/server/pull/2777. For the items 1-3 help would be needed. The armhf situation does not map into a single MDEV. Do we have any Raspberry Pi enthusiasts here who would like to run the mariadb-test-run suite on their Pi and try to fix some of the failures? Overall it is nice to see nearly 6k tests pass on so many architectures! The Debian builds at https://buildd.debian.org/status/package.php?mariadb only run post-build the main suite (~1060 tests), and some architectures don't run any tests (e.g. m68k), just the build, but still nice to see the architecture support situation being very good at the moment for 10.11.5 across official and in-official Debian architectures as well :) Many thanks in particular to Daniel, Marko, Sergei, Tuukka, Faustin and Andrew for helping to hunt for fixes to the MariaDB code and tests to have wide architecture support! - Otto

1 0

Re: 132c56f1325: online alter: extract the source to a separate file
by Sergei Golubchik 29 Sep '23

29 Sep '23

Hi, Nikita, On Sep 29, Nikita Malyavin wrote: > revision-id: 132c56f1325 (mariadb-11.2.1-8-g132c56f1325) > parent(s): a71b13433f9 > author: Nikita Malyavin > committer: Nikita Malyavin > timestamp: 2023-09-23 11:46:36 +0400 > message: > > online alter: extract the source to a separate file > > Move all the functions dedicated to online alter to a newly created > online_alter.cc. Copyright/GPL header in both new files, please. Also one comment below. > With that, make many functions static and simplify the static functions > naming. > > Also, rename binlog_log_row_online_alter -> online_alter_log_row. > > diff --git a/sql/log_event.h b/sql/log_event.h > index 848ad0cead9..35142e034f3 100644 > --- a/sql/log_event.h > +++ b/sql/log_event.h > @@ -729,19 +729,6 @@ enum Log_event_type > }; > > > -/* > - Bit flags for what has been writing to cache. Used to > - discard logs with table map events but not row events and > - nothing else important. This is stored by cache. > -*/ > - > -enum enum_logged_status > -{ > - LOGGED_TABLE_MAP= 1, > - LOGGED_ROW_EVENT= 2, > - LOGGED_NO_DATA= 4, > - LOGGED_CRITICAL= 8 > -}; > > static inline bool LOG_EVENT_IS_QUERY(enum Log_event_type type) > { > @@ -979,6 +966,19 @@ typedef struct st_print_event_info > } PRINT_EVENT_INFO; > #endif > > +/** > + Bit flags for what has been writing to cache. Used to > + discard logs with table map events but not row events and > + nothing else important. This is stored by cache. > +*/ > +enum enum_logged_status > +{ > + LOGGED_TABLE_MAP= 1, > + LOGGED_ROW_EVENT= 2, > + LOGGED_NO_DATA= 4, > + LOGGED_CRITICAL= 8 > +}; > + why did you move it? looks unintentional. Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

1 0

Re: a71b13433f9: MDEV-32126 Assertion fails upon online ALTER and binary log enabled
by Sergei Golubchik 29 Sep '23

29 Sep '23

Hi, Nikita, On Sep 29, Nikita Malyavin wrote: > revision-id: a71b13433f9 (mariadb-11.2.1-7-ga71b13433f9) > parent(s): 883e7e5f7e7 > author: Nikita Malyavin > committer: Nikita Malyavin > timestamp: 2023-09-22 17:55:14 +0400 > message: > > MDEV-32126 Assertion fails upon online ALTER and binary log enabled > > Assertion `!writer.checksum_len || writer.remains == 0' fails upon > concurrent online ALTER and transactions with failing statements and binary > log enabled. > Also another assertion, `pos != (~(my_off_t) 0)', fails in my_seek, upon > reinit_io_cache, on a simplified test. This means that IO_CACHE wasn't > properly initialized, or had an error before. > > The overall problem is a deep interference with the effect of an installed > binlog_hton. As I wrote in Jira, please, provide a better explanation here. I had to debug the issue myself and as far as I can see, the problem happens because you register binlog_hton for a transaction, but doesn't prepare the cache manager for it. This can be fixed simply with @@ -2279,6 +2279,12 @@ int binlog_log_row_online_alter(TABLE* table, const ucha> trans_register_ha(thd, false, binlog_hton, 0); if (thd->in_multi_stmt_transaction_mode()) trans_register_ha(thd, true, binlog_hton, 0); + if (mysql_bin_log.is_open()) + { + binlog_cache_mngr *cache_mngr= thd->binlog_get_cache_mngr(); + if (cache_mngr && cache_mngr->trx_cache.get_prev_position() == MY_OFF_T_UNDEF) + thd->binlog_set_stmt_begin(); + } } which solves all cases from the bug report. Don't get me wrong, I don't suggest this fix, it's kind of hackish, and I liked how other pieces of the puzzle fell into a place after a separate handlerton, like savepoints and online_alter_cache_list. I'm just saying that "magical interference with the binlog_hton" doesn't explain the problem. At all. > Solution: > Extract online alter operations into a separate handlerton. > > --- a/sql/CMakeLists.txt > +++ b/sql/CMakeLists.txt > @@ -218,6 +218,8 @@ MYSQL_ADD_PLUGIN(partition ha_partition.cc STORAGE_ENGINE DEFAULT STATIC_ONLY > RECOMPILE_FOR_EMBEDDED) > MYSQL_ADD_PLUGIN(sql_sequence ha_sequence.cc STORAGE_ENGINE MANDATORY STATIC_ONLY > RECOMPILE_FOR_EMBEDDED) > +MYSQL_ADD_PLUGIN(online_alter_log log.cc STORAGE_ENGINE MANDATORY STATIC_ONLY > +NOT_EMBEDDED) better add it next to builtin_maria_binlog_plugin, instead of via MYSQL_ADD_PLUGIN. > ADD_LIBRARY(sql STATIC ${SQL_SOURCE}) > MAYBE_DISABLE_IPO(sql) > diff --git a/sql/log.cc b/sql/log.cc > index ee78844464e..132ec254adf 100644 > --- a/sql/log.cc > +++ b/sql/log.cc > @@ -12522,3 +12512,61 @@ void wsrep_register_binlog_handler(THD *thd, bool trx) > } > > #endif /* WITH_WSREP */ > + > +static int online_alter_close_connection(handlerton *hton, THD *thd) > +{ > + DBUG_ASSERT(thd->online_alter_cache_list.empty()); > + return 0; > +} > + > +handlerton *online_alter_hton; > + > +int online_alter_log_init(void *p) > +{ > + online_alter_hton= (handlerton *)p; > + online_alter_hton->db_type= DB_TYPE_ONLINE_ALTER; > + online_alter_hton->savepoint_offset= sizeof(my_off_t); > + online_alter_hton->close_connection= online_alter_close_connection; > + > + online_alter_hton->savepoint_set= // Done by online_alter_savepoint_set > + [](handlerton *, THD *, void *){ return 0; }; > + online_alter_hton->savepoint_rollback= // Done by online_alter_savepoint_rollback > + [](handlerton *, THD *, void *){ return 0; }; > + online_alter_hton->savepoint_rollback_can_release_mdl= > + [](handlerton *hton, THD *thd){ return true; }; > + > + online_alter_hton->commit= [](handlerton *, THD *thd, bool all) > + { return binlog_online_alter_end_trans(thd, all, true); }; > + online_alter_hton->rollback= [](handlerton *, THD *thd, bool all) > + { return binlog_online_alter_end_trans(thd, all, false); }; > + online_alter_hton->commit_by_xid= [](handlerton *hton, XID *xid) > + { return binlog_online_alter_end_trans(current_thd, true, true); }; > + online_alter_hton->rollback_by_xid= [](handlerton *hton, XID *xid) > + { return binlog_online_alter_end_trans(current_thd, true, false); }; should it be so? commit_by_xid/rollback_by_xid take XID as an argument ad you ignore it. If commit_by_xid/rollback_by_xid shouldn't be called for online alter, don't set them at all. Or set to something that always fails (and add an assert too). > + > + online_alter_hton->drop_table= [](handlerton *, const char*) { return -1; }; yeah. Like this. > + online_alter_hton->flags= HTON_NOT_USER_SELECTABLE | HTON_HIDDEN > + | HTON_NO_ROLLBACK; better remove HTON_NO_ROLLBACK. As far as I can see it doesn't do anything here anyway, so the only effect it has - everyone reading the code will stop and thing "why HTON_NO_ROLLBACK is here?" Takes at least 5-10 minutes to look through all places where HTON_NO_ROLLBACK is tested to see that online_alter_hton cannot be there. > + return 0; > +} > + > +struct st_mysql_storage_engine online_alter_storage_engine= > +{ MYSQL_HANDLERTON_INTERFACE_VERSION }; > + > +maria_declare_plugin(online_alter_log) > +{ > + MYSQL_STORAGE_ENGINE_PLUGIN, > + &online_alter_storage_engine, > + "online_alter_log", > + "MariaDB PLC", "plc" lowercase > + "This is a pseudo storage engine to represent the online alter log in a transaction", > + PLUGIN_LICENSE_GPL, > + online_alter_log_init, > + NULL, > + 0x0100, // 1.0 > + NULL, // no status vars > + NULL, // no sysvars > + "1.0", > + MariaDB_PLUGIN_MATURITY_STABLE > +} > +maria_declare_plugin_end; Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

1 0

Re: f9c1546bd7d: MDEV-32123: require_secure_transport doesn't allow TCP connections
by Sergei Golubchik 28 Sep '23

28 Sep '23

Hi, Dmitry, On Sep 28, Dmitry Shulga wrote: > revision-id: f9c1546bd7d (mariadb-11.2.1-6-gf9c1546bd7d) > parent(s): c4cbafe0c4c > author: Dmitry Shulga > committer: Dmitry Shulga > timestamp: 2023-09-28 16:20:36 +0700 > message: > > MDEV-32123: require_secure_transport doesn't allow TCP connections > > diff --git a/mysql-test/main/require_secure_transport.test b/mysql-test/main/require_secure_transport.test > index e238e732423..4ad18afb96d 100644 > --- a/mysql-test/main/require_secure_transport.test > +++ b/mysql-test/main/require_secure_transport.test > @@ -5,11 +5,16 @@ SET GLOBAL require_secure_transport=ON; > --error ER_SECURE_TRANSPORT_REQUIRED > connect without_ssl,localhost,root,,,,,TCP NOSSL; > --enable_query_log > + > +connect with_ssl,localhost,root,,,,,TCP SSL; where's a check that a connection was established successfully and it indeed uses SSL with require_secure_transport enabled? also, you didn't add any tests for --require-secure-transport=1 > +disconnect with_ssl; > + > connection default; > SET GLOBAL require_secure_transport=OFF; > --disable_query_log > connect without_ssl,localhost,root,,,,,TCP NOSSL; > --enable_query_log > disconnect without_ssl; > + > connection default; > DROP TABLE t1; > diff --git a/sql/sql_acl.cc b/sql/sql_acl.cc > index 1f00a8670a6..3ce1c363c86 100644 > --- a/sql/sql_acl.cc > +++ b/sql/sql_acl.cc > @@ -14519,6 +14546,22 @@ bool acl_authenticate(THD *thd, uint com_change_user_pkt_len) > > if (initialized) // if not --skip-grant-tables > { > + /* > + Check whether the option require_secure_transport is on and in case > + it is true that the secured connection type is used, that is either > + unix socket or named pipe or ssl is in use. > + */ > + if(check_require_secured_transport(thd)) > + { > + Host_errors errors; > + errors.m_ssl= 1; > + inc_host_errors(mpvio.auth_info.thd->security_ctx->ip, &errors); > + status_var_increment(thd->status_var.access_denied_errors); > + my_error(ER_SECURE_TRANSPORT_REQUIRED, MYF(0)); > + > + DBUG_RETURN(1); > + } No, no. This is wrong. The point of "require_secure_transport" is to make sure that *everything* is sent over the secure transport. In particular, passwords must never be sent if the transport is not secure. You cannot check for it after the authentication, you must abort the connection is soon as possible. (technically, the server can do nothing to prevent the client from sending passwords in plain-text over the insecure connection. but practically clients use mysql_native_password by default, so the password won't leak) Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

1 0

Re: [PATCH] MDEV-31872 Deprecate ENCODE()/DECODE()
by Kristian Nielsen 26 Sep '23

26 Sep '23

Kristian Nielsen <knielsen(a)knielsen-hq.org> writes: > From: Sergei Golubchik <serg(a)mariadb.org> > @@ -2724,6 +2725,10 @@ bool Item_func_encode::seed() > hash_password(rand_nr, key->ptr(), key->length()); > sql_crypt.init(rand_nr); > > + push_warning_printf(thd, Sql_condition::WARN_LEVEL_WARN, ER_WARN_DEPRECATED_SYNTAX, > + ER_THD(thd, ER_WARN_DEPRECATED_SYNTAX_NO_REPLACEMENT), > + func_name_cstring().str); // since 11.3 > + NAK. As I already explained in the MDEV-31872, deprecating these functions is not appropriate and will hurt users, not help them. Don't do this. You need to see this from the user's point of view. She is a DBA, she has an application with columns encoded with ENCODE(). What do you expect her to do, rewrite the application because you or someone does not like these functions? That's very disrespectful. If you want to give a warning for these functions, then you need to create an SQL mode or some other handle for the DBA to disable the warning and keep their application running. And the warning should explain why it's there, and what should be done to avoid it. And you need to carefully justify how burdening the DBA with this task of tracking down the source of the new warning and the way to remove it, is worth the benefit. And deprecation suggests eventual removal, which is even worse, please don't do that. You haven't even updated the documentation https://mariadb.com/kb/en/encode/ to eg. say "encode" or "scramble" instead of encrypt. "base64_encode" also does not do encryption, nor is rot13 cryptographically safe, but that doesn't justify removing them either. Most columns in mariadb databases are not encrypted or encoded at all, how is ENCODE() any less secure that that? - Kristian.

3 10

Re: 025ca90e5df: MDEV-32101 CREATE PACKAGE [BODY] for sql_mode=DEFAULT
by Sergei Golubchik 22 Sep '23

22 Sep '23

Hi, Alexander, On Sep 15, Alexander Barkov wrote: > revision-id: 025ca90e5df (mariadb-11.0.1-231-g025ca90e5df) > parent(s): 8ad1e26b1ba > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2023-09-05 17:33:27 +0400 > message: > > MDEV-32101 CREATE PACKAGE [BODY] for sql_mode=DEFAULT > > This patch adds PACKAGE support with SQL/PSM dialect for sql_mode=DEFAULT: > diff --git a/sql/sql_yacc.yy b/sql/sql_yacc.yy > index 7452f6a083a..22c60bd4277 100644 > --- a/sql/sql_yacc.yy > +++ b/sql/sql_yacc.yy > @@ -18454,6 +18518,22 @@ drop_routine: > if (Lex->stmt_drop_procedure($3, $4)) > MYSQL_YYABORT; > } > + | DROP PACKAGE_MARIADB_SYM opt_if_exists sp_name > + { > + LEX *lex= Lex; > + lex->set_command(SQLCOM_DROP_PACKAGE, $3); > + if (unlikely(lex->sphead)) > + my_yyabort_error((ER_SP_NO_DROP_SP, MYF(0), "PACKAGE")); > + lex->spname= $4; > + } > + | DROP PACKAGE_MARIADB_SYM BODY_MARIADB_SYM opt_if_exists sp_name > + { > + LEX *lex= Lex; > + lex->set_command(SQLCOM_DROP_PACKAGE_BODY, $4); > + if (unlikely(lex->sphead)) > + my_yyabort_error((ER_SP_NO_DROP_SP, MYF(0), "PACKAGE BODY")); > + lex->spname= $5; > + } > ; at least here and in SHOW you can use package: PACKAGE_MARIADB_SYM | PACKAGE_ORACLE_SYM ; and then move drop_routine out of ifdef > > > @@ -18498,6 +18578,248 @@ create_routine: > (Item_result) $8, $10)) > MYSQL_YYABORT; > } > + > + | create_or_replace definer_opt PACKAGE_MARIADB_SYM > + opt_if_not_exists sp_name opt_create_package_chistics_init > + remember_name > + { > + sp_package *pkg; > + if (unlikely(!(pkg= Lex-> > + create_package_start(thd, > + SQLCOM_CREATE_PACKAGE, > + &sp_handler_package_spec, > + $5, $1 | $4)))) > + MYSQL_YYABORT; > + pkg->set_c_chistics(Lex->sp_chistics); > + } > + opt_sqlpsm_package_specification_element_list END > + remember_end_opt > + { > + if (unlikely(Lex->create_package_finalize(thd, $5, NULL, $7, $11))) > + MYSQL_YYABORT; > + } > + | create_or_replace definer_opt PACKAGE_MARIADB_SYM BODY_MARIADB_SYM > + opt_if_not_exists sp_name opt_create_package_chistics_init > + remember_name > + { > + sp_package *pkg; > + if (unlikely(!(pkg= Lex-> > + create_package_start(thd, > + SQLCOM_CREATE_PACKAGE_BODY, > + &sp_handler_package_body, > + $6, $1 | $5)))) > + MYSQL_YYABORT; > + pkg->set_c_chistics(Lex->sp_chistics); > + Lex->sp_block_init(thd); > + } > + sqlpsm_package_implementation_declare_section > + { > + if (unlikely(Lex->sp_block_with_exceptions_finalize_declarations(thd))) > + MYSQL_YYABORT; > + } > + sqlpsm_package_implementation_executable_section > + { > + $10.hndlrs+= $12.hndlrs; > + if (unlikely(Lex->sp_block_finalize(thd, $10))) > + MYSQL_YYABORT; > + } > + remember_end_opt > + { > + if (unlikely(Lex->create_package_finalize(thd, $6, NULL, $8, $14))) > + MYSQL_YYABORT; > + } this can also use `package` rule as above and be moved outside of %ifdef. you'll only need %ifdef MARIADB sp_tail_is: _empty ; opt_sp_name: _empty { $$= NULL; } ; %else sp_tail_is: IS | AS ; opt_sp_name: _empty { $$= NULL; } | sp_name { $$= $1; } ; %endif > + ; > + > + > +opt_sqlpsm_package_specification_element_list: > + _empty > + | sqlpsm_package_specification_element_list > + ; and so on, this can be called simply opt_package_specification_element_list if placed under %ifdef MARIADB > + > +sqlpsm_package_specification_element_list: > + sqlpsm_package_specification_element > + | sqlpsm_package_specification_element_list > + sqlpsm_package_specification_element > + ; ... Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: 14a12f98eb8: MDEV-15935 Clean up string sys_var types.
by Sergei Golubchik 14 Sep '23

14 Sep '23

Hi, Yuchen, ok to push, after making one change (const) below. On Sep 13, Yuchen Pei wrote: > revision-id: 14a12f98eb8 (mariadb-11.0.1-232-g14a12f98eb8) > parent(s): e987b9350cb > author: Yuchen Pei > committer: Yuchen Pei > timestamp: 2023-09-13 11:50:41 +1000 > message: > > MDEV-15935 Clean up string sys_var types. > > Merge sys_var_charptr with sys_var_charptr_base, as well as merge > Sys_var_session_lexstring into Sys_var_lexstring. Also refactored > update methods of sys_var_charptr accordingly. > > Because the class is more generic, session_update() calls > sys_var_charptr::session_update() which does not assume a buffer field > associated with THD, but instead call strdup/free, we get rid of > THD::default_master_connection_buff accordingly You could've mentioned here "this also makes THD smaller by 192 bytes, and there can be many thousands of concurrent THDs" I mean, it's a good result, one of the benefits of your cleanup. > diff --git a/sql/sys_vars.inl b/sql/sys_vars.inl > index 31ebbae01ca..5bac91a0db3 100644 > --- a/sql/sys_vars.inl > +++ b/sql/sys_vars.inl > @@ -492,14 +492,18 @@ class Sys_var_mybool: public Sys_var_typelib > Backing store: char* > > @note > - This class supports only GLOBAL variables, because THD on destruction > - does not destroy individual members of SV, there's no way to free > - allocated string variables for every thread. > + > + Note that the memory management for SESSION_VAR's is manual, the > + value must be strdup'ed in THD::init() and freed in > + plugin_thdvar_cleanup(). TODO: it should be done automatically when > + we'll have more session string variables to justify it. Maybe some > + kind of a loop over all variables, like sys_var_end() in set_var.cc? > */ > -class Sys_var_charptr_base: public sys_var > +class Sys_var_charptr: public sys_var > { > + size_t max_length= 2000; Should be const, right? No need to store max_length in every Sys_var_charptr instance, if you aren't going to modify it per instance. > public: > - Sys_var_charptr_base(const char *name_arg, > + Sys_var_charptr(const char *name_arg, > const char *comment, int flag_args, ptrdiff_t off, size_t size, > CMD_LINE getopt, > const char *def_val, PolyLock *lock=0, Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: 9db04b3dc73: MDEV-15935 Adding global/session system var redirect_url
by Sergei Golubchik 13 Sep '23

13 Sep '23

Hi, Yuchen, Thanks. Ok to push both commits into preview-11.3-preview branch On Sep 13, Yuchen Pei wrote: > revision-id: 9db04b3dc73 (mariadb-11.0.1-233-g9db04b3dc73) > parent(s): 14a12f98eb8 > author: Yuchen Pei > committer: Yuchen Pei > timestamp: 2023-09-13 11:50:42 +1000 > message: > > MDEV-15935 Adding global/session system var redirect_url > > Adding a global/session var `redirect_url' of string type. The initial > value is empty. Can be supplied in mysqld with --redirect-url or set > in --init-connect. A valid redirect_url should be of the format > > {mysql,mariadb}://host[:port] > > where <host> is an arbitrary string not containing colons, and <port> > is a number between 0 and 65535 inclusive. > > The variable will be used by the server to notify clients that they > should connect to another server, specified by the value of the > variable, if not empty. > > The notification is done by the inclusion of the variable in > session_track_system_variable. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: 8ac17273b14: MDEV-15935 Adding global/session system var redirect_url
by Sergei Golubchik 13 Sep '23

13 Sep '23

Hi, Yuchen, Right. This is pretty much the whole feature. Mostly ok, few minor comments below. On Sep 08, Yuchen Pei wrote: > revision-id: 8ac17273b14 (mariadb-11.0.1-104-g8ac17273b14) > parent(s): 365faf7229f > author: Yuchen Pei > committer: Yuchen Pei > timestamp: 2023-09-08 14:46:50 +1000 > message: > > MDEV-15935 Adding global/session system var redirect_url > > Adding a global/session var `redirect_url' of string type. The initial > value is empty. Can be supplied in mysqld with --redirect-url or set > in --init-connect. A valid redirect_url should be of the format > > {mysql,mariadb}://host[:port] > > where <host> is an arbitrary string not containing colons, and <port> > is a number between 0 and 65535 inclusive. > > The variable will be used by the server to notify clients that they > should connect to another server, specified by the value of the > variable, if not empty. > > The notification is done by the inclusion of the variable in > session_track_system_variable. > > diff --git a/mysql-test/suite/sys_vars/r/sysvars_server_embedded.result b/mysql-test/suite/sys_vars/r/sysvars_server_embedded.result > index fa3bf6b4354..85dc924fe5f 100644 > --- a/mysql-test/suite/sys_vars/r/sysvars_server_embedded.result > +++ b/mysql-test/suite/sys_vars/r/sysvars_server_embedded.result > @@ -3232,6 +3232,16 @@ NUMERIC_BLOCK_SIZE 1 > ENUM_VALUE_LIST NULL > READ_ONLY NO > COMMAND_LINE_ARGUMENT REQUIRED > +VARIABLE_NAME REDIRECT_URL > +VARIABLE_SCOPE SESSION > +VARIABLE_TYPE VARCHAR > +VARIABLE_COMMENT URL to redirect to, if not empty see if you can make it a bit more descriptive. if not, ok as is. hmm, and a second thought. it's in the sysvars_server_embedded test. it doesn't make sense there (embedded doesn't accept connections, and doesn't have session tracking, iirc), so, may be hide it under #ifndef EMBEDDED_LIBRARY ? > +NUMERIC_MIN_VALUE NULL > +NUMERIC_MAX_VALUE NULL > +NUMERIC_BLOCK_SIZE NULL > +ENUM_VALUE_LIST NULL > +READ_ONLY NO > +COMMAND_LINE_ARGUMENT REQUIRED > VARIABLE_NAME REQUIRE_SECURE_TRANSPORT > VARIABLE_SCOPE GLOBAL > VARIABLE_TYPE BOOLEAN > diff --git a/sql/sql_class.h b/sql/sql_class.h > index cf2b5b96a22..c86a76f6a08 100644 > --- a/sql/sql_class.h > +++ b/sql/sql_class.h > @@ -878,6 +878,7 @@ typedef struct system_variables > > Time_zone *time_zone; > char *session_track_system_variables; > + char *redirect_url; > > /* Some wsrep variables */ > ulonglong wsrep_trx_fragment_size; > diff --git a/sql/sql_plugin.cc b/sql/sql_plugin.cc > index 4724e2200e6..70aba9cc4d5 100644 > --- a/sql/sql_plugin.cc > +++ b/sql/sql_plugin.cc > @@ -3307,6 +3309,10 @@ void plugin_thdvar_init(THD *thd) > #ifndef EMBEDDED_LIBRARY > thd->session_tracker.sysvars.init(thd); > #endif > + thd->variables.redirect_url= > + my_strdup(PSI_INSTRUMENT_ME, global_system_variables.redirect_url, and here, key_memory_Sys_var_charptr_value > + MYF(MY_WME | MY_THREAD_SPECIFIC)); > + > DBUG_VOID_RETURN; > } > > @@ -3375,6 +3381,8 @@ void plugin_thdvar_cleanup(THD *thd) > #ifndef EMBEDDED_LIBRARY > thd->session_tracker.sysvars.deinit(thd); > #endif > + my_free(thd->variables.redirect_url); > + thd->variables.redirect_url= 0; also, perhaps in cleanup_variables() ? > my_free((char*) thd->variables.default_master_connection.str); > thd->variables.default_master_connection.str= 0; > thd->variables.default_master_connection.length= 0; > diff --git a/sql/sys_vars.cc b/sql/sys_vars.cc > index 064c60ab2b0..a93e68f40d7 100644 > --- a/sql/sys_vars.cc > +++ b/sql/sys_vars.cc > @@ -1343,6 +1343,61 @@ Sys_init_connect( > DEFAULT(""), &PLock_sys_init_connect, NOT_IN_BINLOG, > ON_CHECK(check_init_string)); > > +/** > + Validate a redirect_url string. > + > + A valid string is either empty, or of the format mysql://host[:port], > + where host is an arbitrary string without any colon ':'. > + > + @param str A string to validate > + @param len Length of the string > + @retval false The string is valid > + @retval true The string is invalid > +*/ > +static bool sysvar_validate_redirect_url(sys_var *, THD *, set_var *var) > +{ > + char *str= var->save_result.string_value.str; > + size_t len= var->save_result.string_value.length; > + /* Empty string is valid */ > + if (len == 0) > + return false; > + const char* end= str + len; > + if (!strncmp(str, "mysql://", strlen("mysql://"))) use !strncmp(str, STRING_WITH_LEN("mysql://")) avoids no strlen and no need to write the string twice. same below. > + str+= strlen("mysql://"); ah. Then, I'd suggest instead LEX_CSTRING mysql_prefix={STRING_WITH_LEN("mysql://")}; if (!strncmp(str, mysql_prefix.str, mysql_prefix.length)) str+= mysql_prefix.length; > + else if (!strncmp(str, "mariadb://", strlen("mariadb://"))) > + str+= strlen("mariadb://"); > + else > + return true; > + /* Host name cannot be empty */ > + if (str == end) > + return true; > + /* Find the colon, if any */ > + while (str < end && *str != ':') > + str++; > + /* Found colon */ > + if (str < end) > + { > + /* Should have at least one number after the colon */ > + if (str + 1 == end) > + return true; > + int p= 0; > + while (str < end && isdigit(*++str)) > + p= p * 10 + (*str - '0'); > + /* Should be all numbers after the colon */ > + if (str < end) > + return true; > + /* The port number should be between 0 and 65535 */ > + if (p > 65535) > + return true; check for overflow too. you can either put the p>65535 check inside the loop or at the end check that, like, (str - colon_ptr) < 7. > + } > + return false; > +} > + > +static Sys_var_charptr Sys_redirect_url( > + "redirect_url", "URL to redirect to, if not empty", > + SESSION_VAR(redirect_url), CMD_LINE(REQUIRED_ARG), DEFAULT(""), > + NO_MUTEX_GUARD, NOT_IN_BINLOG, sysvar_validate_redirect_url, 0, 0); By convention, write ON_CHECK(sysvar_validate_redirect_url). And you don't need 0's at the end, so just NO_MUTEX_GUARD, NOT_IN_BINLOG, ON_CHECK(sysvar_validate_redirect_url)); Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 4

Re: [MariaDB commits] [PATCH] MDEV-31872: Add --old_mode=COMPAT_DISCOURAGED
by Sergei Golubchik 12 Sep '23

12 Sep '23

Hi, Kristian, On Sep 12, Kristian Nielsen wrote: > > diff --git a/sql/sys_vars.cc b/sql/sys_vars.cc > index e25d293bbb7..18d08996aa8 100644 > --- a/sql/sys_vars.cc > +++ b/sql/sys_vars.cc > @@ -3863,12 +3863,13 @@ static const char *old_mode_names[]= > "IGNORE_INDEX_ONLY_FOR_JOIN", // deprecated since 11.3 > "COMPAT_5_1_CHECKSUM", // deprecated since 11.3 > "LOCK_ALTER_TABLE_COPY", // deprecated since 11.3 > + "COMPAT_DISCOURAGED", > 0 > }; > > void old_mode_deprecated_warnings(THD *thd, ulonglong v) > { > - v &= ~OLD_MODE_DEFAULT_VALUE; > + v &= ~(OLD_MODE_DEFAULT_VALUE | OLD_MODE_COMPAT_DISCOURAGED); No, this is wrong. old-mode *by definition* is something that is a temporary workaround for something new, old-mode values are by design deprecated from the day one. So, OLD_MODE_COMPAT_DISCOURAGED shouldn't be on the line above, and thus it'll itself be deprecated and will cause a deprecation warning. Which is rather silly, as I wrote in a previous email, and that's why an old-mode is not a good solution for turning deprecated warnings away. > for (uint i=0; old_mode_names[i]; i++) > if ((1ULL<<i) & v) > { > diff --git a/sql/item_strfunc.cc b/sql/item_strfunc.cc > index 4b96271c331..339430a2794 100644 > --- a/sql/item_strfunc.cc > +++ b/sql/item_strfunc.cc > @@ -2720,7 +2720,16 @@ bool Item_func_encode::seed() > hash_password(rand_nr, key->ptr(), key->length()); > sql_crypt.init(rand_nr); > > - warn_deprecated<1103>(current_thd, func_name_cstring().str); > + if (!(current_thd->variables.old_behavior & OLD_MODE_COMPAT_DISCOURAGED)) > + { > + /* > + This function should not be removed despite the deprecation warning. > + That would cause problems for users who can then no longer access > + columns in their database that might have been inserted as > + ENCODE(str,pass_str). > + */ > + warn_deprecated<1103>(current_thd, func_name_cstring().str); > + } No, this is exactly what I wanted to prevent by adding a warn_deprecated<>() wrapper. If something is giving a warning "... is deprecated and will be removed" - it *has to be removed*. We don't want to lie to users to simply scare them. If we promise to do something, we should do it. If we aren't going to remove ENCODE/DECODE functions, we shouldn't deprecate them at all. Which is the result I'm leaning to right now. > > return FALSE; > } Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: 75ea482c4fc: MDEV-30973 GTID in deadlock output
by Sergei Golubchik 11 Sep '23

11 Sep '23

Hi, Aleksey, On Sep 11, Aleksey Midenkov wrote: > revision-id: 75ea482c4fc (mariadb-11.0.1-230-g75ea482c4fc) > parent(s): fe62f70fb4b > author: Aleksey Midenkov > committer: Aleksey Midenkov > timestamp: 2023-09-01 20:30:24 +0300 > message: > > MDEV-30973 GTID in deadlock output > > Example: > > TRANSACTION 40, ACTIVE 0 sec starting index read > mysql tables in use 1, locked 1 > LOCK WAIT 2 lock struct(s), heap size 1152, 1 row lock(s) > MySQL thread id 23, OS thread handle 140201491547840, query id 310, GTID 3-1-1 Updating > delete from tu where a = 9999 and b = 'xxxx' > 2023-04-12 19:53:58 22 [Note] InnoDB: *** (1) WAITING FOR THIS LOCK TO BE GRANTED: > > Deterministic slave_deadlock.test > > Instead of "MDEV-10962 Deadlock with 3 concurrent DELETEs by unique key" > this one uses "MDEV-22698 Deadlock on concurrent acquisition from multiple indexes" > > --- > mysql-test/suite/rpl/t/slave_deadlock.test | 93 ++++++++++++++++++++++++++++++ > sql/sql_show.cc | 10 ++++ please move slave_deadlock.result from the following commit to this one. otherwise ok Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Implementing Binlog GTID Indexes (MDEV-4991)
by Kristian Nielsen 10 Sep '23

10 Sep '23

I am implementing Binlog GTID Indexes, to fix an old performance regression for GTID when slaves connect to the master. I now have a solid design and a working prototype, so I wanted to describe the work to encourage early comments and suggestions. I'm hoping this will make it to the next release (where is the place to actively follow/participate in the release planning?) Ever since I implemented GTID, connecting slaves needed to sequentially scan the leading portion of one binlog file to locate the GTID position to start at. This can be slow as binlog files default to 1GB in size, even worse so when binlog files can now be encrypted. I files MDEV-4991 for this 10 years ago, and it is embarrassing that this has somehow remained unfixed for so long :-/ But now I'm fixing it. The current implementation is in the work-in-progress branch knielsen_mdev4991: https://github.com/MariaDB/server/tree/knielsen_mdev4991 The code is already functional, with the testsuite passing. Still missing is binlog purge, index crash recovery, async path, and general testing and cleanup. The basic idea is for each binlog file master-bin.000001 to write an index file master-bin.000001.idx. The index contains a B+-Tree in which the keys are pairs of (GTID state, binlog offset). A connecting slave's GTID position can be looked up quickly in the tree to find the corresponding binlog offset (and gtid_binlog_state) to start replicating at. Similarly, a non-GTID connecting slave can quickly look up its starting offset (BINLOG_GTID_POS()) to obtain the GTID position corresponding to its starting offset. The index is written out to disk concurrently with the writing of the binlog file. Since the tree is written in sorted order and append-only, the implementation is significantly simplified over a general B+-Tree, and can also be described as a Log-structed Merge Tree with a Btree search structure on top. All GTID state keys (except the first) in the index are delta-compressed, storing only the GTIDs that changed since the last record (which typically will be only one). Additionally the index is sparse, storing only say every 1-in-10 GTIDs in the index (this will be configurable of course). Using a sparse index, the disk space is reduced at the cost only of scanning a few extra events of the binlog file to find the exact position requested. Actual benchmarking of space usage is TBO, but we can expect a typical binlog file of 1 GB containing say 3 million GTIDs to require something like 10 MB of index disk space (1% increase). The file format is page-based (unlike the binlog file), so that it can be written efficiently and easily read using random-access. The index is written in parallel with the binlog file, but no extra fsync()s are needed (except one at the end of the file when the index is closed and synced to disk). The writing of the index will happen asynchroneously from the binlog background thread; this way it will have minimal impact on the performance and scalability of the binlog and the contested LOCK_log. Connecting slaves can read from the currently-being-written ("hot") GTID index by accessing internal memory buffers of pages not yet written to disk. If the server crashes, the existing binlog recovery and binlog checkpoint mechanism will be used to re-create any incomplete indexes as part of the normal recovery scan of binlog files. If somehow a binlog index should be found corrupt or missing (could be eg. old server upgrade or backup script that omits the new .idx files), the code will gracefully fall back to the old (slower) way of sequentially scanning the binlog file to locate the slave starting position. I think this will be a great (if rather late) improvement to one remaining corner of the GTID implementation that currently has sub-optimal performance, and I'm exited to get this completed and reviewed/tested and added to hopefully the next MariaDB release. - Kristian.

1 0

Re: 365faf7229f: MDEV-15935 Clean up string sys_var types.
by Sergei Golubchik 08 Sep '23

08 Sep '23

Hi, Yuchen, On Sep 08, Yuchen Pei wrote: > revision-id: 365faf7229f (mariadb-11.0.1-103-g365faf7229f) > parent(s): 4f39ae75606 > author: Yuchen Pei > committer: Yuchen Pei > timestamp: 2023-09-08 13:12:22 +1000 > message: > > MDEV-15935 Clean up string sys_var types. > > Merge sys_var_charptr with sys_var_charptr_base, as well as merge > Sys_var_session_lexstring into Sys_var_lexstring. Also refactored > update methods of sys_var_charptr accordingly. > > Because the class is more generic, session_update() calls > sys_var_charptr::session_update() which does not assume a buffer field > associated with THD, but instead call strdup/free, we get rid of > THD::default_master_connection_buff accordingly > diff --git a/sql/sql_class.cc b/sql/sql_class.cc > index 18e796655bd..52fa6c27112 100644 > --- a/sql/sql_class.cc > +++ b/sql/sql_class.cc > @@ -1233,10 +1233,12 @@ void THD::init() > avoid temporary tables replication failure. > */ > variables.pseudo_thread_id= thread_id; > - variables.default_master_connection.str= default_master_connection_buff; > - ::strmake(default_master_connection_buff, > - global_system_variables.default_master_connection.str, > - variables.default_master_connection.length); > + variables.default_master_connection.str= > + my_strndup(PSI_INSTRUMENT_ME, may be key_memory_Sys_var_charptr_value instead of PSI_INSTRUMENT_ME? > + global_system_variables.default_master_connection.str, > + global_system_variables.default_master_connection.length, > + MYF(MY_WME | MY_THREAD_SPECIFIC)); > + > mysql_mutex_unlock(&LOCK_global_system_variables); > > user_time.val= start_time= start_time_sec_part= 0; > diff --git a/sql/sql_plugin.cc b/sql/sql_plugin.cc > index d39589ed1ab..4724e2200e6 100644 > --- a/sql/sql_plugin.cc > +++ b/sql/sql_plugin.cc > @@ -3279,6 +3279,9 @@ void plugin_thdvar_init(THD *thd) > #ifndef EMBEDDED_LIBRARY > thd->session_tracker.sysvars.deinit(thd); > #endif > + my_free((char*) thd->variables.default_master_connection.str); > + thd->variables.default_master_connection.str= 0; > + thd->variables.default_master_connection.length= 0; shouldn't this go inside cleanup_variables() ? > > thd->variables= global_system_variables; > > diff --git a/sql/sys_vars.cc b/sql/sys_vars.cc > index 4843450352e..064c60ab2b0 100644 > --- a/sql/sys_vars.cc > +++ b/sql/sys_vars.cc > @@ -1355,12 +1355,11 @@ static bool check_master_connection(sys_var *self, THD *thd, set_var *var) > return false; > } > > -static Sys_var_session_lexstring Sys_default_master_connection( > +static Sys_var_lexstring Sys_default_master_connection( > "default_master_connection", > "Master connection to use for all slave variables and slave commands", > - SESSION_ONLY(default_master_connection), > - NO_CMD_LINE, > - DEFAULT(""), MAX_CONNECTION_NAME, ON_CHECK(check_master_connection)); > + SESSION_ONLY(default_master_connection), NO_CMD_LINE, DEFAULT(""), > + NO_MUTEX_GUARD, NOT_IN_BINLOG, ON_CHECK(check_master_connection)); some length limit would still be nice, I'd say. Doesn't have to be per variable, may be all Sys_var_lexstring could have some common limit, like, arbitrarily, 1K or 10K. > #endif > > static Sys_var_charptr_fscs Sys_init_file( > diff --git a/sql/sys_vars.inl b/sql/sys_vars.inl > index 31ebbae01ca..0f2a07d92fc 100644 > --- a/sql/sys_vars.inl > +++ b/sql/sys_vars.inl > @@ -576,6 +571,21 @@ class Sys_var_charptr_base: public sys_var > new_val= 0; > return new_val; > } > + char *session_update_prepare(set_var *var) do you need this session_update_prepare method? global_update_prepare() and global_update_finish() are used by Sys_var_sesvartrack. Getting rid of them is a task for another day :) > + { > + return update_prepare(var, MYF(MY_WME | MY_THREAD_SPECIFIC)); > + } > + bool session_update(THD *thd, set_var *var) override > + { > + char *new_val= session_update_prepare(var); > + my_free(session_var(thd, char*)); > + session_var(thd, char*)= new_val; > + return (new_val == 0 && var->save_result.string_value.str != 0); > + } > + char *global_update_prepare(set_var *var) > + { > + return update_prepare(var, MYF(MY_WME)); > + } > void global_update_finish(char *new_val) > { > if (flags & ALLOCATED) > @@ -583,50 +593,28 @@ class Sys_var_charptr_base: public sys_var > flags|= ALLOCATED; > global_var(char*)= new_val; > } > - bool global_update(THD *thd, set_var *var) > + bool global_update(THD *thd, set_var *var) override > { > - char *new_val= global_update_prepare(thd, var); > + char *new_val= global_update_prepare(var); > global_update_finish(new_val); > return (new_val == 0 && var->save_result.string_value.str != 0); > } > - void session_save_default(THD *thd, set_var *var)= 0; > - void global_save_default(THD *thd, set_var *var) > + void save_default(set_var *var) > { > char *ptr= (char*)(intptr)option.def_value; > var->save_result.string_value.str= ptr; > var->save_result.string_value.length= ptr ? strlen(ptr) : 0; > } > -}; > - > -class Sys_var_charptr: public Sys_var_charptr_base > -{ > -public: > - Sys_var_charptr(const char *name_arg, > - const char *comment, int flag_args, ptrdiff_t off, size_t size, > - CMD_LINE getopt, > - const char *def_val, PolyLock *lock=0, > - enum binlog_status_enum binlog_status_arg=VARIABLE_NOT_IN_BINLOG, > - on_check_function on_check_func=0, > - on_update_function on_update_func=0, > - const char *substitute=0) : > - Sys_var_charptr_base(name_arg, comment, flag_args, off, size, getopt, > - def_val, lock, binlog_status_arg, > - on_check_func, on_update_func, substitute) > + void session_save_default(THD *, set_var *var) override > { > - SYSVAR_ASSERT(scope() == GLOBAL); > - SYSVAR_ASSERT(size == sizeof(char *)); > + return save_default(var); no, session's default value is global's value. See any other Sys_var_xxx class, e.g. for Sys_var_mybool: void session_save_default(THD *thd, set_var *var) { var->save_result.ulonglong_value= (ulonglong)*(my_bool *)global_value_ptr(thd, 0); } You cannot have a test for that now, but let's have one in the redirect_url commit. > } > - > - bool session_update(THD *thd, set_var *var) > + void global_save_default(THD *, set_var *var) override > { > - DBUG_ASSERT(FALSE); > - return true; > + return save_default(var); > } > - void session_save_default(THD *thd, set_var *var) > - { DBUG_ASSERT(FALSE); } > }; > > - > class Sys_var_charptr_fscs: public Sys_var_charptr > { > using Sys_var_charptr::Sys_var_charptr; > diff --git a/mysql-test/suite/sys_vars/r/mdev_15935.result b/mysql-test/suite/sys_vars/r/mdev_15935.result > new file mode 100644 > index 00000000000..d20c2ad2f19 > --- /dev/null > +++ b/mysql-test/suite/sys_vars/r/mdev_15935.result > @@ -0,0 +1,12 @@ > +# > +# test that the global/session-only enforcement is still in place despite the merge of string sys_var classes > +# > +set session init_connect="whatever"; > +ERROR HY000: Variable 'init_connect' is a GLOBAL variable and should be set with SET GLOBAL > +set session wsrep_auto_increment_control="whatever"; > +ERROR HY000: Variable 'wsrep_auto_increment_control' is a GLOBAL variable and should be set with SET GLOBAL > +set global default_master_connection="whatever"; > +ERROR HY000: Variable 'default_master_connection' is a SESSION variable and can't be used with SET GLOBAL not sure you need that, it's checked in the very base class, you could've have affected that I'd say it's enough if existing tests for Sys_var_session_lexstring and Sys_var_sesvartrack variables succeed and ASAN/MSAN are happy > +# > +# end of test mdev_15935 > +# Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: 04b99a30a3f: MDEV-15935 Adding global/session var redirect_url
by Sergei Golubchik 08 Sep '23

08 Sep '23

Hi, Yuchen, On Sep 01, Yuchen Pei wrote: > revision-id: 04b99a30a3f (mariadb-11.0.1-103-g04b99a30a3f) > parent(s): 4f39ae75606 > author: Yuchen Pei > committer: Yuchen Pei > timestamp: 2023-08-16 11:31:16 +1000 > message: > > MDEV-15935 Adding global/session var redirect_url > > Adding a global/session var `redirect_url' of string type. The initial > value is empty. Can be supplied in mysqld with --redirect-url. A valid > redirect_url should be of the format > > {mysql,mariadb}://host[:port] > > where <host> is an arbitrary string not containing colons, and <port> > is a number between 0 and 65535 inclusive. > > diff --git a/mysql-test/main/redirect.opt b/mysql-test/main/redirect.opt move this to the sys_vars suite, please > --- /dev/null > +++ b/mysql-test/main/redirect.opt > @@ -0,0 +1 @@ > +--redirect_url=mysql://foobar ok, as long as this isn't a complete feature, but just one piece of it. but when client support will be implemented (CONC-599) and the server will actually send this to the connecting clients (MDEV-31609), this test will need to be fixed, as foobar is not something a client can be redirected to. Also, it'd be more interesting to try something like --init-connect='set @@redirect_url="mysql://foobar"' instead of the simple --redirect_url=mysql://foobar because init-connect is much more flexible, it can generate the url on the fly, it can send different users to different locations. So it'd make sense to test init-connect approach. > diff --git a/mysql-test/main/redirect.test b/mysql-test/main/redirect.test > new file mode 100644 > --- /dev/null > +++ b/mysql-test/main/redirect.test > @@ -0,0 +1,56 @@ > +--echo # > +--echo # MDEV-15935 Connection Redirection Mechanism in MariaDB Client/Server Protocol > +--echo # > + > +set @old_redirect_url=@@redirect_url; > +select @@global.redirect_url; > +set global redirect_url=default; > +select @@global.redirect_url; > +--error ER_WRONG_VALUE_FOR_VAR > +set global redirect_url="mariadb.org"; > +--error ER_WRONG_VALUE_FOR_VAR > +set global redirect_url="https://mariadb.org"; before doing all this fun, please remove redirect_url from the session_track_system_variables, or just set it to the empty string. > diff --git a/sql/sql_class.h b/sql/sql_class.h > --- a/sql/sql_class.h > +++ b/sql/sql_class.h > @@ -3585,6 +3587,7 @@ class THD: public THD_count, /* this must be first */ > */ > LEX_CSTRING connection_name; > char default_master_connection_buff[MAX_CONNECTION_NAME+1]; > + char redirect_url_buff[MAX_CONNECTION_NAME+1]; This isn't such a good idea (I'm afraid, I missed it when default_master_connection_buff was added). A couple of years ago Monty have spent significant efforts trying to make THD smaller as it has grown to be too large. Now THD keeps growing again. Let's try not to encourage it, if we can help it. > uint8 password; /* 0, 1 or 2 */ > uint8 failed_com_change_user; > bool slave_thread; > diff --git a/sql/sys_vars.inl b/sql/sys_vars.inl > --- a/sql/sys_vars.inl > +++ b/sql/sys_vars.inl > @@ -637,6 +637,129 @@ class Sys_var_charptr_fscs: public Sys_var_charptr ... > +class Sys_var_redirect_url: public Sys_var_charptr_base please, no. new classes are needed only for variables with very special, unlike anything else, behavior. redirect_url is just a session string variable. See the comment at the top of sys_vars.cc But (!) there are no other session string variables yet. So I'd recomment to fix Sys_var_charptr to support session variables. And then create redirect_url out of it. Practically it might need no changes at all, if you put explicit strmake (or strdup/free) into THD::init like you did anyway. After that, I suspect, you might be able to remove Sys_var_charptr_base, Sys_var_sesvartrack, and Sys_var_session_lexstring, because Sys_var_charptr/Sys_var_lexstring will be able to serve all those use cases. > +{ > + static const size_t max_length = MAX_CONNECTION_NAME; > +public: > + Sys_var_redirect_url(const char *name_arg, > + const char *comment, > + CMD_LINE getopt, > + const char *def_val, PolyLock *lock= 0) : > + Sys_var_charptr_base(name_arg, comment, > + SESSION_VAR(redirect_url), > + getopt, def_val, lock, > + VARIABLE_NOT_IN_BINLOG, 0, 0, 0) > + { > + global_var(LEX_CSTRING).length= strlen(def_val); > + option.var_type|= GET_STR; > + *const_cast<SHOW_TYPE*>(&show_val_type)= SHOW_LEX_STRING; > + } Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: 44e692d0dbc: MDEV-29167 new db-level SHOW CREATE ROUTINE privilege
by Sergei Golubchik 06 Sep '23

06 Sep '23

Hi, Oleksandr, On Sep 04, Oleksandr Byelkin wrote: > revision-id: 44e692d0dbc (mariadb-11.0.1-228-g44e692d0dbc) > parent(s): 9cb75f333fa > author: Oleksandr Byelkin > committer: Oleksandr Byelkin > timestamp: 2023-09-04 14:12:29 +0200 > message: > > MDEV-29167 new db-level SHOW CREATE ROUTINE privilege > > -- Create general_log if CSV is enabled. > diff --git a/scripts/mariadb_system_tables_fix.sql b/scripts/mariadb_system_tables_fix.sql > --- a/scripts/mariadb_system_tables_fix.sql > +++ b/scripts/mariadb_system_tables_fix.sql > @@ -706,6 +706,9 @@ ALTER TABLE db modify Delete_history_priv enum('N','Y') COLLATE utf8mb3_general_ > > UPDATE user SET Delete_history_priv = Super_priv WHERE @had_user_delete_history_priv = 0; > > +ALTER TABLE db ADD Show_create_routine_priv enum('N','Y') COLLATE utf8mb3_general_ci NOT NULL DEFAULT 'N' AFTER Delete_history_priv; > +ALTER TABLE db modify Show_create_routine_priv enum('N','Y') COLLATE utf8mb3_general_ci NOT NULL DEFAULT 'N'; Why two statements? > + > ALTER TABLE user ADD plugin char(64) CHARACTER SET latin1 DEFAULT '' NOT NULL AFTER max_user_connections, > ADD authentication_string TEXT NOT NULL AFTER plugin; > ALTER TABLE user CHANGE auth_string authentication_string TEXT NOT NULL; > diff --git a/sql/sql_acl.cc b/sql/sql_acl.cc > --- a/sql/sql_acl.cc > +++ b/sql/sql_acl.cc > @@ -9003,7 +9008,7 @@ bool check_routine_level_acl(THD *thd, const char *db, const char *name, > NULL, db, > sctx->priv_role, > name, sph, 0))) > - no_routine_acl= !(grant_proc->privs & SHOW_PROC_ACLS); > + no_routine_acl= !(grant_proc->privs & SHOW_PROC_NO_DEF_ACLS); Oh, now I get it. All the time up until this line I thought that SHOW_PROC_NO_DEF_ACLS means "privileges that *not a definer* of the routine needs to have" And now I see that it means to "show procedure without definition". Please, rename to something, like, SHOW_PROC_EXISTANCE or SHOW_PROC_NO_BODY or even SHOW_PROC_WITHOUT_DEFINITION. But SHOW_PROC_NO_DEF is rather misleading. > } > mysql_rwlock_unlock(&LOCK_grant); > return no_routine_acl; > @@ -9237,7 +9242,7 @@ static const char *command_array[]= > "CREATE USER", "EVENT", "TRIGGER", "CREATE TABLESPACE", "DELETE HISTORY", > "SET USER", "FEDERATED ADMIN", "CONNECTION ADMIN", "READ_ONLY ADMIN", > "REPLICATION SLAVE ADMIN", "REPLICATION MASTER ADMIN", "BINLOG ADMIN", > - "BINLOG REPLAY", "SLAVE MONITOR" > + "BINLOG REPLAY", "SLAVE MONITOR", "SHOW CREATE ROUTINE" > }; > > static uint command_lengths[]= > @@ -9250,7 +9255,7 @@ static uint command_lengths[]= > 11, 5, 7, 17, 14, > 8, 15, 16, 15, > 23, 24, 12, > - 13, 13 > + 13, 13, 19 May be you'll change these two arrays into one of LEX_CSTRING's ? in a separate commit, of course. > }; > > diff --git a/sql/sql_show.cc b/sql/sql_show.cc > --- a/sql/sql_show.cc > +++ b/sql/sql_show.cc > @@ -6676,7 +6677,8 @@ int store_schema_params(THD *thd, TABLE *table, TABLE *proc_table, > DBUG_RETURN(0); > > if (!full_access) > - full_access= !strcmp(sp_user, definer.str); > + full_access= !check_db_routine_access(thd, SHOW_CREATE_ROUTINE_ACL, > + db.str, name.str, sph, TRUE); Wait, I thought we agreed that to see a routine body one has to be either the definer or granted SHOW CREATE ROUTINE. As far as I can see check_db_routine_access doesn't check for the user being the definer. > if (!full_access && > check_some_routine_access(thd, db.str, name.str, sph)) > DBUG_RETURN(0); Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

3 3

Re: 82a49812507: MDEV-22597 Add views for periods in information_schema
by Sergei Golubchik 05 Sep '23

05 Sep '23

Hi, Nikita, On Aug 31, Nikita Malyavin wrote: > revision-id: 82a49812507 (mariadb-11.0.1-228-g82a49812507) > parent(s): 23b15d24eb3 > author: Nikita Malyavin > committer: Nikita Malyavin > timestamp: 2023-08-30 22:49:36 +0400 > message: > > MDEV-22597 Add views for periods in information_schema > > Two new information_schema views are added: > * PERIOD table -- columns TABLE_CATALOG, TABLE_SCHEMA, TABLE_NAME, > PERIOD_NAME, START_COLUMN_NAME, END_COLUMN_NAME. > * KEY_PERIOD_USAGE -- works similar to KEY_COLUMN_USAGE, but for periods. > Columns CONSTRAINT_CATALOG, CONSTRAINT_SCHEMA, CONSTRAINT_NAME, > TABLE_CATALOG, TABLE_SCHEMA, TABLE_NAME, PERIOD_NAME > > Two new columns are added to the COLUMNS view: > IS_SYSTEM_TIME_PERIOD_START, IS_SYSTEM_TIME_PERIOD_END - contain YES/NO. > > diff --git a/mysql-test/main/information_schema-big.result b/mysql-test/main/information_schema-big.result > index 94c8d274fad..ad0451293c9 100644 > --- a/mysql-test/main/information_schema-big.result > +++ b/mysql-test/main/information_schema-big.result good, you didn't forget to update big and embedded tests > diff --git a/mysql-test/suite/period/r/i_s_notembedded.result b/mysql-test/suite/period/r/i_s_notembedded.result > new file mode 100644 > index 00000000000..879376cc8b4 > --- /dev/null > +++ b/mysql-test/suite/period/r/i_s_notembedded.result > @@ -0,0 +1,52 @@ > +select * from information_schema.periods; > +TABLE_CATALOG TABLE_SCHEMA TABLE_NAME PERIOD START_COLUMN_NAME END_COLUMN_NAME > +create or replace table t1 (id int primary key, s timestamp(6), e timestamp(6), > +period for mytime(s,e)); > +create or replace table t2 (id int primary key, s timestamp(6), e timestamp(6), > +period for mytime(s,e)) with system versioning; > +show columns from t1; > +Field Type Null Key Default Extra > +id int(11) NO PRI NULL > +s timestamp(6) NO NULL > +e timestamp(6) NO NULL > +select * from information_schema.periods where table_schema = 'test'; > +TABLE_CATALOG TABLE_SCHEMA TABLE_NAME PERIOD START_COLUMN_NAME END_COLUMN_NAME > +def test t2 SYSTEM_TIME row_start row_end I don't think so, they're INVISIBLE_SYSTEM. for all practical purposes they don't exist as columns in the table. Shouldn't be shown either. But don't remove this test, let's keep it to show that row_start/row_end are *not* shown here. And add table t3 with full standard definition of period for system_time. it should be shown all right. > +def test t2 mytime s e > +def test t1 mytime s e > +create user periods_hidden@localhost; ... > diff --git a/sql/sql_show.cc b/sql/sql_show.cc > index 1dc60a01f38..876d6c6f244 100644 > --- a/sql/sql_show.cc > +++ b/sql/sql_show.cc > @@ -6165,11 +6166,83 @@ static void store_variable_type(THD *thd, const sp_variable *spvar, > } > } > > +static int store_schema_period_record(THD *thd, TABLE_LIST *tl, > + TABLE *schema_table, > + const LEX_CSTRING *db_name, > + const LEX_CSTRING *table_name, > + const TABLE_SHARE::period_info_t &period) > +{ > + TABLE_SHARE *s= tl->table->s; > + const CHARSET_INFO *cs= system_charset_info; > + > + int err= schema_table->field[0]->store(STRING_WITH_LEN("def"), cs); > + if(!err) err= schema_table->field[1]->store(db_name, cs); > + if(!err) err= schema_table->field[2]->store(table_name, cs); > + if(!err) err= schema_table->field[3]->store(period.name, cs); well, ok. but as you've seen elsewhere these go without checks, as these calls generally cannot fail, they don't even allocate any memory. > + > + if (err) return err; > + > + int period_field= 4; > + for (auto *field: {period.start_field(s), period.end_field(s)}) > + { > +#ifndef NO_EMBEDDED_ACCESS_CHECKS > + bool col_granted= get_column_grant(thd, &tl->grant, > + db_name->str, table_name->str, > + field->field_name.str) & COL_ACLS; > + if (col_granted) > +#endif > + { > + schema_table->field[period_field]->set_notnull(); > + err= schema_table->field[period_field]->store(field->field_name, cs); > + } > + > + if (err) > + return err; > + period_field++; > + } > + > + return schema_table_store_record(thd, schema_table); > +} > + > + > +static int > +get_schema_period_records(THD *thd, TABLE_LIST *tl, > + TABLE *schema_table, //!< @ref > + //!< Show::periods_fields_info > + bool res, > + const LEX_CSTRING *db_name, > + const LEX_CSTRING *table_name) > +{ > + TABLE *table= tl->table; > + > + if (!table || (!table->s->period.name && !table->versioned())) > + return 0; > + > +#ifndef NO_EMBEDDED_ACCESS_CHECKS > + check_access(thd, SELECT_ACL, db_name->str, > + &tl->grant.privilege, 0, 0, MY_TEST(tl->schema_table)); > + if (is_temporary_table(tl)) > + { > + tl->grant.privilege|= TMP_TABLE_ACLS; > + } > +#endif > + int err= 0; > + if (table->versioned()) > + err= store_schema_period_record(thd, tl, schema_table, db_name, table_name, > + table->s->vers); > + if (!err && table->s->period.name) > + err= store_schema_period_record(thd, tl, schema_table, db_name, table_name, > + table->s->period); > + > + return err; > +} > > -static int get_schema_column_record(THD *thd, TABLE_LIST *tables, > - TABLE *table, bool res, > - const LEX_CSTRING *db_name, > - const LEX_CSTRING *table_name) > +static > +int get_schema_column_record(THD *thd, TABLE_LIST *tables, > + TABLE *table, //!< @ref Show::columns_fields_info > + bool res, > + const LEX_CSTRING *db_name, > + const LEX_CSTRING *table_name) > { > LEX *lex= thd->lex; > const char *wild= lex->wild ? lex->wild->ptr() : NullS; > @@ -7513,30 +7592,41 @@ static int get_schema_triggers_record(THD *thd, TABLE_LIST *tables, > DBUG_RETURN(0); > } > > +static int > +store_key_column_usage(TABLE *table, const LEX_CSTRING &db_name, > + const LEX_CSTRING &table_name, > + const LEX_CSTRING &key_name, > + const LEX_CSTRING &col_name) > +{ > + CHARSET_INFO *cs= system_charset_info; > + static const LEX_CSTRING def{STRING_WITH_LEN("def")}; > + const LEX_CSTRING *values[] { &def, &db_name, &key_name, > + &def, &db_name, &table_name, &col_name }; > + int err = 0; > + for (uint i = 0; i < array_elements(values) && !err; i++) > + table->field[i]->store(values[i], cs); you forgot `err=` here :) but as I wrote above, it's not needed here anyway > + return 0; > +} > > static void > store_key_column_usage(TABLE *table, const LEX_CSTRING *db_name, > const LEX_CSTRING *table_name, const char *key_name, > - size_t key_len, const char *con_type, size_t con_len, > + size_t key_len, const char *col_name, size_t col_len, > longlong idx) > { > - CHARSET_INFO *cs= system_charset_info; > - table->field[0]->store(STRING_WITH_LEN("def"), cs); > - table->field[1]->store(db_name->str, db_name->length, cs); > - table->field[2]->store(key_name, key_len, cs); > - table->field[3]->store(STRING_WITH_LEN("def"), cs); > - table->field[4]->store(db_name->str, db_name->length, cs); > - table->field[5]->store(table_name->str, table_name->length, cs); > - table->field[6]->store(con_type, con_len, cs); > + store_key_column_usage(table, *db_name, *table_name, {key_name, key_len}, > + {col_name, col_len}); pretty, but better, please, put it back. You can use the old store_key_column_usage from your get_schema_key_period_usage_record() just the same. There's no need to have two store_key_column_usage() functions. > table->field[7]->store((longlong) idx, TRUE); > } > > > -static int get_schema_key_column_usage_record(THD *thd, > - TABLE_LIST *tables, > - TABLE *table, bool res, > - const LEX_CSTRING *db_name, > - const LEX_CSTRING *table_name) > +static int > +get_schema_key_column_usage_record(THD *thd, TABLE_LIST *tables, > + TABLE *table, > + //!< @ref Show::key_column_usage_fields_info > + bool res, > + const LEX_CSTRING *db_name, > + const LEX_CSTRING *table_name) > { > DBUG_ENTER("get_schema_key_column_usage_record"); > if (res) > @@ -9703,6 +9824,18 @@ ST_FIELD_INFO key_column_usage_fields_info[]= > CEnd() > }; > > +ST_FIELD_INFO key_period_usage_fields_info[]= > +{ > + Column("CONSTRAINT_CATALOG", Catalog(), NOT_NULL, OPEN_FULL_TABLE), > + Column("CONSTRAINT_SCHEMA", Name(), NOT_NULL, OPEN_FULL_TABLE), > + Column("CONSTRAINT_NAME", Name(), NOT_NULL, OPEN_FULL_TABLE), > + Column("TABLE_CATALOG", Catalog(), NOT_NULL, OPEN_FULL_TABLE), > + Column("TABLE_SCHEMA", Name(), NOT_NULL, OPEN_FULL_TABLE), > + Column("TABLE_NAME", Name(), NOT_NULL, OPEN_FULL_TABLE), > + Column("PERIOD_NAME", Name(), NOT_NULL, OPEN_FULL_TABLE), Cannot they all be OPEN_FRM_ONLY? > + CEnd() > +}; > + > > ST_FIELD_INFO table_names_fields_info[]= > { > @@ -10208,6 +10358,8 @@ ST_SCHEMA_TABLE schema_tables[]= > {0, 0, 0, 0, 0, 0, 0, 0, 0, 0} > }; > > +static_assert(array_elements(schema_tables) == SCH_ENUM_SIZE + 1, > + "Update enum_schema_tables as well."); +1 > > int initialize_schema_table(st_plugin_int *plugin) > { Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 6

Re: 8114fb63bbb: MDEV-31684: Add timezone information to DATE_FORMAT
by Sergei Golubchik 04 Sep '23

04 Sep '23

Hi, Rucha, Few comments, after I saw the commit in the branch: On Sep 04, Rucha Deodhar wrote: > revision-id: 8114fb63bbb (mariadb-11.0.1-229-g8114fb63bbb) > parent(s): 7ba9c7fb84b > author: Rucha Deodhar > committer: Rucha Deodhar > timestamp: 2023-09-01 11:01:09 +0530 > message: > > MDEV-31684: Add timezone information to DATE_FORMAT > > Before starting to go over the format string, prepare the current time > zone information incase '%z' or '%Z' is encountered. > This information can be obtained as given below: > > A) If timezone is not set ( meaning we are working with system timezone): > Get the MYSQL_TIME representation for current time and GMT time using > current thread variable for timezone and timezone variable for UTC > respectively. This MYSQL_TIME variable will be used to calculate time > difference. Also convert current time in second to tm structure to > get system timezone information. > > B) If timezone is set as offset: > Get timezone information using current timezone information and store > in appropriate variable. > > C) If timezone is set as some place (example: Europe/Berlin) > Get timezone information by searching the timezone. During internal > timezone search, information like timeoffset from UTC and abbrevation > is stored in another relevant structure. Hence use the same information. > > diff --git a/mysql-test/main/date_formats.result b/mysql-test/main/date_formats.result > index 463cce39520..23aeffedef7 100644 > --- a/mysql-test/main/date_formats.result > +++ b/mysql-test/main/date_formats.result > @@ -573,3 +573,67 @@ time_format('01 02:02:02', '%T') > select time_format('2001-01-01 02:02:02', '%T'); > time_format('2001-01-01 02:02:02', '%T') > 02:02:02 > +# > +# Beginning of 11.3 test > +# > +# MDEV-31684: Add timezone information to DATE_FORMAT > +# > +SET @old_timezone= @@time_zone; > +SET TIME_ZONE='Japan'; > +SELECT DATE_FORMAT('2009-10-04 22:23:00', '%z %Z') AS current_timezone; > +current_timezone > ++0900 CJT > +SELECT DATE_FORMAT(CURDATE(), '%z %Z') AS current_tz; 1) please, don't use CURDATE/CURRENT_TIMESTAMP/etc. The time zone abbreviation depends on the date: $ TZ=Europe/Berlin date -d '2020-01-01 1:2:3' Wed Jan 1 01:02:03 CET 2020 $ TZ=Europe/Berlin date -d '2020-06-01 1:2:3' Mon Jun 1 01:02:03 CEST 2020 See? Could be CET or CEST. Japan doesn't seem to have daylight saving, so it'll be the same abbreviation throughout the year, but it can still change at some point. So always use a fixed date, like in your first test. 2) why is it CJT at all? $ TZ=Japan date -d '2009-10-04 22:23:00' +'%z %Z' +0900 JST it looks like it should be JST? ... 3) also, please test that CET/CEST example. Or, rather, MET/MEST/MST/MMT example - the data for these abbreviations are already present in mariadb_test_data_timezone.sql That is, add lines like SET @@time_zone='Europe/Moscow'; SELECT DATE_FORMAT('....', '%z %Z'); SELECT DATE_FORMAT('....', '%z %Z'); SELECT DATE_FORMAT('....', '%z %Z'); and with different dates it should return different abbreviations for the same time zone. > +# > +# output depends on system time information. Hence replacing > +# to avoid result diff test failures. > +# > +SET @@time_zone= default; > +SELECT DATE_FORMAT('2009-10+|-HH:MM ABC22:23:00', '%z %Z') AS current_timezone; 1) I don't understand what you're testing here :( 2) Anyway, note that you can put, say, --timezone=Europe/Berlin into the date_formats.opt file and the server will be restarted with this value in TZ. That is, you can force the system time zone to be a specific fixed value. > diff --git a/scripts/mariadb_test_data_timezone.sql b/scripts/mariadb_test_data_timezone.sql > index 8d07d413cef..83bb548478d 100644 > --- a/scripts/mariadb_test_data_timezone.sql > +++ b/scripts/mariadb_test_data_timezone.sql > @@ -14,8 +14,8 @@ > -- along with this program; if not, write to the Free Software > -- Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1335 USA > > -INSERT INTO time_zone_name (Name, Time_Zone_id) ... > -INSERT INTO time_zone (Time_zone_id, Use_leap_seconds) ... > -INSERT INTO time_zone_transition (Time_zone_id, Transition_time, Transition_type_id) ... > -INSERT INTO time_zone_transition_type (Time_zone_id, Transition_type_id, `Offset`, Is_DST, Abbreviation) ... > -INSERT INTO time_zone_leap_second (Transition_time, Correction) ... > +INSERT INTO time_zone_name (Name, Time_Zone_id) VALUES ... ('India/Kolkata', 6); > +INSERT INTO time_zone (Time_zone_id, Use_leap_seconds) ... > +INSERT INTO time_zone_transition (Time_zone_id, Transition_time, Transition_type_id) ... > +INSERT INTO time_zone_transition_type (Time_zone_id, Transition_type_id, `Offset`, Is_DST, Abbreviation) ... > +INSERT INTO time_zone_leap_second (Transition_time, Correction) ... 1. Why? I didn't see where you use this new value in tests. 2. Is it really India/Kolkata? The official https://www.iana.org/time-zones seems to have Asia/Kolkata. > diff --git a/sql/item_timefunc.cc b/sql/item_timefunc.cc > index 26adc4eddaa..f8d3fe7dfe8 100644 > --- a/sql/item_timefunc.cc > +++ b/sql/item_timefunc.cc > @@ -468,20 +468,111 @@ static bool extract_date_time(THD *thd, DATE_TIME_FORMAT *format, > DBUG_RETURN(1); > } > > +/* > + > + Gets local time in MYSQL_TIME structure and tm struct. > + > +*/ > +void get_timezone_offset_information(THD *thd, Time_zone *tz, int &hr, > + int &min, char* abbrevation, > + bool &is_neg) > +{ > + if (!tz) > + return; > + > + if (tz->get_name()->length() == tz_SYSTEM_name.length() && > + !strncmp((const char*)(tz->get_name()->ptr()), > + (const char*)(tz_SYSTEM_name.ptr()), 6)) > + { > + struct tm tm_local_time; > + time_t time_sec= my_time(0); > + MYSQL_TIME local_TIME, gmt_TIME; > + ulonglong seconds_diff; > + ulong microsec_diff; > + > + /* Get local time in MYSQL_TIME struct. */ > + thd->variables.time_zone->gmt_sec_to_TIME(&local_TIME, > + thd->query_start()); > + /* Get GMT time in MYSQL_TIME struct. */ > + my_tz_UTC->gmt_sec_to_TIME(&gmt_TIME, thd->query_start()); > + > + /* We need above two MYSQL_TIME to get time difference. */ > + is_neg= calc_time_diff(&local_TIME, &gmt_TIME, 1, > + &seconds_diff, &microsec_diff); > + > + /* > + once we have time difference in seconds_diff, > + convert it into hr and min. > + */ > + hr= (int)(seconds_diff/3600L); > + int temp= (int)(seconds_diff%3600L); > + min= temp/60L; > + > + /* > + Get timezone abbrevation. It is stored in tm structure, > + so convert time to tm struct and copy it. > + */ 1) tm structure also has the offset, you don't need to calculate it manually. 2) you should use not my_time(0) but the value of the first DATE_FORMAT() argument > + localtime_r(&time_sec, &tm_local_time); > +# ifdef __USE_MISC > + int len= strlen((tm_local_time.tm_zone)); > + strncpy(abbrevation, (tm_local_time.tm_zone), len); > + abbrevation[len]= '\0'; > +# endif > + } > + else > + { > + /* > + Information for "named" timezone and offset can be obtained from their > + relevant class. So get the timezone information first. > + */ > + struct tz tmp; > + tz->get_timezone_information(&tmp); > + is_neg= tmp.is_behind; > + > + /* Get time in hour and min. */ > + time_t time_in_sec= abs(tmp.seconds_offset); > + hr= (int)(time_in_sec/3600L); > + int temp= (int)(time_in_sec%3600L); > + min= temp/60L; > + > + /* Copy the abbrevation. */ > + strncpy(abbrevation, (const char*)(tmp.abbrevation), > + strlen((const char*)(tmp.abbrevation))); > + abbrevation[strlen((const char*)(tmp.abbrevation))+1]='\0'; > + } > + > + return; > +} > + > > /** > Create a formatted date/time value in a string. > */ > > -static bool make_date_time(const String *format, const MYSQL_TIME *l_time, > - timestamp_type type, const MY_LOCALE *locale, > - String *str) > +static bool make_date_time(THD *thd, const String *format, > + const MYSQL_TIME *l_time, timestamp_type type, > + const MY_LOCALE *locale, String *str) > { > char intbuff[15]; > uint hours_i; > uint weekday; > ulong length; > const char *ptr, *end; > + int diff_hr=0, diff_min=0; > + bool curr_time_behind_utc= false; > + char abbrevation[8]; > + > + Time_zone* curr_timezone= my_tz_find(thd, > + thd->variables.time_zone->get_name()); > + > + /* > + precalculate local time and timezone information because > + %z or %Z maybe present in the format. > + */ > + memset(abbrevation, 0, sizeof(abbrevation)); > + get_timezone_offset_information(thd, curr_timezone, diff_hr, > + diff_min, abbrevation, > + curr_time_behind_utc); Just a thought. get_timezone_offset_information() doesn't seem to be very cheap. May be do it only when %z or %Z is requested? Like case 'z': if (!have_tz_info) { memset(abbrevation, 0, sizeof(abbrevation)); get_timezone_offset_information(thd, curr_timezone, diff_hr, diff_min, abbrevation, curr_time_behind_utc); have_tz_info= true; } Or better something with less code duplication :) > str->length(0); > > @@ -699,6 +790,31 @@ static bool make_date_time(const String *format, const MYSQL_TIME *l_time, > str->append_with_prefill(intbuff, length, 1, '0'); > break; > > + case 'z': > + { > + if (curr_time_behind_utc) > + str->append("-", 1, system_charset_info); > + else > + str->append("+", 1, system_charset_info); > + > + if (diff_hr/10 == 0) > + str->append("0", 1, system_charset_info); > + length= (uint) (int10_to_str(diff_hr, intbuff, 10) - intbuff); > + str->append_with_prefill(intbuff, length, 1, '0'); > + if (diff_min/10 == 0) > + str->append("0", 1, system_charset_info); > + length= (uint) (int10_to_str(diff_min, intbuff, 10) - intbuff); > + str->append_with_prefill(intbuff, length, 1, '0'); why do you append "0" manually, if you use append_with_prefill(), which, I suppose, does it automatically? or, if you add "0" manually, you should use simple append(). > + } > + break; > + > + case 'Z': > + { > + str->append(abbrevation, > + (uint) strlen(abbrevation), > + system_charset_info); > + } > + break; > default: > str->append(*ptr); > break; > @@ -1912,7 +2028,7 @@ String *Item_func_date_format::val_str(String *str) > > /* Create the result string */ > str->set_charset(collation.collation); > - if (!make_date_time(format, &l_time, > + if (!make_date_time(thd, format, &l_time, > is_time_format ? MYSQL_TIMESTAMP_TIME : > MYSQL_TIMESTAMP_DATE, > lc, str)) > diff --git a/sql/tztime.cc b/sql/tztime.cc > index e77a7529332..4662fa082e0 100644 > --- a/sql/tztime.cc > +++ b/sql/tztime.cc > @@ -55,15 +55,10 @@ > > /* > Now we don't use abbreviations in server but we will do this in future. > + Edit: Startd needing abbrevation in server as part of task MDEV-31684. typo > */ > -#if defined(TZINFO2SQL) || defined(TESTTIME) > -#define ABBR_ARE_USED > -#else > -#if !defined(DBUG_OFF) > -/* Let use abbreviations for debug purposes */ > -#undef ABBR_ARE_USED > -#define ABBR_ARE_USED > -#endif /* !defined(DBUG_OFF) */ > +#ifndef ABBR_ARE_USED > +#define ABBR_ARE_USED > #endif /* defined(TZINFO2SQL) || defined(TESTTIME) */ > > /* Structure describing local time type (e.g. Moscow summer time (MSD)) */ > @@ -1008,11 +1003,6 @@ TIME_to_gmt_sec(const MYSQL_TIME *t, const TIME_ZONE_INFO *sp, uint *error_code) > > #if !defined(TESTTIME) && !defined(TZINFO2SQL) > > -/* > - String with names of SYSTEM time zone. > -*/ > -static const String tz_SYSTEM_name("SYSTEM", 6, &my_charset_latin1); > - > > /* > Instance of this class represents local time zone used on this system > @@ -1031,6 +1021,7 @@ class Time_zone_system : public Time_zone > virtual my_time_t TIME_to_gmt_sec(const MYSQL_TIME *t, uint *error_code) const; > virtual void gmt_sec_to_TIME(MYSQL_TIME *tmp, my_time_t t) const; > virtual const String * get_name() const; > + virtual void get_timezone_information(struct tz* curr_tz) const; > }; > > > @@ -1097,6 +1088,12 @@ Time_zone_system::gmt_sec_to_TIME(MYSQL_TIME *tmp, my_time_t t) const > adjust_leap_second(tmp); > } > > +void > +Time_zone_system::get_timezone_information(struct tz* curr_tz) const > +{ this is weird. You have a special if (tz->get_name()->length() == tz_SYSTEM_name.length() && !strncmp((const char*)(tz->get_name()->ptr()), (const char*)(tz_SYSTEM_name.ptr()), 6)) to detect SYSTEM time zone by name in get_timezone_offset_information(). why not to put that logic here, in the dedicated method, like you did for other time zones? > + return; > +} > + > > /* > Get name of time zone > @@ -1128,6 +1125,7 @@ class Time_zone_utc : public Time_zone > uint *error_code) const; > virtual void gmt_sec_to_TIME(MYSQL_TIME *tmp, my_time_t t) const; > virtual const String * get_name() const; > + virtual void get_timezone_information(struct tz* curr_tz) const; > }; > > > @@ -1175,6 +1173,12 @@ Time_zone_utc::gmt_sec_to_TIME(MYSQL_TIME *tmp, my_time_t t) const > adjust_leap_second(tmp); > } > > +void > +Time_zone_utc::get_timezone_information(struct tz* curr_tz) const > +{ what is printed for UTC? Please, add a test for that. > + return; > +} > + > > /* > Get name of time zone > @@ -1210,6 +1214,7 @@ class Time_zone_db : public Time_zone > virtual my_time_t TIME_to_gmt_sec(const MYSQL_TIME *t, uint *error_code) const; > virtual void gmt_sec_to_TIME(MYSQL_TIME *tmp, my_time_t t) const; > virtual const String * get_name() const; > + virtual void get_timezone_information(struct tz* curr_tz) const; > private: > TIME_ZONE_INFO *tz_info; > const String *tz_name; > @@ -1280,6 +1285,16 @@ Time_zone_db::gmt_sec_to_TIME(MYSQL_TIME *tmp, my_time_t t) const > adjust_leap_second(tmp); > } > > +void > +Time_zone_db::get_timezone_information(struct tz* curr_tz) const > +{ > + curr_tz->seconds_offset= tz_info->revtis->rt_offset; > + curr_tz->is_behind= tz_info->revtis->rt_offset < 0 ? true : false; > + strncpy((char*)(curr_tz->abbrevation), (const char*)(tz_info->chars), > + strlen((const char*)(tz_info->chars))+1); > + curr_tz->abbrevation[strlen((const char*)(tz_info->chars))+1]='\0'; here and below (in Time_zone_offset) 1. why casts? 2. you use strncpy incorrectly, the third argument specifies the size of the buffer, like strncpy(curr_tz->abbrevation, tz_info->chars, sizeof(curr_tz->abbrevation)); but better use strmake, it'll do the zero-termination for you strmake(curr_tz->abbrevation, tz_info->chars, sizeof(curr_tz->abbrevation)-1); > +} > + > > /* > Get name of time zone > @@ -1309,6 +1324,7 @@ class Time_zone_offset : public Time_zone > uint *error_code) const; > virtual void gmt_sec_to_TIME(MYSQL_TIME *tmp, my_time_t t) const; > virtual const String * get_name() const; > + virtual void get_timezone_information(struct tz* curr_tz) const; > /* > This have to be public because we want to be able to access it from > my_offset_tzs_get_key() function > @@ -1434,6 +1450,16 @@ Time_zone_offset::get_name() const > return &name; > } > > +void > +Time_zone_offset::get_timezone_information(struct tz* curr_tz) const > +{ > + curr_tz->seconds_offset= offset; > + const char *name= get_name()->ptr(); > + curr_tz->is_behind= name[0] == '+' ? false : true; > + strncpy((char*)(curr_tz->abbrevation), name, this->get_name()->length()); > + curr_tz->abbrevation[this->get_name()->length()]= '\0'; > +} > + > > static Time_zone_utc tz_UTC; > static Time_zone_system tz_SYSTEM; > diff --git a/sql/tztime.h b/sql/tztime.h > index 6d8af62ecd4..57e4aa2021b 100644 > --- a/sql/tztime.h > +++ b/sql/tztime.h > @@ -26,6 +26,11 @@ > #include "sql_list.h" /* Sql_alloc */ > #include "sql_string.h" /* String */ > > +/* > + String with names of SYSTEM time zone. > +*/ > +static const String tz_SYSTEM_name("SYSTEM", 6, &my_charset_latin1); No, sorry. It's a .h file included in many .cc files. You'll have a separate `static const String` in every single source file that includes this header. > + > class THD; Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: 23b15d24eb3: cleanup: add Field::store_yesno
by Sergei Golubchik 31 Aug '23

31 Aug '23

Hi, Nikita, On Aug 31, Nikita Malyavin wrote: > revision-id: 23b15d24eb3 (mariadb-11.0.1-227-g23b15d24eb3) > parent(s): cb37c99dd87 > author: Nikita Malyavin > committer: Nikita Malyavin > timestamp: 2023-08-30 13:18:52 +0400 > message: > > cleanup: add Field::store_yesno I agree, it's a good idea to have a helper for storing YES/NO in the field. But I think Field class itself is a wrong place for it. YES/NO is exclusively INFORMATION_SCHEMA thing and it feels kind of wrong to have it in the base class. I'd do a separate function, like store_yes_no(). Or something. Also, note that you missed few places where it's needed, like pos=(uchar*) ((field->flags & NOT_NULL_FLAG) ? "NO" : "YES"); table->field[6]->store((const char*) pos, strlen((const char*) pos), cs); or if (updatable_view) table->field[5]->store(STRING_WITH_LEN("YES"), cs); else table->field[5]->store(STRING_WITH_LEN("NO"), cs); both in sql_show.cc. There are more. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: Client-server protocol improvement proposal: Connection redirection
by Otto Kekäläinen 28 Aug '23

28 Aug '23

Hi Pali! On Fri, 18 Aug 2023 at 11:47, Pali <pali(a)cpan.org> wrote: > CCing choroba (for Perl's DBD::MariaDB) and replying to developers@ ML. > > I have read proposal and the first thing which worries me is how the > proposal handles security aspect of the whole solution. > ... > If TLS is used then any protocol redirection information must happen > inside of the TLS tunnel. Thanks for the comments! Actually Daniel's submission has already this covered and the redirect works only if the connection is TLS protected: * https://github.com/MariaDB/server/pull/2681, server-side implementation * https://github.com/mariadb-corporation/mariadb-connector-c/pull/226, MariaDB Connector/C implementation

1 0

Re: [MariaDB commits] [PATCH 4/4] MDEV-31273: Precompute binlog checksums
by Marko Mäkelä 25 Aug '23

25 Aug '23

On Fri, Aug 25, 2023 at 10:16 AM Kristian Nielsen via commits <commits(a)lists.mariadb.org> wrote: > > Compute binlog checksums (when enabled) already when writing events > into the statement or transaction caches, where before it was done > when the caches are copied to the real binlog file. This moves the > checksum computation outside of holding LOCK_log, improving > scalabitily. > > At stmt/trx cache write time, the final end_log_pos values are not > known, so with this patch these will be set to 0. [...] > Checksums cannot be pre-computed when binlog encryption is enabled, as > encryption relies on correct end_log_pos to provide part of the > nonce/IV. Checksum pre-computation is also disabled for WSREP/Galera. Have you thought about a format change that would address these issues? In MDEV-14425 https://github.com/MariaDB/server/commit/685d958e38b825ad9829be311f26729ccc… I redesigned the InnoDB log file format in a way that allows checksums to be computed before the final position of the records (the log sequence number of LSN) is known. For encryption, an additional 64-bit nonce will be written; this currently is the LSN at the time the encryption was scheduled. Tablespace identifiers, page numbers and file names are not encrypted, because these were never fully encrypted in innodb_encrypt_tables=ON either. This allows the metadata to be used as initialization vectors, and also avoids the need for backup tools to know any encryption keys. Thanks to MDEV-27774, not only InnoDB computes checksums while not holding any mutex, but multiple threads can copy their mini-transactions to the shared log_sys.buf while holding a shared lock_sys.latch. An exclusive latch will only be needed during a log checkpoint when we need to guarantee that all changes up to a particular LSN have been written to the log buffer. This would not be possible without the format change that was implemented in MDEV-14425. > +/* Convenience macros since ulong is 32 or 64 bit depending on platform. */ > +#if SIZEOF_LONG == 4 > +#define my_atomic_storeul(P, D) my_atomic_store32((int32 volatile *)(P), (D)) In all currently supported MariaDB Server versions, C++11 is available. Is there a particular reason why you would not use std::atomic<size_t> here? Marko -- Marko Mäkelä, Lead Developer InnoDB MariaDB plc

2 3

Client-server protocol improvement proposal: Connection redirection
by Lenski, Daniel 18 Aug '23

18 Aug '23

Hi! I would like to propose a new feature in the MariaDB client-server protocol<https://mariadb.com/kb/en/clientserver-protocol>: application-layer redirection of client connections. We want the MariaDB server to be able to tell clients connecting to it, “Sorry, this server is unavailable. Connect to an alternate server instead.” This mechanism is inspired by HTTP 302 (“temporary redirect”) mechanism familiar to all developers of web applications, and is intended to have similar semantics and security properties, since these have now been widely deployed and tested for decades. I have submitted a minimal but viable implementation of this at: * https://github.com/MariaDB/server/pull/2681, server-side implementation * https://github.com/mariadb-corporation/mariadb-connector-c/pull/226, MariaDB Connector/C implementation I am seeking advice and opinions on these, and will proceed to finalize this in the following 1-2 months. In its current form, this implementation allows the MariaDB server administrator to set two variables: * SET GLOBAL SERVER_REDIRECT_MODE={ON,OFF} (the default is OFF) * SET GLOBAL SERVER_REDIRECT_TARGET='my-new-server.example.com' (or 192.168.0.123:3307, or new-server.com:3308, etc) When SERVER_REDIRECT_MODE is set to on, the server will stop accepting new TCP-based connections (unless they are to the extra port<https://mariadb.com/kb/en/thread-pool-in-mariadb/#configuring-the-extra-port>, which is intended for emergency administrative use), and it will instead respond to all clients with an error packet<https://mariadb.com/kb/en/err_packet/>: $ mariadb --host my-server.example.com ERROR 4189 (HY000): |Server is redirecting clients to 'my-new-server.example.com:3307'|my-new-server.example.com:3307 When an appropriately-updated client receives this error packet (error code 4196, message formatted as |Human readable message|<value of SERVER_REDIRECT_TARGET>), it will parse the redirection target out from the error message and attempt a new connection: $ updated-mariadb --host my-server.example.com -e 'select @@hostname' Got server redirect to 'my-new-server.example.com' (port 3307) +---------------------------+ | @@hostname | +---------------------------+ | my-new-server.example.com | +---------------------------+ The feature is gracefully backwards-compatible. Old clients that do not support redirection will fail and show the message in a form from which humans can deduce what happened. New clients that do support it will follow it to the new server. The feature is also designed to be extensible in a forwards-compatible way. The SERVER_REDIRECT_MODE system variable could be expanded to support additional redirection modes. For example, it could be augmented to redirect new connections only from specific IP subnets, or to direct the client to interpret the SERVER_REDIRECT_TARGET variable in a new form (such as a<https://dev.mysql.com/doc/connector-j/8.0/en/connector-j-reference-jdbc-url…> JDBC-compatible mysql:// URL<https://dev.mysql.com/doc/connector-j/8.0/en/connector-j-reference-jdbc-url…> as proposed in MDEV-15935<https://jira.mariadb.org/browse/MDEV-15935>). While researching and designing this connection redirection mechanism, I discovered or rediscovered a couple of major TLS-related security vulnerabilities in the MariaDB Connector/C client [1, 2]. These vulnerabilities affect all MariaDB users who care about the confidentiality and authenticity of their client/server connections, and they become even more severe if a connection redirection mechanism exists — because this entails the client trusting the server to send it instructions for creating a new network connection. Unless these are resolved, any connection redirection mechanism will be susceptible to hijacking and redirection to an attacker-controlled server even if the client thinks it is using TLS/SSL. What are the use cases for this connection redirection feature? Even in this simple, initial form, the feature is already useful for database administrators who want to gracefully shut down a MariaDB server (for example, to upgrade it), and to redirect new client connections to a replica or alternate server that can accept the traffic. Server-side connection redirection can be combined with existing MariaDB server features, such as SET GLOBAL READ_ONLY=ON and KILLing of idle or long-running connections, in order to build a full switchover solution for shunting traffic from one server to another as gracefully as possible. Currently, MariaDB database administrators tend to use proxies and wrappers (such as HikariCP) to redirect client connections from one MariaDB server to another. Supporting this redirection mechanism directly in the server, and in the client-server protocol, will allow many simple use cases of redirection without extra software, making MariaDB easier and simpler to manage for minimal downtime situations. Please post on this mailing list what you think about this suggestion. I am looking forward to polishing this feature and submitting it in a form that meets the quality requirements for new features from MariaDB core developers. Thank you! Daniel Lenski Amazon RDS MySQL/MariaDB engine team --- Notes on TLS vulnerabilities in MariaDB Connector/C: [1] CONC-648<https://jira.mariadb.org/browse/CONC-648>, reported by me on 5 June: when a Connector/C-based client is using connections secured by TLS/SSL, Connector/C will nevertheless receive plain-text, unencrypted error packets prior to the TLS handshake, and present them to the client in a way that’s indistinguishable from TLS-protected error packets sent by the server after the TLS handshake. This longstanding vulnerability is a consequence of the tangled and unsystematic way that the Connector/C codebase interleaves the MariaDB application-layer authentication protocol with the transport-layer security protocol. My standalone mariadb-connector-c PR#223<https://github.com/mariadb-corporation/mariadb-connector-c/pull/223> fixes the most straightforward manifestation of this vulnerability. Until this issue is resolved in Connector/C, it will be trivial for MITM attackers to inject plaintext connection redirection packets into MariaDB client/server connections, and thereby redirect clients to attacker-controlled servers despite those clients’ intention to use only TLS-secured connections. [2] MDEV-28634<https://jira.mariadb.org/browse/MDEV-28634>, reported almost 3 years ago by Geoff Montee: Connector/C clients which specify --ssl without --ssl-verify-server-cert will allow their connections to be silently downgraded to unencrypted, plaintext connections. This vulnerability can be used by on-path attackers to trivially downgrade most MariaDB client-server connections from TLS-encrypted to plaintext. My standalone mariadb-connector-c PR#224<https://github.com/mariadb-corporation/mariadb-connector-c/pull/224> fixes this. It is a backwards-incompatible change in that clients which specify --ssl (and thus clearly expect TLS-secured connections) will no longer succeed in connecting to servers which don’t support TLS. This should be considered a good thing: MariaDB Connector/C-based clients have been misleading their users about whether their connections are guaranteed to be encrypted, and will no longer do so. Essentially all software that purports to secure network connections with TLS have implemented similar changes, most of it about a decade ago when the pervasiveness of such attacks<https://en.wikipedia.org/wiki/Global_surveillance_disclosures_(2013%E2%80%9…> (by ISPs, censors, intelligenceagencies, etc) became very widely-known.

8 28

Re: [Commits] cb06612a9da: MDEV-31655: Parallel replication deadlock victim preference code errorneously removed
by Kristian Nielsen 15 Aug '23

15 Aug '23

Hi Marko, I'd like to ask you to review the below patch for MDEV-31655. https://jira.mariadb.org/browse/MDEV-31655 https://github.com/MariaDB/server/commit/cb06612a9da09a7981ada84768793f2ff3… This patch restores code in InnoDB that was originally part of the parallel replication implementation. It makes InnoDB choose the right victim when two transactions in parallel replication deadlock with each other. The code was erroneously removed as part of the VATS work in InnoDB. I saw that you later deprecated and eventually removed the VATS code. I saw that the InnoDB deadlock code is substantially changed in 10.6, so I will need to rework the patch in that version. But I wanted to ask your opinion about this 10.4 version first. In the patch, note in particular the modification to has_higher_priority(), which I think has the effect of choosing which of two waiting transactions to grant a lock to. I don't remember adding this myself, so it might have been added as part of VATS work. But it sounds appropriate; if T1 and T2 are both waiting for a lock, there is no point in granting it to T2 over T1, as then T2 will just be immediately killed and rolled back by replication. - Kristian. Kristian Nielsen via commits <commits(a)lists.mariadb.org> writes: > revision-id: cb06612a9da09a7981ada84768793f2ff3ef842c (mariadb-10.4.30-3-gcb06612a9da) > parent(s): dbe5c20b755b87f67d87990c3baabc866667e41b > author: Kristian Nielsen > committer: Kristian Nielsen > timestamp: 2023-07-11 00:31:29 +0200 > message: > > MDEV-31655: Parallel replication deadlock victim preference code errorneously removed > > Restore code to make InnoDB choose the second transaction as a deadlock > victim if two transactions deadlock that need to commit in-order for > parallel replication. This code was erroneously removed when VATS was > implemented in InnoDB. > > Also add a test case for InnoDB choosing the right deadlock victim. > > Signed-off-by: Kristian Nielsen <knielsen(a)knielsen-hq.org> > > --- > .../suite/binlog_encryption/rpl_parallel.result | 42 ++++++++++++- > mysql-test/suite/rpl/r/rpl_parallel.result | 42 ++++++++++++- > mysql-test/suite/rpl/t/rpl_parallel.test | 71 +++++++++++++++++++++- > sql/rpl_parallel.cc | 7 ++- > sql/sql_class.cc | 43 +++++++++++++ > storage/innobase/lock/lock0lock.cc | 12 ++++ > storage/innobase/trx/trx0trx.cc | 12 ++++ > 7 files changed, 225 insertions(+), 4 deletions(-) > > diff --git a/mysql-test/suite/binlog_encryption/rpl_parallel.result b/mysql-test/suite/binlog_encryption/rpl_parallel.result > index b75a66a634a..b24ff7ba53d 100644 > --- a/mysql-test/suite/binlog_encryption/rpl_parallel.result > +++ b/mysql-test/suite/binlog_encryption/rpl_parallel.result > @@ -2,6 +2,7 @@ include/master-slave.inc > [connection master] > connection server_2; > SET @old_parallel_threads=@@GLOBAL.slave_parallel_threads; > +SET @old_parallel_mode=@@GLOBAL.slave_parallel_mode; > SET GLOBAL slave_parallel_threads=10; > ERROR HY000: This operation cannot be performed as you have a running slave ''; run STOP SLAVE '' first > include/stop_slave.inc > @@ -1680,13 +1681,52 @@ a > 2000 > SELECT * FROM t2 WHERE a>=2000 ORDER BY a; > a > +MDEV-31655: Parallel replication deadlock victim preference code erroneously removed > +connection server_1; > +CREATE TABLE t7 (a INT PRIMARY KEY, b INT) ENGINE=InnoDB; > +BEGIN; > +COMMIT; > +include/save_master_gtid.inc > +connection server_2; > +include/sync_with_master_gtid.inc > +include/stop_slave.inc > +set @@global.slave_parallel_threads= 5; > +set @@global.slave_parallel_mode= conservative; > +SET @old_dbug= @@GLOBAL.debug_dbug; > +SET GLOBAL debug_dbug= "+d,rpl_mdev31655_zero_retries"; > +connection master; > +SET @old_dbug= @@SESSION.debug_dbug; > +SET SESSION debug_dbug="+d,binlog_force_commit_id"; > +SET @commit_id= 1+1000; > +SET @commit_id= 2+1000; > +SET @commit_id= 3+1000; > +SET @commit_id= 4+1000; > +SET @commit_id= 5+1000; > +SET @commit_id= 6+1000; > +SET @commit_id= 7+1000; > +SET @commit_id= 8+1000; > +SET @commit_id= 9+1000; > +SET @commit_id= 10+1000; > +SET SESSION debug_dbug= @old_dbug; > +SELECT COUNT(*), SUM(a*100*b) FROM t7; > +COUNT(*) SUM(a*100*b) > +10 225000 > +include/save_master_gtid.inc > +connection server_2; > +include/start_slave.inc > +include/sync_with_master_gtid.inc > +SET GLOBAL debug_dbug= @old_dbug; > +SELECT COUNT(*), SUM(a*100*b) FROM t7; > +COUNT(*) SUM(a*100*b) > +10 225000 > connection server_2; > include/stop_slave.inc > SET GLOBAL slave_parallel_threads=@old_parallel_threads; > +SET GLOBAL slave_parallel_mode=@old_parallel_mode; > include/start_slave.inc > SET DEBUG_SYNC= 'RESET'; > connection server_1; > DROP function foo; > -DROP TABLE t1,t2,t3,t4,t5,t6; > +DROP TABLE t1,t2,t3,t4,t5,t6,t7; > SET DEBUG_SYNC= 'RESET'; > include/rpl_end.inc > diff --git a/mysql-test/suite/rpl/r/rpl_parallel.result b/mysql-test/suite/rpl/r/rpl_parallel.result > index 9b2e68d366e..ef89d954faa 100644 > --- a/mysql-test/suite/rpl/r/rpl_parallel.result > +++ b/mysql-test/suite/rpl/r/rpl_parallel.result > @@ -2,6 +2,7 @@ include/master-slave.inc > [connection master] > connection server_2; > SET @old_parallel_threads=@@GLOBAL.slave_parallel_threads; > +SET @old_parallel_mode=@@GLOBAL.slave_parallel_mode; > SET GLOBAL slave_parallel_threads=10; > ERROR HY000: This operation cannot be performed as you have a running slave ''; run STOP SLAVE '' first > include/stop_slave.inc > @@ -1679,13 +1680,52 @@ a > 2000 > SELECT * FROM t2 WHERE a>=2000 ORDER BY a; > a > +MDEV-31655: Parallel replication deadlock victim preference code erroneously removed > +connection server_1; > +CREATE TABLE t7 (a INT PRIMARY KEY, b INT) ENGINE=InnoDB; > +BEGIN; > +COMMIT; > +include/save_master_gtid.inc > +connection server_2; > +include/sync_with_master_gtid.inc > +include/stop_slave.inc > +set @@global.slave_parallel_threads= 5; > +set @@global.slave_parallel_mode= conservative; > +SET @old_dbug= @@GLOBAL.debug_dbug; > +SET GLOBAL debug_dbug= "+d,rpl_mdev31655_zero_retries"; > +connection master; > +SET @old_dbug= @@SESSION.debug_dbug; > +SET SESSION debug_dbug="+d,binlog_force_commit_id"; > +SET @commit_id= 1+1000; > +SET @commit_id= 2+1000; > +SET @commit_id= 3+1000; > +SET @commit_id= 4+1000; > +SET @commit_id= 5+1000; > +SET @commit_id= 6+1000; > +SET @commit_id= 7+1000; > +SET @commit_id= 8+1000; > +SET @commit_id= 9+1000; > +SET @commit_id= 10+1000; > +SET SESSION debug_dbug= @old_dbug; > +SELECT COUNT(*), SUM(a*100*b) FROM t7; > +COUNT(*) SUM(a*100*b) > +10 225000 > +include/save_master_gtid.inc > +connection server_2; > +include/start_slave.inc > +include/sync_with_master_gtid.inc > +SET GLOBAL debug_dbug= @old_dbug; > +SELECT COUNT(*), SUM(a*100*b) FROM t7; > +COUNT(*) SUM(a*100*b) > +10 225000 > connection server_2; > include/stop_slave.inc > SET GLOBAL slave_parallel_threads=@old_parallel_threads; > +SET GLOBAL slave_parallel_mode=@old_parallel_mode; > include/start_slave.inc > SET DEBUG_SYNC= 'RESET'; > connection server_1; > DROP function foo; > -DROP TABLE t1,t2,t3,t4,t5,t6; > +DROP TABLE t1,t2,t3,t4,t5,t6,t7; > SET DEBUG_SYNC= 'RESET'; > include/rpl_end.inc > diff --git a/mysql-test/suite/rpl/t/rpl_parallel.test b/mysql-test/suite/rpl/t/rpl_parallel.test > index 9ba7a30f2eb..d43cec4df34 100644 > --- a/mysql-test/suite/rpl/t/rpl_parallel.test > +++ b/mysql-test/suite/rpl/t/rpl_parallel.test > @@ -13,6 +13,7 @@ > > --connection server_2 > SET @old_parallel_threads=@@GLOBAL.slave_parallel_threads; > +SET @old_parallel_mode=@@GLOBAL.slave_parallel_mode; > --error ER_SLAVE_MUST_STOP > SET GLOBAL slave_parallel_threads=10; > --source include/stop_slave.inc > @@ -2203,16 +2204,84 @@ SELECT * FROM t1 WHERE a>=2000 ORDER BY a; > SELECT * FROM t2 WHERE a>=2000 ORDER BY a; > > > +--echo MDEV-31655: Parallel replication deadlock victim preference code erroneously removed > +# The problem was that InnoDB would choose the wrong deadlock victim. > +# Create a lot of transactions that can cause deadlocks, and use error > +# injection to check that the first transactions in each group is never > +# selected as deadlock victim. > +--let $rows= 10 > +--let $transactions= 5 > +--let $gcos= 10 > + > +--connection server_1 > +CREATE TABLE t7 (a INT PRIMARY KEY, b INT) ENGINE=InnoDB; > +BEGIN; > +--disable_query_log > +--let $i= 0 > +while ($i < 10) { > + eval INSERT INTO t7 VALUES ($i, 0); > + inc $i; > +} > +--enable_query_log > +COMMIT; > +--source include/save_master_gtid.inc > + > +--connection server_2 > +--source include/sync_with_master_gtid.inc > +--source include/stop_slave.inc > +eval set @@global.slave_parallel_threads= $transactions; > +set @@global.slave_parallel_mode= conservative; > +SET @old_dbug= @@GLOBAL.debug_dbug; > +# This error injection will allow no retries for GTIDs divisible by 1000. > +SET GLOBAL debug_dbug= "+d,rpl_mdev31655_zero_retries"; > + > +--connection master > +SET @old_dbug= @@SESSION.debug_dbug; > +SET SESSION debug_dbug="+d,binlog_force_commit_id"; > + > +--let $j= 1 > +while ($j <= $gcos) { > + eval SET @commit_id= $j+1000; > + --let $i= 0 > + while ($i < $transactions) { > + --disable_query_log > + eval SET SESSION gtid_seq_no= 1000 + 1000*$j + $i; > + BEGIN; > + --let $k= 0 > + while ($k < $rows) { > + eval UPDATE t7 SET b=b+1 WHERE a=(($i+$k) MOD $rows); > + inc $k; > + } > + COMMIT; > + --enable_query_log > + inc $i; > + } > + inc $j; > +} > + > +SET SESSION debug_dbug= @old_dbug; > +SELECT COUNT(*), SUM(a*100*b) FROM t7; > + > +--source include/save_master_gtid.inc > + > +--connection server_2 > +--source include/start_slave.inc > +--source include/sync_with_master_gtid.inc > +SET GLOBAL debug_dbug= @old_dbug; > +SELECT COUNT(*), SUM(a*100*b) FROM t7; > + > + > # Clean up. > --connection server_2 > --source include/stop_slave.inc > SET GLOBAL slave_parallel_threads=@old_parallel_threads; > +SET GLOBAL slave_parallel_mode=@old_parallel_mode; > --source include/start_slave.inc > SET DEBUG_SYNC= 'RESET'; > > --connection server_1 > DROP function foo; > -DROP TABLE t1,t2,t3,t4,t5,t6; > +DROP TABLE t1,t2,t3,t4,t5,t6,t7; > SET DEBUG_SYNC= 'RESET'; > > --source include/rpl_end.inc > diff --git a/sql/rpl_parallel.cc b/sql/rpl_parallel.cc > index b550315d69f..1aeb1257c4a 100644 > --- a/sql/rpl_parallel.cc > +++ b/sql/rpl_parallel.cc > @@ -1385,8 +1385,13 @@ handle_rpl_parallel_thread(void *arg) > err= dbug_simulate_tmp_error(rgi, thd);); > if (unlikely(err)) > { > + ulong max_retries= slave_trans_retries; > convert_kill_to_deadlock_error(rgi); > - if (has_temporary_error(thd) && slave_trans_retries > 0) > + DBUG_EXECUTE_IF("rpl_mdev31655_zero_retries", > + if ((rgi->current_gtid.seq_no % 1000) == 0) > + max_retries= 0; > + ); > + if (has_temporary_error(thd) && max_retries > 0) > err= retry_event_group(rgi, rpt, qev); > } > } > diff --git a/sql/sql_class.cc b/sql/sql_class.cc > index 8ed3f8a9c5e..e6ed7ca1cc4 100644 > --- a/sql/sql_class.cc > +++ b/sql/sql_class.cc > @@ -5247,6 +5247,49 @@ thd_need_ordering_with(const MYSQL_THD thd, const MYSQL_THD other_thd) > return 0; > } > > + > +/* > + If the storage engine detects a deadlock, and needs to choose a victim > + transaction to roll back, it can call this function to ask the upper > + server layer for which of two possible transactions is prefered to be > + aborted and rolled back. > + > + In parallel replication, if two transactions are running in parallel and > + one is fixed to commit before the other, then the one that commits later > + will be prefered as the victim - chosing the early transaction as a victim > + will not resolve the deadlock anyway, as the later transaction still needs > + to wait for the earlier to commit. > + > + The return value is -1 if the first transaction is prefered as a deadlock > + victim, 1 if the second transaction is prefered, or 0 for no preference (in > + which case the storage engine can make the choice as it prefers). > +*/ > +extern "C" int > +thd_deadlock_victim_preference(const MYSQL_THD thd1, const MYSQL_THD thd2) > +{ > + rpl_group_info *rgi1, *rgi2; > + > + if (!thd1 || !thd2) > + return 0; > + > + /* > + If the transactions are participating in the same replication domain in > + parallel replication, then request to select the one that will commit > + later (in the fixed commit order from the master) as the deadlock victim. > + */ > + rgi1= thd1->rgi_slave; > + rgi2= thd2->rgi_slave; > + if (rgi1 && rgi2 && > + rgi1->is_parallel_exec && > + rgi1->rli == rgi2->rli && > + rgi1->current_gtid.domain_id == rgi2->current_gtid.domain_id) > + return rgi1->gtid_sub_id < rgi2->gtid_sub_id ? 1 : -1; > + > + /* No preferences, let the storage engine decide. */ > + return 0; > +} > + > + > extern "C" int thd_non_transactional_update(const MYSQL_THD thd) > { > return(thd->transaction.all.modified_non_trans_table); > diff --git a/storage/innobase/lock/lock0lock.cc b/storage/innobase/lock/lock0lock.cc > index b1cf2152cd6..469d03eaa06 100644 > --- a/storage/innobase/lock/lock0lock.cc > +++ b/storage/innobase/lock/lock0lock.cc > @@ -49,6 +49,8 @@ Created 5/7/1996 Heikki Tuuri > #include <mysql/service_wsrep.h> > #endif /* WITH_WSREP */ > > +extern "C" int thd_deadlock_victim_preference(const MYSQL_THD thd1, const MYSQL_THD thd2); > + > /** Lock scheduling algorithm */ > ulong innodb_lock_schedule_algorithm; > > @@ -1538,6 +1540,16 @@ static bool has_higher_priority(lock_t *lock1, lock_t *lock2) > } else if (!lock_get_wait(lock2)) { > return false; > } > + // Ask the upper server layer if any of the two trx should be prefered. > + int preference = thd_deadlock_victim_preference(lock1->trx->mysql_thd, > + lock2->trx->mysql_thd); > + if (preference == -1) { > + // lock1 is preferred as a victim, so lock2 has higher priority > + return false; > + } else if (preference == 1) { > + // lock2 is preferred as a victim, so lock1 has higher priority > + return true; > + } > return lock1->trx->start_time_micro <= lock2->trx->start_time_micro; > } > > diff --git a/storage/innobase/trx/trx0trx.cc b/storage/innobase/trx/trx0trx.cc > index 7cd95878b0c..0771d764fb6 100644 > --- a/storage/innobase/trx/trx0trx.cc > +++ b/storage/innobase/trx/trx0trx.cc > @@ -52,6 +52,9 @@ Created 3/26/1996 Heikki Tuuri > #include <set> > #include <new> > > +extern "C" > +int thd_deadlock_victim_preference(const MYSQL_THD thd1, const MYSQL_THD thd2); > + > /** The bit pattern corresponding to TRX_ID_MAX */ > const byte trx_id_max_bytes[8] = { > 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff > @@ -1906,6 +1909,15 @@ trx_weight_ge( > { > ibool a_notrans_edit; > ibool b_notrans_edit; > + int pref; > + > + /* First ask the upper server layer if it has any preference for which > + to prefer as a deadlock victim. */ > + pref= thd_deadlock_victim_preference(a->mysql_thd, b->mysql_thd); > + if (pref < 0) > + return FALSE; > + else if (pref > 0) > + return TRUE; > > /* If mysql_thd is NULL for a transaction we assume that it has > not edited non-transactional tables. */ > _______________________________________________ > commits mailing list -- commits(a)lists.mariadb.org > To unsubscribe send an email to commits-leave(a)lists.mariadb.org

2 7

Re: 9c52eede2c8: MDEV-31646 Online alter applies binlog cache limit to cache writes
by Sergei Golubchik 15 Aug '23

15 Aug '23

Hi, Nikita, On Aug 14, Nikita Malyavin wrote: > revision-id: 9c52eede2c8 (mariadb-11.0.1-175-g9c52eede2c8) > parent(s): 209c894b431 > author: Nikita Malyavin > committer: Nikita Malyavin > timestamp: 2023-08-11 16:29:37 +0400 > message: > > MDEV-31646 Online alter applies binlog cache limit to cache writes > > 1. Make online disk writes unlimited, same as filesort does. > 2. Make proper error handling -- in 32-bit build IO_CACHE capacity limit is > 4GB, so it is quite possible to overfill there. > 3. Event_log::write_cache complicated with event reparsing, and as it was > proven by QA, contains some mistakes. Rewrite introbuce a simpler and much > faster version, not featuring reparsing and therefore copying a whole > buffer at once. This also disables checksums and crypto. > > As a result, online alter is untied of several binlog variables, which was > a second aim of this patch. > > diff --git a/mysql-test/main/alter_table_online_debug.result b/mysql-test/main/alter_table_online_debug.result > index 021267a15f5..3a615372ed5 100644 > --- a/mysql-test/main/alter_table_online_debug.result > +++ b/mysql-test/main/alter_table_online_debug.result > @@ -409,23 +409,23 @@ set debug_sync= 'now SIGNAL start_replication'; > set debug_sync= 'now WAIT_FOR applied'; > select stage, progress from INFORMATION_SCHEMA.PROCESSLIST where id = @con; > stage progress > -3 51.220 > +3 53.390 why is that? > set debug_sync= 'now SIGNAL proceed WAIT_FOR applied'; > select stage, progress from INFORMATION_SCHEMA.PROCESSLIST where id = @con; > stage progress > diff --git a/sql/log.cc b/sql/log.cc > index d9e81700463..8c57487d57e 100644 > --- a/sql/log.cc > +++ b/sql/log.cc > @@ -2286,13 +2286,21 @@ int binlog_log_row_online_alter(TABLE* table, const uchar *before_record, > MY_BITMAP *old_rpl_write_set= table->rpl_write_set; > table->rpl_write_set= &table->s->all_set; > > + table->online_alter_cache->store_prev_position(); > int error= (*log_func)(thd, table, table->s->online_alter_binlog, > - table->online_alter_cache, true, > + table->online_alter_cache, > + table->file->has_transactions_and_rollback(), > before_record, after_record); > > table->rpl_write_set= old_rpl_write_set; > > - return unlikely(error) ? HA_ERR_RBR_LOGGING_FAILED : 0; > + if (unlikely(error)) > + { > + table->online_alter_cache->restore_prev_position(); what is this for? > + return HA_ERR_RBR_LOGGING_FAILED; > + } > + > + return 0; > } > > static void > @@ -7748,7 +7763,7 @@ static int binlog_online_alter_end_trans(THD *thd, bool all, bool commit) > { > DBUG_ASSERT(cache.cache_log.type != READ_CACHE); > mysql_mutex_lock(binlog->get_log_lock()); > - error= binlog->write_cache(thd, &cache.cache_log); > + error= binlog->write_cache_raw(thd, &cache.cache_log); > mysql_mutex_unlock(binlog->get_log_lock()); > } > } Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: ca64ddcc709: MDEV-31646 Online alter applies binlog cache limit to cache writes
by Sergei Golubchik 10 Aug '23

10 Aug '23

Hi, Nikita, This is a review of all the three commits On Jul 20, Nikita Malyavin wrote: > revision-id: ca64ddcc709 (mariadb-11.0.1-144-gca64ddcc709) > parent(s): f34c7419cb8 > author: Nikita Malyavin > committer: Nikita Malyavin > timestamp: 2023-07-19 02:31:32 +0400 > message: > > MDEV-31646 Online alter applies binlog cache limit to cache writes > diff --git a/sql/log.cc b/sql/log.cc > index afd0643fe82..7466d63d325 100644 > --- a/sql/log.cc > +++ b/sql/log.cc > @@ -2291,8 +2294,14 @@ int binlog_log_row_online_alter(TABLE* table, const uchar *before_record, > before_record, after_record); > > table->rpl_write_set= old_rpl_write_set; > + thd->pop_internal_handler(); > + > + if (unlikely(error)) > + push_warning_printf(thd, Sql_condition::WARN_LEVEL_NOTE, ER_DISK_FULL, > + "Broken online alter log. " > + "ALTER TABLE will finish with error."); I don't think so. Why would a DML statement get a warning about the concurrently running ALTER TABLE? It didn't do anything wrong, I don't think there should be a warning here Like, someone is running a series of DML statements, or the same statement many times. At some point another connection runs ALTER TABLE and suddenly those perfectly good DML statements start spewing out warnings. ALTER ends and all is clear again - looks wrong to me > > - return unlikely(error) ? HA_ERR_RBR_LOGGING_FAILED : 0; > + return 0; > } > > static void > @@ -2592,7 +2601,7 @@ void Event_log::set_write_error(THD *thd, bool is_transactional) > DBUG_VOID_RETURN; > } > > -bool Event_log::check_write_error(THD *thd) > +bool MYSQL_BIN_LOG::check_write_error(THD *thd) What's the difference? it's a static method anyway, FWIW it could be outside of any class. What's the point of moving it from one class to another? > { > DBUG_ENTER("MYSQL_BIN_LOG::check_write_error"); > > @@ -3806,9 +3815,9 @@ bool MYSQL_BIN_LOG::open_index_file(const char *index_file_name_arg, > } > > > -bool Event_log::open(enum cache_type io_cache_type_arg) > +bool Event_log::open(enum cache_type io_cache_type_arg, size_t buffer_size) > { > - bool error= init_io_cache(&log_file, -1, LOG_BIN_IO_SIZE, io_cache_type_arg, > + bool error= init_io_cache(&log_file, -1, buffer_size, io_cache_type_arg, why? it's only used by online alter, where would you need a different buffer size? only in your DBUG_EXECUTE_IF? > 0, 0, MYF(MY_WME | MY_NABP | MY_WAIT_IF_FULL)); > > log_state= LOG_OPENED; > @@ -6544,7 +6562,8 @@ Event_log::flush_and_set_pending_rows_event(THD *thd, Rows_log_event* event, > if (writer.write(pending)) > { > set_write_error(thd, is_transactional); > - if (check_write_error(thd) && cache_data && > + if (dynamic_cast<MYSQL_BIN_LOG*>(this) && why? > + MYSQL_BIN_LOG::check_write_error(thd) && cache_data && > stmt_has_updated_non_trans_table(thd)) > cache_data->set_incident(); > delete pending; > @@ -7721,8 +7749,11 @@ static int binlog_online_alter_end_trans(THD *thd, bool all, bool commit) > { > DBUG_ASSERT(cache.cache_log.type != READ_CACHE); > mysql_mutex_lock(binlog->get_log_lock()); > - error= binlog->write_cache(thd, &cache.cache_log); > + error= binlog->write_cache_raw(thd, &cache.cache_log); > mysql_mutex_unlock(binlog->get_log_lock()); > + > + if (unlikely(error)) > + binlog->set_write_error(my_errno); shouldn't binlog->write_cache() call set_write_error() internally? why would the caller have to do it? > } > } > else if (!commit) // rollback > @@ -7736,14 +7767,14 @@ static int binlog_online_alter_end_trans(THD *thd, bool all, bool commit) > cache.store_prev_position(); > } > > + thd->pop_internal_handler(); > > if (error) > { > - my_error(ER_ERROR_ON_WRITE, MYF(ME_ERROR_LOG), > - binlog->get_name(), errno); > - binlog_online_alter_cleanup(thd->online_alter_cache_list, > - is_ending_transaction); > - DBUG_RETURN(error); > + push_warning_printf(thd, Sql_state_errno_level::WARN_LEVEL_WARN, > + ER_ERROR_ON_WRITE, ER_THD(thd, ER_ERROR_ON_WRITE), > + binlog->get_name(), errno); > + DBUG_ASSERT(binlog->get_write_error()); same comment as above about not showing ALTER problems in concurrent threads > } > } > > diff --git a/sql/log.h b/sql/log.h > index 83e323cb6fb..39512b01991 100644 > --- a/sql/log.h > +++ b/sql/log.h > @@ -444,17 +445,20 @@ class Event_log: public MYSQL_LOG > */ > class Cache_flip_event_log: public Event_log { > IO_CACHE alt_buf; > +IF_DBUG(public:,) yuck I don't see you using online_write_error or ref_count in DBUG macros > IO_CACHE *current, *alt; > std::atomic<uint> ref_count; > + std::atomic<int> online_write_error; > public: > > Cache_flip_event_log() : Event_log(), alt_buf{}, > - current(&log_file), alt(&alt_buf), ref_count(1) {} > - bool open(enum cache_type io_cache_type_arg) > + current(&log_file), alt(&alt_buf), ref_count(1), > + online_write_error(0) {} > + bool open(size_t buffer_size) looks like Cache_flip_event_log::open() doesn't need any arguments at all > { > log_file.dir= mysql_tmpdir; > alt_buf.dir= log_file.dir; > - bool res= Event_log::open(io_cache_type_arg); > + bool res= Event_log::open(WRITE_CACHE); > if (res) > return res; > > diff --git a/sql/log_event.cc b/sql/log_event.cc > index 10180a95d99..0c4c284a40f 100644 > --- a/sql/log_event.cc > +++ b/sql/log_event.cc > @@ -761,16 +761,21 @@ Log_event::Log_event(const uchar *buf, > > int Log_event::read_log_event(IO_CACHE* file, String* packet, > const Format_description_log_event *fdle, > - enum enum_binlog_checksum_alg checksum_alg_arg) > + enum enum_binlog_checksum_alg checksum_alg_arg, > + size_t max_allowed_packet) > { > ulong data_len; > char buf[LOG_EVENT_MINIMAL_HEADER_LEN]; > uchar ev_offset= packet->length(); > #if !defined(MYSQL_CLIENT) > - THD *thd=current_thd; > - ulong max_allowed_packet= thd ? thd->slave_thread ? slave_max_allowed_packet > - : thd->variables.max_allowed_packet > - : ~(uint)0; > + if (max_allowed_packet == 0) may be better pass THD as an argument? or even correct value of max_allowed_packet. As ulong max_allowed_packet(THD *thd) { return thd ? ... : ~(uint)0; } on the second thought, do you need to ignore max_allowed_packet here for online ALTER? Is there no event size limit when the event is written into binlog? > + { > + THD *thd=current_thd; > + max_allowed_packet= thd ? thd->slave_thread > + ? slave_max_allowed_packet > + : thd->variables.max_allowed_packet > + : ~(uint)0; > + } > #endif > DBUG_ENTER("Log_event::read_log_event(IO_CACHE*,String*...)"); > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 7

Re: 88a77860a8b: MDEV-31804 Assertion `thd->m_transaction_psi == __null' fails
by Sergei Golubchik 09 Aug '23

09 Aug '23

Hi, Nikita, On Aug 08, Nikita Malyavin wrote: > revision-id: 88a77860a8b (mariadb-11.0.1-194-g88a77860a8b) > parent(s): 73a677f0306 > author: Nikita Malyavin > committer: Nikita Malyavin > timestamp: 2023-08-07 22:45:11 +0400 > message: > > MDEV-31804 Assertion `thd->m_transaction_psi == __null' fails > > ... upon replicating online ALTER > > When an online event is applied and slave_exec_mode is idempotent, > Write_rows_log_event::do_before_row_operations had reset > thd->lex->sql_command to SQLCOM_REPLACE. > > This led to that a statement was detected as a row-type during > binlogging, and was logged as not standalone. > > So the corresponding Gtid_log_event, when applied on replica, did not > exit early and created a new PSI transaction. Hence the difference > with non-online ALTER. Nice find. I vaguely believe I've seen in one of the commits I've reviewed a change that makes every row event set thd->lex->sql_command to the corresponding SQLCOM_INSERT/UPDATE/DELETE. But be it was in an unpushed commit. Anyway, I think it's a bit risky to assume that row events never change thd->lex->sql_command (and never will). Perhaps you should simply restore it after the applying part? Like --- a/sql/sql_table.cc +++ b/sql/sql_table.cc @@ -12181,6 +12181,7 @@ copy_data_between_tables(THD *thd, TABLE *from, TABLE *> if (error) from->s->tdc->flush_unused(1); // to free the binlog to->pos_in_table_list= NULL; // Safety + thd->lex->sql_command= SQLCOM_ALTER_TABLE; } else if (online) // error was on copy stage { Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 5

Re: 63c93f89af9: MDEV-31804 Assertion `thd->m_transaction_psi == __null' fails
by Sergei Golubchik 07 Aug '23

07 Aug '23

Hi, Nikita, On Aug 07, Nikita Malyavin wrote: > revision-id: 63c93f89af9 (mariadb-11.0.1-192-g63c93f89af9) > parent(s): a75ba4caa51 > author: Nikita Malyavin > committer: Nikita Malyavin > timestamp: 2023-08-05 20:17:41 +0400 > message: > > MDEV-31804 Assertion `thd->m_transaction_psi == __null' fails ... > index 1c8d56cafd1..66b2b1ee4cc 100644 > --- a/sql/handler.cc > +++ b/sql/handler.cc > @@ -1459,7 +1459,7 @@ void trans_register_ha(THD *thd, bool all, handlerton *ht_arg, ulonglong trxid) > Do not register transactions in which binary log is the only participating > transactional storage engine. > */ > - if (thd->m_transaction_psi == NULL && ht_arg->db_type != DB_TYPE_BINLOG) > + if (thd->m_transaction_psi == NULL && ht_arg != binlog_hton) it would've been better to fix it in 10.4, but if it doesn't show before online alter then ok > { > thd->m_transaction_psi= MYSQL_START_TRANSACTION(&thd->m_transaction_state, > thd->get_xid(), trxid, thd->tx_isolation, thd->tx_read_only, > diff --git a/sql/sql_table.cc b/sql/sql_table.cc > index 88302096272..944807ea422 100644 > --- a/sql/sql_table.cc > +++ b/sql/sql_table.cc > @@ -65,6 +65,7 @@ > #include "rpl_rli.h" > #include "log.h" > #include "sql_debug.h" > +#include "mysql/psi/mysql_transaction.h" > > #ifdef _WIN32 > #include <io.h> > @@ -11653,6 +11654,9 @@ bool mysql_trans_commit_alter_copy_data(THD *thd) > > DEBUG_SYNC(thd, "alter_table_copy_trans_commit"); > > + MYSQL_COMMIT_TRANSACTION(thd->m_transaction_psi); > + thd->m_transaction_psi= NULL; I don't understand it. ha_enable_transaction() commits the transaction it calls ha_commit_trans(). And ha_commit_trans() does the above psi manipulations. Why do you need to do that explicitly? Also, as a cleanup, I suspect, you can remove explicit commit that follows ha_enable_transaction() in mysql_trans_commit_alter_copy_data(). Seems to be redundant. > if (ha_enable_transaction(thd, TRUE)) > DBUG_RETURN(TRUE); Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 2

Re: b508107c7a3: MDEV-31631 Adding auto-increment to table with history online misbehaves
by Sergei Golubchik 04 Aug '23

04 Aug '23

Hi, Nikita, Looks good. One comment below: On Aug 04, Nikita Malyavin wrote: > revision-id: b508107c7a3 (mariadb-11.0.1-189-gb508107c7a3) > parent(s): e9a75c98781 > author: Nikita Malyavin > committer: Nikita Malyavin > timestamp: 2023-07-29 00:13:48 +0400 > message: > > MDEV-31631 Adding auto-increment to table with history online misbehaves > > diff --git a/sql/sql_table.cc b/sql/sql_table.cc > index eabee13e8d3..3a70ed672a6 100644 > --- a/sql/sql_table.cc > +++ b/sql/sql_table.cc > @@ -9953,6 +9953,37 @@ const char *online_alter_check_supported(const THD *thd, > } > } > > + for (auto &c: alter_info->create_list) > + { > + *online= c.field || !(c.flags & AUTO_INCREMENT_FLAG); > + if (!*online) > + return "ADD COLUMN ... AUTO_INCREMENT"; > + > + auto *def= c.default_value; > + *online= !(def && def->flags & VCOL_NEXTVAL > + // either it's a new field, or a NULL -> NOT NULL change > + && (!c.field || (!(c.field->flags & NOT_NULL_FLAG) > + && (c.flags & NOT_NULL_FLAG)))); > + if (!*online) > + { > + const char *fmt= ER_THD(thd, ER_GENERATED_COLUMN_FUNCTION_IS_NOT_ALLOWED); > + > + Item::vcol_func_processor_result res; > + IF_DBUG(bool ret=,) > + def->expr->walk(&Item::check_vcol_func_processor, 0, &res); I don't think you need to walk here, res.name is always "nextval()" isn't it? > + DBUG_ASSERT(!ret); > + DBUG_ASSERT(res.errors & VCOL_NEXTVAL); > + > + LEX_CSTRING dflt{STRING_WITH_LEN("DEFAULT")}; > + size_t len= strlen(fmt) + strlen(res.name) + c.field_name.length > + + dflt.length; > + char *resp= (char*)thd->alloc(len); > + // expression %s cannot be used in the %s clause of %`s > + my_snprintf(resp, len, fmt, res.name, dflt.str, c.field_name.str); > + return resp; > + } > + } Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Running msqld with thread sanitizer
by Thejaka Amila J Kanewala 04 Aug '23

04 Aug '23

Hello, I am trying to build and run a thread sanitized mysqld binary. In the CMake configuration, I see the WITH_ASAN option, however, I do not see an option to build mysqld with the thread sanitizer. After making some custom configurations, I was able to build a mysqld binary with the thread sanitizer (by linking static thread sanitizer), however, when trying to run it, I got the following error: "ThreadSanitizer: sync allocator overflow (65536*1024). Dying." This error seems to be coming from the tsan library and seems this error is caused by too many synchronizations [1]. Reducing the number of threads also didn't help me with this error. So my questions to you experts: 1. Do you run mysqld with the thread sanitizer ? 2. If yes, could you please share the details. (e.g., CMake configurations, compiler/linker flags etc.) 3. If not, any suggestions to overcome the above error ? [1] https://groups.google.com/g/thread-sanitizer/c/AX5twD_IPP4 Thank you. Best Regards, Thejaka Amila Kanewala, PhD https://github.com/thejkane/agm

2 3

Re: 44b4a3e64fa: MDEV-31777 ER_GET_ERRNO upon online alter on CONNECT table
by Sergei Golubchik 03 Aug '23

03 Aug '23

Hi, Nikita, Generally, please, prefer table flags over new virtual function. E.g. *online= new_table->file->ha_table_flags() & HA_CAN_ONLINE_ALTER; But in this particular case there's no reason for a engine to not support online alter, it's just CONNECT being very weird. Like you wrote, it doesn't support REPLACE either and we didn't introduce a flag for that. This should be solved inside CONNECT. E.g it can throw an error just like it does for REPLACE. It can upgrade the lock: --- a/storage/connect/ha_connect.cc +++ b/storage/connect/ha_connect.cc @@ -5125,10 +5125,12 @@ @see get_lock_data() in lock.cc */ -THR_LOCK_DATA **ha_connect::store_lock(THD *, +THR_LOCK_DATA **ha_connect::store_lock(THD *thd, THR_LOCK_DATA **to, enum thr_lock_type lock_type) { + if (thd_sql_command(thd) == SQLCOM_ALTER_TABLE && lock_type == TL_READ) + lock_type= TL_READ_NO_INSERT; if (lock_type != TL_IGNORE && lock.type == TL_UNLOCK) lock.type=lock_type; *to++= &lock; or it can implement reading and inserting to the same table :) Either way, I don't think we need to extend the storage engine API for this specific case. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org On Aug 03, Nikita Malyavin wrote: > revision-id: 44b4a3e64fa (mariadb-11.0.1-190-g44b4a3e64fa) > parent(s): b508107c7a3 > author: Nikita Malyavin > committer: Nikita Malyavin > timestamp: 2023-07-29 00:13:48 +0400 > message: > > MDEV-31777 ER_GET_ERRNO upon online alter on CONNECT table > > Forbid Online for CONNECT. > > diff --git a/sql/handler.h b/sql/handler.h > index 480c79017ab..4a1363d2e93 100644 > --- a/sql/handler.h > +++ b/sql/handler.h > @@ -4911,6 +4911,8 @@ class handler :public Sql_alloc > bool commit); > > > + virtual bool is_online_alter_supported() const { return true; } > + > protected:

1 0

Re: dbe5c20b755: MDEV-31482: Lock wait timeout with INSERT-SELECT, autoinc, and statement-based replication
by Kristian Nielsen 03 Aug '23

03 Aug '23

Hi Marko, I'd like to ask your opinion/review of the below patch for MDEV-31482: https://github.com/MariaDB/server/commit/dbe5c20b755b87f67d87990c3baabc8666… https://jira.mariadb.org/browse/MDEV-31482 As you know, in-order parallel replication has InnoDB report lock waits. If T1 must replicate before T2, and InnoDB reports that T1 is waiting for a lock held by T2, then T2 will be killed (and retried later) so that T1 can replicate. As an optimisation, auto-increment lock waits where _not_ reported, but this turns out to be a bug. This patch just removes the exception so that such waits are reported like other waits. My only concern is the possible performance impact of this. I'm hoping it will not be very big, because in many cases the auto-increment lock is not taken for replicated transactions? My understanding is: - It is not taken for INSERT INTO t VALUES (...) (and maybe not for multi-row "INSERT ... VALUES (...), (...), ..." ?) - SELECT ... INSERT is not safe for statement-based replication, so will be logged as row in MIXED mode and give a warning in STATEMENT. But I wanted to hear you if you know any case and/or MariaDB version where a significant amount of auto-increment lock waits could be expected in parallel replication? And if so, maybe this fix should be pushed only to later MariaDB versions? - Kristian. Kristian Nielsen <knielsen(a)knielsen-hq.org> writes: > revision-id: dbe5c20b755b87f67d87990c3baabc866667e41b (mariadb-10.4.30-2-gdbe5c20b755) > parent(s): a6114df595eeb7aeed4b050c9a3f4640c4320b5f > author: Kristian Nielsen > committer: Kristian Nielsen > timestamp: 2023-07-09 16:45:47 +0200 > message: > > MDEV-31482: Lock wait timeout with INSERT-SELECT, autoinc, and statement-based replication > > Remove the exception that InnoDB does not report auto-increment locks waits > to the parallel replication. > > There was an assumption that these waits could not cause conflicts with > in-order parallel replication and thus need not be reported. However, this > assumption is wrong and it is possible to get conflicts that lead to hangs > for the duration of --innodb-lock-wait-timeout. This can be seen with three > transactions: > > 1. T1 is waiting for T3 on an autoinc lock > 2. T2 is waiting for T1 to commit > 3. T3 is waiting on a normal row lock held by T2 > > Here, T3 needs to be deadlock killed on the wait by T1. > > Signed-off-by: Kristian Nielsen <knielsen(a)knielsen-hq.org> > > --- > mysql-test/suite/rpl/r/rpl_parallel_autoinc.result | 95 ++++++++++++++ > mysql-test/suite/rpl/t/rpl_parallel_autoinc.test | 140 +++++++++++++++++++++ > sql/sql_class.cc | 6 - > storage/innobase/lock/lock0lock.cc | 8 +- > 4 files changed, 236 insertions(+), 13 deletions(-) > > diff --git a/mysql-test/suite/rpl/r/rpl_parallel_autoinc.result b/mysql-test/suite/rpl/r/rpl_parallel_autoinc.result > new file mode 100644 > index 00000000000..c1829bafa1a > --- /dev/null > +++ b/mysql-test/suite/rpl/r/rpl_parallel_autoinc.result > @@ -0,0 +1,95 @@ > +include/master-slave.inc > +[connection master] > +MDEV-31482: Lock wait timeout with INSERT-SELECT, autoinc, and statement-based replication > +include/rpl_connect.inc [creating slave2] > +include/rpl_connect.inc [creating slave3] > +connection master; > +ALTER TABLE mysql.gtid_slave_pos ENGINE=InnoDB; > +CREATE TABLE t1 (a INT PRIMARY KEY AUTO_INCREMENT, b INT, c INT, INDEX (c)) ENGINE=InnoDB; > +INSERT INTO t1 (b,c) VALUES (0, 1), (0, 1), (0, 2), (0,3), (0, 5), (0, 7), (0, 8); > +CREATE TABLE t2 (a INT PRIMARY KEY, b INT) ENGINE=InnoDB; > +INSERT INTO t2 VALUES (10,1), (20,2), (30,3), (40,4), (50,5); > +CREATE TABLE t3 (a VARCHAR(20) PRIMARY KEY, b INT) ENGINE=InnoDB; > +INSERT INTO t3 VALUES ('row for T1', 0), ('row for T2', 0), ('row for T3', 0); > +include/save_master_gtid.inc > +connection slave; > +include/sync_with_master_gtid.inc > +include/stop_slave.inc > +set @@global.slave_parallel_threads= 3; > +set @@global.slave_parallel_mode= OPTIMISTIC; > +set @@global.innodb_lock_wait_timeout= 20; > +connection master; > +BEGIN; > +UPDATE t3 SET b=b+1 where a="row for T1"; > +INSERT INTO t1(b, c) SELECT 1, t2.b FROM t2 WHERE a=10; > +Warnings: > +Note 1592 Unsafe statement written to the binary log using statement format since BINLOG_FORMAT = STATEMENT. Statements writing to a table with an auto-increment column after selecting from another table are unsafe because the order in which rows are retrieved determines what (if any) rows will be written. This order cannot be predicted and may differ on master and the slave > +COMMIT; > +DELETE FROM t1 WHERE c >= 4 and c < 6; > +BEGIN; > +UPDATE t3 SET b=b+1 where a="row for T3"; > +INSERT INTO t1(b, c) SELECT 3, t2.b FROM t2 WHERE a >= 20 AND a <= 40; > +Warnings: > +Note 1592 Unsafe statement written to the binary log using statement format since BINLOG_FORMAT = STATEMENT. Statements writing to a table with an auto-increment column after selecting from another table are unsafe because the order in which rows are retrieved determines what (if any) rows will be written. This order cannot be predicted and may differ on master and the slave > +COMMIT; > +include/save_master_gtid.inc > +connection slave1; > +BEGIN; > +SELECT * FROM t3 WHERE a="row for T1" FOR UPDATE; > +a b > +row for T1 0 > +connection slave2; > +BEGIN; > +SELECT * FROM t3 WHERE a="row for T3" FOR UPDATE; > +a b > +row for T3 0 > +connection slave3; > +BEGIN; > +DELETE FROM t2 WHERE a=30; > +connection slave; > +include/start_slave.inc > +connection slave2; > +ROLLBACK; > +connection slave1; > +ROLLBACK; > +connection slave3; > +ROLLBACK; > +connection slave; > +include/sync_with_master_gtid.inc > +SELECT * FROM t1 ORDER BY a; > +a b c > +1 0 1 > +2 0 1 > +3 0 2 > +4 0 3 > +6 0 7 > +7 0 8 > +8 1 1 > +9 3 2 > +10 3 3 > +11 3 4 > +SELECT * FROM t2 ORDER BY a; > +a b > +10 1 > +20 2 > +30 3 > +40 4 > +50 5 > +SELECT * FROM t3 ORDER BY a; > +a b > +row for T1 1 > +row for T2 0 > +row for T3 1 > +connection master; > +CALL mtr.add_suppression("Unsafe statement written to the binary log using statement format"); > +DROP TABLE t1, t2, t3; > +connection slave; > +include/stop_slave.inc > +SET @@global.slave_parallel_threads= 0; > +SET @@global.slave_parallel_mode= conservative; > +SET @@global.innodb_lock_wait_timeout= 50; > +include/start_slave.inc > +SELECT @@GLOBAL.innodb_autoinc_lock_mode; > +@@GLOBAL.innodb_autoinc_lock_mode > +1 > +include/rpl_end.inc > diff --git a/mysql-test/suite/rpl/t/rpl_parallel_autoinc.test b/mysql-test/suite/rpl/t/rpl_parallel_autoinc.test > new file mode 100644 > index 00000000000..0e96b4dfb80 > --- /dev/null > +++ b/mysql-test/suite/rpl/t/rpl_parallel_autoinc.test > @@ -0,0 +1,140 @@ > +--source include/have_binlog_format_statement.inc > +--source include/have_innodb.inc > +--source include/master-slave.inc > + > +--echo MDEV-31482: Lock wait timeout with INSERT-SELECT, autoinc, and statement-based replication > + > +# The scenario is transactions T1, T2, T3: > +# > +# T1 is waiting for T3 on an autoinc lock > +# T2 is waiting for T1 to commit > +# T3 is waiting on a normal row lock held by T2 > +# > +# This caused a hang until innodb_lock_wait_timeout, because autoinc > +# locks were not reported to the in-order parallel replication, so T3 > +# was not deadlock killed. > + > +--let $lock_wait_timeout=20 > + > +--let $rpl_connection_name= slave2 > +--let $rpl_server_number= 2 > +--source include/rpl_connect.inc > + > +--let $rpl_connection_name= slave3 > +--let $rpl_server_number= 2 > +--source include/rpl_connect.inc > + > +--connection master > +ALTER TABLE mysql.gtid_slave_pos ENGINE=InnoDB; > + > +# A table as destination for INSERT-SELECT > +CREATE TABLE t1 (a INT PRIMARY KEY AUTO_INCREMENT, b INT, c INT, INDEX (c)) ENGINE=InnoDB; > +INSERT INTO t1 (b,c) VALUES (0, 1), (0, 1), (0, 2), (0,3), (0, 5), (0, 7), (0, 8); > + > +# A table as source for INSERT-SELECT. > +CREATE TABLE t2 (a INT PRIMARY KEY, b INT) ENGINE=InnoDB; > +INSERT INTO t2 VALUES (10,1), (20,2), (30,3), (40,4), (50,5); > + > +# A table to help order slave worker threads to setup the desired scenario. > +CREATE TABLE t3 (a VARCHAR(20) PRIMARY KEY, b INT) ENGINE=InnoDB; > +INSERT INTO t3 VALUES ('row for T1', 0), ('row for T2', 0), ('row for T3', 0); > +--source include/save_master_gtid.inc > + > +--connection slave > +--source include/sync_with_master_gtid.inc > +--source include/stop_slave.inc > +--let $save_innodb_lock_wait_timeout= `SELECT @@global.innodb_lock_wait_timeout` > +--let $save_slave_parallel_threads= `SELECT @@global.slave_parallel_threads` > +--let $save_slave_parallel_mode= `SELECT @@global.slave_parallel_mode` > +set @@global.slave_parallel_threads= 3; > +set @@global.slave_parallel_mode= OPTIMISTIC; > +eval set @@global.innodb_lock_wait_timeout= $lock_wait_timeout; > + > +--connection master > +# Transaction T1. > +BEGIN; > +UPDATE t3 SET b=b+1 where a="row for T1"; > +INSERT INTO t1(b, c) SELECT 1, t2.b FROM t2 WHERE a=10; > +COMMIT; > + > +# Transaction T2. > +DELETE FROM t1 WHERE c >= 4 and c < 6; > + > +# Transaction T3. > +BEGIN; > +UPDATE t3 SET b=b+1 where a="row for T3"; > +INSERT INTO t1(b, c) SELECT 3, t2.b FROM t2 WHERE a >= 20 AND a <= 40; > +COMMIT; > + > +--source include/save_master_gtid.inc > + > +--connection slave1 > +# Temporarily block T1 to create the scheduling that triggers the bug. > +BEGIN; > +SELECT * FROM t3 WHERE a="row for T1" FOR UPDATE; > + > +--connection slave2 > +# Temporarily block T3 from starting (so T2 can reach commit). > +BEGIN; > +SELECT * FROM t3 WHERE a="row for T3" FOR UPDATE; > + > +--connection slave3 > +# This critical step blocks T3 after it has inserted its first row, > +# and thus taken the auto-increment lock, but before it has reached > +# the point where it gets a row lock wait on T2. Even though > +# auto-increment lock waits were not reported due to the bug, > +# transitive lock waits (T1 waits on autoinc of T3 which waits on row > +# on T2) _were_ reported as T1 waiting on T2, and thus a deadlock kill > +# happened and the bug was not triggered. > +BEGIN; > +DELETE FROM t2 WHERE a=30; > + > +--connection slave > +--source include/start_slave.inc > + > +# First let T2 complete until it is waiting for T1 to commit. > +--let $wait_condition= SELECT count(*)=1 FROM information_schema.processlist WHERE state='Waiting for prior transaction to commit' and command LIKE 'Slave_worker'; > +--source include/wait_condition.inc > + > +# Then let T3 reach the point where it has obtained the autoinc lock, > +# but it is not yet waiting for a row lock held by T2. > +--connection slave2 > +ROLLBACK; > +--let $wait_condition= SELECT count(*)=1 FROM information_schema.processlist WHERE state='Sending data' and info LIKE 'INSERT INTO t1(b, c) SELECT 3, t2.b%' and time_ms > 500 and command LIKE 'Slave_worker'; > +--source include/wait_condition.inc > + > +# Now let T1 continue, while T3 is holding the autoinc lock but before > +# it is waiting for T2. Wait a short while to give the hang a chance to > +# happen; T1 needs to get to request the autoinc lock before we let T3 > +# continue. (There's a small chance the sleep will be too small, which will > +# let the test occasionally pass on non-fixed server). > +--connection slave1 > +ROLLBACK; > +--sleep 0.5 > + > +# Now let T3 continue; the bug was that this lead to an undetected > +# deadlock that remained until innodb lock wait timeout. > +--connection slave3 > +ROLLBACK; > + > +--connection slave > +--let $slave_timeout= `SELECT $lock_wait_timeout/2` > +--source include/sync_with_master_gtid.inc > +--let $slave_timeout= > +SELECT * FROM t1 ORDER BY a; > +SELECT * FROM t2 ORDER BY a; > +SELECT * FROM t3 ORDER BY a; > + > +# Cleanup. > +--connection master > +CALL mtr.add_suppression("Unsafe statement written to the binary log using statement format"); > +DROP TABLE t1, t2, t3; > + > +--connection slave > +--source include/stop_slave.inc > +eval SET @@global.slave_parallel_threads= $save_slave_parallel_threads; > +eval SET @@global.slave_parallel_mode= $save_slave_parallel_mode; > +eval SET @@global.innodb_lock_wait_timeout= $save_innodb_lock_wait_timeout; > +--source include/start_slave.inc > +SELECT @@GLOBAL.innodb_autoinc_lock_mode; > +--source include/rpl_end.inc > diff --git a/sql/sql_class.cc b/sql/sql_class.cc > index 73bb654080a..8ed3f8a9c5e 100644 > --- a/sql/sql_class.cc > +++ b/sql/sql_class.cc > @@ -5119,12 +5119,6 @@ thd_need_wait_reports(const MYSQL_THD thd) > deadlock with the pre-determined commit order, we kill the later > transaction, and later re-try it, to resolve the deadlock. > > - This call need only receive reports about waits for locks that will remain > - until the holding transaction commits. InnoDB auto-increment locks, > - for example, are released earlier, and so need not be reported. (Such false > - positives are not harmful, but could lead to unnecessary kill and retry, so > - best avoided). > - > Returns 1 if the OTHER_THD will be killed to resolve deadlock, 0 if not. The > actual kill will happen later, asynchronously from another thread. The > caller does not need to take any actions on the return value if the > diff --git a/storage/innobase/lock/lock0lock.cc b/storage/innobase/lock/lock0lock.cc > index 26388ad95e2..b1cf2152cd6 100644 > --- a/storage/innobase/lock/lock0lock.cc > +++ b/storage/innobase/lock/lock0lock.cc > @@ -6872,13 +6872,7 @@ DeadlockChecker::search() > return m_start; > } > > - /* We do not need to report autoinc locks to the upper > - layer. These locks are released before commit, so they > - can not cause deadlocks with binlog-fixed commit > - order. */ > - if (m_report_waiters > - && (lock_get_type_low(lock) != LOCK_TABLE > - || lock_get_mode(lock) != LOCK_AUTO_INC)) { > + if (m_report_waiters) { > thd_rpl_deadlock_check(m_start->mysql_thd, > lock->trx->mysql_thd); > }

2 1

Proposal: mariadbd should automatically upgrade system tables and data directory on startup
by Otto Kekäläinen 03 Aug '23

03 Aug '23

Hi! I frequently see users running into various problems after upgrading MariaDB and starting the upgraded mariadbd server: - Server might refuse to start, asking users to manually run mariadb-upgrade - Server might start, but misbehave as on-disk data format for regular tables or the system tables has not updated - On systems where mariadb-upgrade runs automatically on MariaDB service startup there might be unnecessary delay if mariadb-upgrade is run multiple times - If running mariadb-upgrade fails or is slow, users might end up in a situation where they re-restart the server and have two mariadb-upgrade scripts running and blocking each other - In some cases user might attempt to run mariadb-upgrade but it fails it the MariaDB server was not running, forcing user to figure out how to manually start the server for upgrades - Users might end up starting server in a mode that does not accept connections with the intent to only run mariadb-upgrade, but then don't understand they need to restart the server to have it serve connections again - Users planning upgrades get confused because because reading about Fedora based distros the DBA is requires to manually run mariadb-upgrade themselves, and reading about Debian based distros DBAs are not required to do this as it is automatic, but they might end up trying to run it manually anyway because of prior experience with Fedora Some of the above issues can be remediated by running 'mariadb_upgrade --check-if-upgrade-is-needed' before actually running mariadb_upgrade in service scripts, but it does not work if server is offline (https://jira.mariadb.org/browse/MDEV-27636) Making users aware that their database is running but mariadb-upgrade has not run can be done via https://github.com/MariaDB/server/pull/2464. However this are just duct tape to the original problem. All of the above could be permanently solved if mariadbd simply did the upgrade by itself automatically. Can we please have this? Can we agree that this would be a desired feature in next MariaDB 11.x version, and if somebody implements this, it would be timely reviewed and merged? There is already design outlined at https://jira.mariadb.org/browse/MDEV-30499

8 19

Problem Upgrading Database
by jashley＠deadacated.com 25 Jul '23

25 Jul '23

Greetings, MariaDB 10.11.4 on Server 2022 Upgrading from 10.11.4 to 11.0.2 or 11.1.1 fails on Phase 5/10: Waiting for startup to complete. Is this a known problem? Thank you, mux

3 2

Re: 64ac6ed3e35: MDEV-31742 incorrect examined rows in case of stored function usage
by Sergei Golubchik 21 Jul '23

21 Jul '23

Hi, Oleksandr, On Jul 21, Oleksandr Byelkin wrote: > revision-id: 64ac6ed3e35 (mariadb-10.4.30-31-g64ac6ed3e35) > parent(s): 55a27bac80c > author: Oleksandr Byelkin > committer: Oleksandr Byelkin > timestamp: 2023-07-19 16:14:39 +0200 > message: > > MDEV-31742 incorrect examined rows in case of stored function usage > > The counter is global so we do not need add backup to it > if we do not zero it after taking the backup. > > diff --git a/sql/sql_class.cc b/sql/sql_class.cc > index c9e86b6142c..7911c78b6e0 100644 > --- a/sql/sql_class.cc > +++ b/sql/sql_class.cc > @@ -5568,7 +5568,6 @@ void THD::restore_sub_statement_state(Sub_statement_state *backup) > The following is added to the old values as we are interested in the > total complexity of the query > */ > - inc_examined_row_count(backup->examined_row_count); may be backup->examined_row_count should be simply removed? Sub_statement_state::examined_row_count, I mean. Not saved, not restored? There's also void THD::add_slow_query_state(Sub_statement_state *backup) { affected_rows+= backup->affected_rows; bytes_sent_old= backup->bytes_sent_old; m_examined_row_count+= backup->examined_row_count; it's not even using inc_examined_row_count(). Also examined_row_count is mentioned in comments in sql_class.cc, if you'll be removing it from Sub_statement_state, don't forget to fix comments to match. > cuted_fields+= backup->cuted_fields; > DBUG_VOID_RETURN; > } Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 2

Re: 08403d85b00: MDEV-31677 Assertion failed upon online ALTER with binlog_row_image=NOBLOB
by Sergei Golubchik 21 Jul '23

21 Jul '23

Hi, Nikita, ok to push after making few small changes, see inline: On Jul 20, Nikita Malyavin wrote: > revision-id: 08403d85b00 (mariadb-11.0.1-147-g08403d85b00) > parent(s): a31b2a52119 > author: Nikita Malyavin > committer: Nikita Malyavin > timestamp: 2023-07-19 19:08:51 +0400 > message: > > MDEV-31677 Assertion failed upon online ALTER with binlog_row_image=NOBLOB > > Make binlog_prepare_row_images accept image type as an argument. > > diff --git a/sql/sql_class.cc b/sql/sql_class.cc > index d4ad0efbf3a..bb2453af6a0 100644 > --- a/sql/sql_class.cc > +++ b/sql/sql_class.cc > @@ -7070,7 +7071,7 @@ int THD::binlog_update_row(TABLE* table, Event_log *bin_log, > not needed for binlogging. This is done according to the: > binlog-row-image option. the comment is outdated, should say This is done according to the row_image argument > */ > - binlog_prepare_row_images(table); > + binlog_prepare_row_images(table, row_image); > > size_t const before_maxlen= max_row_length(table, table->read_set, > before_record); > @@ -7194,7 +7195,7 @@ void THD::binlog_prepare_row_images(TABLE *table) > and the handler involved supports this. > */ > if (table->s->primary_key < MAX_KEY && > - (thd->variables.binlog_row_image < BINLOG_ROW_IMAGE_FULL) && > + row_image < BINLOG_ROW_IMAGE_FULL && > !ha_check_storage_engine_flag(table->s->db_type(), > HTON_NO_BINLOG_ROW_OPT)) > { please also repace thd->variables.binlog_row_image in the switch() below. doesn't make any practical difference, just for clarity Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Debugging MDEV-28776, the rare rpl.rpl_mark_optimize_tbl_ddl test failure
by Kristian Nielsen 21 Jul '23

21 Jul '23

Hi Brandon, The test failure in https://jira.mariadb.org/browse/MDEV-28776 (the one with 10 failed retries after deadlock, not the freebsd failures) seems like it could be something serious. But it has been very difficult to track down. I believe you have tried different things, and I also tried hard to reproduce it, so far without success. But it's still failing occasionally in actual buildbot runs. What do you think about pushing the below to the different branches? It makes the test case log InnoDB information about deadlocks encountered. This way, when it fails the next time in buildbot we can see 1) what kind of InnoDB deadlock is triggering the problem, if any; and 2) if it actually gets a deadlock inside InnoDB 10 times in a row, or if it is something else. - Kristian. ----------------------------------------------------------------------- Author: Kristian Nielsen <knielsen(a)knielsen-hq.org> Date: Sun Jul 9 15:18:03 2023 +0200 MDEV-28776: rpl.rpl_mark_optimize_tbl_ddl fails with timeout on sync_with_master This commit just extends the testcase to include some more information when the test failure occurs. The failure has so far only been reproducible in buildbot runs. Signed-off-by: Kristian Nielsen <knielsen(a)knielsen-hq.org> ----------------------------------------------------------------------- diff --git a/mysql-test/suite/rpl/t/rpl_mark_optimize_tbl_ddl.opt b/mysql-test/suite/rpl/t/rpl_mark_optimize_tbl_ddl.opt new file mode 100644 index 00000000000..3c53d5257b4 --- /dev/null +++ b/mysql-test/suite/rpl/t/rpl_mark_optimize_tbl_ddl.opt @@ -0,0 +1 @@ +--innodb-print-all-deadlocks=1

3 6

Re: 534031b77e0: MDEV-31400 Simple plugin dependency resolution
by Sergei Golubchik 20 Jul '23

20 Jul '23

Hi, Yuchen, Thanks. Conceptually all good, the approach is correct. See comments below - about the state and about user visible warnings, nothing major. On Jul 18, Yuchen Pei wrote: > revision-id: 534031b77e0 (mariadb-10.9.5-17-g534031b77e0) > parent(s): d8997f875e2 > author: Yuchen Pei > committer: Yuchen Pei > timestamp: 2023-07-03 17:54:56 +1000 > message: > > MDEV-31400 Simple plugin dependency resolution > > diff --git a/sql/sql_plugin.cc b/sql/sql_plugin.cc > index 5a077a934ac..15717352040 100644 > --- a/sql/sql_plugin.cc > +++ b/sql/sql_plugin.cc > @@ -1435,6 +1435,47 @@ void plugin_unlock_list(THD *thd, plugin_ref *list, size_t count) > DBUG_VOID_RETURN; > } > > +static int plugin_do_initialize(struct st_plugin_int *plugin, uint &state) > +{ > + DBUG_ENTER("plugin_do_initialize"); > + mysql_mutex_assert_not_owner(&LOCK_plugin); > + plugin_type_init init= plugin_type_initialize[plugin->plugin->type]; > + if (!init) > + init= (plugin_type_init) plugin->plugin->init; > + if (init) > + if (int ret= init(plugin)) > + { > + /* Plugin init failed but requested a retry if possible */ > + if (unlikely(ret== HA_ERR_RETRY_INIT)) > + sql_print_warning("Plugin '%s' registration as a %s failed.", > + plugin->name.str, plugin_type_names[plugin->plugin->type].str); > + else > + sql_print_error("Plugin '%s' registration as a %s failed.", > + plugin->name.str, plugin_type_names[plugin->plugin->type].str); > + DBUG_RETURN(ret); > + } > + state= PLUGIN_IS_READY; // plugin->init() succeeded > + > + if (plugin->plugin->status_vars) > + { > + /* > + historical ndb behavior caused MySQL plugins to specify > + status var names in full, with the plugin name prefix. > + this was never fixed in MySQL. > + MariaDB fixes that but supports MySQL style too. > + */ > + SHOW_VAR *show_vars= plugin->plugin->status_vars; > + SHOW_VAR tmp_array[2]= {{plugin->plugin->name, > + (char *) plugin->plugin->status_vars, SHOW_ARRAY}, > + {0, 0, SHOW_UNDEF}}; > + if (strncasecmp(show_vars->name, plugin->name.str, plugin->name.length)) > + show_vars= tmp_array; > + > + if (add_status_vars(show_vars)) > + DBUG_RETURN(1); > + } > + DBUG_RETURN(0); > +} > > static int plugin_initialize(MEM_ROOT *tmp_root, struct st_plugin_int *plugin, > int *argc, char **argv, bool options_only) > @@ -1595,7 +1594,7 @@ int plugin_init(int *argc, char **argv, int flags) > size_t i; > struct st_maria_plugin **builtins; > struct st_maria_plugin *plugin; > - struct st_plugin_int tmp, *plugin_ptr, **reap; > + struct st_plugin_int tmp, *plugin_ptr, **reap, **retry_end, **retry_start; > MEM_ROOT tmp_root; > bool reaped_mandatory_plugin= false; > bool mandatory= true; > @@ -1737,11 +1736,16 @@ int plugin_init(int *argc, char **argv, int flags) > */ > > mysql_mutex_lock(&LOCK_plugin); > + /* List of plugins to reap */ > reap= (st_plugin_int **) my_alloca((plugin_array.elements+1) * sizeof(void*)); > *(reap++)= NULL; > + /* List of plugins to retry */ > + retry_start= retry_end= > + (st_plugin_int **) my_alloca((plugin_array.elements+1) * sizeof(void*)); > > for(;;) > { > + int error; > for (i=0; i < MYSQL_MAX_PLUGIN_TYPE_NUM; i++) > { > HASH *hash= plugin_hash + plugin_type_initialization_order[i]; > @@ -1753,16 +1757,46 @@ int plugin_init(int *argc, char **argv, int flags) > bool plugin_table_engine= lex_string_eq(&plugin_table_engine_name, > &plugin_ptr->name); > bool opts_only= flags & PLUGIN_INIT_SKIP_INITIALIZATION && > - (flags & PLUGIN_INIT_SKIP_PLUGIN_TABLE || > - !plugin_table_engine); > - if (plugin_initialize(&tmp_root, plugin_ptr, argc, argv, opts_only)) > + (flags & PLUGIN_INIT_SKIP_PLUGIN_TABLE || !plugin_table_engine); > + error= plugin_initialize(&tmp_root, plugin_ptr, argc, argv, > + opts_only); > + if (error) > { > plugin_ptr->state= PLUGIN_IS_DYING; > - *(reap++)= plugin_ptr; > + /* The plugin wants a retry of the initialisation, > + possibly due to dependency on other plugins */ > + if (unlikely(error == HA_ERR_RETRY_INIT)) > + *(retry_end++)= plugin_ptr; > + else > + *(reap++)= plugin_ptr; > } > } > } > } > + /* Retry plugins that asked for it */ > + while (retry_start < retry_end) > + { > + st_plugin_int **to_re_retry, **retrying; > + for (to_re_retry= retrying= retry_start; retrying < retry_end; retrying++) > + { > + plugin_ptr= *retrying; > + uint state= plugin_ptr->state; > + mysql_mutex_unlock(&LOCK_plugin); you might want to add here sql_print_information("plugin XXX registration is retried") otherwise it would be confusing to see "plugin XXX registration failed" and then see the plugin perferctly initialized and usable. Alternatively (preferable, even) don't print "plugin registration failed" above, print it when adding a plugin to the reap list. So that the user won't see fail/retry loop at all, this is the inner working of the plugin subsystem, no user serviceable parts inside. > + error= plugin_do_initialize(plugin_ptr, state); > + mysql_mutex_lock(&LOCK_plugin); > + plugin_ptr->state= state; why do you need to manipulate (save/restore) the state here? Is that even correct? You set the state to PLUGIN_IS_DYING earlier and plugin_do_initialize() an change it to PLUGIN_IS_READY on success. You wouldn't want to change PLUGIN_IS_READY to PLUGIN_IS_DYING, would you? If this is the case, then you don't need to save the state at all, instead you need something like if (error) plugin_ptr->state= PLUGIN_IS_DYING; > + if (error == HA_ERR_RETRY_INIT) > + *(to_re_retry++)= plugin_ptr; > + else if (error) > + *(reap++)= plugin_ptr; > + } > + /* If the retry list has not changed, i.e. if all retry attempts > + result in another retry request, empty the retry list */ > + if (to_re_retry == retry_end) > + while (to_re_retry > retry_start) > + *(reap++)= *(--to_re_retry); > + retry_end= to_re_retry; > + } > > /* load and init plugins from the plugin table (unless done already) */ > if (flags & PLUGIN_INIT_SKIP_PLUGIN_TABLE) > @@ -1775,7 +1809,8 @@ int plugin_init(int *argc, char **argv, int flags) > } > > /* > - Check if any plugins have to be reaped > + Merge the retry list to the reap list, then reap the failed > + plugins. Note that during the merge we reverse the order in retry seems like this comment is obsolete > */ > while ((plugin_ptr= *(--reap))) > { > @@ -1788,6 +1823,7 @@ int plugin_init(int *argc, char **argv, int flags) > } > > mysql_mutex_unlock(&LOCK_plugin); > + my_afree(retry_start); > my_afree(reap); > if (reaped_mandatory_plugin && !opt_help) > goto err; > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: Client-server protocol improvement proposal: Connection redirection
by Andrey Sidorov 19 Jul '23

19 Jul '23

Hi Otto, thanks for bringing this to my attention. Happy to add redirect support to mysql2, looks like its relatively straightforward and sits on top of the existing protocol Andrey On Wed, 19 Jul 2023 at 15:44, Otto Kekäläinen <otto(a)kekalainen.net> wrote: > Hi Andrey! > > I am reaching out to you because you maintain > https://github.com/sidorares/node-mysql2 > > Have you heard about this proposal on the MariaDB developers list? > > It suggests adding a new feature to the MariaDB/MySQL protocol where > the server would be able to tell clients "No accepting connections - > connect to this server instead". The implementation would be work by > sending a new error type ERR_REDIRECT along with a client the > IP/hostname of a new server, similar to how HTTP 301 redirect works- > > Full thread visible at > > https://lists.mariadb.org/hyperkitty/list/developers@lists.mariadb.org/thre… > > If you want to participate in discussion, just reply to this message > and change recipient to developers(a)lists.mariadb.org. > > Thanks! > > > ---------- Forwarded message --------- > From: Lenski, Daniel via developers <developers(a)lists.mariadb.org> > Date: Wed, 28 Jun 2023 at 15:24 > Subject: [MariaDB developers] Client-server protocol improvement > proposal: Connection redirection > To: developers(a)lists.mariadb.org <developers(a)lists.mariadb.org> > > > Hi! > > I would like to propose a new feature in the MariaDB client-server > protocol: application-layer redirection of client connections. > > We want the MariaDB server to be able to tell clients connecting to > it, “Sorry, this server is unavailable. Connect to an alternate server > instead.” This mechanism is inspired by HTTP 302 (“temporary > redirect”) mechanism familiar to all developers of web applications, > and is intended to have similar semantics and security properties, > since these have now been widely deployed and tested for decades. > > I have submitted a minimal but viable implementation of this at: > https://github.com/MariaDB/server/pull/2681, server-side implementation > https://github.com/mariadb-corporation/mariadb-connector-c/pull/226, > MariaDB Connector/C implementation > > > I am seeking advice and opinions on these, and will proceed to > finalize this in the following 1-2 months. > > In its current form, this implementation allows the MariaDB server > administrator to set two variables: > > SET GLOBAL SERVER_REDIRECT_MODE={ON,OFF} (the default is OFF) > SET GLOBAL SERVER_REDIRECT_TARGET='my-new-server.example.com' (or > 192.168.0.123:3307, or new-server.com:3308, etc) > > When SERVER_REDIRECT_MODE is set to on, the server will stop accepting > new TCP-based connections (unless they are to the extra port, which is > intended for emergency administrative use), and it will instead > respond to all clients with an error packet: > > $ mariadb --host my-server.example.com > ERROR 4189 (HY000): |Server is redirecting clients to > 'my-new-server.example.com:3307'|my-new-server.example.com:3307 > > When an appropriately-updated client receives this error packet (error > code 4196, message formatted as |Human readable message|<value of > SERVER_REDIRECT_TARGET>), it will parse the redirection target out > from the error message and attempt a new connection: > > $ updated-mariadb --host my-server.example.com -e 'select @@hostname' > Got server redirect to 'my-new-server.example.com' (port 3307) > +---------------------------+ > | @@hostname | > +---------------------------+ > | my-new-server.example.com | > +---------------------------+ > > The feature is gracefully backwards-compatible. Old clients that do > not support redirection will fail and show the message in a form from > which humans can deduce what happened. New clients that do support it > will follow it to the new server. > > [...] >

1 0

Re: 845fbe9ce01: MDEV-30820 slow log Rows_examined out of range
by Sergei Golubchik 18 Jul '23

18 Jul '23

Hi, Sanja, The patch is ok, but the test case is rather weird. What does it do? How long does it run? On Jul 17, Oleksandr Byelkin wrote: > revision-id: 845fbe9ce01 (mariadb-10.4.30-30-g845fbe9ce01) > parent(s): 02cd3675c4d > author: Oleksandr Byelkin > committer: Oleksandr Byelkin > timestamp: 2023-07-17 10:22:54 +0200 > message: > > MDEV-30820 slow log Rows_examined out of range > > Fix row counters to be able to get any possible value. > > diff --git a/mysql-test/main/log_slow.test b/mysql-test/main/log_slow.test > --- a/mysql-test/main/log_slow.test > +++ b/mysql-test/main/log_slow.test > @@ -119,3 +119,98 @@ drop table t; > --echo # > --echo # End of 10.3 tests > --echo # > + > + > +--echo # > +--echo # MDEV-30820: slow log Rows_examined out of range > +--echo # > + > +set @log_output.save= @@global.log_output, @slow_log.save= @@global.slow_query_log; > +set global log_output= 'TABLE', slow_query_log= ON; > +set long_query_time= 0.000001; > +create database db; > +use db; > + > +--delimiter // > +CREATE OR REPLACE FUNCTION `get_id`(`INPUT_STRING` VARCHAR(50), `DECRYPT_KEY` CHAR(20)) RETURNS int(11) > +BEGIN > + DECLARE > + REQUEST_ID INT ; > + SET > + REQUEST_ID = > + IF( > + ( > + LOCATE('#$#$#$ LHP_ID_', INPUT_STRING) > + ) > 0 AND > + ( > + INPUT_STRING REGEXP DECRYPT_KEY > + )>0, > + ( > + SPLIT_STRING( > + SUBSTRING_INDEX( > + SUBSTRING_INDEX(INPUT_STRING, '#$#$#$ LHP_ID_', -1), > + '#$#$#$', > + 1 > + ), > + '-', > + 2 > + ) > + ), > + NULL > + ) ; > + SET > + REQUEST_ID=IF( > + ISNULL(REQUEST_ID)OR(REQUEST_ID='')OR(REQUEST_ID=0), > + NULL, > + REQUEST_ID > + > + ); > + RETURN(REQUEST_ID) ; > +END > +// > + > +CREATE OR REPLACE FUNCTION `SPLIT_STRING`(`str` VARCHAR(255), `delim` VARCHAR(12), `pos` INT) RETURNS varchar(255) CHARSET utf8mb4 > +RETURN REPLACE( > + SUBSTRING( > + SUBSTRING_INDEX(str , delim , pos) , > + CHAR_LENGTH( > + SUBSTRING_INDEX(str , delim , pos - 1) > + ) + 1 > + ) , > + delim , > + '' > +) > +// > + > +CREATE TABLE `tab1` ( > +`ID` int(11) NOT NULL AUTO_INCREMENT, > +`NAME_F` varchar(50) DEFAULT NULL, > + PRIMARY KEY (`ID`) > +) DEFAULT CHARSET=utf8mb4 // > + > + CREATE TABLE `tab2` ( > + `ID` int(11) NOT NULL AUTO_INCREMENT, > + `TAB1_ID` int(11) DEFAULT NULL, > + PRIMARY KEY (`id`) > +) DEFAULT CHARSET=utf8mb4 // > + > +for i in 1..100 do insert into tab1 values (null,'qwerty'); end for ; // > +for i in 1..1000 do insert into tab2 values (null,round(rand()*10000)); end for ; // > + > +--delimiter ; > + > +SELECT > + get_id(CAST(aes_decrypt(tab1.NAME_F,'V41iNM0n4') AS char),'KM_ID_PL') as get_string, > + (CASE WHEN (SELECT ID FROM tab2 where tab2.TAB1_ID = tab1.ID LIMIT 1) IS NULL THEN 0 ELSE 1 END) AS IS_ATT > +FROM > + tab1 > +ORDER BY 2 DESC > +LIMIT 2; > + > +set global log_output= @log_output.save, slow_query_log= @slow_log.save; > +drop database db; > + > +--echo # > +--echo # End of 10.4 tests > +--echo # Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 5

Re: 5102d86beae: MDEV-30164 System variable for default collations
by Sergei Golubchik 17 Jul '23

17 Jul '23

Hi, Alexander, Looks good, thanks. ok to push. a couple of comments below - but really, on remnants of the previous version of the commit. On Jul 17, Alexander Barkov wrote: > revision-id: 5102d86beae (mariadb-10.11.2-47-g5102d86beae) > parent(s): 3430767e006 > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2023-07-10 13:17:45 +0400 > message: > > MDEV-30164 System variable for default collations > > This patch adds a way to override default collations > (or "character set collations") for desired character sets. > > diff --git a/mysql-test/main/mysqld--help.result b/mysql-test/main/mysqld--help.result > index 77df8536fa2..5c361e3da88 100644 > --- a/mysql-test/main/mysqld--help.result > +++ b/mysql-test/main/mysqld--help.result > @@ -144,6 +144,8 @@ The following specify which files/extra groups are read (specified before remain > Don't ignore client side character set value sent during > handshake. > (Defaults to on; use --skip-character-set-client-handshake to disable.) > + --character-set-collations=name > + Set default collations for character sets. shouldn't it say "overrides"? > --character-set-filesystem=name > Set the filesystem character set. > -C, --character-set-server=name > diff --git a/sql/sql_parse.cc b/sql/sql_parse.cc > index 607877dfb83..d35869e7e39 100644 > --- a/sql/sql_parse.cc > +++ b/sql/sql_parse.cc > @@ -6061,6 +6061,32 @@ mysql_execute_command(THD *thd, bool is_called_from_prepared_stmt) > } > thd->reset_kill_query(); > } > + > + /* > + Binary logging is now done. Unset the "used" flags to avoid > + flags leaking to the next event (and to the COMMIT statement > + in the end of the current event). > + > + Example: > + > + Suppose a non-default collation (in @@character_set_collations) > + was used during the statement, the mysqlbinlog output for > + the current statement will contain a sequence like this: > + > + SET character_set_collations='utf8mb3=utf8mb3_bin'; > + INSERT INTO t1 VALUES (_utf8mb3'test'); > + COMMIT; > + > + The statment (INSERT in this example) is already in binlog at this point, and the > + and the "SET character_set_collations" is written inside a > + Q_CHARACTER_SET_COLLATIONS chunk in its log entry header. > + The flag CHARACTER_SET_COLLATIONS_USED is not needed any more. > + The COMMIT can be printed without "SET character_set_collations". > + > + The same logic applies to the other _USED flags. That's a bit too verbose in my opinion. The example was good in the previous version of the patch, but now the first paragraph of the comment pretty much explains it all. but do it as you prefer. > + */ > + thd->used= 0; > + > if (unlikely(thd->is_error()) || > (thd->variables.option_bits & OPTION_MASTER_SQL_ERROR)) > { Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: 8e9c68b49dd: MDEV-25237 Assertion `global_system_variables.session_track_system_variables' failed in Session_sysvars_tracker::init | SIGSEGV's in __strlen_avx2 | UBSAN: runtime error: null pointer passed as argument 1, which is declared to never be null in my_strdup
by Sergei Golubchik 15 Jul '23

15 Jul '23

Hi, Sanja, ok to push. but see comments below: On Jul 15, Oleksandr Byelkin wrote: > revision-id: 8e9c68b49dd (mariadb-10.4.30-9-g8e9c68b49dd) > parent(s): 423c28f0aa4 > author: Oleksandr Byelkin > committer: Oleksandr Byelkin > timestamp: 2023-07-10 13:40:46 +0200 > message: > > MDEV-25237 Assertion `global_system_variables.session_track_system_variables' failed in Session_sysvars_tracker::init | SIGSEGV's in __strlen_avx2 | UBSAN: runtime error: null pointer passed as argument 1, which is declared to never be null in my_strdup again, if you see a bug with a very long summary, particularly if it basically makes no sense to a user - in that case, please, don't hesitate to make it shorter and more meaningful. For example "crash after setting session_track_system_variables to an invalid value" > Fix of typo in checking variable list corectness. > > Fix of error handling in case of variable list parse error > > diff --git a/sql/session_tracker.cc b/sql/session_tracker.cc > index 2f6c5e29bf2..3e1558f6759 100644 > --- a/sql/session_tracker.cc > +++ b/sql/session_tracker.cc > @@ -221,7 +221,7 @@ bool sysvartrack_validate_value(THD *thd, const char *str, size_t len) > /* Remove leading/trailing whitespace. */ > trim_whitespace(system_charset_info, &var); > > - if (!strcmp(var.str, "*") && !find_sys_var(thd, var.str, var.length)) > + if (strcmp(var.str, "*") && !find_sys_var(thd, var.str, var.length)) > return true; > > if (lasts) > @@ -331,9 +331,10 @@ void Session_sysvars_tracker::init(THD *thd) > mysql_mutex_assert_owner(&LOCK_global_system_variables); > DBUG_ASSERT(thd->variables.session_track_system_variables == > global_system_variables.session_track_system_variables); > - DBUG_ASSERT(global_system_variables.session_track_system_variables); > thd->variables.session_track_system_variables= > - my_strdup(global_system_variables.session_track_system_variables, > + my_strdup((global_system_variables.session_track_system_variables? > + global_system_variables.session_track_system_variables: > + ""), this is such a common pattern that we have a helper for that, use my_strdup(safe_str(global_system_variables.session_track_system_variables)) > MYF(MY_WME | MY_THREAD_SPECIFIC)); > } > > @@ -572,6 +573,12 @@ bool sysvartrack_global_update(THD *thd, char *str, size_t len) > { > LEX_STRING tmp= { str, len }; > Session_sysvars_tracker::vars_list dummy; > + DBUG_EXECUTE_IF("dbug_session_tracker_parse_error", > + { > + my_error(ER_OUTOFMEMORY, MYF(0), 1); > + return true; > + }); > + > if (!dummy.parse_var_list(thd, tmp, false, system_charset_info)) > { > dummy.construct_var_list(str, len + 1); > diff --git a/sql/sys_vars.inl b/sql/sys_vars.inl > index 84d1cd6b331..3e282de439a 100644 > --- a/sql/sys_vars.inl > +++ b/sql/sys_vars.inl > @@ -620,7 +620,11 @@ class Sys_var_sesvartrack: public Sys_var_charptr_base > { > if (sysvartrack_global_update(thd, new_val, > var->save_result.string_value.length)) > + { > + if (new_val) > + my_free(new_val); it's ok, as you like. But technically you don't need an `if` here, my_free() just as free() works with NULL pointers fine. > new_val= 0; > + } > } > global_update_finish(new_val); > return (new_val == 0 && var->save_result.string_value.str != 0); > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: 8d51c6d234b: MDEV-30164 System variable for default collations
by Sergei Golubchik 10 Jul '23

10 Jul '23

Hi, Alexander, Please, see comments below. Sorry, it took a while. On Jun 23, Alexander Barkov wrote: > revision-id: 8d51c6d234b (mariadb-10.11.2-13-g8d51c6d234b) > parent(s): ceb0e7f944b > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2023-03-21 10:07:57 +0400 > message: > > MDEV-30164 System variable for default collations > > diff --git a/mysql-test/main/ctype_collate_implicit.test b/mysql-test/main/ctype_collate_implicit.test > --- /dev/null > +++ b/mysql-test/main/ctype_collate_implicit.test > @@ -0,0 +1,209 @@ > +--source include/have_utf8.inc > +--source include/have_utf8mb4.inc > + > +--echo # > +--echo # MDEV-30164 System variable for default collations > +--echo # > + > +SET @@character_set_collations= ' utf8mb3 = utf8mb3_bin , LATIN1 = LATIN1_BIN '; also, please, test that parsing of commas is relaxed too. Like set @@character_set_collations= ',,,utf8mb3 = utf8mb3_bin,,latin1 = latin1_bin,,,'; it makes it easier for the user to edit the value. > +SELECT @@character_set_collations; > +SET @@character_set_collations=''; > +SELECT @@character_set_collations; > + > +SET @@character_set_collations='utf8mb3=utf8mb3_bin'; > +SELECT @@character_set_collations; > +SET @@character_set_collations=''; > + > +--error ER_COLLATION_CHARSET_MISMATCH > +SET @@character_set_collations='utf8mb3=utf8mb4_general_ci'; > +SELECT @@character_set_collations; > + > +--error ER_COLLATION_CHARSET_MISMATCH > +SET @@character_set_collations='utf8mb4=utf8mb3_general_ci'; > +SELECT @@character_set_collations; > + > +SET @@character_set_collations='utf8mb3=utf8mb3_general_ci'; > +SELECT @@character_set_collations; > + > +SET @@character_set_collations='utf8mb4=utf8mb4_general_ci,latin1=latin1_bin'; > +SELECT @@character_set_collations; > + > +--error ER_COLLATION_CHARSET_MISMATCH > +SET @@character_set_collations='utf8mb4=uca1400_ai_ci,latin1=uca1400_ai_ci'; > + > +# All or nothing is set. "Nothing" in this case because of the error on latin1. > +# The "uca1400_ai_ci FOR utf8mb4" part was ignored. > +SELECT @@character_set_collations; > +SELECT @@character_set_collations RLIKE 'utf8mb4=utf8mb4_general_ci' AS expect_true; > + > + > +SET @@character_set_collations='utf8mb4=uca1400_ai_ci'; > +SELECT @@character_set_collations; > + > +SET NAMES utf8mb4; > +SELECT @@collation_connection; > + > +# We have to disable --view-protocol for the following statement. > +# 'mtr --view-protocol' creates a separate connection for these statements: > +# CREATE VIEW mysqltest_tmp_sp AS ...; > +# DROP VIEW mysqltest_tmp_sp; > +# The current @@character_set_collations does not affect this connection. > +# So --view-protocol would return the hard-coded character set collation here, > +# instead of utf8mb4_uca1400_ai_ci > + > +--disable_view_protocol > +SELECT collation('literal'); > +--enable_view_protocol > +EXECUTE IMMEDIATE 'SELECT COLLATION(?)' USING 'literal'; > + > +CREATE VIEW v1 AS SELECT 'literal', collation('literal') as cl; > +SHOW CREATE VIEW v1; > +SELECT * FROM v1; > +DROP VIEW v1; > + > + > +# Override @@collation_connection to utf8mb4_general_ci. > +# Make sure that CREATE statements does not use @@collation_connection. > +# to detect implicit collations. > +# Implicit collations are detected using @@character_set_collations! > + > +SET NAMES utf8mb4 COLLATE utf8mb4_general_ci; > + > +CREATE TABLE t1 (a TEXT CHARACTER SET utf8mb4); > +SHOW CREATE TABLE t1; > +DROP TABLE t1; > + > +CREATE TABLE t1 (a TEXT CHARACTER SET utf8mb4 COLLATE DEFAULT); > +SHOW CREATE TABLE t1; > +DROP TABLE t1; > + > +CREATE TABLE t1 (a TEXT COLLATE DEFAULT) CHARACTER SET utf8mb4; > +SHOW CREATE TABLE t1; > +DROP TABLE t1; > + > +CREATE TABLE t1 (a TEXT) CHARACTER SET utf8mb4; > +SHOW CREATE TABLE t1; > +DROP TABLE t1; > + > +CREATE DATABASE db1 CHARACTER SET utf8mb4; > +SHOW CREATE DATABASE db1; > +DROP DATABASE db1; > + > + > +# Test how @@character_set_collations affects various expressions > +# with implicit collations. > + > + > +let query=SELECT > + @@collation_connection AS conn, > + COLLATION('a') AS lit, > + COLLATION(CONCAT(1)) AS num, > + COLLATION(CAST(123 AS CHAR)) AS casti, > + COLLATION(_utf8mb4'a') AS litu, > + COLLATION(_utf8mb4 0x62) AS lituh, > + COLLATION(_utf8mb4 X'63') AS lituhs, > + COLLATION(CAST(123 AS CHAR CHARACTER SET utf8mb4)) AS castic, > + COLLATION(CHAR(0x61 USING utf8mb4)) AS chr, > + COLLATION(CONVERT('a' USING utf8mb4)) AS conv; > + > +# The below SET NAMES sets @@collation_connection to utf8mb4_general_ci. > +# But @@character_set_collations still contains utf8mb4=uca1400_ai_ci. > + > +SET NAMES utf8mb4 COLLATE utf8mb4_general_ci; > + > +# Columns expected to print utf8mb4_general_ci > +# because they use @@collation_connection: > +# - String literals without introducers > +# - Automatic number-to-string conversions > +# - CAST(AS CHAR) - without USING > +# > +# Columns expected to print utf8mb4_uca1400_ai_ci > +# because they use the current session default collation > +# for the character set (as specified in @@collation_connection) > +# - String literals with introducers > +# - CAST(AS CHAR USING cs) > +# - CHAR() > +# - CONVERT() > + > +--vertical_results > +--eval $query; > +--horizontal_results > + > +# This sets collation_connection to utf8mb4_uca1400_ai_ci > +# according to @@character_set_collations. > +SET NAMES utf8mb4; > + > +# Now all columns are expected to print utf8mb4_uca1400_ai_ci: > +# - Some columns because @@collation_connection says so > +# - Some columns because @@character_set_collations says so. > + > +--vertical_results > +--eval $query; > +--horizontal_results > + > + > +# > +# INFORMATION_SCHEMA > +# > + > +SET character_set_collations='latin1=latin1_bin,utf8mb4=uca1400_ai_ci'; > +SHOW CHARACTER SET LIKE 'latin1'; > +SELECT * FROM INFORMATION_SCHEMA.CHARACTER_SETS > +WHERE CHARACTER_SET_NAME='latin1'; > + > +SHOW COLLATION LIKE 'latin1%'; > +SELECT COLLATION_NAME, IS_DEFAULT > +FROM INFORMATION_SCHEMA.COLLATIONS > +WHERE CHARACTER_SET_NAME LIKE 'latin1%'; > +SELECT COLLATION_NAME, FULL_COLLATION_NAME, IS_DEFAULT > +FROM INFORMATION_SCHEMA.COLLATION_CHARACTER_SET_APPLICABILITY > +WHERE COLLATION_NAME LIKE 'latin1%'; > + > +SHOW CHARACTER SET LIKE 'utf8mb4'; > +SELECT * FROM INFORMATION_SCHEMA.CHARACTER_SETS > +WHERE CHARACTER_SET_NAME='utf8mb4'; > + > +SHOW COLLATION LIKE '%uca1400_ai_ci%'; > +SELECT COLLATION_NAME, IS_DEFAULT > +FROM INFORMATION_SCHEMA.COLLATIONS > +WHERE COLLATION_NAME LIKE '%uca1400_ai_ci%'; > +SELECT COLLATION_NAME, FULL_COLLATION_NAME, IS_DEFAULT > +FROM INFORMATION_SCHEMA.COLLATION_CHARACTER_SET_APPLICABILITY > +WHERE COLLATION_NAME LIKE '%uca1400_ai_ci%'; > + > +# > +# Prepared statements: reprepare on @@character_set_collations change. > +# > + > +SET @@character_set_collations=''; > +PREPARE stmt FROM 'SELECT ' > + 'COLLATION(CAST("x" AS CHAR CHARACTER SET utf8mb3)) AS a, ' > + 'COLLATION(_utf8mb3"x") AS b'; > +EXECUTE stmt; > +SET @@character_set_collations='utf8mb3=utf8mb3_bin'; > +EXECUTE stmt; > + > +SET @@character_set_collations='utf8mb3=utf8mb3_bin'; > +PREPARE stmt FROM 'SELECT ' > + 'COLLATION(CAST("x" AS CHAR CHARACTER SET utf8mb3)) AS a, ' > + 'COLLATION(_utf8mb3"x") AS b'; > +EXECUTE stmt; > +SET @@character_set_collations=DEFAULT; > +EXECUTE stmt; > + > +SET NAMES utf8mb3; > +SET @@character_set_collations=''; > +PREPARE stmt FROM 'CREATE TABLE t1 ' > + '(a TEXT CHARACTER SET utf8mb3 COLLATE DEFAULT COLLATE utf8mb3_general_ci)'; > +EXECUTE stmt; > +SHOW CREATE TABLE t1; > +DROP TABLE t1; > + > +SET @@character_set_collations='utf8mb3=utf8mb3_bin'; > +--error ER_CONFLICTING_DECLARATIONS > +EXECUTE stmt; > + > +SET @@character_set_collations=''; > +EXECUTE stmt; > +SHOW CREATE TABLE t1; > +DROP TABLE t1; > diff --git a/mysql-test/suite/sys_vars/r/sysvars_server_notembedded.result b/mysql-test/suite/sys_vars/r/sysvars_server_notembedded.result > --- a/mysql-test/suite/sys_vars/r/sysvars_server_notembedded.result > +++ b/mysql-test/suite/sys_vars/r/sysvars_server_notembedded.result > @@ -502,6 +502,16 @@ NUMERIC_BLOCK_SIZE NULL > ENUM_VALUE_LIST NULL > READ_ONLY NO > COMMAND_LINE_ARGUMENT NULL > +VARIABLE_NAME CHARACTER_SET_COLLATIONS > +VARIABLE_SCOPE SESSION > +VARIABLE_TYPE VARCHAR > +VARIABLE_COMMENT Default collations for character sets Better to emphasize somehow that it's not a complete list of default collations, but adjustments to the compiled-in list of default collations. If this variable is empty, it doesn't mean that character sets have no default collations. I'm not sure how to put this succintly in the help text though :) > +NUMERIC_MIN_VALUE NULL > +NUMERIC_MAX_VALUE NULL > +NUMERIC_BLOCK_SIZE NULL > +ENUM_VALUE_LIST NULL > +READ_ONLY NO > +COMMAND_LINE_ARGUMENT NULL > VARIABLE_NAME CHARACTER_SET_CONNECTION > VARIABLE_SCOPE SESSION > VARIABLE_TYPE ENUM > diff --git a/sql/lex_charset.cc b/sql/lex_charset.cc > --- a/sql/lex_charset.cc > +++ b/sql/lex_charset.cc > @@ -447,6 +451,17 @@ Lex_exact_charset_opt_extended_collate::find_default_collation() const > } > > > +CHARSET_INFO * > +Lex_exact_charset_opt_extended_collate:: > + find_mapped_default_collation(Charset_collation_map_st::Used *used, > + const Charset_collation_map_st &map) const > +{ > + CHARSET_INFO *cs= find_compiled_default_collation(); > + if (!cs) > + return nullptr; > + return map.get_collation_for_charset(used, cs); This seems a bit redundant. find_compiled_default_collation() will do get_charset_by_csname(m_ci, MY_CS_PRIMARY) and only then you'll be able to search, because you map default compiled collation id to use specified collation id. but conceptually you need to map a charset to a collation, not a collation to a collation. If you'd map, say, cs->cs_name.str (just the pointer, not the string value) or, say, cs->cset, then you'd be mapping a charset to a collation. and here you'd be able to do just return map.get_collation_for_charset(used, m_ci) with find_compiled_default_collation() being done inside get_collation_for_charset() and only when Elem_st wasn't found > +} > + > /* > Resolve an empty or a contextually typed collation according to the > upper level default character set (and optionally a collation), e.g.: > diff --git a/sql/lex_charset.h b/sql/lex_charset.h > --- a/sql/lex_charset.h > +++ b/sql/lex_charset.h > @@ -544,6 +561,21 @@ struct Lex_exact_charset_extended_collation_attrs_st > { > return m_ci; > } > + CHARSET_INFO *charset_info(Charset_collation_map_st::Used *used, > + const Charset_collation_map_st &map) const > + { > + switch (m_type) > + { > + case TYPE_CHARACTER_SET: > + return map.get_collation_for_charset(used, m_ci); Why do you need this special charset_info(used,map) that differs from charset_info() when m_type==TYPE_CHARACTER_SET, and is used in set_charset_collation_attrs() only? > + case TYPE_EMPTY: > + case TYPE_CHARACTER_SET_COLLATE_EXACT: > + case TYPE_COLLATE_CONTEXTUALLY_TYPED: > + case TYPE_COLLATE_EXACT: > + break; > + } > + return m_ci; > + } > Type type() const > { > return m_type; > diff --git a/sql/log_event_server.cc b/sql/log_event_server.cc > --- a/sql/log_event_server.cc > +++ b/sql/log_event_server.cc > @@ -1989,6 +1997,16 @@ int Query_log_event::do_apply_event(rpl_group_info *rgi, > thd->variables.sql_mode= > (sql_mode_t) ((thd->variables.sql_mode & MODE_NO_DIR_IN_CREATE) | > (sql_mode & ~(sql_mode_t) MODE_NO_DIR_IN_CREATE)); > + > + size_t cslen= thd->variables.character_set_collations.from_binary( > + character_set_collations.str, > + character_set_collations.length); > + if (cslen != character_set_collations.length) > + { > + thd->variables.character_set_collations.init(); > + goto compare_errors; // QQ: report an error here? is it corrupted event or unknown collation? > + } > + > if (charset_inited) > { > rpl_sql_thread_info *sql_info= thd->system_thread_info.rpl_sql_info; > diff --git a/sql/sql_parse.cc b/sql/sql_parse.cc > --- a/sql/sql_parse.cc > +++ b/sql/sql_parse.cc > @@ -6049,6 +6049,27 @@ mysql_execute_command(THD *thd, bool is_called_from_prepared_stmt) > } > thd->reset_kill_query(); > } > + > + /* > + If a non-default collation (in @@character_set_collations) > + was used during the statement, the mysqlbinlog output for > + the current statement will contain a sequence like this: > + > + SET character_set_collations='utf8mb3=utf8mb3_bin'; > + INSERT INTO t1 VALUES (_utf8mb3'test'); > + COMMIT; > + > + The statment (INSERT in this example) is already in binlog at this point, and the > + and the "SET character_set_collations" is written inside a > + Q_CHARACTER_SET_COLLATIONS chunk in its log entry header. > + The flag CHARACTER_SET_COLLATIONS_USED is not needed any more. > + > + Let's suppress the flag to avoid a Q_CHARACTER_SET_COLLATIONS chunk > + inside the COMMIT log entry header - it would be useless and would > + only waste space in the binary log. > + */ > + thd->used&= ~THD::CHARACTER_SET_COLLATIONS_USED; Wouldn't that logic apply to other THD::xxx_USED flags? May be you should reset thd->used=0 here? > + > if (unlikely(thd->is_error()) || > (thd->variables.option_bits & OPTION_MASTER_SQL_ERROR)) > { > diff --git a/sql/sql_table.cc b/sql/sql_table.cc > --- a/sql/sql_table.cc > +++ b/sql/sql_table.cc > @@ -12457,8 +12459,10 @@ bool HA_CREATE_INFO:: I'd prefer you didn't put a line break after :: see how useful it makes the hunk header? > { > // Make sure we don't do double resolution in direct SQL execution > DBUG_ASSERT(!default_table_charset || thd->stmt_arena->is_stmt_execute()); > + Character_set_collations_used used(thd); > if (!(default_table_charset= > - default_cscl.resolved_to_context(ctx))) > + default_cscl.resolved_to_context(&used, > + thd->variables.character_set_collations, ctx))) > return true; > } > > diff --git a/sql/sql_class.h b/sql/sql_class.h > --- a/sql/sql_class.h > +++ b/sql/sql_class.h > @@ -5620,6 +5624,29 @@ class THD: public THD_count, /* this must be first */ > }; > > > +class Character_set_collations_used: public Charset_collation_map_st::Used > +{ > + THD *m_thd; > +public: > + Character_set_collations_used(THD *thd) > + :m_thd(thd) > + { } > + ~Character_set_collations_used() > + { > + /* > + Mark THD that the collation map was used, > + no matter if a compiled or a mapped collation was why no matter? as far as I understand, you didn't plan to change compiled-in defaults > + found during charset->collation resolution. > + Even if the map was empty, we still need to print > + SET @@session.character_set_collations=''; > + in mariadb-binlog output. > + */ > + if (m_used) > + m_thd->used|= THD::CHARACTER_SET_COLLATIONS_USED; > + } > +}; > + > + > /* > Start a new independent transaction for the THD. > The old one is stored in this object and restored when calling > diff --git a/sql/sql_prepare.cc b/sql/sql_prepare.cc > --- a/sql/sql_prepare.cc > +++ b/sql/sql_prepare.cc > @@ -3538,6 +3544,13 @@ static void mysql_stmt_execute_common(THD *thd, > DBUG_VOID_RETURN; > } > > + if (stmt->prepare_time_charset_collation_map_version() != > + thd->variables.character_set_collations.version()) > + { > + if (stmt->reprepare()) > + DBUG_VOID_RETURN; > + } why do you reprepare here and now where reprepare() is normally happens, in the Prepared_statement::execute_loop()? > + > /* > In case of direct execution application decides how many parameters > to send. > @@ -5220,6 +5252,13 @@ bool Prepared_statement::execute(String *expanded_query, bool open_cursor) > MYSQL_QUERY_EXEC_START(thd->query(), thd->thread_id, thd->get_db(), > &thd->security_ctx->priv_user[0], > (char *) thd->security_ctx->host_or_ip, 1); > + /* > + If PREPARE used @@character_set_collations, > + then we need to make sure binary log writes > + the map in the event header. > + */ why is it different from other THD::xx_USED flags? > + thd->used|= m_prepare_time_thd_used_flags & > + THD::CHARACTER_SET_COLLATIONS_USED; > error= mysql_execute_command(thd, true); > MYSQL_QUERY_EXEC_DONE(error); > thd->update_server_status(); > diff --git a/sql/sys_vars.cc b/sql/sys_vars.cc > --- a/sql/sys_vars.cc > +++ b/sql/sys_vars.cc > @@ -436,6 +436,115 @@ static bool update_auto_increment_increment (sys_var *self, THD *thd, enum_var_t > > #endif /* WITH_WSREP */ > > + > +class Sys_var_charset_collation_map: public sys_var in sys_var.inl, please > +{ > +public: > + Sys_var_charset_collation_map(const char *name_arg, const char *comment, > + int flag_args, ptrdiff_t off, size_t size, > + CMD_LINE getopt, > + enum binlog_status_enum binlog_status_arg) > + :sys_var(&all_sys_vars, name_arg, comment, > + flag_args, off, getopt.id, getopt.arg_type, > + SHOW_CHAR, > + DEFAULT(0), nullptr, binlog_status_arg, > + nullptr, nullptr, nullptr) > + { > + option.var_type|= GET_STR; > + } > + > +private: > + > + static bool charset_collation_map_from_item(Charset_collation_map_st *map, > + Item *item, > + myf utf8_flag) > + { > + String *value, buffer; > + if (!(value= item->val_str_ascii(&buffer))) > + return true; > + return map->from_text(value->to_lex_cstring(), utf8_flag); > + } > + > + static const uchar *make_value_ptr(THD *thd, > + const Charset_collation_map_st &map) > + { > + size_t nbytes= map.text_format_nbytes_needed(); > + char *buf= (char *) thd->alloc(nbytes); > + size_t length= map.print(buf, nbytes); > + return (uchar *) thd->strmake(buf, length); Eh, sorry. I don't understand. You allocate memory twice? for two copies of the string? > + } > + > + > +private: > + > + bool do_check(THD *thd, set_var *var) override > + { > + Charset_collation_map_st map; > + return charset_collation_map_from_item(&map, var->value, > + thd->get_utf8_flag()); > + } > + > + void session_save_default(THD *thd, set_var *var) override > + { > + thd->variables.character_set_collations.set( > + global_system_variables.character_set_collations, 1); > + } > + > + void global_save_default(THD *thd, set_var *var) override > + { > + global_system_variables.character_set_collations.init(); > + } > + > + bool session_update(THD *thd, set_var *var) override > + { > + Charset_collation_map_st map; > + if (!var->value) > + { > + session_save_default(thd, var); > + return false; > + } > + if (charset_collation_map_from_item(&map, var->value, thd->get_utf8_flag())) The idea is that var->value is only evaluated in check and is not reevaluated in update. check is supposed to store the value in set_var *var for update to use. > + return true; > + thd->variables.character_set_collations.set(map, 1); > + return false; > + } > + > + bool global_update(THD *thd, set_var *var) override > + { > + Charset_collation_map_st map; > + if (!var->value) > + { > + global_save_default(thd, var); > + return false; > + } > + if (charset_collation_map_from_item(&map, var->value, thd->get_utf8_flag())) > + return true; > + global_system_variables.character_set_collations= map; > + return false; > + } > + > + const uchar * > + session_value_ptr(THD *thd, const LEX_CSTRING *base) const override > + { > + return make_value_ptr(thd, thd->variables.character_set_collations); > + } > + > + const uchar * > + global_value_ptr(THD *thd, const LEX_CSTRING *base) const override > + { > + return make_value_ptr(thd, global_system_variables. > + character_set_collations); > + } > +}; > + > + > +static Sys_var_charset_collation_map Sys_character_set_collations( > + "character_set_collations", > + "Default collations for character sets", > + SESSION_VAR(character_set_collations), > + NO_CMD_LINE, NOT_IN_BINLOG); > + > + > static Sys_var_double Sys_analyze_sample_percentage( > "analyze_sample_percentage", > "Percentage of rows from the table ANALYZE TABLE will sample " > diff --git a/sql/sql_yacc.yy b/sql/sql_yacc.yy > --- a/sql/sql_yacc.yy > +++ b/sql/sql_yacc.yy > @@ -9686,6 +9713,9 @@ column_default_non_parenthesized_expr: > } > | CONVERT_SYM '(' expr USING charset_name ')' > { > + Character_set_collations_used used(thd); > + $5= thd->variables.character_set_collations. > + get_collation_for_charset(&used, $5); this is rather awkward pattern that you use everywhere, with Character_set_collations_used used(thd); why not to pass thd as an argument instead? - Character_set_collations_used used(thd); $5= thd->variables.character_set_collations. - get_collation_for_charset(&used, $5); + get_collation_for_charset(thd, $5); > $$= new (thd->mem_root) Item_func_conv_charset(thd, $3, $5); > if (unlikely($$ == NULL)) > MYSQL_YYABORT; > diff --git a/sql/simple_tokenizer.h b/sql/simple_tokenizer.h > --- /dev/null > +++ b/sql/simple_tokenizer.h > @@ -0,0 +1,83 @@ > +/* Copyright (c) 2022, MariaDB Corporation. 2023 > + > + This program is free software; you can redistribute it and/or modify > + it under the terms of the GNU General Public License as published by > + the Free Software Foundation; version 2 of the License. > + > + This program is distributed in the hope that it will be useful, > + but WITHOUT ANY WARRANTY; without even the implied warranty of > + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > + GNU General Public License for more details. > + > + You should have received a copy of the GNU General Public License > + along with this program; if not, write to the Free Software > + Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1335 USA */ > + > +#ifndef SIMPLE_TOKENIZER_INCLUDED > +#define SIMPLE_TOKENIZER_INCLUDED > + > + > +class Simple_tokenizer > +{ > + const char *m_ptr; > + const char *m_end; > +public: > + Simple_tokenizer(const char *str, size_t length) > + :m_ptr(str), m_end(str + length) > + { } > + const char *ptr() const > + { > + return m_ptr; > + } > + bool eof() const > + { > + return m_ptr >= m_end; > + } > + void get_spaces() > + { > + for ( ; !eof(); m_ptr++) > + { > + if (m_ptr[0] != ' ') > + break; > + } > + } > + bool is_ident_start(char ch) const > + { > + return (ch >= 'a' && ch <= 'z') || > + (ch >= 'A' && ch <= 'Z') || > + ch == '_'; > + } > + bool is_ident_body(char ch) const > + { > + return is_ident_start(ch) || > + (ch >= '0' && ch <= '9'); > + } > + bool is_ident_start() const > + { > + return !eof() && is_ident_start(*m_ptr); > + } > + bool is_ident_body() const > + { > + return !eof() && is_ident_body(*m_ptr); > + } > + LEX_CSTRING get_ident() > + { > + if (!is_ident_start()) > + return {m_ptr,0}; > + const char *start= m_ptr++; > + for ( ; is_ident_body(); m_ptr++) > + { } > + LEX_CSTRING res= {start, (size_t) (m_ptr - start)}; > + return res; > + } > + bool get_char(char ch) > + { > + if (eof() || *m_ptr != ch) > + return true; > + m_ptr++; > + return false; > + } > +}; that's too big and complex, but also not complex enough. note that we have quite a few of those already, in patricular in typelib.c, in strfunc.cc, item_func.cc (find_in_set) etc. and this one could've been 2-3 times smaller for @@character_set_collations only, and it still cannot replace any other parsers from above. I'd keep just two methods: get_token() - which skips whitespaces and returns the next substring of non-white-space and not ',' or '=' characters. multibyte-safe. And get_char, which returns the next non-white-space character. this parser could be used in all use cases from above. > + > + > +#endif // SIMPLE_TOKENIZER_INCLUDED > diff --git a/sql/charset_collations.h b/sql/charset_collations.h > --- /dev/null > +++ b/sql/charset_collations.h > @@ -0,0 +1,265 @@ > +/* Copyright (c) 2023, MariaDB Corporation. > + > + This program is free software; you can redistribute it and/or modify > + it under the terms of the GNU General Public License as published by > + the Free Software Foundation; version 2 of the License. > + > + This program is distributed in the hope that it will be useful, > + but WITHOUT ANY WARRANTY; without even the implied warranty of > + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > + GNU General Public License for more details. > + > + You should have received a copy of the GNU General Public License > + along with this program; if not, write to the Free Software > + Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1335 USA */ > + > +#ifndef LEX_CHARSET_COLLATIONS_INCLUDED > +#define LEX_CHARSET_COLLATIONS_INCLUDED > + > +struct Charset_collation_map_st > +{ > +public: > + > + class Used > + { > + public: > + enum map_used_t > + { > + USED_NONE= 0, > + USED_COMPILED_COLLATION= 1 << 0, > + USED_MAPPED_COLLATION= 1 << 1 > + }; > + protected: > + map_used_t m_used; > + public: > + Used() > + :m_used(USED_NONE) > + { } > + void add(map_used_t flag) > + { > + m_used= (map_used_t) ((uint) m_used | (uint) flag); > + } > + }; > + > + struct Elem_st > + { > + protected: > + CHARSET_INFO *m_charset; > + CHARSET_INFO *m_collation; What is the difference between charset and collation? I was under impression that any CHARSET_INFO describes a collation, there is no special structure just for a charset. Am I wrong? > + static size_t print_lex_string(char *dst, const LEX_CSTRING &str) > + { > + memcpy(dst, str.str, str.length); > + return str.length; > + } > + public: > + /* > + Size in text format: 'utf8mb4=utf8mb4_unicode_ai_ci' > + */ > + static constexpr size_t text_size_max() > + { > + return MY_CS_CHARACTER_SET_NAME_SIZE + 1 + > + MY_CS_COLLATION_NAME_SIZE; > + } > + CHARSET_INFO *charset() const > + { > + return m_charset; > + } > + CHARSET_INFO *collation() const > + { > + return m_collation; > + } > + void set_collation(CHARSET_INFO *cl) > + { > + m_collation= cl; > + } > + size_t print(char *dst) const > + { > + const char *dst0= dst; > + dst+= print_lex_string(dst, m_charset->cs_name); > + *dst++= '='; > + dst+= print_lex_string(dst, m_collation->coll_name); > + return (size_t) (dst - dst0); > + } > + int cmp_by_charset_id(const Elem_st &rhs) const > + { > + return m_charset->number < rhs.m_charset->number ? -1 : > + m_charset->number > rhs.m_charset->number ? +1 : 0; > + } > + }; > + class Elem: public Elem_st > + { > + public: > + Elem(CHARSET_INFO *charset, CHARSET_INFO *collation) > + { > + m_charset= charset; > + m_collation= collation; > + } > + }; > +protected: > + Elem_st m_element[8]; // Should be enough for now > + uint m_count; > + uint m_version; > + > + static int cmp_by_charset_id(const void *a, const void *b) > + { > + return static_cast<const Elem_st*>(a)-> > + cmp_by_charset_id(*static_cast<const Elem_st*>(b)); > + } > + > + void sort() > + { > + qsort(m_element, m_count, sizeof(Elem_st), cmp_by_charset_id); > + } > + > + const Elem_st *find_elem_by_charset_id(uint id) const > + { > + if (!m_count) > + return NULL; > + int first= 0, last= ((int) m_count) - 1; > + for ( ; first <= last; ) > + { > + const int middle= (first + last) / 2; > + DBUG_ASSERT(middle >= 0); > + DBUG_ASSERT(middle < (int) m_count); > + const uint middle_id= m_element[middle].charset()->number; > + if (middle_id == id) > + return &m_element[middle]; > + if (middle_id < id) > + first= middle + 1; > + else > + last= middle - 1; > + } > + return NULL; > + } > + > + bool insert(const Elem_st &elem) > + { > + DBUG_ASSERT(elem.charset()->state & MY_CS_PRIMARY); > + if (m_count >= array_elements(m_element)) > + return true; > + m_element[m_count]= elem; > + m_count++; > + sort(); > + return false; > + } > + > + bool insert_or_replace(const Elem_st &elem) > + { > + DBUG_ASSERT(elem.charset()->state & MY_CS_PRIMARY); > + const Elem_st *found= find_elem_by_charset_id(elem.charset()->number); > + if (found) > + { > + const_cast<Elem_st*>(found)->set_collation(elem.collation()); > + return false; > + } > + return insert(elem); > + } > + > +public: > + void init() > + { > + m_count= 0; > + m_version= 0; > + } > + uint count() const > + { > + return m_count; > + } > + uint version() const > + { > + return m_version; > + } > + void set(const Charset_collation_map_st &rhs, uint version_increment) > + { > + uint version= m_version; > + *this= rhs; > + m_version= version + version_increment; > + } > + const Elem_st & operator[](uint pos) const > + { > + DBUG_ASSERT(pos < m_count); > + return m_element[pos]; > + } > + bool insert_or_replace(const class Lex_exact_charset &cs, > + const class Lex_extended_collation &cl, > + bool error_on_conflicting_duplicate); > + CHARSET_INFO *get_collation_for_charset(Used *used, > + CHARSET_INFO *cs) const > + { > + DBUG_ASSERT(cs->state & MY_CS_PRIMARY); > + const Elem_st *elem= find_elem_by_charset_id(cs->number); > + if (elem) > + { > + used->add(Used::USED_MAPPED_COLLATION); > + return elem->collation(); > + } > + used->add(Used::USED_COMPILED_COLLATION); > + return cs; > + } > + size_t text_format_nbytes_needed() const > + { > + return (Elem_st::text_size_max() + 1/* for ',' */) * m_count; > + } > + size_t print(char *dst, size_t nbytes_available) const > + { > + const char *dst0= dst; > + const char *end= dst + nbytes_available; > + for (uint i= 0; i < m_count; i++) > + { > + if (Elem_st::text_size_max() + 1/* for ',' */ > (size_t) (end - dst)) > + break; > + if (i > 0) > + *dst++= ','; > + dst+= m_element[i].print(dst); > + } > + return dst - dst0; > + } > + static constexpr size_t binary_size_max() > + { > + return 1/*count*/ + 4 * array_elements(m_element); > + } > + size_t to_binary(char *dst) const > + { > + const char *dst0= dst; > + *dst++= (char) (uchar) m_count; > + for (uint i= 0; i < m_count; i++) > + { > + int2store(dst, (uint16) m_element[i].charset()->number); > + dst+= 2; > + int2store(dst, (uint16) m_element[i].collation()->number); > + dst+= 2; > + } > + return (size_t) (dst - dst0); > + } > + size_t from_binary(const char *src, size_t srclen) > + { > + const char *src0= src; > + init(); > + if (!srclen) > + return 0; // Empty > + uint count= (uchar) *src++; > + if (srclen < 1 + 4 * count) > + return 0; > + for (uint i= 0; i < count; i++, src+= 4) > + { > + CHARSET_INFO *cs, *cl; > + if (!(cs= get_charset(uint2korr(src), MYF(0))) || > + !(cl= get_charset(uint2korr(src + 2), MYF(0)))) > + { > + /* > + Unpacking from binary format happens on the slave side. > + If for some reasons the slave does not know about a > + character set or a collation, just skip the pair here. > + This pair might not even be needed. > + */ > + continue; > + } > + insert_or_replace(Elem(cs, cl)); > + } > + return src - src0; > + } > + bool from_text(const LEX_CSTRING &str, myf utf8_flag); > +}; > + > + > +#endif // LEX_CHARSET_COLLATIONS_INCLUDED Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 3

Re: d8e04ef367b: MDEV-25237 Assertion `global_system_variables.session_track_system_variables' failed in Session_sysvars_tracker::init | SIGSEGV's in __strlen_avx2 | UBSAN: runtime error: null pointer passed as argument 1, which is declared to never be null in my_strdup
by Sergei Golubchik 07 Jul '23

07 Jul '23

Hi, Oleksandr, On Jul 07, Oleksandr Byelkin wrote: > revision-id: d8e04ef367b (mariadb-10.4.30-9-gd8e04ef367b) > parent(s): 423c28f0aa4 > author: Oleksandr Byelkin > committer: Oleksandr Byelkin > timestamp: 2023-07-03 17:50:49 +0200 > message: > > MDEV-25237 Assertion `global_system_variables.session_track_system_variables' failed in Session_sysvars_tracker::init | SIGSEGV's in __strlen_avx2 | UBSAN: runtime error: null pointer passed as argument 1, which is declared to never be null in my_strdup first, if you see a bug with a very long summary, particularly if it's basically makes no sense to a user - in that case, please, don't hesitate to make it shorter and more meaningful. For example "crash after setting session_track_system_variables to an invalid value" > --- a/sql/sys_vars.inl > +++ b/sql/sys_vars.inl > @@ -620,7 +620,12 @@ class Sys_var_sesvartrack: public Sys_var_charptr_base > { > if (sysvartrack_global_update(thd, new_val, > var->save_result.string_value.length)) > - new_val= 0; > + { > + if (new_val) > + my_free(new_val); > + // keep the old value in case of error > + new_val= (char*)my_strdup(global_var(const char*), MYF(MY_WME)); > + } No, this is wrong. Well, not wrong, but an incorrect bug fix. update should not fail, this is the contract. check() is supposed to verify that the value is valid, so basically update must succeed. technically it can fail because of OOM, I suppose, so restoring the value here might be needed still. But in the case of OOM, strdup() might be not such a great idea. Perhaps you can make sure that OOM is impossible and update truly cannot fail? Anyway, the actual fix for MDEV-25237 is to make check fail on an invalid value. In fact, it was a simple typo: @@ -221,7 +221,7 @@ bool sysvartrack_validate_value(THD *thd, const char *str, > /* Remove leading/trailing whitespace. */ trim_whitespace(system_charset_info, &var); - if (!strcmp(var.str, "*") && !find_sys_var(thd, var.str, var.length)) + if (strcmp(var.str, "*") && !find_sys_var(thd, var.str, var.length)) return true; > } > global_update_finish(new_val); > return (new_val == 0 && var->save_result.string_value.str != 0); Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: d23555b1920: MDEV-30984 Online ALTER table is denied with non-informative error messages
by Sergei Golubchik 06 Jul '23

06 Jul '23

Hi, Nikita, On Jun 24, Nikita Malyavin wrote: > revision-id: d23555b1920 (mariadb-11.0.1-138-gd23555b1920) > parent(s): df90bf34644 > author: Nikita Malyavin > committer: Nikita Malyavin > timestamp: 2023-06-20 16:10:48 +0300 > message: > > MDEV-30984 Online ALTER table is denied with non-informative error messages > > diff --git a/mysql-test/main/alter_table_online.result b/mysql-test/main/alter_table_online.result > --- a/mysql-test/main/alter_table_online.result > +++ b/mysql-test/main/alter_table_online.result > @@ -179,25 +179,27 @@ t CREATE TABLE `t` ( > UNIQUE KEY `b` (`b`) > ) ENGINE=InnoDB DEFAULT CHARSET=latin1 COLLATE=latin1_swedish_ci > alter table t drop b, change c c serial, algorithm=copy, lock=none; > -ERROR 0A000: LOCK=NONE is not supported. Reason: COPY algorithm requires a lock. Try LOCK=SHARED > +ERROR 0A000: LOCK=NONE is not supported. Reason: CHANGE COLUMN ... AUTO_INCREMENT. Try LOCK=SHARED > # Check existed unique keys. > create or replace table t(a int, b int not null, c int not null, d int); > # No unique in the old table; > alter table t add unique(b, c), modify d int auto_increment, add key(d), > algorithm=copy, lock=none; > -ERROR 0A000: LOCK=NONE is not supported. Reason: COPY algorithm requires a lock. Try LOCK=SHARED > +ERROR 0A000: LOCK=NONE is not supported. Reason: CHANGE COLUMN ... AUTO_INCREMENT. Try LOCK=SHARED > alter table t add unique(a, b); > # Unique in the old table has nulls; > alter table t modify d int auto_increment, add key(d), > algorithm=copy, lock=none; > -ERROR 0A000: LOCK=NONE is not supported. Reason: COPY algorithm requires a lock. Try LOCK=SHARED > +ERROR 0A000: LOCK=NONE is not supported. Reason: CHANGE COLUMN ... AUTO_INCREMENT. Try LOCK=SHARED > alter table t add unique(b, c); > # Change unique'scolumn; Typo > -alter table t change b x int, modify d int auto_increment, add key(d), > +alter table t change b x bigint, modify d int auto_increment, add key(d), > algorithm=copy, lock=none; > -ERROR 0A000: LOCK=NONE is not supported. Reason: COPY algorithm requires a lock. Try LOCK=SHARED > -# Finally good. > -alter table t modify d int auto_increment, add key(d), > +ERROR 0A000: LOCK=NONE is not supported. Reason: CHANGE COLUMN ... AUTO_INCREMENT. Try LOCK=SHARED > +# Finally good. Simple renames with a type unchenged will not affect typo s/unchenged/unchanged/ > +# the result. Also NOT NULL -> NULL transform is fine. > +alter table t modify d int auto_increment, add key(d), > +change b x int null, > algorithm=copy, lock=none; > drop table t; > # MDEV-31172 Server crash or ASAN errors in online_alter_check_autoinc > diff --git a/sql/sql_class.h b/sql/sql_class.h > --- a/sql/sql_class.h > +++ b/sql/sql_class.h > @@ -1228,6 +1228,15 @@ class Query_arena > { return strdup_root(mem_root,str); } > inline char *strmake(const char *str, size_t size) const > { return strmake_root(mem_root,str,size); } > + inline LEX_CSTRING strcat(const LEX_CSTRING &a, const LEX_CSTRING &b) const > + { > + char *buf= (char*)alloc(a.length + b.length + 1); > + if (unlikely(!buf)) > + return null_clex_str; > + strncpy(buf, a.str, a.length); memcpy both ↑ and ↓ (and then you might need to append \0 explicitly) > + strncpy(buf + a.length, b.str, b.length + 1); > + return {buf, a.length + b.length}; > + } > inline void *memdup(const void *str, size_t size) const > { return memdup_root(mem_root,str,size); } > inline void *memdup_w_gap(const void *str, size_t size, size_t gap) const > diff --git a/sql/sql_table.cc b/sql/sql_table.cc > --- a/sql/sql_table.cc > +++ b/sql/sql_table.cc > @@ -9869,26 +9869,21 @@ bool online_alter_check_autoinc(const THD *thd, const Alter_info *alter_info, > */ > for (uint k= 0; k < table->s->keys; k++) > { > - const KEY &key= table->s->key_info[k]; > + const KEY &key= table->key_info[k]; This is generally incorrect. User specified keys are in table->s->key_info[] Keys in table->key_info[] can be modified to match what indexes are actually created, in particular, for long uniques table->key_info[] differs from table->s->key_info[] > if ((key.flags & HA_NOSAME) == 0 || key.flags & HA_NULL_PART_KEY) > continue; > bool key_parts_good= true; > for (uint kp= 0; kp < key.user_defined_key_parts && key_parts_good; kp++) > { > - const char *field_name= key.key_part[kp].field->field_name.str; > - for (const auto &c: alter_info->drop_list) > - if (c.type == Alter_drop::COLUMN > - && my_strcasecmp(system_charset_info, c.name, field_name) == 0) > - { > - key_parts_good= false; > - break; > - } > + const Field *f= key.key_part[kp].field; > + // tmp_set contains dropped fields after mysql_prepare_alter_table > + key_parts_good= !bitmap_is_set(&table->tmp_set, f->field_index); > + > if (key_parts_good) > for (const auto &c: alter_info->create_list) > - if (c.change.str && my_strcasecmp(system_charset_info, c.change.str, > - field_name) == 0) > + if (c.field == f) > { > - key_parts_good= false; > + key_parts_good= f->is_equal(c); > break; > } > } > @@ -9896,25 +9891,63 @@ bool online_alter_check_autoinc(const THD *thd, const Alter_info *alter_info, > return true; > } > > - const char *old_autoinc= table->found_next_number_field > - ? table->found_next_number_field->field_name.str > - : ""; > - bool online= true; > for (const auto &c: alter_info->create_list) > { > - if (c.change.str && c.flags & AUTO_INCREMENT_FLAG) > + if (c.flags & AUTO_INCREMENT_FLAG) > { > - if (my_strcasecmp(system_charset_info, c.change.str, old_autoinc) != 0) > + if (c.field && !(c.field->flags & AUTO_INCREMENT_FLAG)) > + return false; > + break; > + } > + } > + return true; > +} > + > +static > +const char *online_alter_check_supported(const THD *thd, > + const Alter_info *alter_info, > + const TABLE *table, bool *online) > { > - if (c.create_if_not_exists // check IF EXISTS option > - && table->find_field_by_name(&c.change) == NULL) > - continue; > - online= false; > + DBUG_ASSERT(!table->s->tmp_table); that's confusing. *online can be false here already, then return "DROP SYSTEM VERSIONING" will be wrong and you use some other condition to decide whether "DROP SYSTEM VERSIONING" is correct or not. it'd be much clearer not to call online_alter_check_supported in this case. and add here DBUG_ASSERT(*online); > + > + *online= *online && (alter_info->flags & ALTER_DROP_SYSTEM_VERSIONING) == 0; and then you won't need to use && above, just as you don't use it below > + if (!*online) > + return "DROP SYSTEM VERSIONING"; > + > + *online= !thd->lex->ignore; > + if (!*online) > + return "ALTER IGNORE TABLE"; > + > + *online= !table->versioned(VERS_TRX_ID); > + if (!*online) > + return "BIGINT GENERATED ALWAYS AS ROW_START"; > + > + List<FOREIGN_KEY_INFO> fk_list; > + table->file->get_foreign_key_list(thd, &fk_list); > + for (auto &fk: fk_list) > + { > + if (fk_modifies_child(fk.delete_method) || > + fk_modifies_child(fk.update_method)) > + { > + *online= false; > + // Don't fall to a common unsupported case to avoid heavy string ops. > + if (alter_info->requested_lock == Alter_info::ALTER_TABLE_LOCK_NONE) > + { > + return fk_modifies_child(fk.delete_method) > + ? thd->strcat({STRING_WITH_LEN("ON DELETE ")}, > + *fk_option_name(fk.delete_method)).str > + : thd->strcat({STRING_WITH_LEN("ON UPDATE ")}, > + *fk_option_name(fk.update_method)).str; > } > - break; > + return NULL; > } > } > - return online; > + > + *online= online_alter_check_autoinc(thd, alter_info, table); > + if (!*online) > + return "CHANGE COLUMN ... AUTO_INCREMENT"; > + > + return NULL; > } > > > @@ -10979,14 +10994,14 @@ do_continue:; > > if (!table->s->tmp_table) > { > - // COPY algorithm doesn't work with concurrent writes. > + auto *reason= online_alter_check_supported(thd, alter_info, table, &online); > + // COPY algorithm works with concurrent writes only when online is true. as I wrote above, better put it inside if (online) { > if (!online && > alter_info->requested_lock == Alter_info::ALTER_TABLE_LOCK_NONE) > { > + DBUG_ASSERT(reason); > my_error(ER_ALTER_OPERATION_NOT_SUPPORTED_REASON, MYF(0), > - "LOCK=NONE", > - ER_THD(thd, ER_ALTER_OPERATION_NOT_SUPPORTED_REASON_COPY), > - "LOCK=SHARED"); > + "LOCK=NONE", reason, "LOCK=SHARED"); > goto err_new_table_cleanup; > } > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

3 7

[MariaDB developers]Re: a2e71bc8e89: MDEV-22979 MDEV-31400 Fixing spider init bugs
by Yuchen Pei 03 Jul '23

03 Jul '23

Hi Sergei, Thanks for the comments. I have addressed them and the url of the new patch can be found in the new request for review comment in the ticket. On Mon 2023-06-12 18:07:03 +0200, Sergei Golubchik wrote: > Hi, Yuchen, > > See comments inline. Here I'm only reviewing MDEV-31400, > please, tell me if you'd like me to review spider part too > Given that Alexey (cc'd) has already been working on the review of the spider init bug fix, I think it makes sense to ask Alexey to continue on the spider part, which I just did on MDEV-22979. Given that the spider part requires the patch for MDEV-31400 to work, I have the spider part on top of the init dependency part. > On Jun 12, Yuchen Pei wrote: >> revision-id: a2e71bc8e89 (mariadb-10.9.5-19-ga2e71bc8e89) >> parent(s): cb0e0c915f6 >> author: Yuchen Pei >> committer: Yuchen Pei >> timestamp: 2023-06-08 16:13:56 +1000 >> message: >> >> MDEV-22979 MDEV-31400 Fixing spider init bugs > > please, make a separate commit for MDEV-31400 Done. > >> diff --git a/sql/handler.cc b/sql/handler.cc >> index a12e9ea18f5..60080f1da6a 100644 >> --- a/sql/handler.cc >> +++ b/sql/handler.cc >> @@ -646,10 +648,14 @@ int ha_initialize_handlerton(st_plugin_int *plugin) >> hton->slot= HA_SLOT_UNDEF; >> /* Historical Requirement */ >> plugin->data= hton; // shortcut for the future >> - if (plugin->plugin->init && plugin->plugin->init(hton)) >> + if (plugin->plugin->init && (ret= plugin->plugin->init(hton))) >> { >> - sql_print_error("Plugin '%s' init function returned error.", >> - plugin->name.str); >> + if (unlikely(ret == -1)) >> + sql_print_warning("Plugin '%s' init function returned error but >> may be retried.", >> + plugin->name.str); >> + else >> + sql_print_error("Plugin '%s' init function returned error.", >> + plugin->name.str); > > let's treat all plugins identically. Meaning, in particular, no warnings > or errors in ha_initialize_handlerton(). They belong to sql_plugin.cc > Done. >> goto err; >> } >> >> diff --git a/sql/mysqld.cc b/sql/mysqld.cc >> index 32392ab882e..caf85736770 100644 >> --- a/sql/mysqld.cc >> +++ b/sql/mysqld.cc >> @@ -365,6 +365,7 @@ bool opt_disable_networking=0, opt_skip_show_db=0; >> bool opt_skip_name_resolve=0; >> my_bool opt_character_set_client_handshake= 1; >> bool opt_endinfo, using_udf_functions; >> +bool udf_initialized= 0; > > I don't think you need that. You can use exising mysqld_server_started. > That won't work, because mysqld_server_started will only be turned on very late, towards the end of mysqld_main(). If spider is initialised between udf_init() and mysqld_server_started is 1 (for example in an init-file), then the udf will fail. See https://github.com/MariaDB/server/commit/91176a7be7c where the test udf_mysql_func_early fails with "query 'SELECT SPIDER_DIRECT_SQL('select * from tbl_a', 'results', 'srv "s_2_1", database "auto_test_remote"')' failed: ER_SP_DOES_NOT_EXIST (1305): FUNCTION auto_test_local.SPIDER_DIRECT_SQL does not exist" because of this. Now, I assume MDEV-31401, the task to include the udf insertion trick in early calls CREATE FUNCTION, will fix this issue without introduce any new global booleans because it should has access to the internal variable `initialized`, though I do not know how complex or doable this task is. So I guess there's a trade-off here between fixing the spider init bugs now and investigating MDEV-31401. Would you like me to explore MDEV-31401 first? >> my_bool locked_in_memory; >> bool opt_using_transactions; >> bool volatile abort_loop; >> diff --git a/sql/sql_plugin.cc b/sql/sql_plugin.cc >> index 5a077a934ac..e564ab5a38d 100644 >> --- a/sql/sql_plugin.cc >> +++ b/sql/sql_plugin.cc >> @@ -1462,10 +1462,16 @@ static int plugin_initialize(MEM_ROOT *tmp_root, struct st_plugin_int *plugin, >> >> if (plugin_type_initialize[plugin->plugin->type]) >> { >> - if ((*plugin_type_initialize[plugin->plugin->type])(plugin)) >> + ret= (*plugin_type_initialize[plugin->plugin->type])(plugin); >> + if (ret) >> { >> - sql_print_error("Plugin '%s' registration as a %s failed.", >> - plugin->name.str, plugin_type_names[plugin->plugin->type].str); >> + /* Plugin init failed but requested a retry if possible */ >> + if (unlikely(ret == -1)) >> + sql_print_warning("Plugin '%s' registration as a %s failed but may be retried.", >> + plugin->name.str, plugin_type_names[plugin->plugin->type].str); >> + else >> + sql_print_error("Plugin '%s' registration as a %s failed.", >> + plugin->name.str, plugin_type_names[plugin->plugin->type].str); > > 1. "may be" is confusing, we should be less vague in the error > messages Done. I have expanded it a bit more to the following but we still really use "will be" here without either losing accuracy or being overly verbose. Even if it fails and returns the retry code, the retry will not happen if no new plugins are successfully initialised this round. --8<---------------cut here---------------start------------->8--- sql_print_warning("Plugin '%s' registration as a %s failed and may " "be retried provided it is being loaded with " "plugin-load[-add]", plugin->name.str, plugin_type_names[plugin->plugin->type].str); --8<---------------cut here---------------end--------------->8--- > > 2. about (ret == -1). I looked at what plugin init functions return now: > 33 plugins don't have an init function > 76 plugins always return 0 > 13 plugins return 0 or 1 > 3 plugins return 0 or -1 > 3 plugins return 0 or HA_ERR_INITIALIZATION, HA_ERR_OUT_OF_MEM > and innodb inconsistently return 0, HA_ERR_xxx as above, or 1 > > so, I suggest to introduce, like, HA_ERR_DEPENDENCIES (or whatever) > and only retry plugins that return that specific error code. > Done. The HA_ERR_ namespace is tightly packed and bounded by HA_ERR_FIRST (120) and HA_ERR_LAST (198), and I crammed in a #define HA_ERR_RETRY_INIT 129. >> goto err; >> } >> } >> @@ -1462,8 +1462,14 @@ static int plugin_initialize(MEM_ROOT *tmp_root, struct st_plugin_int *plugin, >> >> if (plugin_type_initialize[plugin->plugin->type]) >> { >> - if ((*plugin_type_initialize[plugin->plugin->type])(plugin)) >> + ret= (*plugin_type_initialize[plugin->plugin->type])(plugin); > > there's a second branch below, with plugin->plugin->init(), > please, handle it too. May be, do as with deinit? > > plugin_type_init deinit= plugin_type_deinitialize[plugin->plugin->type]; > if (!deinit) > deinit= (plugin_type_init)(plugin->plugin->deinit); > > if (deinit && deinit(plugin)) > > Less code duplication here. > Done. >> + if (ret) >> { >> + /* Plugin init failed but requested a retry if possible */ >> + if (unlikely(ret == -1)) >> + sql_print_warning("Plugin '%s' registration as a %s failed but may be retried.", >> + plugin->name.str, plugin_type_names[plugin->plugin->type].str); >> + else >> sql_print_error("Plugin '%s' registration as a %s failed.", >> plugin->name.str, plugin_type_names[plugin->plugin->type].str); >> goto err; >> @@ -1737,11 +1743,34 @@ int plugin_init(int *argc, char **argv, int flags) >> */ >> >> mysql_mutex_lock(&LOCK_plugin); >> + /* List of plugins to reap */ >> reap= (st_plugin_int **) my_alloca((plugin_array.elements+1) * sizeof(void*)); >> *(reap++)= NULL; >> + /* List of plugins to retry */ >> + retry= (st_plugin_int **) my_alloca((plugin_array.elements+1) * sizeof(void*)); >> + *(retry++)= NULL; >> >> for(;;) >> { >> + /* Number of plugins that is successfully initialised in a round */ >> + int num_initialized; >> + do >> + { >> + num_initialized= 0; >> + /* If any plugins failed and requested a retry, clean up before >> + retry */ >> + while ((plugin_ptr= *(--retry))) >> + { >> + mysql_mutex_unlock(&LOCK_plugin); >> + plugin_deinitialize(plugin_ptr, true); >> + mysql_mutex_lock(&LOCK_plugin); >> + /** Needed to satisfy assertions in `test_plugin_options()` */ > > Hmm, I don't think it'll work. I mean, test_plugin_options() - as far as > I remember - will remove processed options from the command line array, > meaning you cannot call it twice for the same plugin. > > May be it'd be better to repeat only plugin init function call and not > the complete initialize/deinitialize cycle? Done. I separated out a plugin_do_initialize() function that skips the test_options() part and largely mirrors the plugin_deinitialize() function, and calls this function instead of the full plugin_initialize() during a retry. I also made sure the code path outside of plugin_init() does not change, i.e. if plugin_initialize() is called from an INSTALL statement. The contract of init and deinit is not clear, but I suppose it is a reasonable expectation that if a plugin init wants to be retried, it will clean up before returning init instead of relying the server to call the deinit before retrying. > >> + my_afree(plugin_ptr->ptr_backup); >> + plugin_ptr->ptr_backup= NULL; >> + plugin_ptr->nbackups= 0; >> + plugin_ptr->state= PLUGIN_IS_UNINITIALIZED; >> + } >> + retry++; >> for (i=0; i < MYSQL_MAX_PLUGIN_TYPE_NUM; i++) >> { >> HASH *hash= plugin_hash + plugin_type_initialization_order[i]; > > Regards, > Sergei > VP of MariaDB Server Engineering > and security(a)mariadb.org Best, Yuchen

2 3

Re: 8b5de389ab1: MDEV-31400 Simple plugin dependency resolution
by Sergei Golubchik 03 Jul '23

03 Jul '23

Hi, Yuchen, On Jul 01, Yuchen Pei wrote: > revision-id: 8b5de389ab1 (mariadb-10.9.5-17-g8b5de389ab1) > parent(s): d8997f875e2 > author: Yuchen Pei > committer: Yuchen Pei > timestamp: 2023-06-13 20:08:28 +1000 > message: > > MDEV-31400 Simple plugin dependency resolution > > We introduce simple plugin dependency. A plugin init function may > return HA_ERR_RETRY_INIT. If this happens during server startup when > the server is trying to initialise all plugins, the failed plugins > will be retried, until no more plugins succeed in initialisation or > want to be retried. > > This will fix spider init bugs which is caused in part by its > dependency on Aria for initialisation. > > The reason we need a new return code, instead of treating every > failure as a request for retry, is that it may be impossible to clean > up after a failed plugin initialisation. Take InnoDB for example, it > has a global variable `buf_page_cleaner_is_active`, which may not > satisfy an assertion during a second initialisation try, probably > because InnoDB does not expect the initialisation to be called > twice. A test that may fail because of this is > `encryption.corrupted_during_recovery`, see for example[1], which is > tested at 73835f64b7fc245d38812380685aca03bef72bb5, a previous commit again, commits that you are going to push into the main branche should not refer to commits that you aren't going to push. these are dangling references leading nowhere. > where we retry every failed plugin. > > [1] https://buildbot.mariadb.org/#/builders/369/builds/10107/steps/7/logs/stdio nor references to buildbot logs, they're very short lived, while I can see commits in the server git repo going to 2000. > > diff --git a/sql/sql_plugin.cc b/sql/sql_plugin.cc > index 5a077a934ac..92d74aa51e8 100644 > --- a/sql/sql_plugin.cc > +++ b/sql/sql_plugin.cc > @@ -1435,6 +1435,49 @@ void plugin_unlock_list(THD *thd, plugin_ref *list, size_t count) > DBUG_VOID_RETURN; > } > > +static int plugin_do_initialize(struct st_plugin_int *plugin, uint &state) > +{ > + DBUG_ENTER("plugin_do_initialize"); > + mysql_mutex_assert_not_owner(&LOCK_plugin); > + plugin_type_init init= plugin_type_initialize[plugin->plugin->type]; > + if (!init) > + init= (plugin_type_init) plugin->plugin->init; > + if (init) > + if (int ret= init(plugin)) > + { > + /* Plugin init failed but requested a retry if possible */ > + if (unlikely(ret== HA_ERR_RETRY_INIT)) > + sql_print_warning("Plugin '%s' registration as a %s failed and may " please, don't write "may". It's saying "eh, I don't know, you, user, figure it out". But you should know better than the user. So either say "will be retried" or "won't be retried" (in which case, you can just use the error message below and not mention retry at all). > + "be retried provided it is being loaded with " > + "plugin-load[-add]", > + plugin->name.str, plugin_type_names[plugin->plugin->type].str); > + else > + sql_print_error("Plugin '%s' registration as a %s failed.", > + plugin->name.str, plugin_type_names[plugin->plugin->type].str); > + DBUG_RETURN(ret); > + } > + state= PLUGIN_IS_READY; // plugin->init() succeeded > + > + if (plugin->plugin->status_vars) > + { > + /* > + historical ndb behavior caused MySQL plugins to specify > + status var names in full, with the plugin name prefix. > + this was never fixed in MySQL. > + MariaDB fixes that but supports MySQL style too. > + */ > + SHOW_VAR *show_vars= plugin->plugin->status_vars; > + SHOW_VAR tmp_array[2]= {{plugin->plugin->name, > + (char *) plugin->plugin->status_vars, SHOW_ARRAY}, > + {0, 0, SHOW_UNDEF}}; > + if (strncasecmp(show_vars->name, plugin->name.str, plugin->name.length)) > + show_vars= tmp_array; > + > + if (add_status_vars(show_vars)) > + DBUG_RETURN(1); > + } > + DBUG_RETURN(0); > +} > > static int plugin_initialize(MEM_ROOT *tmp_root, struct st_plugin_int *plugin, > int *argc, char **argv, bool options_only) > @@ -1737,32 +1738,71 @@ int plugin_init(int *argc, char **argv, int flags) > */ > > mysql_mutex_lock(&LOCK_plugin); > + /* List of plugins to reap */ > reap= (st_plugin_int **) my_alloca((plugin_array.elements+1) * sizeof(void*)); > *(reap++)= NULL; > + /* List of plugins to retry */ > + retry= (st_plugin_int **) my_alloca((plugin_array.elements+1) * sizeof(void*)); > + *(retry++)= NULL; > > for(;;) > { > + /* Number of plugins that is successfully initialised in a round */ > + int num_initialized; > + do > + { > + num_initialized= 0; > + /* If any plugins failed and requested a retry, clean up before > + retry */ > + while ((plugin_ptr= *(--retry))) > + plugin_ptr->state= PLUGIN_IS_TO_BE_RETRIED; Why do you need a special state for that? You haven't fixed the SHOW PLUGINS command to show it correctly. And I didn't check other conditions, may be some of them need to take the new state into account too, I don't know. Wouldn't it be easier to have a special retry look after the main for() without all its complexity and without a new state. Like (pseudocode, and note that here I've changed the creative trick of walking retry[] backwards until NULL, so now it iterated forward until end): while (retry_start < retry_end) { retry_ptr= retry= retry_start; while ((plugin_ptr= *(retry++))) { mysql_mutex_unlock(&LOCK_plugin); error= plugin_do_initialize(plugin_ptr, state); mysql_mutex_lock(&LOCK_plugin); if (error == HA_ERR_RETRY_INIT) *(retry_ptr++)= plugin_ptr; else if (error) *(reap++)= plugin_ptr; } if (retry == retry_ptr) while (retry_ptr > retry_start) *(reap++)= *(--retry_ptr); retry_end= retry_ptr; } > + retry++; > for (i=0; i < MYSQL_MAX_PLUGIN_TYPE_NUM; i++) > { > HASH *hash= plugin_hash + plugin_type_initialization_order[i]; > for (uint idx= 0; idx < hash->records; idx++) > { > plugin_ptr= (struct st_plugin_int *) my_hash_element(hash, idx); > + if (plugin_ptr->state == PLUGIN_IS_UNINITIALIZED || > + plugin_ptr->state == PLUGIN_IS_TO_BE_RETRIED) > + { > + int error; > if (plugin_ptr->state == PLUGIN_IS_UNINITIALIZED) > { > bool plugin_table_engine= > lex_string_eq(&plugin_table_engine_name, &plugin_ptr->name); > bool opts_only= flags & PLUGIN_INIT_SKIP_INITIALIZATION && > (flags & PLUGIN_INIT_SKIP_PLUGIN_TABLE || > !plugin_table_engine); > - if (plugin_initialize(&tmp_root, plugin_ptr, argc, argv, opts_only)) > + error= plugin_initialize(&tmp_root, plugin_ptr, argc, argv, > + opts_only); > + } else > { > + uint state= plugin_ptr->state; > + mysql_mutex_unlock(&LOCK_plugin); > + error= plugin_do_initialize(plugin_ptr, state); > + mysql_mutex_lock(&LOCK_plugin); > + plugin_ptr->state= state; > + } > + if (error) > + { > plugin_ptr->state= PLUGIN_IS_DYING; > + /* The plugin wants a retry of the initialisation, > + possibly due to dependency on other plugins */ > + if (unlikely(error == HA_ERR_RETRY_INIT)) > + *(retry++)= plugin_ptr; > + else > *(reap++)= plugin_ptr; > } > + else > + num_initialized++; > } > } > } > + /* Only retry if at least one plugin has been initialised > + successfully and at least one has requested a retry during this > + round */ > + } while (num_initialized > 0 && *(retry - 1)); > > /* load and init plugins from the plugin table (unless done already) */ > if (flags & PLUGIN_INIT_SKIP_PLUGIN_TABLE) > @@ -1775,8 +1815,11 @@ int plugin_init(int *argc, char **argv, int flags) > } > > /* > - Check if any plugins have to be reaped > + Merge the retry list to the reap list, then reap the failed > + plugins. Note that during the merge we reverse the order in retry > */ > + while ((plugin_ptr= *(--retry))) > + *(reap++) = plugin_ptr; I already have that in my pseudocode above > while ((plugin_ptr= *(--reap))) > { > mysql_mutex_unlock(&LOCK_plugin); Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Numeric datatype with large decimal values
by brajmohan saxena 23 Jun '23

23 Jun '23

Hi Folks, Do we have Or someone already working on creating a Numeric (decimal) data type plugin in MariaDB, Which supports large decimal digits and may be upto infinite. It would be great if someone already has some pointers on it. And let me know the best approach to achieve this type of data type implementation in MariaDB. Thanks in advance. Thanks Braj mohan saxena

1 0

Re: f3aa4d0a215: MDEV-29447 MDEV-26285 MDEV-31338 Refactor spider_db_mbase_util::open_item_func
by Sergei Golubchik 22 Jun '23

22 Jun '23

Hi, Yuchen, On Jun 20, Yuchen Pei wrote: > revision-id: f3aa4d0a215 (mariadb-10.4.30-4-gf3aa4d0a215) > parent(s): f5dceafd0ba > author: Yuchen Pei > committer: Yuchen Pei > timestamp: 2023-06-14 15:02:38 +1000 > message: > > MDEV-29447 MDEV-26285 MDEV-31338 Refactor spider_db_mbase_util::open_item_func > > spider_db_mbase_util::open_item_func() is a monster function. > It is difficult to maintain while it is expected that we need to > modify it when a new SQL function or a new func_type is added. > > We split the function into two distinct functions: one handles the > case of str != NULL and the other handles the case of str == NULL. > > This refactoring was done in a conservative way because we do not > have comprehensive tests on the function. > > It also fixes MDEV-29447 and MDEV-31338 where field items that are > arguments of a func item may be used before created / initialised. > > Note this commit is a port of > 3836098c29ef1b7ff9d5fbde99b690eab73a0df1 (MDEV-26285) to current > versions 10.3+. As I wrote in the first review, please, don't mention transient commit hashes in a permanent commit comment. This commit will, supposedly, be pushed into the main branch. While 3836098c29ef1 will, on the other hand, disappear, as it's not in any of the main branches. You'll have a dangling pointer > Signed-off-by: Yuchen Pei <yuchen.pei(a)mariadb.com> also, it looks strange to sign-off your own commits > > diff --git a/storage/spider/spd_db_mysql.cc b/storage/spider/spd_db_mysql.cc > index a51c1497b02..1ed733105c6 100644 > --- a/storage/spider/spd_db_mysql.cc > +++ b/storage/spider/spd_db_mysql.cc > @@ -5848,94 +5992,12 @@ int spider_db_mbase_util::open_item_func( > { > if (!strncasecmp("utc_timestamp", func_name, func_name_length)) > { > - if (str) > str->length(str->length() - SPIDER_SQL_OPEN_PAREN_LEN); > DBUG_RETURN(spider_db_open_item_string(item_func, NULL, spider, str, > alias, alias_length, dbton_id, use_fields, fields)); > } else if (!strncasecmp("timestampdiff", func_name, func_name_length)) > { > -#ifdef ITEM_FUNC_TIMESTAMPDIFF_ARE_PUBLIC Why did you removed this and not the #else part? > - Item_func_timestamp_diff *item_func_timestamp_diff = > - (Item_func_timestamp_diff *) item_func; > - if (str) > - { > - const char *interval_str; > - uint interval_len; > - switch (item_func_timestamp_diff->int_type) > - { > - case INTERVAL_YEAR: > - interval_str = SPIDER_SQL_YEAR_STR; > - interval_len = SPIDER_SQL_YEAR_LEN; > - break; > - case INTERVAL_QUARTER: > - interval_str = SPIDER_SQL_QUARTER_STR; > - interval_len = SPIDER_SQL_QUARTER_LEN; > - break; > - case INTERVAL_MONTH: > - interval_str = SPIDER_SQL_MONTH_STR; > - interval_len = SPIDER_SQL_MONTH_LEN; > - break; > - case INTERVAL_WEEK: > - interval_str = SPIDER_SQL_WEEK_STR; > - interval_len = SPIDER_SQL_WEEK_LEN; > - break; > - case INTERVAL_DAY: > - interval_str = SPIDER_SQL_DAY_STR; > - interval_len = SPIDER_SQL_DAY_LEN; > - break; > - case INTERVAL_HOUR: > - interval_str = SPIDER_SQL_HOUR_STR; > - interval_len = SPIDER_SQL_HOUR_LEN; > - break; > - case INTERVAL_MINUTE: > - interval_str = SPIDER_SQL_MINUTE_STR; > - interval_len = SPIDER_SQL_MINUTE_LEN; > - break; > - case INTERVAL_SECOND: > - interval_str = SPIDER_SQL_SECOND_STR; > - interval_len = SPIDER_SQL_SECOND_LEN; > - break; > - case INTERVAL_MICROSECOND: > - interval_str = SPIDER_SQL_MICROSECOND_STR; > - interval_len = SPIDER_SQL_MICROSECOND_LEN; > - break; > - default: > - interval_str = ""; > - interval_len = 0; > - break; > - } > - str->length(str->length() - SPIDER_SQL_OPEN_PAREN_LEN); > - if (str->reserve(func_name_length + SPIDER_SQL_OPEN_PAREN_LEN + > - interval_len + SPIDER_SQL_COMMA_LEN)) > - DBUG_RETURN(HA_ERR_OUT_OF_MEM); > - str->q_append(func_name, func_name_length); > - str->q_append(SPIDER_SQL_OPEN_PAREN_STR, SPIDER_SQL_OPEN_PAREN_LEN); > - str->q_append(interval_str, interval_len); > - str->q_append(SPIDER_SQL_COMMA_STR, SPIDER_SQL_COMMA_LEN); > - } > - if ((error_num = spider_db_print_item_type(item_list[0], NULL, spider, > - str, alias, alias_length, dbton_id, use_fields, fields))) > - DBUG_RETURN(error_num); > - if (str) > - { > - if (str->reserve(SPIDER_SQL_COMMA_LEN)) > - DBUG_RETURN(HA_ERR_OUT_OF_MEM); > - str->q_append(SPIDER_SQL_COMMA_STR, SPIDER_SQL_COMMA_LEN); > - } > - if ((error_num = spider_db_print_item_type(item_list[1], NULL, spider, > - str, alias, alias_length, dbton_id, use_fields, fields))) > - DBUG_RETURN(error_num); > - if (str) > - { > - if (str->reserve(SPIDER_SQL_CLOSE_PAREN_LEN)) > - DBUG_RETURN(HA_ERR_OUT_OF_MEM); > - str->q_append(SPIDER_SQL_CLOSE_PAREN_STR, > - SPIDER_SQL_CLOSE_PAREN_LEN); > - } > - DBUG_RETURN(0); > -#else > DBUG_RETURN(ER_SPIDER_COND_SKIP_NUM); > -#endif > } > } else if (func_name_length == 14) > { Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 3

DBA in Tokyo
by James Tobin 20 Jun '23

20 Jun '23

Hello, I'm working with an employer that is looking to hire a DBA in Tokyo. Consequently, I had hoped some members of this list may like to discuss. I can be reached using "JamesBTobin (at) Gmail (dot) Com". Kind regards, James

1 0

[Maria-developers] Review input for: MDEV-22534 Decorrelate IN subquery
by Sergey Petrunia 01 Jun '23

01 Jun '23

Hi Yuchen, Please find below review input for > commit ba9cf8efeb0b1e1c132d565adb79c98156783f15 > Author: Yuchen Pei <yuchen.pei(a)mariadb.com> > Date: Fri May 19 20:06:31 2023 +1000 > > MDEV-22534 Decorrelate IN subquery > First, general questions: Q1: It seems there is some overlap between Item_in_subselect::exists2in_processor() and Item_exists_subselect::exists2in_processor() Did you consider factoring out common code rather than copying it? (if yes, I'd like to know why it couldn't be done). Q2: Where is the coe that removes the correlated equalities from the subquery's WHERE? (I assume Item_exists_subselect code does it?) > diff --git a/sql/item_subselect.cc b/sql/item_subselect.cc > index 9e6c205ca76..8d893794da3 100644 > --- a/sql/item_subselect.cc > +++ b/sql/item_subselect.cc > @@ -3099,6 +3099,132 @@ static bool find_inner_outer_equalities(Item **conds, > return TRUE; > } > > +/* Checks whether item tree intersects with the free list */ > +static bool intersects_free_list(Item *item, THD *thd) > +{ > + for (const Item *to_find= thd->free_list; to_find; to_find= to_find->next) > + if (item->walk(&Item::find_item_processor, 1, (void *) to_find)) > + return true; > + return false; > +} > + Please try moving this function into a separate file. First name that came to my mind: opt_subq_decorrelate.cc Feel free to suggest better names. The idea is that we should aim for better structure. (This request is only valid if the factoring out suggested above doesn't work out). > +/* De-correlate where conditions in an IN subquery */ This needs a bigger comment describing what the rewrite does. > +bool Item_in_subselect::exists2in_processor(void *opt_arg) > +{ > + THD *thd= (THD *)opt_arg; > + SELECT_LEX *first_select= unit->first_select(); > + JOIN *join= first_select->join; > + bool will_be_correlated; > + Dynamic_array<EQ_FIELD_OUTER> eqs(PSI_INSTRUMENT_MEM, 5, 5); > + List<Item> outer; > + Query_arena *arena= NULL, backup; > + int res= FALSE; > + DBUG_ENTER("Item_in_subselect::exists2in_processor"); > + Please move comments out of the if statement and number the conditions. For examples, see best_access_path(), large if-statement starting with "Don't test table scan if it can't be better." or "EXISTS-to-IN coversion and ORDER BY ... LIMIT clause" in Item_exists_subselect::exists2in_processor(). > + if (!optimizer_flag(thd, OPTIMIZER_SWITCH_EXISTS_TO_IN) || > + /* proceed only if I'm a toplevel IN or a toplevel NOT IN */ > + !(is_top_level_item() || > + (upper_not && upper_not->is_top_level_item())) || what is the reason behind this condition? If we're able to handle top-level NOT IN, why can't we handle subquery in any context? Can we really handle the NOT IN? (I suspect not, it will suffer from the semantics of IN subqueries with NULLs.. I can't find the name for this, it's described e.g. here: https://petrunia.net/2006/11/16/inany-subqueries-null-woes/ ) > + first_select->is_part_of_union() || /* skip if part of a union */ > + first_select->group_list.elements || /* skip if with group by */ > + first_select->with_sum_func || /* skip if aggregation */ > + join->having || /* skip if with having */ > + !join->conds || /* skip if no conds */ > + /* skip if left expr is a single nonscalar subselect */ > + (left_expr->type() == Item::SUBSELECT_ITEM && > + !left_expr->type_handler()->is_scalar_type())) The above seems to be some exotic SQL. Is it something like (select col1,lol2 from t1) IN (select col3,col4 from t2 where ..) and does MariaDB really support this? Please add a comment about this. > + DBUG_RETURN(FALSE); > + /* iterate over conditions, and check whether they can be moved out. */ > + if (find_inner_outer_equalities(&join->conds, eqs)) > + DBUG_RETURN(FALSE); > + /* If we are in the execution of a prepared statement, check for > + intersection with the free list to avoid segfault. Note that the > + check for prepared statement execution is necessary, otherwise it > + will likely always find intersection thus abort the > + transformation. > + > + fixme: this only works when HAVE_PSI_STATEMENT_INTERFACE is > + defined. It causes CI's like amd64-debian-10-debug-embedded to > + fail. Are there other ways to find out we are in the execution of a > + prepared statement? */ I'm looking at the code of activate_stmt_arena_if_needed(). It seems "stmt_arena->is_conventional()" is the check you're looking for. > + if (thd->m_statement_state.m_parent_prepared_stmt) > + { > + for (uint i= 0; i < (uint) eqs.elements(); i++) > + { > + if (intersects_free_list(*eqs.at(i).eq_ref, thd)) > + DBUG_RETURN(FALSE); This check doesn't look good. As far as I understand, it is a temporary measure until related re-execution bug(s) are fixed by Igor (and/or Sanja)? > + } > + } > + /* Determines whether the result will be correlated */ > + { > + List<Item> unused; > + Collect_deps_prm prm= {&unused, // parameters > + unit->first_select()->nest_level_base, // nest_level_base > + 0, // count > + unit->first_select()->nest_level, // nest_level > + FALSE // collect > + }; > + walk(&Item::collect_outer_ref_processor, TRUE, &prm); > + DBUG_ASSERT(prm.count > 0); > + DBUG_ASSERT(prm.count >= (uint)eqs.elements()); > + will_be_correlated= prm.count > (uint)eqs.elements(); See the example pasted in the MDEV: https://jira.mariadb.org/browse/MDEV-22534?focusedCommentId=259705&page=com… Somehow, I get here will_be_correlated==FALSE but Materialization is still not considered for the subquery? > + } > + /* Back up the free list */ > + arena= thd->activate_stmt_arena_if_needed(&backup); > + /* Construct the items for left_expr */ > + if (left_expr->type() == Item::ROW_ITEM) > + for (uint i= 0; i < left_expr->cols(); i++) > + outer.push_back(left_expr->element_index(i)); > + else > + outer.push_back(left_expr); > + const uint offset= first_select->item_list.elements; > + /* Move items to outer and select item list */ > + for (uint i= 0; i < (uint)eqs.elements(); i++) > + { > + Item **eq_ref= eqs.at(i).eq_ref; > + Item_ident *local_field= eqs.at(i).local_field; > + Item *outer_exp= eqs.at(i).outer_exp; > + first_select->item_list.push_back(local_field, thd->mem_root); > + first_select->ref_pointer_array[offset + i]= (Item *)local_field; How do we know that ref_pointer_array has enough room for the new elements? (We can discuss this with Sanja, he's an expert on the topic). I see this check in Item_exists_subselect::exists2in_processor if ((uint)eqs.elements() > (first_select->item_list.elements + first_select->select_n_reserved)) Is it how Item_exists_subselect handles this issue? > + outer.push_back(outer_exp); > + *eq_ref= new (thd->mem_root) Item_int(thd, 1); > + if((*eq_ref)->fix_fields(thd, (Item **)eq_ref)) > + { > + res= TRUE; > + goto out; > + } > + } > + /* Update the left_expr */ > + left_expr= new (thd->mem_root) Item_row(thd, outer); > + if (left_expr->fix_fields(thd, &left_expr)) > + { > + res= TRUE; > + goto out; > + } > + left_expr_orig= left_expr; > + is_correlated= will_be_correlated; > + /* Update any Item_in_optimizer wrapper if exists */ > + if (optimizer) > + { > + optimizer->reset_cache(); > + if (optimizer->fix_left(thd)) > + { > + res= TRUE; > + goto out; > + } > + } > + { > + OPT_TRACE_TRANSFORM(thd, trace_wrapper, trace_transform, > + get_select_lex()->select_number, "IN (SELECT)", > + "decorrelation"); > + } > +out: > + /* Restores the free list */ > + if (arena) > + thd->restore_active_arena(arena, &backup); > + DBUG_RETURN(res); > +} On Mon, Dec 20, 2021 at 09:29:47PM +0300, Sergey Petrunia wrote: > Hello Igor, > > I'm trying examples with this patch: > > create table t1 (a int not null, key(a)); > insert into t1 select seq from seq_1_to_1000; > > First, I debug this example: > > explain select * from t1 where a <10 or a>=10; > > The execution enters key_or(), passes through the new code this patch has added > and key_or() returns NULL. Good. > > But if I modify the example slightly: > > explain select * from t1 where a <20 or a>=10 ; > > Here, key_or() still returns a SEL_ARG object that represents an infinite > range: > > (gdb) fini > Run till exit from #0 key_or (...) > 0x0000555555f1fca2 in tree_or (...) > Value returned is $24 = (SEL_ARG *) 0x7fff70064d88 > (gdb) p dbug_print_sel_arg($24) > $26 = 0x7fff70024390 "SEL_ARG(-inf<=a<=+inf)" > > Should this case be fixed as well? > > On Fri, Dec 17, 2021 at 02:11:40PM -0800, IgorBabaev wrote: > > revision-id: 677643a80986107491b7886441f2828384f0494b (mariadb-10.2.31-1286-g677643a) > > parent(s): 8bb55633699612279744c055e22eeca8d4058273 > > author: Igor Babaev > > committer: Igor Babaev > > timestamp: 2021-12-17 14:11:39 -0800 > > message: > > > > MDEV-27262 Unexpected index intersection with full index scan for an index > > > > If when extracting a range condition foran index from the WHERE condition > > I think the comment wording could be improved also. > > > Range Optimizer sees that the range condition covers the whole index then > > such condition should be discarded because cannot it be used in any range > > scan. In some cases Range Optimizer really does it, but there remained some > > conditions for which it was not done. As a result the optimizer could > > produce index merge plans with the full index scan for one of the indexes > > participating in the index merge. > > This could be observed in one of the test cases from index_merge1.inc > > where a plan with index_merge_sort_union was produced and in the test case > > reported for this bug where a plan with index_merge_sort_intersect was > > produced. In both cases one of two index scans participating in index merge > > ran over the whole index. > > The patch slightly changes the original above mentioned test case from > > index_merge1.inc to be able to produce an intended plan employing > > index_merge_sort_union. The original query was left to show that index > > merge is not used for it anymore. > > It should be noted that for the plan with index_merge_sort_intersect could > > be chosen for execution only due to a defect in the InnoDB code that > > returns wrong estimates for the cardinality of big ranges. > > > > This bug led to serious problems in 10.4+ where the optimization using > > Rowid filters is employed (see mdev-26446). > > > > Approved by Oleksandr Byelkin <sanja(a)mariadb.com> > > > > --- > > mysql-test/include/index_merge1.inc | 8 +++- > > mysql-test/r/index_merge_myisam.result | 12 +++-- > > mysql-test/r/range_innodb.result | 81 ++++++++++++++++++++++++++++++++++ > > mysql-test/t/range_innodb.test | 78 ++++++++++++++++++++++++++++++++ > > sql/opt_range.cc | 7 +++ > > 5 files changed, 181 insertions(+), 5 deletions(-) > > > > diff --git a/mysql-test/include/index_merge1.inc b/mysql-test/include/index_merge1.inc > > index b168a76..440f1f7 100644 > > --- a/mysql-test/include/index_merge1.inc > > +++ b/mysql-test/include/index_merge1.inc > > @@ -150,15 +150,19 @@ explain select * from t0 where > > (((key3 <7 and key7 < 6) or key5 < 2) and (key5 < 5 or key6 < 6)); > > > > explain select * from t0 where > > - ((key3 <5 or key5 < 4) and (key1 < 4 or key2 < 4)) > > + ((key3 < 4 or key5 < 4) and (key1 < 4 or key2 < 4)) > > or > > ((key3 >=5 or key5 < 2) and (key5 < 5 or key6 < 6)); > > > > explain select * from t0 force index(i1, i2, i3, i4, i5, i6 ) where > > - ((key3 <5 or key5 < 4) and (key1 < 4 or key2 < 4)) > > + ((key3 < 4 or key5 < 4) and (key1 < 4 or key2 < 4)) > > or > > ((key3 >=5 or key5 < 2) and (key5 < 5 or key6 < 6)); > > > > +explain select * from t0 force index(i1, i2, i3, i4, i5, i6 ) where > > + ((key3 < 5 or key5 < 4) and (key1 < 4 or key2 < 4)) > > + or > > + ((key3 >=5 or key5 < 2) and (key5 < 5 or key6 < 6)); > > # 8. Verify that "order by" after index merge uses filesort > > select * from t0 where key1 < 5 or key8 < 4 order by key1; > > > > diff --git a/mysql-test/r/index_merge_myisam.result b/mysql-test/r/index_merge_myisam.result > > index 5a23092..a096c34 100644 > > --- a/mysql-test/r/index_merge_myisam.result > > +++ b/mysql-test/r/index_merge_myisam.result > > @@ -173,17 +173,23 @@ or > > id select_type table type possible_keys key key_len ref rows Extra > > 1 SIMPLE t0 index_merge i1,i2,i3,i5,i6,i7 i3,i5 4,4 NULL 11 Using sort_union(i3,i5); Using where > > explain select * from t0 where > > -((key3 <5 or key5 < 4) and (key1 < 4 or key2 < 4)) > > +((key3 < 4 or key5 < 4) and (key1 < 4 or key2 < 4)) > > or > > ((key3 >=5 or key5 < 2) and (key5 < 5 or key6 < 6)); > > id select_type table type possible_keys key key_len ref rows Extra > > 1 SIMPLE t0 ALL i1,i2,i3,i5,i6 NULL NULL NULL 1024 Using where > > explain select * from t0 force index(i1, i2, i3, i4, i5, i6 ) where > > -((key3 <5 or key5 < 4) and (key1 < 4 or key2 < 4)) > > +((key3 < 4 or key5 < 4) and (key1 < 4 or key2 < 4)) > > or > > ((key3 >=5 or key5 < 2) and (key5 < 5 or key6 < 6)); > > id select_type table type possible_keys key key_len ref rows Extra > > -1 SIMPLE t0 index_merge i1,i2,i3,i5,i6 i3,i5 0,4 NULL 1024 Using sort_union(i3,i5); Using where > > +1 SIMPLE t0 index_merge i1,i2,i3,i5,i6 i3,i5 4,4 NULL 1024 Using sort_union(i3,i5); Using where > > +explain select * from t0 force index(i1, i2, i3, i4, i5, i6 ) where > > +((key3 < 5 or key5 < 4) and (key1 < 4 or key2 < 4)) > > +or > > +((key3 >=5 or key5 < 2) and (key5 < 5 or key6 < 6)); > > +id select_type table type possible_keys key key_len ref rows Extra > > +1 SIMPLE t0 ALL i1,i2,i3,i5,i6 NULL NULL NULL 1024 Using where > > select * from t0 where key1 < 5 or key8 < 4 order by key1; > > key1 key2 key3 key4 key5 key6 key7 key8 > > 1 1 1 1 1 1 1 1023 > > diff --git a/mysql-test/r/range_innodb.result b/mysql-test/r/range_innodb.result > > index f2349f2..ccb6da3 100644 > > --- a/mysql-test/r/range_innodb.result > > +++ b/mysql-test/r/range_innodb.result > > @@ -108,3 +108,84 @@ DROP TABLE t0,t1; > > SET @@GLOBAL.debug_dbug = @saved_dbug; > > set @@optimizer_switch= @optimizer_switch_save; > > # End of 10.1 tests > > +# > > +# MDEV-27262: Index intersection with full scan over an index > > +# > > +CREATE TABLE t1 ( > > +id int(10) unsigned NOT NULL AUTO_INCREMENT, > > +p char(32) DEFAULT NULL, > > +es tinyint(3) unsigned NOT NULL DEFAULT 0, > > +er tinyint(3) unsigned NOT NULL DEFAULT 0, > > +x mediumint(8) unsigned NOT NULL DEFAULT 0, > > +PRIMARY KEY (id), > > +INDEX es (es), > > +INDEX x (x), > > +INDEX er (er,x), > > +INDEX p (p) > > +) ENGINE=InnoDB DEFAULT CHARSET=latin1; > > +insert into t1(es,er) select 0, 1 from seq_1_to_45; > > +insert into t1(es,er) select 0, 2 from seq_1_to_49; > > +insert into t1(es,er) select 0, 3 from seq_1_to_951; > > +insert into t1(es,er) select 0, 3 from seq_1_to_1054; > > +insert into t1(es,er) select 0, 6 from seq_1_to_25; > > +insert into t1(es,er) select 0, 11 from seq_1_to_1; > > +insert into t1(es,er) select 1, 1 from seq_1_to_45; > > +insert into t1(es,er) select 1, 2 from seq_1_to_16; > > +insert into t1(es,er) select 1, 3 from seq_1_to_511; > > +insert into t1(es,er) select 1, 4 from seq_1_to_687; > > +insert into t1(es,er) select 1, 6 from seq_1_to_50; > > +insert into t1(es,er) select 1, 7 from seq_1_to_4; > > +insert into t1(es,er) select 1, 11 from seq_1_to_1; > > +insert into t1(es,er) select 2, 1 from seq_1_to_82; > > +insert into t1(es,er) select 2, 2 from seq_1_to_82; > > +insert into t1(es,er) select 2, 3 from seq_1_to_1626; > > +insert into t1(es,er) select 2, 4 from seq_1_to_977; > > +insert into t1(es,er) select 2, 6 from seq_1_to_33; > > +insert into t1(es,er) select 2, 11 from seq_1_to_1; > > +insert into t1(es,er) select 3, 1 from seq_1_to_245; > > +insert into t1(es,er) select 3, 2 from seq_1_to_81; > > +insert into t1(es,er) select 3, 3 from seq_1_to_852; > > +insert into t1(es,er) select 3, 4 from seq_1_to_2243; > > +insert into t1(es,er) select 3, 6 from seq_1_to_44; > > +insert into t1(es,er) select 3, 11 from seq_1_to_1; > > +insert into t1(es,er) select 4, 1 from seq_1_to_91; > > +insert into t1(es,er) select 4, 2 from seq_1_to_83; > > +insert into t1(es,er) select 4, 3 from seq_1_to_297; > > +insert into t1(es,er) select 4, 4 from seq_1_to_2456; > > +insert into t1(es,er) select 4, 6 from seq_1_to_19; > > +insert into t1(es,er) select 4, 11 from seq_1_to_1; > > +update t1 set p='foobar'; > > +update t1 set x=0; > > +set @save_isp=@@innodb_stats_persistent; > > +set global innodb_stats_persistent= 1; > > +analyze table t1; > > +Table Op Msg_type Msg_text > > +test.t1 analyze status OK > > +set optimizer_switch='index_merge_sort_intersection=on'; > > +SELECT * FROM t1 > > +WHERE ((p = 'foo' AND er != 4) OR er = 4 ) AND (es >= 4) LIMIT 2; > > +id p es er x > > +14645 foobar 4 4 0 > > +14646 foobar 4 4 0 > > +EXPLAIN EXTENDED SELECT * FROM t1 > > +WHERE ((p = 'foo' AND er != 4) OR er = 4 ) AND (es >= 4) LIMIT 2; > > +id select_type table type possible_keys key key_len ref rows filtered Extra > > +1 SIMPLE t1 range es,er,p es 1 NULL # 100.00 Using index condition; Using where > > +Warnings: > > +Note 1003 select `test`.`t1`.`id` AS `id`,`test`.`t1`.`p` AS `p`,`test`.`t1`.`es` AS `es`,`test`.`t1`.`er` AS `er`,`test`.`t1`.`x` AS `x` from `test`.`t1` where (`test`.`t1`.`p` = 'foo' and `test`.`t1`.`er` <> 4 or `test`.`t1`.`er` = 4) and `test`.`t1`.`es` >= 4 limit 2 > > +set optimizer_switch='index_merge_sort_intersection=off'; > > +SELECT * FROM t1 > > +WHERE ((p = 'foo' AND er != 4) OR er = 4 ) AND (es >= 4) LIMIT 2; > > +id p es er x > > +14645 foobar 4 4 0 > > +14646 foobar 4 4 0 > > +EXPLAIN EXTENDED SELECT * FROM t1 > > +WHERE ((p = 'foo' AND er != 4) OR er = 4 ) AND (es >= 4) LIMIT 2; > > +id select_type table type possible_keys key key_len ref rows filtered Extra > > +1 SIMPLE t1 range es,er,p es 1 NULL # 100.00 Using index condition; Using where > > +Warnings: > > +Note 1003 select `test`.`t1`.`id` AS `id`,`test`.`t1`.`p` AS `p`,`test`.`t1`.`es` AS `es`,`test`.`t1`.`er` AS `er`,`test`.`t1`.`x` AS `x` from `test`.`t1` where (`test`.`t1`.`p` = 'foo' and `test`.`t1`.`er` <> 4 or `test`.`t1`.`er` = 4) and `test`.`t1`.`es` >= 4 limit 2 > > +set optimizer_switch='index_merge_sort_intersection=default'; > > +set global innodb_stats_persistent= @save_isp; > > +DROP TABLE t1; > > +# End of 10.2 tests > > diff --git a/mysql-test/t/range_innodb.test b/mysql-test/t/range_innodb.test > > index 428e5c2..7420e72 100644 > > --- a/mysql-test/t/range_innodb.test > > +++ b/mysql-test/t/range_innodb.test > > @@ -116,3 +116,81 @@ SET @@GLOBAL.debug_dbug = @saved_dbug; > > set @@optimizer_switch= @optimizer_switch_save; > > > > --echo # End of 10.1 tests > > + > > +--echo # > > +--echo # MDEV-27262: Index intersection with full scan over an index > > +--echo # > > + > > +--source include/have_sequence.inc > > + > > +CREATE TABLE t1 ( > > + id int(10) unsigned NOT NULL AUTO_INCREMENT, > > + p char(32) DEFAULT NULL, > > + es tinyint(3) unsigned NOT NULL DEFAULT 0, > > + er tinyint(3) unsigned NOT NULL DEFAULT 0, > > + x mediumint(8) unsigned NOT NULL DEFAULT 0, > > + PRIMARY KEY (id), > > + INDEX es (es), > > + INDEX x (x), > > + INDEX er (er,x), > > + INDEX p (p) > > +) ENGINE=InnoDB DEFAULT CHARSET=latin1; > > + > > +insert into t1(es,er) select 0, 1 from seq_1_to_45; > > +insert into t1(es,er) select 0, 2 from seq_1_to_49; > > +insert into t1(es,er) select 0, 3 from seq_1_to_951; > > +insert into t1(es,er) select 0, 3 from seq_1_to_1054; > > +insert into t1(es,er) select 0, 6 from seq_1_to_25; > > +insert into t1(es,er) select 0, 11 from seq_1_to_1; > > +insert into t1(es,er) select 1, 1 from seq_1_to_45; > > +insert into t1(es,er) select 1, 2 from seq_1_to_16; > > +insert into t1(es,er) select 1, 3 from seq_1_to_511; > > +insert into t1(es,er) select 1, 4 from seq_1_to_687; > > +insert into t1(es,er) select 1, 6 from seq_1_to_50; > > +insert into t1(es,er) select 1, 7 from seq_1_to_4; > > +insert into t1(es,er) select 1, 11 from seq_1_to_1; > > +insert into t1(es,er) select 2, 1 from seq_1_to_82; > > +insert into t1(es,er) select 2, 2 from seq_1_to_82; > > +insert into t1(es,er) select 2, 3 from seq_1_to_1626; > > +insert into t1(es,er) select 2, 4 from seq_1_to_977; > > +insert into t1(es,er) select 2, 6 from seq_1_to_33; > > +insert into t1(es,er) select 2, 11 from seq_1_to_1; > > +insert into t1(es,er) select 3, 1 from seq_1_to_245; > > +insert into t1(es,er) select 3, 2 from seq_1_to_81; > > +insert into t1(es,er) select 3, 3 from seq_1_to_852; > > +insert into t1(es,er) select 3, 4 from seq_1_to_2243; > > +insert into t1(es,er) select 3, 6 from seq_1_to_44; > > +insert into t1(es,er) select 3, 11 from seq_1_to_1; > > +insert into t1(es,er) select 4, 1 from seq_1_to_91; > > +insert into t1(es,er) select 4, 2 from seq_1_to_83; > > +insert into t1(es,er) select 4, 3 from seq_1_to_297; > > +insert into t1(es,er) select 4, 4 from seq_1_to_2456; > > +insert into t1(es,er) select 4, 6 from seq_1_to_19; > > +insert into t1(es,er) select 4, 11 from seq_1_to_1; > > +update t1 set p='foobar'; > > +update t1 set x=0; > > +set @save_isp=@@innodb_stats_persistent; > > +set global innodb_stats_persistent= 1; > > +analyze table t1; > > + > > +let $q= > > +SELECT * FROM t1 > > + WHERE ((p = 'foo' AND er != 4) OR er = 4 ) AND (es >= 4) LIMIT 2; > > + > > +set optimizer_switch='index_merge_sort_intersection=on'; > > +eval $q; > > +--replace_column 9 # > > +eval EXPLAIN EXTENDED $q; > > + > > +set optimizer_switch='index_merge_sort_intersection=off'; > > +# execution of $q and explain for it led to an assertion failure in 10.4 > > +# (with the optimizer switch rowid_filter set to 'on') > > +eval $q; > > +--replace_column 9 # > > +eval EXPLAIN EXTENDED $q; > > +set optimizer_switch='index_merge_sort_intersection=default'; > > + > > +set global innodb_stats_persistent= @save_isp; > > +DROP TABLE t1; > > + > > +--echo # End of 10.2 tests > > diff --git a/sql/opt_range.cc b/sql/opt_range.cc > > index f3f1843..2e05b88 100644 > > --- a/sql/opt_range.cc > > +++ b/sql/opt_range.cc > > @@ -9413,6 +9413,13 @@ key_or(RANGE_OPT_PARAM *param, SEL_ARG *key1,SEL_ARG *key2) > > key2->copy_min(tmp); > > if (!(key1=key1->tree_delete(tmp))) > > { // Only one key in tree > > + if (key2->min_flag & NO_MIN_RANGE && > > + key2->max_flag & NO_MAX_RANGE) > > + { > > + if (key2->maybe_flag) > > + return new SEL_ARG(SEL_ARG::MAYBE_KEY); > > + return 0; // Always true OR > > + } > > key1=key2; > > key1->make_root(); > > key2=key2_next; > > BR > Sergei > -- > Sergei Petrunia, Software Developer > MariaDB Corporation | Skype: sergefp | Blog: http://petrunia.net > > -- BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://petrunia.net

2 3

Re: [Maria-developers] c457f237511: MDEV-30984 Online ALTER table is denied with non-informative error messages
by Sergei Golubchik 29 May '23

29 May '23

Hi, Nikita, comments below refer to a combined diff of both commits On May 23, Nikita Malyavin wrote: > revision-id: c457f237511 (mariadb-11.0.1-124-gc457f237511) > parent(s): f4b04ec534d > author: Nikita Malyavin > committer: Nikita Malyavin > timestamp: 2023-05-15 21:07:50 +0300 > message: > > MDEV-30984 Online ALTER table is denied with non-informative error messages Two main thoughts. First, users will generally don't do show warnings after an error - mariadb already puts some useful info into a warning after an error in some (few) cases, and it's always a surprise when I write in a bug report "use SHOW WARNINGS". It's not very intuitive. And in your case it's even unnecessary, because your error message is LOCK=NONE is not supported. Reason: %s. Try LOCK=SHARED and "Reason" part is perfectly suited for an explanation that you generate, instead of a misleading "COPY algorithm requires a lock" (because it doesn't). Second thought, your messages are hard-coded and won't be translated even if the user will configure the server to use, say, spanish or Japanese error messages. See how it was solved for "COPY algorithm requires a lock", ER_ALTER_OPERATION_NOT_SUPPORTED_REASON_COPY. But I think the message will still be comprehensible if you'll simply print the offending sql clause as the reason, this won't need any localization. See below where I've shown what your error messages would look like if you do that. If you think that's too cryptic, then, please, use the ER_ALTER_OPERATION_NOT_SUPPORTED_REASON_COPY approach. Two more comments at the very end but I'll repeat them here, because they're easy to overlook: AUTO_INCREMENT instead of AUTOINC, and a test for DROP SYSTEM VERSIONING. > diff --git a/mysql-test/main/alter_table_online.result b/mysql-test/main/alter_table_online.result > index 8b726ccb591..bcf7afea9df 100644 > --- a/mysql-test/main/alter_table_online.result > +++ b/mysql-test/main/alter_table_online.result > @@ -4,6 +4,10 @@ > create table t (a int); > alter ignore table t add primary key (a), algorithm=copy, lock=none; > ERROR 0A000: LOCK=NONE is not supported. Reason: COPY algorithm requires a lock. Try LOCK=SHARED ERROR 0A000: LOCK=NONE is not supported. Reason: ALTER IGNORE. Try LOCK=SHARED > +show warnings; > +Level Code Message > +Note 1846 ALTER IGNORE TABLE is incompatible with LOCK=NONE, ALGORITHM=COPY > +Error 1846 LOCK=NONE is not supported. Reason: COPY algorithm requires a lock. Try LOCK=SHARED > drop table t; > # > # MDEV-28771 Assertion `table->in_use&&tdc->flushed' failed after ALTER > @@ -92,12 +96,20 @@ on update cascade) engine=InnoDB; > insert into t2 values (1),(2),(3); > alter table t2 add c int, algorithm=copy, lock=none; > ERROR 0A000: LOCK=NONE is not supported. Reason: COPY algorithm requires a lock. Try LOCK=SHARED ERROR 0A000: LOCK=NONE is not supported. Reason: FOREIGN KEY ... ON UPDATE CASCADE. Try LOCK=SHARED > +show warnings; > +Level Code Message > +Note 1846 Tables with CASCADE/SET NULL foreign keys are incompatible with LOCK=NONE, ALGORITHM=COPY > +Error 1846 LOCK=NONE is not supported. Reason: COPY algorithm requires a lock. Try LOCK=SHARED > alter table t2 add c int, algorithm=inplace, lock=none; > create or replace table t2 (b int, foreign key (b) > references t1 (a) > on delete set null) engine=InnoDB; > alter table t2 add c int, algorithm=copy, lock=none; > ERROR 0A000: LOCK=NONE is not supported. Reason: COPY algorithm requires a lock. Try LOCK=SHARED ERROR 0A000: LOCK=NONE is not supported. Reason: FOREIGN KEY ... ON DELETE SET NULL. Try LOCK=SHARED > +show warnings; > +Level Code Message > +Note 1846 Tables with CASCADE/SET NULL foreign keys are incompatible with LOCK=NONE, ALGORITHM=COPY > +Error 1846 LOCK=NONE is not supported. Reason: COPY algorithm requires a lock. Try LOCK=SHARED > alter table t2 add c int, algorithm=inplace, lock=none; > create or replace table t2 (b int, foreign key (b) > references t1 (a) > @@ -117,6 +129,10 @@ b int references t1 (b) on update cascade) engine=InnoDB; > insert into t2 values (1, 1),(2, 2); > alter table t2 add c int, algorithm=copy, lock=none; > ERROR 0A000: LOCK=NONE is not supported. Reason: COPY algorithm requires a lock. Try LOCK=SHARED ERROR 0A000: LOCK=NONE is not supported. Reason: FOREIGN KEY ... ON DELETE SET NULL. Try LOCK=SHARED > +show warnings; > +Level Code Message > +Note 1846 Tables with CASCADE/SET NULL foreign keys are incompatible with LOCK=NONE, ALGORITHM=COPY > +Error 1846 LOCK=NONE is not supported. Reason: COPY algorithm requires a lock. Try LOCK=SHARED > alter table t2 add c int, algorithm=copy; > alter table t2 add d int, algorithm=inplace; > drop table t2, t1; > @@ -132,6 +148,10 @@ period for system_time (row_start, row_end)) > engine=innodb with system versioning; > alter table t1 add c int, algorithm=copy, lock=none; > ERROR 0A000: LOCK=NONE is not supported. Reason: COPY algorithm requires a lock. Try LOCK=SHARED ERROR 0A000: LOCK=NONE is not supported. Reason: BIGINT GENERATED ALWAYS AS ROW_START. Try LOCK=SHARED > +show warnings; > +Level Code Message > +Note 1846 Transaction-versioned tables are incompatible with LOCK=NONE, ALGORITHM=COPY > +Error 1846 LOCK=NONE is not supported. Reason: COPY algorithm requires a lock. Try LOCK=SHARED > alter table t1 add c int, algorithm=inplace; > alter table t1 add d int, lock=none; > set system_versioning_alter_history= default; > @@ -143,6 +163,10 @@ drop table t1; > create table t (a serial, b int) engine=innodb; > alter table t drop a, modify b serial, algorithm=copy, lock=none; > ERROR 0A000: LOCK=NONE is not supported. Reason: COPY algorithm requires a lock. Try LOCK=SHARED ERROR 0A000: LOCK=NONE is not supported. Reason: MODIFY COLUMN ... SERIAL. Try LOCK=SHARED > +show warnings; > +Level Code Message > +Note 1846 Adding AUTOINC to an existing column for a table without a primary key is incompatible with LOCK=NONE, ALGORITHM=COPY > +Error 1846 LOCK=NONE is not supported. Reason: COPY algorithm requires a lock. Try LOCK=SHARED > set statement sql_mode= NO_AUTO_VALUE_ON_ZERO for > alter table t drop a, modify b serial, algorithm=copy, lock=none; > create or replace table t (a serial, b int) engine=innodb; > @@ -157,6 +181,10 @@ t CREATE TABLE `t` ( > # Only unsafe approach is fine because of possible collisions. > alter table t modify a int, modify b serial, algorithm=copy, lock=none; > ERROR 0A000: LOCK=NONE is not supported. Reason: COPY algorithm requires a lock. Try LOCK=SHARED ERROR 0A000: LOCK=NONE is not supported. Reason: MODIFY COLUMN ... SERIAL. Try LOCK=SHARED > +show warnings; > +Level Code Message > +Note 1846 Adding AUTOINC to an existing column for a table without a primary key is incompatible with LOCK=NONE, ALGORITHM=COPY > +Error 1846 LOCK=NONE is not supported. Reason: COPY algorithm requires a lock. Try LOCK=SHARED > # > # Check that we treat autoinc columns correctly modify old autoinc is > # fine, adding new autoinc for existed column is unsafe. > @@ -180,22 +208,38 @@ t CREATE TABLE `t` ( > ) ENGINE=InnoDB DEFAULT CHARSET=latin1 COLLATE=latin1_swedish_ci > alter table t drop b, change c c serial, algorithm=copy, lock=none; > ERROR 0A000: LOCK=NONE is not supported. Reason: COPY algorithm requires a lock. Try LOCK=SHARED ERROR 0A000: LOCK=NONE is not supported. Reason: MODIFY COLUMN ... SERIAL. Try LOCK=SHARED > +show warnings; > +Level Code Message > +Note 1846 Adding AUTOINC to an existing column for a table without a primary key is incompatible with LOCK=NONE, ALGORITHM=COPY > +Error 1846 LOCK=NONE is not supported. Reason: COPY algorithm requires a lock. Try LOCK=SHARED > # Check existed unique keys. > create or replace table t(a int, b int not null, c int not null, d int); > # No unique in the old table; > alter table t add unique(b, c), modify d int auto_increment, add key(d), > algorithm=copy, lock=none; > ERROR 0A000: LOCK=NONE is not supported. Reason: COPY algorithm requires a lock. Try LOCK=SHARED ERROR 0A000: LOCK=NONE is not supported. Reason: MODIFY COLUMN ... AUTO_INCREMENT. Try LOCK=SHARED > +show warnings; > +Level Code Message > +Note 1846 Adding AUTOINC to an existing column for a table without a primary key is incompatible with LOCK=NONE, ALGORITHM=COPY > +Error 1846 LOCK=NONE is not supported. Reason: COPY algorithm requires a lock. Try LOCK=SHARED > alter table t add unique(a, b); > # Unique in the old table has nulls; > alter table t modify d int auto_increment, add key(d), > algorithm=copy, lock=none; > ERROR 0A000: LOCK=NONE is not supported. Reason: COPY algorithm requires a lock. Try LOCK=SHARED ERROR 0A000: LOCK=NONE is not supported. Reason: MODIFY COLUMN ... AUTO_INCREMENT. Try LOCK=SHARED > +show warnings; > +Level Code Message > +Note 1846 Adding AUTOINC to an existing column for a table without a primary key is incompatible with LOCK=NONE, ALGORITHM=COPY > +Error 1846 LOCK=NONE is not supported. Reason: COPY algorithm requires a lock. Try LOCK=SHARED > alter table t add unique(b, c); > # Change unique'scolumn; > alter table t change b x int, modify d int auto_increment, add key(d), > algorithm=copy, lock=none; > ERROR 0A000: LOCK=NONE is not supported. Reason: COPY algorithm requires a lock. Try LOCK=SHARED ERROR 0A000: LOCK=NONE is not supported. Reason: MODIFY COLUMN ... AUTO_INCREMENT. Try LOCK=SHARED > +show warnings; > +Level Code Message > +Note 1846 Adding AUTOINC to an existing column for a table without a primary key is incompatible with LOCK=NONE, ALGORITHM=COPY > +Error 1846 LOCK=NONE is not supported. Reason: COPY algorithm requires a lock. Try LOCK=SHARED > # Finally good. > alter table t modify d int auto_increment, add key(d), > algorithm=copy, lock=none; > diff --git a/sql/sql_table.cc b/sql/sql_table.cc > index 99570008ddb..22447db42af 100644 > --- a/sql/sql_table.cc > +++ b/sql/sql_table.cc > @@ -9916,6 +9916,67 @@ bool online_alter_check_autoinc(const THD *thd, const Alter_info *alter_info, > return online; > } > > +bool online_alter_is_supported(THD *thd, const Alter_info *alter_info, > + const TABLE *table) > +{ > + const char *reason= NULL; > + List<FOREIGN_KEY_INFO> fk_list; > + > + if (alter_info->flags & ALTER_DROP_SYSTEM_VERSIONING) > + { > + reason= "DROP SYSTEM VERSIONING is"; no test for that > + goto unsupported; > + } > + > + if (thd->lex->ignore) > + { > + reason= "ALTER IGNORE TABLE is"; > + goto unsupported; > + } > + > + if (table->s->tmp_table) > + { > + reason= "Temporary tables are"; > + goto unsupported; > + } > + > + if (table->versioned(VERS_TRX_ID)) > + { > + reason= "Transaction-versioned tables are"; > + goto unsupported; > + } > + > + table->file->get_foreign_key_list(thd, &fk_list); > + for (auto &fk: fk_list) > + { > + if (fk_modifies_child(fk.delete_method) > + || fk_modifies_child(fk.update_method)) > + { > + reason= "Tables with CASCADE/SET NULL foreign keys are"; > + goto unsupported; > + } > + } > + > + if (!online_alter_check_autoinc(thd, alter_info, table)) > + { > + reason= "Adding AUTOINC to an existing column for a table without a " AUTO_INCREMENT, please, don't abbrev user visible messages > + "primary key is"; > + goto unsupported; > + } > + > + return true; > + > +unsupported: > + if (alter_info->requested_lock == Alter_info::ALTER_TABLE_LOCK_NONE > + && alter_info->algorithm(thd) == Alter_info::ALTER_TABLE_ALGORITHM_COPY) > + push_warning_printf(thd, Sql_condition::WARN_LEVEL_NOTE, > + ER_ALTER_OPERATION_NOT_SUPPORTED_REASON, > + "%s incompatible with " > + "LOCK=NONE, ALGORITHM=COPY", > + reason); > + return false; > +} > + > > /** > Alter table Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 4

Re: [Maria-developers] 56e2cd20ed3: MDEV-31033 ER_KEY_NOT_FOUND upon online COPY ALTER on a partitioned table
by Sergei Golubchik 29 May '23

29 May '23

Hi, Nikita, On May 24, Nikita Malyavin wrote: > revision-id: 56e2cd20ed3 (mariadb-11.0.1-114-g56e2cd20ed3) > parent(s): 47f5c7ae7e4 > author: Nikita Malyavin > committer: Nikita Malyavin > timestamp: 2023-05-10 01:49:42 +0300 > message: > > MDEV-31033 ER_KEY_NOT_FOUND upon online COPY ALTER on a partitioned table > > The row events were applied "twice": once for the ha_partition, and one > more time for the underlying storage engine. > > There's no such problem in binlog/rpl, because ha_partiton::row_logging > is normally set to false. > > The fix makes the events replicate only when the handler is a root handler. > We will try to *guess* this by comparing it to table->file. The same > approach is used in the MDEV-21540 fix, 231feabd. The assumption is made, > that the row methods are only called for table->file (and never for a > cloned handler), hence the assertions are added in ha_innobase and > ha_myisam to make sure that this is true at least for those engines > > Also closes MDEV-31040, however the test is not included, since we have no > convenient way to construct a deterministic version. > > diff --git a/sql/handler.cc b/sql/handler.cc > index 3fcbb5171d6..d8aca5bc80b 100644 > --- a/sql/handler.cc > +++ b/sql/handler.cc > @@ -7280,7 +7280,7 @@ int handler::binlog_log_row(const uchar *before_record, > log_func, row_logging_has_trans); > > #ifdef HAVE_REPLICATION > - if (unlikely(!error && table->s->online_alter_binlog)) > + if (unlikely(!error && table->s->online_alter_binlog && is_root_handler())) > error= binlog_log_row_online_alter(table, before_record, after_record, > log_func); > #endif // HAVE_REPLICATION > @@ -7800,7 +7800,7 @@ int handler::ha_write_row(const uchar *buf) > DBUG_RETURN(error); > > /* > - NOTE: this != table->file is true in 3 cases: > + NOTE: is_root_handler() is false in 3 cases: may be move this comment closer to is_root_handler() ? > > 1. under copy_partitions() (REORGANIZE PARTITION): that does not > require long unique check as it does not introduce new rows or new index. > @@ -7810,7 +7810,7 @@ int handler::ha_write_row(const uchar *buf) > 3. under ha_mroonga::wrapper_write_row() > */ > > - if (table->s->long_unique_table && this == table->file) > + if (table->s->long_unique_table && is_root_handler()) > { > DBUG_ASSERT(inited == NONE || lookup_handler != this); > if ((error= check_duplicate_long_entries(buf))) > @@ -7862,13 +7862,13 @@ int handler::ha_update_row(const uchar *old_data, const uchar *new_data) > error= ha_check_overlaps(old_data, new_data); > > /* > - NOTE: this != table->file is true under partition's ha_update_row(): > + NOTE: is_root_handler() is false under partition's ha_update_row(): > check_duplicate_long_entries_update() was already done by > ha_partition::ha_update_row(), no need to check it again for each single > partition. Same applies to ha_mroonga wrapper. > */ > > - if (!error && table->s->long_unique_table && this == table->file) > + if (!error && table->s->long_unique_table && is_root_handler()) > error= check_duplicate_long_entries_update(new_data); > table->status= saved_status; > > @@ -7913,6 +7913,12 @@ int handler::ha_update_row(const uchar *old_data, const uchar *new_data) > return error; > } > > + > +bool handler::is_root_handler() const > +{ > + return this == table->file; > +} better make it inline in handler.h > + > /* > Update first row. Only used by sequence tables > */ > diff --git a/sql/handler.h b/sql/handler.h > index fb0eba83c38..9b903e7ebce 100644 > --- a/sql/handler.h > +++ b/sql/handler.h > @@ -3502,6 +3502,10 @@ class handler :public Sql_alloc > return extra(HA_EXTRA_NO_KEYREAD); > } > > +protected: > + bool is_root_handler() const; > + > +public: > int check_collation_compatibility(); > int check_long_hash_compatibility() const; > int ha_check_for_upgrade(HA_CHECK_OPT *check_opt); > diff --git a/storage/innobase/handler/ha_innodb.cc b/storage/innobase/handler/ha_innodb.cc > index f035437b1d7..94d9da3486b 100644 > --- a/storage/innobase/handler/ha_innodb.cc > +++ b/storage/innobase/handler/ha_innodb.cc > @@ -7750,6 +7750,8 @@ ha_innobase::write_row( > > DBUG_ENTER("ha_innobase::write_row"); > > + DBUG_ASSERT(is_root_handler() || table->file->ht != innodb_hton_ptr); instead of putting this assert into every handler's implementation and in many methods, why not to have it in handler ha_ methods or may be even in the is_root_handler itself? You can compare this->ht with table->file->ht. For example, like DBUG_ASSERT(this == table->file || this->ht != table->file->ht); > + > trx_t* trx = thd_to_trx(m_user_thd); > > /* Validation checks before we commence write_row operation. */ > @@ -8479,6 +8481,8 @@ ha_innobase::update_row( > > DBUG_ENTER("ha_innobase::update_row"); > > + DBUG_ASSERT(is_root_handler() || table->file->ht != innodb_hton_ptr); > + > if (is_read_only()) { > DBUG_RETURN(HA_ERR_TABLE_READONLY); > } else if (!trx_is_started(trx)) { > @@ -8653,6 +8657,8 @@ ha_innobase::delete_row( > > DBUG_ENTER("ha_innobase::delete_row"); > > + DBUG_ASSERT(is_root_handler() || table->file->ht != innodb_hton_ptr); > + > if (is_read_only()) { > DBUG_RETURN(HA_ERR_TABLE_READONLY); > } else if (!trx_is_started(trx)) { > diff --git a/storage/myisam/ha_myisam.cc b/storage/myisam/ha_myisam.cc > index 4057c3ea859..b7fadd86a23 100644 > --- a/storage/myisam/ha_myisam.cc > +++ b/storage/myisam/ha_myisam.cc > @@ -983,6 +983,7 @@ int ha_myisam::close(void) > > int ha_myisam::write_row(const uchar *buf) > { > + DBUG_ASSERT(is_root_handler() || table->file->ht != ht); > /* > If we have an auto_increment column and we are writing a changed row > or a new row, then update the auto_increment value in the record. > @@ -1957,11 +1958,13 @@ bool ha_myisam::is_crashed() const > > int ha_myisam::update_row(const uchar *old_data, const uchar *new_data) > { > + DBUG_ASSERT(is_root_handler() || table->file->ht != ht); > return mi_update(file,old_data,new_data); > } > > int ha_myisam::delete_row(const uchar *buf) > { > + DBUG_ASSERT(is_root_handler() || table->file->ht != ht); > return mi_delete(file,buf); > } > > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 5

[Maria-developers] MTR fails on undefined {cpus} array in My/SysInfo.pm on armhf and armel
by Otto Kekäläinen 28 May '23

28 May '23

Hi! I noticed that the autopkgtests on Debian on armhf and armel that run the mariadb-test-run fail on error: starting mysql-test-tun.pl... Logging: ./mysql-test-run.pl --force --testcase-timeout=120 --suite-timeout=540 --retry=3 -- ... Collecting tests... Installing system database... Can't use an undefined value as an ARRAY reference at lib/My/SysInfo.pm line 166. Logs for full context: https://ci.debian.net/data/autopkgtest/unstable/armel/m/mariadb/33870767/lo… https://ci.debian.net/data/autopkgtest/unstable/armhf/m/mariadb/33870768/lo… This line 166 in mysql-test/lib/My/SysInfo.pm has: # Return the number of cpus found sub num_cpus { my ($self)= @_; return int(@{$self->{cpus}}) or confess "INTERNAL ERROR: No cpus in list"; } Can somebody with strong Perl skills help me understand 1) Where this cpus array is inherited from? 2) Why might it have now regressed in MariaDB 10.11.3 and why is this erroring only on armel/armhf but not other architectures? 3) What should I do wrap this section in so that if cpus is not set, it would just default to '1'? - Otto

3 6

Re: [Maria-developers] c75f774d278: MDEV-31058 ER_KEY_NOT_FOUND upon concurrent CHANGE column autoinc and DML
by Sergei Golubchik 27 May '23

27 May '23

Hi, Nikita, On May 24, Nikita Malyavin wrote: > revision-id: c75f774d278 (mariadb-11.0.1-118-gc75f774d278) > parent(s): 319207175bd > author: Nikita Malyavin > committer: Nikita Malyavin > timestamp: 2023-05-15 19:43:49 +0300 > message: > > MDEV-31058 ER_KEY_NOT_FOUND upon concurrent CHANGE column autoinc and DML > > When column is changed to autoinc, ALTER TABLE may update zero/NULL values, > if NO_AUTO_VALUE_ON_ZERO mode is not enabled. > > Forbid this for LOCK=NONE for the unreliable cases. > The cases are described in online_alter_check_autoinc. just disable online for any ALTER that adds an autoinc attribute to an existing column. See MDEV. > diff --git a/mysql-test/main/alter_table_online.test b/mysql-test/main/alter_table_online.test > index b9e147c812d..7ebe553aabb 100644 > --- a/mysql-test/main/alter_table_online.test > +++ b/mysql-test/main/alter_table_online.test > @@ -135,3 +135,71 @@ alter table t1 add d int, lock=none; > set system_versioning_alter_history= default; > drop table t1; > > +--echo # > +--echo # MDEV-31058 ER_KEY_NOT_FOUND upon concurrent CHANGE column autoinc > +--echo # and DML > +--echo # > +create table t (a serial, b int) engine=innodb; > +--error ER_ALTER_OPERATION_NOT_SUPPORTED_REASON > +alter table t drop a, modify b serial, algorithm=copy, lock=none; > + > +set statement sql_mode= NO_AUTO_VALUE_ON_ZERO for > +alter table t drop a, modify b serial, algorithm=copy, lock=none; I don't think it should work, see my comment in the MDEV > +--echo # Finally good. > +alter table t modify d int auto_increment, add key(d), > + algorithm=copy, lock=none; no, it's not good either, as far as I understand > + > +drop table t; lots of unvisible spaces at line ends, please remove them > + > +--echo # MDEV-31172 Server crash or ASAN errors in online_alter_check_autoinc > +create table t (a int, b int, c char(8), key(a,b,c)); > +alter table t modify c int auto_increment key, algorithm=copy; Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

[Maria-developers] Regression in crossbuilding MariaDB
by Otto Kekäläinen 26 May '23

26 May '23

Hello! While working on https://salsa.debian.org/mariadb-team/mariadb-server/-/merge_requests/46 (Import new upstream version 10.11.3 to Debian) I noticed that amd64->arm64 crossbuilding no longer works. The CI shows that Salsa-CI job `test-crossbuild-arm64` no longer passes. Based on Salsa-CI runs I can see that it [still worked in upstream commit e55e761e](https://salsa.debian.org/otto/mariadb-server/-/jobs/4217220) but [failed in upstream commit 06b443be](https://salsa.debian.org/otto/mariadb-server/-/jobs/4217664), thus some of these commits in between regressed it: ``` 06b443be34e Nes CC 3.3 4a668c18926 MDEV-29401 InnoDB history list length increased in 10.6 compared to 10.5 f272463b028 Cleanup of MDEV-14974: --port ignored for --host=localhost bb1d1dc8465 Merge 10.5 into 10.6 c5e50c48bba MDEV-30837: Remove usage of AWK in autobake-debs.sh f99a891858a MDEV-30837: Remove usage of AWK from Debian init and postinst scripts 5740638c4c3 MDEV-31132 Deadlock between DDL and purge of InnoDB history d4265fbde58 MDEV-26055: Correct the formula for adaptive flushing 898320b5f8a MDEV-30804 addendum for 10.6+ branches c22ab93f8af MDEV-26827 fixup: Prevent a hang in LRU eviction 818d5e48146 Merge 10.5 into 10.6 0976afec889 MDEV-31114 Assertion !...is_waiting() failed in os_aio_wait_until_no_pending_writes() 2c567b2fa37 MDEV-30996 insert.. select in presence of full text index freezes all other commits at commit time 51e62cb3b3b MDEV-26782 InnoDB temporary tablespace: reclaiming of free space does not work 204e7225dce Cleanup: MONITOR_EXISTING trx_undo_slots_used, trx_undo_slots_cached 86767bcc0f1 MDEV-29593 Purge misses a chance to free not-yet-reused undo pages 40eff3f8681 MDEV-26827 fixup: hangs and !os_aio_pending_writes() assertion failures ``` Example of error in (cross)build log: ``` Building CXX object CMakeFiles/cmTC_9b8d6.dir/src.cxx.o /usr/lib/ccache/c++ -DHAVE_INCONSISTENT_MISSING_DESTRUCTOR_OVERRIDE -g -O2 -ffile-prefix-map=/builds/otto/mariadb-server/debian/output/source_dir=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -Wdate-time -D_FORTIFY_SOURCE=2 -pie -fPIC -fstack-protector --param=ssp-buffer-size=4 -Wall -Wextra -Woverloaded-virtual -Wconversion -g -Wsuggest-override -Winconsistent-missing-destructor-override -o CMakeFiles/cmTC_9b8d6.dir/src.cxx.o -c /builds/otto/mariadb-server/debian/output/source_dir/builddir-native/CMakeFiles/CMakeScratch/TryCompile-Nn98w6/src.cxx c++: error: unrecognized command-line option '-Winconsistent-missing-destructor-override' gmake[3]: *** [CMakeFiles/cmTC_9b8d6.dir/build.make:78: CMakeFiles/cmTC_9b8d6.dir/src.cxx.o] Error 1 gmake[3]: Leaving directory '/builds/otto/mariadb-server/debian/output/source_dir/builddir-native/CMakeFiles/CMakeScratch/TryCompile-Nn98w6' gmake[2]: *** [Makefile:127: cmTC_9b8d6/fast] Error 2 ``` Is this enough information for somebody to guess what regressed? - Otto

4 6

[Maria-developers] Understanding binlog group commit (MDEV-232, MDEV-532, MDEV-25611, MDEV-18959)
by Marko Mäkelä 22 May '23

22 May '23

Hi Kristian, I am happy to see you being active on the maria-developers list. I don’t think we have collaborated in the past. I have been working on the InnoDB storage engine under MySQL 4.0 through 5.7 (and some 8.0) since 2003, and since 2016 under MariaDB Server, mostly 10.1 and later. I understand that you were an early contributor to MariaDB and implemented some durability I/O optimization between the "distributed transaction" of binlog and the InnoDB write-ahead log (redo log, ib_logfile0). I think that some improvement in this area is long overdue. I find it hard to read the high-level descriptions. Nowhere in https://jira.mariadb.org/browse/MDEV-232 or https://jira.mariadb.org/browse/MDEV-532 I see any mention of any key terminology: * InnoDB log sequence number (LSN): a logical point of time for any persistent InnoDB changes. Measured in bytes. Mini-transactions (atomic changes of InnoDB pages) start and end at some LSN. Corresponds to the Oracle system change number (SCN). * Binlog offset: a logical point of time for any changes at the replication level. Measured in bytes. Replication events such as statements or row-level events start and end at some binlog offset. * The LSN or binlog offset of a COMMIT, XA PREPARE, XA ROLLBACK, or XA COMMIT. There is quite a bit of code in InnoDB that I would like to get rid of. Some of it is related to the data members trx_t::flush_log_later and trx_t::must_flush_log_later. There is also the confusingly named handlerton::commit_checkpoint_request that has nothing to do with log checkpoints, but with some kind of a durability request that fails to identify the minimum logical time up to which everything is desired to be durable. (My talk https://fosdem.org/2022/schedule/event/mariadb_innodb/ explains what a log checkpoint is and how recovery works.) Some years ago, in MDEV-21534 Vladislav Vaintroub introduced write_lock and flush_lock in InnoDB. This works pretty well: a Sysbench workload with innodb_flush_log_at_trx_commit=1 will linearly improve throughput as you add more client connection threads. I think that something similar would be beneficial for the binlog, to keep track of the latest written offset and the latest write offset after which fdatasync() has completed. I am not familiar with the binlog code. A colleague recently claimed to me that LOCK_commit_ordered is a huge bottleneck. As you can see, the write_lock and flush_lock are better because the lock requests include a notion of logical time: If everything up to "my" time has been written or durably written, I do not care to wait. In MDEV-27774 (MariaDB Server 10.8) this was improved further: multiple mini-transactions can copy their local log buffer to the shared log_sys.buf concurrently. It seems to me that the missing tracking of logical time in the data structures is causing hangs like MDEV-25611. I feel that the design is based on a wrong assumption that the binlog-covered transactions are the only source of InnoDB log writes. That has never been the case: For example, there have always been background threads or tasks that purge the history of committed transactions. The task MDEV-18959 (ensuring that one fdatasync() per durable transaction suffices) is also related to this. I think that the simplest solution (assuming that moving the binlog to a bunch of transactional tables is out of the question) is to ensure that one of the logs (binlog and other write-ahead logs) is always ahead. If we choose that the binlog must be ahead, then in InnoDB the write_lock acquisition would have to be preceded by a binlog_write_lock acquisition that will ensure that any relevant changes of persistent state (COMMIT, XA PREPARE, XA ROLLBACK, XA COMMIT) that could be part of the InnoDB log write must have been durably written to the binlog. With best regards, Marko -- Marko Mäkelä, Lead Developer InnoDB MariaDB plc

3 7

Re: [Maria-developers] Understanding binlog group commit (MDEV-232, MDEV-532, MDEV-25611, MDEV-18959)
by Marko Mäkelä 16 May '23

16 May '23

On Tue, May 16, 2023 at 1:37 PM Andrei Elkin <andrei.elkin(a)mariadb.com> wrote: > Can you comment on chances for `flush_lsn` never advance to the value of > `write_lsn` when/because Innodb has nothing to commit (so at such a time > RM (or PURGE or FLUSH LOGS that have been mentioned too) might be run)? If the last write to the log was an InnoDB internal one, it could be done by log_write_up_to(lsn, false). The log_sys.flushed_to_disk_lsn would only be advanced (to somewhere closer to log_sys.lsn) if there was a subsequent call to log_write_up_to(lsn, true). Log checkpoint would advance log_sys.flushed_to_disk_lsn, as should a user transaction commit. This could be a possible explanation. -- Marko Mäkelä, Lead Developer InnoDB MariaDB plc

1 0

Re: [Maria-developers] d67c3f88883: MDEV-27744 InnoDB: Failing assertion: !cursor->index->is_committed() in row0ins.cc (from row_ins_sec_index_entry_by_modify) | Assertion `0' failed in row_upd_sec_index_entry (debug) | Corruption
by Sergei Golubchik 11 May '23

11 May '23

Hi, Alexander, A question: if the frm will *always* be created in the canonical, non-ORACLE mode and always parsed in the same mode, will we even need these schema-qualified native functions? okay, the schema-qualified trick will help to avoid creating REGEX_REPLACE_ORACLE and so on. But is it needed to fix MDEV-27744? More comments below On Apr 27, Alexander Barkov wrote: > revision-id: d67c3f88883 (mariadb-10.3.33-61-gd67c3f88883) > parent(s): 7355f7b1f5c > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2022-04-22 15:35:16 +0400 > message: > > MDEV-27744 InnoDB: Failing assertion: !cursor->index->is_committed() in row0ins.cc (from row_ins_sec_index_entry_by_modify) | Assertion `0' failed in row_upd_sec_index_entry (debug) | Corruption > > The crash happened with an indexed virtual column whose > value is evaluated using a function that has a different meaning > in sql_mode='' vs sql_mode=ORACLE: ... good comment > diff --git a/sql/sql_partition.cc b/sql/sql_partition.cc > index 59d2ea60715..8e83ee02782 100644 > --- a/sql/sql_partition.cc > +++ b/sql/sql_partition.cc > @@ -2591,11 +2591,9 @@ char *generate_partition_syntax_for_frm(THD *thd, partition_info *part_info, > HA_CREATE_INFO *create_info, > Alter_info *alter_info) > { > - sql_mode_t old_mode= thd->variables.sql_mode; > - thd->variables.sql_mode &= ~MODE_ANSI_QUOTES; > + Sql_mode_save_for_frm_handling sql_mode_save(thd); right. thanks > char *res= generate_partition_syntax(thd, part_info, buf_length, > true, create_info, alter_info); > - thd->variables.sql_mode= old_mode; > return res; > } > > diff --git a/sql/unireg.cc b/sql/unireg.cc > index efbb5e773b4..3b37944afa5 100644 > --- a/sql/unireg.cc > +++ b/sql/unireg.cc > @@ -652,6 +649,7 @@ static bool pack_expression(String *buf, Virtual_column_info *vcol, > static bool pack_vcols(String *buf, List<Create_field> &create_fields, > List<Virtual_column_info> *check_constraint_list) > { > + Sql_mode_save_for_frm_handling sql_mode_save(current_thd); well, it was done above pack_vcols(), because the caller had access to the thd. Why not to keep it that way? > List_iterator<Create_field> it(create_fields); > Create_field *field; > > diff --git a/sql/item_cmpfunc.cc b/sql/item_cmpfunc.cc > index f41414f8ae9..0f5c7519d07 100644 > --- a/sql/item_cmpfunc.cc > +++ b/sql/item_cmpfunc.cc > @@ -3292,9 +3292,24 @@ void Item_func_case_simple::print(String *str, enum_query_type query_type) > } > > > +Item_func_decode_oracle * > +Item_func_decode_oracle::create(THD *thd, const LEX_CSTRING &name, > + List<Item> *item_list) > +{ > + if (unlikely(!item_list || item_list->elements < 3)) > + { > + wrong_param_count_error(oracle_schema_ref.name(), name); > + return NULL; > + } > + return new (thd->mem_root) Item_func_decode_oracle(thd, *item_list); > +} why? items are normally created in the Create_xxx::create* methods, defined in item_create.cc. Why did you delegate that to static methods in Item_func_xxx ? > + > + > void Item_func_decode_oracle::print(String *str, enum_query_type query_type) > { > - str->append(func_name()); > + print_sql_mode_dependent_name(str, query_type, > + oracle_schema_ref, > + Item_func_decode_oracle::func_name()); > str->append('('); > args[0]->print(str, query_type); > for (uint i= 1, count= when_count() ; i <= count; i++) > diff --git a/sql/mysqld.h b/sql/mysqld.h > index d40e1d170d0..54069c3cd5a 100644 > --- a/sql/mysqld.h > +++ b/sql/mysqld.h > @@ -741,12 +741,25 @@ enum enum_query_type > QT_ITEM_SUBSELECT_ID_ONLY, > > QT_SHOW_SELECT_NUMBER= (1<<10), > + > + //QT_ITEM_IDENT_DISABLE_DB_TABLE_NAMES= (1 <<11), -- this is taken in 10.5 > + > + /// Print sql_mode-dependent functions with the schema qualifier > + /// even if the currently implied (by sql_mode) schema is equal to > + /// to the function schema, e.g. mariadb_schema.concat('a'). > + QT_ITEM_FUNC_FORCE_SCHEMA_NAME= (1 << 12), > + > + /// Print for FRM file. Focus on parse-back. > + /// e.g. VIEW expressions and virtual column expressions > + QT_FOR_FRM= (1 << 13), > + > /// This is used for EXPLAIN EXTENDED extra warnings / Be more detailed > /// Be more detailed than QT_EXPLAIN. > /// Perhaps we should eventually include QT_ITEM_IDENT_SKIP_CURRENT_DATABASE > /// here, as it would give better readable results > QT_EXPLAIN_EXTENDED= QT_TO_SYSTEM_CHARSET| > - QT_SHOW_SELECT_NUMBER, > + QT_SHOW_SELECT_NUMBER| > + QT_ITEM_FUNC_FORCE_SCHEMA_NAME, don't do this. let's try to keep the behavior *exactly as before* for people who don't switch between sql_modes all the time on the second thought. why do you need QT_ITEM_FUNC_FORCE_SCHEMA_NAME at all? Besides EXPLAIN EXTENDED (where we'd better not use it), it's only used in generating view's sql for I_S. It's not supposed to be used for generating views, so why would it need QT_ITEM_FUNC_FORCE_SCHEMA_NAME? May be it can be deleted altogether? > > // If an expression is constant, print the expression, not the value > // it evaluates to. Should be used for error messages, so that they > diff --git a/sql/sql_class.h b/sql/sql_class.h > index 8268ebe0bcd..6b988938f0b 100644 > --- a/sql/sql_class.h > +++ b/sql/sql_class.h > @@ -6589,6 +6589,50 @@ class Sql_mode_save > sql_mode_t old_mode; // SQL mode saved at construction time. > }; > > + > +/* > + Save the current sql_mode. Switch off sql_mode flags which can prevent > + normal parsing of VIEWs, expressions in generated columns. > + Restore the old sql_mode on destructor. > +*/ > +class Sql_mode_save_for_frm_handling: public Sql_mode_save > +{ > +public: > + Sql_mode_save_for_frm_handling(THD *thd) > + :Sql_mode_save(thd) > + { > + /* > + - MODE_REAL_AS_FLOAT affect only CREATE TABLE parsing > + + MODE_PIPES_AS_CONCAT affect expression parsing > + + MODE_ANSI_QUOTES affect expression parsing > + + MODE_IGNORE_SPACE affect expression parsing > + - MODE_IGNORE_BAD_TABLE_OPTIONS affect only CREATE/ALTER TABLE parsing > + * MODE_ONLY_FULL_GROUP_BY affect execution > + * MODE_NO_UNSIGNED_SUBTRACTION affect execution > + - MODE_NO_DIR_IN_CREATE affect table creation only > + - MODE_POSTGRESQL compounded from other modes > + - MODE_ORACLE affects Item creation (e.g for CONCAT) should be '+' for MODE_ORACLE, not '-', right? > + - MODE_MSSQL compounded from other modes > + - MODE_DB2 compounded from other modes > + - MODE_MAXDB affect only CREATE TABLE parsing > + - MODE_NO_KEY_OPTIONS affect only SHOW > + - MODE_NO_TABLE_OPTIONS affect only SHOW > + - MODE_NO_FIELD_OPTIONS affect only SHOW > + - MODE_MYSQL323 affect only SHOW > + - MODE_MYSQL40 affect only SHOW > + - MODE_ANSI compounded from other modes > + (+ transaction mode) > + ? MODE_NO_AUTO_VALUE_ON_ZERO affect UPDATEs > + + MODE_NO_BACKSLASH_ESCAPES affect expression parsing > + + MODE_EMPTY_STRING_IS_NULL affect expression parsing > + */ > + thd->variables.sql_mode&= ~(MODE_PIPES_AS_CONCAT | MODE_ANSI_QUOTES | > + MODE_IGNORE_SPACE | MODE_NO_BACKSLASH_ESCAPES | > + MODE_ORACLE | MODE_EMPTY_STRING_IS_NULL); > + }; > +}; > + > + > class Abort_on_warning_instant_set > { > THD *m_thd; > diff --git a/sql/sql_schema.h b/sql/sql_schema.h > index 7c8f284d526..36914e36990 100644 > --- a/sql/sql_schema.h > +++ b/sql/sql_schema.h > @@ -33,6 +36,34 @@ class Schema > { > return src; > } > + > + > + /** > + Find the native function builder associated with a given function name. > + @param thd The current thread > + @param name The native function name > + @return The native function builder associated with the name, or NULL > + */ > + virtual Create_func *find_native_function_builder(THD *thd, > + const LEX_CSTRING &name) > + const; > + /** > + Find a native function builder, return an error if not found, > + build an Item otherwise. > + */ > + Item *make_item_func_call_native(THD *thd, > + const Lex_ident_sys &name, > + List<Item> *args) const; > + > + // Builders for native SQL function with a special syntax in sql_yacc.yy > + virtual Item *make_item_func_substr(THD *thd, > + const Lex_substring_spec_st &spec) const; > + > + virtual Item *make_item_func_trim(THD *thd, const Lex_trim_st &spec) const; > + virtual Item *make_item_func_replace(THD *thd, > + Item *subj, > + Item *find, > + Item *replace) const; Why no Lex_replace_spec_st ? > /* > For now we have *hard-coded* compatibility schemas: > schema_mariadb, schema_oracle, schema_maxdb. > @@ -66,5 +97,6 @@ class Schema > > > extern Schema mariadb_schema; > +extern const Schema &oracle_schema_ref; ??? why did you declare mariadb_schema extern, but oracle_schema_ref only via a const ref? > > #endif // SQL_SCHEMA_H_INCLUDED > diff --git a/sql/item_create.h b/sql/item_create.h > index 894e9777b8d..3fadaecb090 100644 > --- a/sql/item_create.h > +++ b/sql/item_create.h > @@ -215,9 +205,48 @@ struct Native_func_registry > Create_func *builder; > }; > > + > +class Native_functions_hash: public HASH > +{ > +public: > + Native_functions_hash() > + { > + bzero(this, sizeof(*this)); > + } > + ~Native_functions_hash() > + { > + /* > + No automatic free. > + The the upper level code should call cleanup() explicitly. why is that? > + */ > + DBUG_ASSERT(!records); > + } > + bool init(size_t count); > + bool init(const Native_func_registry array[], size_t count) > + { > + return init(count) || append(array); > + } > + bool append(const Native_func_registry array[]); > + bool remove(const Native_func_registry array[]); > + void cleanup(); > + /** > + Find the native function builder associated with a given function name. > + @param thd The current thread > + @param name The native function name > + @return The native function builder associated with the name, or NULL > + */ > + Create_func *find(THD *thd, const LEX_CSTRING &name) const; > +}; > + > +extern Native_functions_hash native_functions_hash; > +extern Native_functions_hash native_functions_hash_oracle_overrides; > + > +extern const Native_func_registry func_array[]; > +extern const size_t func_array_length; > + > int item_create_init(); > -int item_create_append(Native_func_registry array[]); > -int item_create_remove(Native_func_registry array[]); > +int item_create_append(const Native_func_registry array[]); > +int item_create_remove(const Native_func_registry array[]); > void item_create_cleanup(); > > Item *create_func_dyncol_create(THD *thd, List<DYNCALL_CREATE_DEF> &list); > diff --git a/sql/sql_schema.cc b/sql/sql_schema.cc > index 0bf4a63c2f8..e2993f51c6d 100644 > --- a/sql/sql_schema.cc > +++ b/sql/sql_schema.cc > @@ -78,3 +89,86 @@ Schema *Schema::find_implied(THD *thd) > return &maxdb_schema; > return &mariadb_schema; > } > + > + > +Create_func * > +Schema::find_native_function_builder(THD *thd, const LEX_CSTRING &name) const > +{ > + return native_functions_hash.find(thd, name); > +} > + > + > +Create_func * > +Schema_oracle::find_native_function_builder(THD *thd, > + const LEX_CSTRING &name) const > +{ > + Create_func *create= native_functions_hash_oracle_overrides.find(thd, name); > + if (create) > + return create; > + return native_functions_hash.find(thd, name); may be populate native_functions_hash_oracle will all names to do only one lookup here? this will be used often in oracle mode > +} > + > + > +Item *Schema::make_item_func_call_native(THD *thd, > + const Lex_ident_sys &name, > + List<Item> *args) const > +{ > + Create_func *builder= find_native_function_builder(thd, name); > + if (builder) > + return builder->create_func(thd, &name, args); > + my_error(ER_FUNCTION_NOT_DEFINED, MYF(0), name.str); > + return NULL; > +} > + > + > +Item *Schema::make_item_func_trim(THD *thd, const Lex_trim_st &spec) const > +{ > + return spec.make_item_func_trim_std(thd); why does trim is constructed via helper in spec, but substr and replace are not? > +} > + > + > +Item *Schema::make_item_func_substr(THD *thd, > + const Lex_substring_spec_st &spec) const > +{ > + return spec.m_for ? > + new (thd->mem_root) Item_func_substr(thd, spec.m_subject, spec.m_from, > + spec.m_for) : > + new (thd->mem_root) Item_func_substr(thd, spec.m_subject, spec.m_from); > +} > + > + > +Item *Schema_oracle::make_item_func_substr(THD *thd, > + const Lex_substring_spec_st &spec) const > +{ > + return spec.m_for ? > + new (thd->mem_root) Item_func_substr_oracle(thd, spec.m_subject, > + spec.m_from, > + spec.m_for) : > + new (thd->mem_root) Item_func_substr_oracle(thd, spec.m_subject, > + spec.m_from); > +} > + > + > +Item *Schema_oracle::make_item_func_trim(THD *thd, > + const Lex_trim_st &spec) const > +{ > + return spec.make_item_func_trim_oracle(thd); > +} > + > + > +Item *Schema::make_item_func_replace(THD *thd, > + Item *subj, > + Item *find, > + Item *replace) const > +{ > + return new (thd->mem_root) Item_func_replace(thd, subj, find, replace); > +} > + > + > +Item *Schema_oracle::make_item_func_replace(THD *thd, > + Item *subj, > + Item *find, > + Item *replace) const > +{ > + return new (thd->mem_root) Item_func_replace_oracle(thd, subj, find, replace); > +} > diff --git a/sql/item_func.h b/sql/item_func.h > index 754b1cd1eb2..52bf7f661d1 100644 > --- a/sql/item_func.h > +++ b/sql/item_func.h > @@ -56,8 +56,117 @@ class Item_func :public Item_func_or_sum > bool check_argument_types_can_return_text(uint start, uint end) const; > bool check_argument_types_can_return_date(uint start, uint end) const; > bool check_argument_types_can_return_time(uint start, uint end) const; > + > + void print_schema_qualified_name(String *to, > + const LEX_CSTRING &schema_name, > + const char *function_name) const > + { > + // e.g. oracle_schema.func() > + to->append(schema_name); > + to->append('.'); > + to->append(function_name); > + } > + > + void print_name_with_optional_suffix(String *to, > + const char *function_name, > + const LEX_CSTRING &suffix) const > + { > + // e.g. func_oracle() > + to->append(function_name); > + if (suffix.length) > + { > + to->append("_"); > + to->append(suffix); > + } > + } > + > + void print_sql_mode_dependent_name(String *to, enum_query_type query_type, > + const Schema &schema, > + const char *function_name, > + const LEX_CSTRING &suffix) const > + { > + if (query_type & QT_ITEM_FUNC_FORCE_SCHEMA_NAME) > + { > + DBUG_ASSERT((query_type & QT_FOR_FRM) == 0); > + print_schema_qualified_name(to, schema.name(), function_name); > + } > + else if (query_type & QT_FOR_FRM) > + { > + // e.g. substr_oracle() > + DBUG_ASSERT((query_type & QT_ITEM_FUNC_FORCE_SCHEMA_NAME) == 0); > + print_name_with_optional_suffix(to, function_name, suffix); > + } > + else if (&schema != Schema::find_implied(current_thd)) > + print_schema_qualified_name(to, schema.name(), function_name); > + else > + to->append(function_name); > + } > + > + void print_sql_mode_dependent_name(String *to, enum_query_type query_type, > + const Schema &schema, > + const char *function_name) const > + { > + static const LEX_CSTRING oracle= {STRING_WITH_LEN("oracle")}; can it be oracle_schema.name() instead? > + static const LEX_CSTRING empty= {NULL, 0}; > + const LEX_CSTRING suffix= &schema == &oracle_schema_ref ? oracle : empty; > + print_sql_mode_dependent_name(to, query_type, > + schema, function_name, suffix); > + } > + void print_sql_mode_dependent_name(String *to, enum_query_type query_type, > + const char *function_name) const > + { > + print_sql_mode_dependent_name(to, query_type, *schema(), function_name); > + } > + void print_sql_mode_dependent(String *to, enum_query_type query_type) > + { > + print_sql_mode_dependent_name(to, query_type, func_name()); > + print_args_parenthesized(to, query_type); > + } > public: > > + // Print an error message for a builtin-schema qualified function call > + static void wrong_param_count_error(const LEX_CSTRING &schema_name, > + const LEX_CSTRING &func_name); > + > + // Check that the number of arguments is greater or equal to "expected" > + static bool create_check_args_ge(const LEX_CSTRING &name, > + const List<Item> *item_list, > + uint expected) > + { > + DBUG_ASSERT(expected > 0); > + if (!item_list || item_list->elements < expected) > + { > + my_error(ER_WRONG_PARAMCOUNT_TO_NATIVE_FCT, MYF(0), name.str); > + return true; > + } > + return false; > + } > + > + // Check that the number of arguments is less or equal to "expected" > + static bool create_check_args_le(const LEX_CSTRING &name, > + const List<Item> *item_list, > + uint expected) > + { > + DBUG_ASSERT(expected > 0); > + if (!item_list || item_list->elements > expected) > + { > + my_error(ER_WRONG_PARAMCOUNT_TO_NATIVE_FCT, MYF(0), name.str); > + return true; > + } > + return false; > + } > + > + // Check that the number of arguments is in the allowed range > + static bool create_check_args_between(const LEX_CSTRING &name, > + const List<Item> *item_list, > + uint min_arg_count, > + uint max_arg_count) > + { > + DBUG_ASSERT(min_arg_count < max_arg_count); > + return create_check_args_ge(name, item_list, min_arg_count) || > + create_check_args_le(name, item_list, max_arg_count); > + } > + > table_map not_null_tables_cache; > > enum Functype { UNKNOWN_FUNC,EQ_FUNC,EQUAL_FUNC,NE_FUNC,LT_FUNC,LE_FUNC, > diff --git a/sql/sql_yacc.yy b/sql/sql_yacc.yy > index 3770488f3f4..e0bafc75a36 100644 > --- a/sql/sql_yacc.yy > +++ b/sql/sql_yacc.yy > @@ -11235,6 +11236,34 @@ function_call_generic: > if (unlikely(!($$= Lex->make_item_func_call_generic(thd, &$1, &$3, &$5, $7)))) > MYSQL_YYABORT; > } > + | ident_cli '.' REPLACE '(' expr ',' expr ',' expr ')' > + { > + if (unlikely(!($$= Lex->make_item_func_replace(thd, $1, $3, > + $5, $7, $9)))) > + MYSQL_YYABORT; > + } > + | ident_cli '.' SUBSTRING '(' substring_operands ')' > + { > + if (unlikely(!($$= Lex->make_item_func_substr(thd, $1, $3, $5)))) > + MYSQL_YYABORT; > + } > + | ident_cli '.' TRIM '(' trim_operands ')' > + { > + if (unlikely(!($$= $5.make_item_func_trim(thd, $1, $3)))) > + MYSQL_YYABORT; > + } > + /* > + We don't add a qualified syntax for TRIM_ORACLE here, > + as this syntax is not absolutely required: > + SELECT mariadb_schema.TRIM_ORACLE(..); > + What absolutely required is only: > + SELECT mariadb_schema.TRIM(..); > + Adding a qualified syntax for TRIM_ORACLE would be tricky because > + it is a non-reserved keyword. To avoid new shift/reduce conflicts > + it would require grammar changes, like introducing a new rule > + ident_step2_cli (which would include everything that ident_cli > + includes but TRIM_ORACLE). Did you add a qualified syntax for any other xxx_ORACLE functions? > + */ > ; > > fulltext_options: > diff --git a/sql/sql_lex.cc b/sql/sql_lex.cc > index 7f1b362d91d..6b0a351f76e 100644 > --- a/sql/sql_lex.cc > +++ b/sql/sql_lex.cc > @@ -2008,7 +2023,59 @@ bool Lex_input_stream::get_7bit_or_8bit_ident(THD *thd, uchar *last_char) > } > > > +/* > + Resolve special SQL functions that have a qualified syntax in sql_yacc.yy. > + These functions are not listed in the native function registry > + because of a special syntax, or a reserved keyword: > + > + mariadb_schema.SUBSTRING('a' FROM 1 FOR 2) -- Special syntax > + mariadb_schema.TRIM(BOTH ' ' FROM 'a') -- Special syntax > + mariadb_schema.REPLACE('a','b','c') -- Verb keyword > +*/ > + > +int Lex_input_stream::find_keyword_qualified_special_func(Lex_ident_cli_st *str, > + uint length) const > +{ > + /* > + There are many other special functions, see the following grammar rules: > + function_call_keyword > + function_call_nonkeyword > + Here we resolve only those that have a qualified syntax to handle > + different behavior in different @@sql_mode settings. > + > + Other special functions do not work in qualified context: > + SELECT mariadb_schema.year(now()); -- Function year is not defined > + SELECT mariadb_schema.now(); -- Function now is not defined > + > + We don't resolve TRIM_ORACLE here, because it does not have > + a qualified syntax yet. Search for "trim_operands" in sql_yacc.yy > + to find more comments. > + */ > + static LEX_CSTRING funcs[]= > + { > + {STRING_WITH_LEN("SUBSTRING")}, > + {STRING_WITH_LEN("SUBSTR")}, > + {STRING_WITH_LEN("TRIM")}, > + {STRING_WITH_LEN("REPLACE")} > + }; > + > + int tokval= find_keyword(str, length, true); > + if (!tokval) > + return 0; > + for (size_t i= 0; i < array_elements(funcs); i++) > + { > + CHARSET_INFO *cs= system_charset_info; > + if (!cs->coll->strnncollsp(cs, may be strcasecmp? or with cs=latin1? you know the repertoire is ascii, so you can cut corners here. > + (const uchar *) m_tok_start, length, > + (const uchar *) funcs[i].str, funcs[i].length)) > + return tokval; > + } > + return 0; > +} > + > + > -int Lex_input_stream::scan_ident_sysvar(THD *thd, Lex_ident_cli_st *str) > +int Lex_input_stream::scan_ident_common(THD *thd, Lex_ident_cli_st *str, > + Ident_mode mode) > { > uchar last_char; > uint length; > @@ -8035,30 +8133,49 @@ bool LEX::add_grant_command(THD *thd, enum_sql_command sql_command_arg, > } > > > +const Schema * > +LEX::find_func_schema_by_name_or_error(const Lex_ident_sys &schema, > + const Lex_ident_sys &func) > +{ > + Schema *res= Schema::find_by_name(schema); > + if (res) > + return res; > + Database_qualified_name qname(schema, func); > + my_error(ER_FUNCTION_NOT_DEFINED, MYF(0), ErrConvDQName(&qname).ptr()); eh, not quite. I can define a function with the same name as a native function in any db. E.g. create test.replace(a int) returns int and it'll be callable as test.replace(1) So ER_FUNCTION_NOT_DEFINED is wrong here. But note that even though test.substr() is possible, it should not allow substr() special syntax. > + return NULL; > +} > -Item *LEX::make_item_func_substr(THD *thd, Item *a, Item *b, Item *c) > -{ > - return (thd->variables.sql_mode & MODE_ORACLE) ? > - new (thd->mem_root) Item_func_substr_oracle(thd, a, b, c) : > - new (thd->mem_root) Item_func_substr(thd, a, b, c); > -} > > > -Item *LEX::make_item_func_substr(THD *thd, Item *a, Item *b) > -{ > - return (thd->variables.sql_mode & MODE_ORACLE) ? > - new (thd->mem_root) Item_func_substr_oracle(thd, a, b) : > - new (thd->mem_root) Item_func_substr(thd, a, b); > -} > +Item *LEX::make_item_func_substr(THD *thd, > + const Lex_ident_cli_st &schema_name_cli, > + const Lex_ident_cli_st &func_name_cli, > + const Lex_substring_spec_st &spec) > +{ > + Lex_ident_sys schema_name(thd, &schema_name_cli); > + Lex_ident_sys func_name(thd, &func_name_cli); > + if (schema_name.is_null() || func_name.is_null()) > + return NULL; // EOM > + const Schema *schema= find_func_schema_by_name_or_error(schema_name, > + func_name); > + return schema ? schema->make_item_func_substr(thd, spec) : NULL; > +} > > > Item *LEX::make_item_func_replace(THD *thd, > + const Lex_ident_cli_st &schema_name_cli, > + const Lex_ident_cli_st &func_name_cli, > Item *org, > Item *find, > Item *replace) > { > - return (thd->variables.sql_mode & MODE_ORACLE) ? > - new (thd->mem_root) Item_func_replace_oracle(thd, org, find, replace) : > - new (thd->mem_root) Item_func_replace(thd, org, find, replace); > + Lex_ident_sys schema_name(thd, &schema_name_cli); > + Lex_ident_sys func_name(thd, &func_name_cli); > + if (schema_name.is_null() || func_name.is_null()) > + return NULL; // EOM > + const Schema *schema= find_func_schema_by_name_or_error(schema_name, > + func_name); > + return schema ? schema->make_item_func_replace(thd, org, find, replace) : > + NULL; > } > > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 2

[Maria-developers] START ALTER replication test cases
by Kristian Nielsen 10 May '23

10 May '23

Hi Sachin, Andrei, I'm looking at test cases for START ALTER / Lag-free Alter on Slave. And I happened to notice something in test case rpl.rpl_start_alter_bugs: --connection master set global binlog_alter_two_phase=true; # ... ALTER TABLE t1 DROP PRIMARY KEY; ALTER TABLE t1 ADD UNIQUE KEY ui (i); ALTER TABLE t1 ADD PRIMARY KEY (i); Note the set *global* binlog_alter_two_phase. This will not affect the testcase at all, as the ALTER TABLE queries are using the *session* value (which defaults to false). This can be verified by checking the binlog after running the testcase, mysql-test/var/mysqld.1/data/master-bin.000001 . It has only normal ALTER TABLE query events, no START ALTER / COMMIT ALTER. So it looks strange that a start_alter test doesn't use START ALTER, and I just wanted to point this out to you in case it is a mistake and maybe the test coverage of START ALTER is less than intended? A quick grep shows similar SET GLOBAL in other start_alter tests, but I did not investigate if they have the same issue. Hope this helps, - Kristian.

2 2

Re: [Maria-developers] da5a72f32d4: MDEV-31033 ER_KEY_NOT_FOUND upon online COPY ALTER on a partitioned table
by Sergei Golubchik 10 May '23

10 May '23

Hi, Nikita, On May 05, Nikita Malyavin wrote: > revision-id: da5a72f32d4 (mariadb-11.0.1-114-gda5a72f32d4) > parent(s): 0d6584c019c > author: Nikita Malyavin > committer: Nikita Malyavin > timestamp: 2023-05-01 01:15:41 +0300 > message: > > MDEV-31033 ER_KEY_NOT_FOUND upon online COPY ALTER on a partitioned table > > The row events were applied "twice": once for the ha_partition, and > one more time for the underlying storage engine. > > There's no such problem in binlog/rpl, because > ha_partiton::row_logging is normally set to false. s/ha_partiton/ha_partition/ > The proposed fix adds table->skip_online_logging field, preventing the > underlying engines from replicating online events. Only upper layer > engine should do this. > > table->skip_online_logging is initialized to 0 together with the table > itself. Normally it is always 0, except for the intermediate > overriding part, once it is checked in handler::binlog_log_row. > > Note on MDEV-31040 (Server crashes in MYSQL_LOG::is_open upon ALTER on > partitioned table): > First the solution was to override table->s->online_alter_binlog, and > set it to NULL during the partitioning operation, which was > ridiculously wrong: no-one is supposed to modify that field outside > ALTER! That field write access is protected by EXCLUSIVE metadata > lock. I don't think you need to list your different (non-working) attemps to fix the bug in the commit comment. Just explain the fix that works, not the fix that doesn't. > In comparison, table->skip_online_logging is a connection-local field. > > The test is a non-deterministic one. It wasn't convenient to construct > a deterministic test here, besides for the case that will never be > repeated. Is it still the case? Your test in alter_table_online_debug.test with debug_sync - is it non-deterministic? > diff --git a/sql/handler.cc b/sql/handler.cc > index 3fcbb5171d6..a7e229e8f8d 100644 > --- a/sql/handler.cc > +++ b/sql/handler.cc > @@ -7280,7 +7280,8 @@ int handler::binlog_log_row(const uchar *before_record, > log_func, row_logging_has_trans); > > #ifdef HAVE_REPLICATION > - if (unlikely(!error && table->s->online_alter_binlog)) > + if (unlikely(!error && table->s->online_alter_binlog && > + !table->skip_online_logging)) this doesn't need any juggling with skip_online_logging. This problem is already solved for, say, long uniques. You can use this == table->file condition to filter out individual partitions and have your binlog_log_row_online_alter called only for the main ha_partition handler. > error= binlog_log_row_online_alter(table, before_record, after_record, > log_func); > #endif // HAVE_REPLICATION Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 5

Re: [Maria-developers] 24c653be25a: unpack_row: unpack a correct number of fields
by Sergei Golubchik 06 May '23

06 May '23

Hi, Nikita, On May 05, Nikita Malyavin wrote: > revision-id: 24c653be25a (mariadb-11.0.1-123-g24c653be25a) > parent(s): c1cbda5a2c8 > author: Nikita Malyavin > committer: Nikita Malyavin > timestamp: 2023-05-04 20:49:43 +0300 > message: > > unpack_row: unpack a correct number of fields > > diff --git a/sql/rpl_record.cc b/sql/rpl_record.cc > index 908d2c39d97..30bd80bdfc7 100644 > --- a/sql/rpl_record.cc > +++ b/sql/rpl_record.cc > @@ -228,24 +228,30 @@ int unpack_row(const rpl_group_info *rgi, TABLE *table, uint const colcnt, > const TABLE *conv_table= rpl_data.conv_table; > DBUG_PRINT("debug", ("Table data: tabldef: %p, conv_table: %p", > tabledef, conv_table)); > - > + bool is_online_alter= rpl_data.is_online_alter(); > DBUG_ASSERT(rgi); > > - for (field_ptr= begin_ptr; field_ptr < end_ptr && *field_ptr; ++field_ptr) > + for (field_ptr= begin_ptr; field_ptr < end_ptr > + /* In Online Alter conv_table can be wider than > + original table, but we need to unpack it all. */ > + && (*field_ptr || is_online_alter); I failed to understand this, could you please explain? The comment didn't help much, unfortunately. The test case didn't either. > + ++field_ptr) > { > + intptr fidx= field_ptr - begin_ptr; intptr is a variant of int with sizeof(intptr) == sizeof(void*). it's needed to cast a pointer to int, do some math, cast back. here you need my_ptrdiff_t or ptrdiff_t > /* > If there is a conversion table, we pick up the field pointer to > the conversion table. If the conversion table or the field > pointer is NULL, no conversions are necessary. > */ > - Field *conv_field= conv_table ? conv_table->field[field_ptr - begin_ptr] : NULL; > + Field *conv_field= conv_table ? conv_table->field[fidx] : NULL; > Field *const f= conv_field ? conv_field : *field_ptr; > +#ifdef DBUG_TRACE > + Field *dbg= is_online_alter ? f : *field_ptr; > +#endif > DBUG_PRINT("debug", ("Conversion %srequired for field '%s' (#%ld)", > conv_field ? "" : "not ", > - (*field_ptr)->field_name.str, > - (long) (field_ptr - begin_ptr))); > + dbg->field_name.str, > + (long) ())); eh? you surely meant `(long) fidx` ? how did it even compile? > DBUG_ASSERT(f != NULL); Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] c5ce597f06a: MDEV-31043 ER_KEY_NOT_FOUND upon concurrent ALTER and transaction
by Sergei Golubchik 06 May '23

06 May '23

Hi, Nikita, On May 03, Nikita Malyavin wrote: > revision-id: c5ce597f06a (mariadb-11.0.1-115-gc5ce597f06a) > parent(s): da5a72f32d4 > author: Nikita Malyavin > committer: Nikita Malyavin > timestamp: 2023-05-01 01:15:41 +0300 > message: > > MDEV-31043 ER_KEY_NOT_FOUND upon concurrent ALTER and transaction > > Non-transactional engines changes are visible immediately the row operation > succeeds, so we should apply the changes to the online buffer immediately > after the statement ends. > > diff --git a/sql/log.cc b/sql/log.cc > index f30ce3bcbd1..1d98455c2b3 100644 > --- a/sql/log.cc > +++ b/sql/log.cc > @@ -7701,23 +7701,35 @@ static int binlog_online_alter_end_trans(THD *thd, bool all, bool commit) > auto *binlog= cache.sink_log; > DBUG_ASSERT(binlog); > > - bool do_commit= commit || !cache.hton->rollback || > - cache.hton->flags & HTON_NO_ROLLBACK; // Aria > + bool do_commit= (commit && is_ending_transaction) > + || cache.hton->flags & HTON_NO_ROLLBACK // Aria > + || !cache.hton->rollback; > + > + error= binlog_flush_pending_rows_event(thd, false, true, binlog, &cache); you've lost the useful comment about STMT_END. and you flush events for rollback too that seems like a waste. I personally would've kept the old structure of if()'s. Like - bool do_commit= commit || !cache.hton->rollback || + bool non_trans= !cache.hton->rollback || and - if (do_commit) + if (commit || non_trans) with - if (is_ending_transaction) + if (is_ending_transaction || non_trans) That would've called binlog flush only where it needs to be. > + > if (do_commit) > { > - // do not set STMT_END for last event to leave table open in altering thd > - error= binlog_flush_pending_rows_event(thd, false, true, binlog, &cache); > - if (is_ending_transaction) > + /* > + If the cache wasn't reinited to write, then it remains empty after > + the last write. > + */ > + if (cache.cache_log.type != READ_CACHE && !error) this is a confusing new condition. are you trying to avoid locking a mutex for an empty cache? If yes, you can check my_b_bytes_in_cache(), that'd be more clear. > { > mysql_mutex_lock(binlog->get_log_lock()); > error= binlog->write_cache(thd, &cache.cache_log); > mysql_mutex_unlock(binlog->get_log_lock()); > } > - else > - cache.store_prev_position(); > } > - else if (!is_ending_transaction) > + else if (!commit) // rollback > + { > cache.restore_prev_position(); > + } > + else add // trans engine, end of statement > + { > + DBUG_ASSERT(!is_ending_transaction); > + cache.store_prev_position(); > + } Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 3

Re: [Maria-developers] d25439f1cbb: MDEV-31140: FLUSH BINARY LOGS DELETE_DOMAIN_ID=(D) can errorneously delete active domains
by Kristian Nielsen 02 May '23

02 May '23

Andrei Elkin <andrei.elkin(a)mariadb.com> writes: >> Want to review a small patch for https://jira.mariadb.org/browse/MDEV-31140 ? > Thanks for a very good piece of analysis and the refinement! > I agree with both. Great, thanks for review! Do you have an opinion into which branch I should push it? The fix should be low-risk. I'm thinking maybe 11.0 or 10.11 is fine, since the bug does not affect normal operation. But I'm open to other suggestions, whatever is the current policy for this kind of bug? > The existed test was more than a bit unlucky, to confess, > to let the problem stay unnoticed. Yes, there is a good test coverage, so a bit unlucky, agree. - Kristian. > On the occasion of the International workers' day, let me wish you more > contribution, and inspiration for how to make not just this server > better alone :-)!!! Thanks, and all the best to you too :-) - Kristian. >> Kristian Nielsen <knielsen(a)knielsen-hq.org> writes: >> >>> revision-id: d25439f1cbb79e5467b4249792130bfe524e3f10 (mariadb-10.11.2-21-gd25439f1cbb) >>> parent(s): 656c2e18b1e9ea5d0314745f3988d126eedbc22a >>> author: Kristian Nielsen >>> committer: Kristian Nielsen >>> timestamp: 2023-04-27 12:16:07 +0200 >>> message: >>> >>> MDEV-31140: FLUSH BINARY LOGS DELETE_DOMAIN_ID=(D) can errorneously delete active domains >>> >>> Fix the code in rpl_binlog_state::drop_domain(), so that _all_ entries for >>> the domain in the binlog state must match an entry in the initial GTID_LIST, >>> not just one entry match. >>> >>> Signed-off-by: Kristian Nielsen <knielsen(a)knielsen-hq.org> >>> >>> --- >>> .../r/binlog_flush_binlogs_delete_domain.result | 14 ++++++++++--- >>> .../t/binlog_flush_binlogs_delete_domain.test | 13 +++++++++++- >>> sql/rpl_gtid.cc | 24 ++++++++++++---------- >>> 3 files changed, 36 insertions(+), 15 deletions(-) >>> >>> diff --git a/mysql-test/suite/binlog/r/binlog_flush_binlogs_delete_domain.result b/mysql-test/suite/binlog/r/binlog_flush_binlogs_delete_domain.result >>> index fdcfb4bfa01..1c11191802f 100644 >>> --- a/mysql-test/suite/binlog/r/binlog_flush_binlogs_delete_domain.result >>> +++ b/mysql-test/suite/binlog/r/binlog_flush_binlogs_delete_domain.result >>> @@ -46,15 +46,23 @@ Warning 1076 The current gtid binlog state is incompatible with a former one mis >>> Warning 1076 The gtid domain being deleted ('1') is not in the current binlog state >>> FLUSH BINARY LOGS DELETE_DOMAIN_ID = (1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 0); >>> ERROR HY000: Could not delete gtid domain. Reason: binlog files may contain gtids from the domain ('1') being deleted. Make sure to first purge those files. >>> +MDEV-31140: Missing error from DELETE_DOMAIN_ID when gtid_binlog_state partially matches GTID_LIST. >>> FLUSH BINARY LOGS; >>> PURGE BINARY LOGS TO 'master-bin.000005'; >>> +SET @@SESSION.gtid_domain_id=8; >>> +SET @@SESSION.server_id=10*8 + 1; >>> +INSERT INTO t SELECT 1+MAX(a) FROM t; >>> +FLUSH BINARY LOGS DELETE_DOMAIN_ID = (1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 0); >>> +ERROR HY000: Could not delete gtid domain. Reason: binlog files may contain gtids from the domain ('8') being deleted. Make sure to first purge those files. >>> +FLUSH BINARY LOGS; >>> +PURGE BINARY LOGS TO 'master-bin.000006'; >>> FLUSH BINARY LOGS DELETE_DOMAIN_ID = (1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 0); >>> Warnings: >>> Warning 1076 The gtid domain being deleted ('0') is not in the current binlog state >>> Gtid_list of the current binlog does not contain 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 0: >>> -show binlog events in 'master-bin.000006' limit 1,1; >>> +show binlog events in 'master-bin.000007' limit 1,1; >>> Log_name Pos Event_type Server_id End_log_pos Info >>> -master-bin.000006 # Gtid_list 1 # [] >>> +master-bin.000007 # Gtid_list 1 # [] >>> SET @@SESSION.gtid_domain_id=1;; >>> SET @@SESSION.server_id=1; >>> SET @@SESSION.gtid_seq_no=1; >>> @@ -75,7 +83,7 @@ INSERT INTO t SET a=1; >>> SELECT @gtid_binlog_state_saved "as original state", @@GLOBAL.gtid_binlog_state as "out of order for 11 domain state"; >>> as original state out of order for 11 domain state >>> 1-1-1,1-2-2,11-11-11 1-1-1,1-2-2,11-11-1 >>> -PURGE BINARY LOGS TO 'master-bin.000007'; >>> +PURGE BINARY LOGS TO 'master-bin.000008'; >>> the following command succeeds with warnings >>> FLUSH BINARY LOGS DELETE_DOMAIN_ID = (1); >>> Warnings: >>> diff --git a/mysql-test/suite/binlog/t/binlog_flush_binlogs_delete_domain.test b/mysql-test/suite/binlog/t/binlog_flush_binlogs_delete_domain.test >>> index 8311f4bd800..1643ecff72d 100644 >>> --- a/mysql-test/suite/binlog/t/binlog_flush_binlogs_delete_domain.test >>> +++ b/mysql-test/suite/binlog/t/binlog_flush_binlogs_delete_domain.test >>> @@ -21,7 +21,6 @@ FLUSH BINARY LOGS DELETE_DOMAIN_ID = (); >>> --echo but with a warning >>> --let $binlog_pre_flush=query_get_value(SHOW MASTER STATUS, Position, 1) >>> FLUSH BINARY LOGS DELETE_DOMAIN_ID = (99); >>> ---let $binlog_start=$binlog_pre_flush >>> --source include/show_binary_logs.inc >>> >>> # Log one event in a specified domain and try to delete the domain >>> @@ -62,6 +61,8 @@ FLUSH BINARY LOGS DELETE_DOMAIN_ID = (1); >>> # expected overrun of the static buffers of underlying dynamic arrays is doing. >>> --let $domain_cnt=17 >>> --let $server_in_domain_cnt=3 >>> +--let $err_domain_id=`SELECT FLOOR($domain_cnt/2)` >>> +--let $err_server_id=`SELECT FLOOR($server_in_domain_cnt/2)` >>> --let $domain_list= >>> --disable_query_log >>> while ($domain_cnt) >>> @@ -86,6 +87,16 @@ while ($domain_cnt) >>> --error ER_BINLOG_CANT_DELETE_GTID_DOMAIN >>> --eval FLUSH BINARY LOGS DELETE_DOMAIN_ID = ($domain_list) >>> >>> +--echo MDEV-31140: Missing error from DELETE_DOMAIN_ID when gtid_binlog_state partially matches GTID_LIST. >>> +FLUSH BINARY LOGS; >>> +--let $purge_to_binlog= query_get_value(SHOW MASTER STATUS, File, 1) >>> +--eval PURGE BINARY LOGS TO '$purge_to_binlog' >>> +--eval SET @@SESSION.gtid_domain_id=$err_domain_id >>> +--eval SET @@SESSION.server_id=10*$err_domain_id + $err_server_id >>> +eval INSERT INTO t SELECT 1+MAX(a) FROM t; >>> +--error ER_BINLOG_CANT_DELETE_GTID_DOMAIN >>> +--eval FLUSH BINARY LOGS DELETE_DOMAIN_ID = ($domain_list) >>> + >>> # Now satisfy the safety condtion to purge log files containing $domain list >>> FLUSH BINARY LOGS; >>> --let $purge_to_binlog= query_get_value(SHOW MASTER STATUS, File, 1) >>> diff --git a/sql/rpl_gtid.cc b/sql/rpl_gtid.cc >>> index c4e5c75b10a..7b67a83b3dd 100644 >>> --- a/sql/rpl_gtid.cc >>> +++ b/sql/rpl_gtid.cc >>> @@ -2209,18 +2209,16 @@ rpl_binlog_state::drop_domain(DYNAMIC_ARRAY *ids, >>> >>> /* >>> For each domain_id from ids >>> - when no such domain in binlog state >>> - warn && continue >>> - For each domain.server's last gtid >>> - when not locate the last gtid in glev.list >>> - error out binlog state can't change >>> - otherwise continue >>> + If the domain is already absent from the binlog state >>> + Warn && continue >>> + If any GTID with that domain in binlog state is missing from glev.list >>> + Error out binlog state can't change >>> */ >>> for (ulong i= 0; i < ids->elements; i++) >>> { >>> rpl_binlog_state::element *elem= NULL; >>> uint32 *ptr_domain_id; >>> - bool not_match; >>> + bool all_found; >>> >>> ptr_domain_id= (uint32*) dynamic_array_ptr(ids, i); >>> elem= (rpl_binlog_state::element *) >>> @@ -2235,14 +2233,18 @@ rpl_binlog_state::drop_domain(DYNAMIC_ARRAY *ids, >>> continue; >>> } >>> >>> - for (not_match= true, k= 0; k < elem->hash.records; k++) >>> + all_found= true; >>> + for (k= 0; k < elem->hash.records && all_found; k++) >>> { >>> rpl_gtid *d_gtid= (rpl_gtid *)my_hash_element(&elem->hash, k); >>> - for (ulong l= 0; l < glev->count && not_match; l++) >>> - not_match= !(*d_gtid == glev->list[l]); >>> + bool match_found= false; >>> + for (ulong l= 0; l < glev->count && !match_found; l++) >>> + match_found= match_found || (*d_gtid == glev->list[l]); >>> + if (!match_found) >>> + all_found= false; >>> } >>> >>> - if (not_match) >>> + if (!all_found) >>> { >>> sprintf(errbuf, "binlog files may contain gtids from the domain ('%u') " >>> "being deleted. Make sure to first purge those files", > > modified sql/rpl_gtid.cc > @@ -2220,7 +2220,7 @@ rpl_binlog_state::drop_domain(DYNAMIC_ARRAY *ids, > { > rpl_binlog_state::element *elem= NULL; > uint32 *ptr_domain_id; > - bool not_match; > + ulong match; // count-down for matches found > > ptr_domain_id= (uint32*) dynamic_array_ptr(ids, i); > elem= (rpl_binlog_state::element *) > @@ -2235,14 +2235,15 @@ rpl_binlog_state::drop_domain(DYNAMIC_ARRAY *ids, > continue; > } > > - for (not_match= true, k= 0; k < elem->hash.records; k++) > + for (match= elem->hash.records, k= 0; > + k < elem->hash.records && match + k == elem->hash.records; k++) > { > rpl_gtid *d_gtid= (rpl_gtid *)my_hash_element(&elem->hash, k); > - for (ulong l= 0; l < glev->count && not_match; l++) > - not_match= !(*d_gtid == glev->list[l]); > + for (ulong l= 0, same= 0; l < glev->count && !same; l++) > + match-= (same= (*d_gtid == glev->list[l])); > } > > - if (not_match) > + if (match) > { > sprintf(errbuf, "binlog files may contain gtids from the domain ('%u') " > "being deleted. Make sure to first purge those files",

1 0

Re: [Maria-developers] MDEV-26301: Obvious wrong query result
by Sergey Petrunia 28 Apr '23

28 Apr '23

Hello Igor, I'm looking at your latest patch for MDEV-26301 in bb-10.4-igor and I see an obvious bug in the execution part: Split-materialized table re-fill is triggered by this code in sub_select(): if (join_tab->split_derived_to_update && !end_of_records) { table_map tab_map= join_tab->split_derived_to_update; for (uint i= 0; tab_map; i++, tab_map>>= 1) { if (tab_map & 1) join->map2table[i]->preread_init_done= false; } } (did you see Table_map_iterator class btw?) but what if the table in question uses join buffer and so uses sub_select_cache(), not sub_select()? Please find the testcase below: # 5 values create table t1(a int, b int); insert into t1 select seq,seq from seq_1_to_5; # 5 value groups of size 2 each create table t2(a int, b int, key(a)); insert into t2 select A.seq,B.seq from seq_1_to_25 A, seq_1_to_2 B; # 5 value groups of size 3 each create table t3(a int, b int, key(a)); insert into t3 select A.seq,B.seq from seq_1_to_5 A, seq_1_to_3 B; analyze table t1,t2,t3 persistent for all; create table t10 ( grp_id int, col1 int, key(grp_id) ); # 100 groups of 100 values each insert into t10 select A.seq, B.seq from seq_1_to_100 A, seq_1_to_100 B; # and X10 multiplier create table t11 ( col1 int, col2 int ); insert into t11 select A.seq, A.seq from seq_1_to_10 A; analyze table t10,t11 persistent for all; explain select * from ( (t1 left join t2 on t2.a=t1.b) left join t3 on t3.a=t1.b ) left join (select grp_id, count(*) from t10 left join t11 on t11.col1=t10.col1 group by grp_id) T on T.grp_id=t1.b; +------+-----------------+------------+------+---------------+--------+---------+---------+------+-------------------------------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +------+-----------------+------------+------+---------------+--------+---------+---------+------+-------------------------------------------------+ | 1 | PRIMARY | t1 | ALL | NULL | NULL | NULL | NULL | 5 | | | 1 | PRIMARY | t2 | ref | a | a | 5 | j1.t1.b | 2 | Using where | | 1 | PRIMARY | t3 | ref | a | a | 5 | j1.t1.b | 3 | Using where | | 1 | PRIMARY | <derived2> | ref | key0 | key0 | 5 | j1.t1.b | 10 | Using where | | 2 | LATERAL DERIVED | t10 | ref | grp_id | grp_id | 5 | j1.t1.b | 100 | | | 2 | LATERAL DERIVED | t11 | ALL | NULL | NULL | NULL | NULL | 10 | Using where; Using join buffer (flat, BNL join) | +------+-----------------+------------+------+---------------+--------+---------+---------+------+-------------------------------------------------+ Ok Note the query result: MariaDB [j1]> select * from -> ( -> (t1 left join t2 on t2.a=t1.b) -> left join t3 on t3.a=t1.b -> ) left join (select grp_id, count(*) -> from t10 left join t11 on t11.col1=t10.col1 -> group by grp_id) T on T.grp_id=t1.b; +------+------+------+------+------+------+--------+----------+ | a | b | a | b | a | b | grp_id | count(*) | +------+------+------+------+------+------+--------+----------+ | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 100 | | 1 | 1 | 1 | 1 | 1 | 2 | 1 | 100 | | 1 | 1 | 1 | 1 | 1 | 3 | 1 | 100 | | 1 | 1 | 1 | 2 | 1 | 1 | 1 | 100 | | 1 | 1 | 1 | 2 | 1 | 2 | 1 | 100 | | 1 | 1 | 1 | 2 | 1 | 3 | 1 | 100 | | 2 | 2 | 2 | 1 | 2 | 1 | 2 | 100 | | 2 | 2 | 2 | 1 | 2 | 2 | 2 | 100 | | 2 | 2 | 2 | 1 | 2 | 3 | 2 | 100 | | 2 | 2 | 2 | 2 | 2 | 1 | 2 | 100 | | 2 | 2 | 2 | 2 | 2 | 2 | 2 | 100 | | 2 | 2 | 2 | 2 | 2 | 3 | 2 | 100 | | 3 | 3 | 3 | 1 | 3 | 1 | 3 | 100 | | 3 | 3 | 3 | 1 | 3 | 2 | 3 | 100 | | 3 | 3 | 3 | 1 | 3 | 3 | 3 | 100 | | 3 | 3 | 3 | 2 | 3 | 1 | 3 | 100 | | 3 | 3 | 3 | 2 | 3 | 2 | 3 | 100 | | 3 | 3 | 3 | 2 | 3 | 3 | 3 | 100 | | 4 | 4 | 4 | 1 | 4 | 1 | 4 | 100 | | 4 | 4 | 4 | 1 | 4 | 2 | 4 | 100 | | 4 | 4 | 4 | 1 | 4 | 3 | 4 | 100 | | 4 | 4 | 4 | 2 | 4 | 1 | 4 | 100 | | 4 | 4 | 4 | 2 | 4 | 2 | 4 | 100 | | 4 | 4 | 4 | 2 | 4 | 3 | 4 | 100 | | 5 | 5 | 5 | 1 | 5 | 1 | 5 | 100 | | 5 | 5 | 5 | 1 | 5 | 2 | 5 | 100 | | 5 | 5 | 5 | 1 | 5 | 3 | 5 | 100 | | 5 | 5 | 5 | 2 | 5 | 1 | 5 | 100 | | 5 | 5 | 5 | 2 | 5 | 2 | 5 | 100 | | 5 | 5 | 5 | 2 | 5 | 3 | 5 | 100 | +------+------+------+------+------+------+--------+----------+ 30 rows in set (0.015 sec) Now, let's make the tables not use indexes and use join buffer in the top-level select: explain select * from ( (t1 left join t2 use index() on t2.a=t1.b) left join t3 use index() on t3.a=t1.b ) left join (select grp_id, count(*) from t10 left join t11 on t11.col1=t10.col1 group by grp_id) T on T.grp_id=t1.b; +------+-----------------+------------+------+---------------+--------+---------+---------+------+--------------------------------------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +------+-----------------+------------+------+---------------+--------+---------+---------+------+--------------------------------------------------------+ | 1 | PRIMARY | t1 | ALL | NULL | NULL | NULL | NULL | 5 | | | 1 | PRIMARY | t2 | ALL | NULL | NULL | NULL | NULL | 50 | Using where; Using join buffer (flat, BNL join) | | 1 | PRIMARY | t3 | ALL | NULL | NULL | NULL | NULL | 15 | Using where; Using join buffer (incremental, BNL join) | | 1 | PRIMARY | <derived2> | ref | key0 | key0 | 5 | j1.t1.b | 10 | Using where | | 2 | LATERAL DERIVED | t10 | ref | grp_id | grp_id | 5 | j1.t1.b | 100 | | | 2 | LATERAL DERIVED | t11 | ALL | NULL | NULL | NULL | NULL | 10 | Using where; Using join buffer (flat, BNL join) | +------+-----------------+------------+------+---------------+--------+---------+---------+------+--------------------------------------------------------+ select * from ( (t1 left join t2 use index() on t2.a=t1.b) left join t3 use index() on t3.a=t1.b ) left join (select grp_id, count(*) from t10 left join t11 on t11.col1=t10.col1 group by grp_id) T on T.grp_id=t1.b; +------+------+------+------+------+------+--------+----------+ | a | b | a | b | a | b | grp_id | count(*) | +------+------+------+------+------+------+--------+----------+ | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 100 | | 1 | 1 | 1 | 2 | 1 | 1 | 1 | 100 | | 1 | 1 | 1 | 1 | 1 | 2 | 1 | 100 | | 1 | 1 | 1 | 2 | 1 | 2 | 1 | 100 | | 1 | 1 | 1 | 1 | 1 | 3 | 1 | 100 | | 1 | 1 | 1 | 2 | 1 | 3 | 1 | 100 | | 2 | 2 | 2 | 1 | 2 | 1 | NULL | NULL | | 2 | 2 | 2 | 2 | 2 | 1 | NULL | NULL | | 2 | 2 | 2 | 1 | 2 | 2 | NULL | NULL | | 2 | 2 | 2 | 2 | 2 | 2 | NULL | NULL | | 2 | 2 | 2 | 1 | 2 | 3 | NULL | NULL | | 2 | 2 | 2 | 2 | 2 | 3 | NULL | NULL | | 3 | 3 | 3 | 1 | 3 | 1 | NULL | NULL | | 3 | 3 | 3 | 2 | 3 | 1 | NULL | NULL | | 3 | 3 | 3 | 1 | 3 | 2 | NULL | NULL | | 3 | 3 | 3 | 2 | 3 | 2 | NULL | NULL | | 3 | 3 | 3 | 1 | 3 | 3 | NULL | NULL | | 3 | 3 | 3 | 2 | 3 | 3 | NULL | NULL | | 4 | 4 | 4 | 1 | 4 | 1 | NULL | NULL | | 4 | 4 | 4 | 2 | 4 | 1 | NULL | NULL | | 4 | 4 | 4 | 1 | 4 | 2 | NULL | NULL | | 4 | 4 | 4 | 2 | 4 | 2 | NULL | NULL | | 4 | 4 | 4 | 1 | 4 | 3 | NULL | NULL | | 4 | 4 | 4 | 2 | 4 | 3 | NULL | NULL | | 5 | 5 | 5 | 1 | 5 | 1 | NULL | NULL | | 5 | 5 | 5 | 2 | 5 | 1 | NULL | NULL | | 5 | 5 | 5 | 1 | 5 | 2 | NULL | NULL | | 5 | 5 | 5 | 2 | 5 | 2 | NULL | NULL | | 5 | 5 | 5 | 1 | 5 | 3 | NULL | NULL | | 5 | 5 | 5 | 2 | 5 | 3 | NULL | NULL | +------+------+------+------+------+------+--------+----------+ 30 rows in set (0.008 sec) Bummer. BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://petrunia.net

1 0

Re: [Maria-developers] d25439f1cbb: MDEV-31140: FLUSH BINARY LOGS DELETE_DOMAIN_ID=(D) can errorneously delete active domains
by Kristian Nielsen 27 Apr '23

27 Apr '23

Hi Andrei, Want to review a small patch for https://jira.mariadb.org/browse/MDEV-31140 ? I found this problem while trying to find the root cause of https://jira.mariadb.org/browse/MDEV-30386 . However this bug seems to be unrelated to MDEV-30386. Basically the logic for comparing the gtid_binlog_state with the GTID_LIST event was flawed, making it possible to delete a domain_id from the binlog state when this should not be allowed. - Kristian. Kristian Nielsen <knielsen(a)knielsen-hq.org> writes: > revision-id: d25439f1cbb79e5467b4249792130bfe524e3f10 (mariadb-10.11.2-21-gd25439f1cbb) > parent(s): 656c2e18b1e9ea5d0314745f3988d126eedbc22a > author: Kristian Nielsen > committer: Kristian Nielsen > timestamp: 2023-04-27 12:16:07 +0200 > message: > > MDEV-31140: FLUSH BINARY LOGS DELETE_DOMAIN_ID=(D) can errorneously delete active domains > > Fix the code in rpl_binlog_state::drop_domain(), so that _all_ entries for > the domain in the binlog state must match an entry in the initial GTID_LIST, > not just one entry match. > > Signed-off-by: Kristian Nielsen <knielsen(a)knielsen-hq.org> > > --- > .../r/binlog_flush_binlogs_delete_domain.result | 14 ++++++++++--- > .../t/binlog_flush_binlogs_delete_domain.test | 13 +++++++++++- > sql/rpl_gtid.cc | 24 ++++++++++++---------- > 3 files changed, 36 insertions(+), 15 deletions(-) > > diff --git a/mysql-test/suite/binlog/r/binlog_flush_binlogs_delete_domain.result b/mysql-test/suite/binlog/r/binlog_flush_binlogs_delete_domain.result > index fdcfb4bfa01..1c11191802f 100644 > --- a/mysql-test/suite/binlog/r/binlog_flush_binlogs_delete_domain.result > +++ b/mysql-test/suite/binlog/r/binlog_flush_binlogs_delete_domain.result > @@ -46,15 +46,23 @@ Warning 1076 The current gtid binlog state is incompatible with a former one mis > Warning 1076 The gtid domain being deleted ('1') is not in the current binlog state > FLUSH BINARY LOGS DELETE_DOMAIN_ID = (1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 0); > ERROR HY000: Could not delete gtid domain. Reason: binlog files may contain gtids from the domain ('1') being deleted. Make sure to first purge those files. > +MDEV-31140: Missing error from DELETE_DOMAIN_ID when gtid_binlog_state partially matches GTID_LIST. > FLUSH BINARY LOGS; > PURGE BINARY LOGS TO 'master-bin.000005'; > +SET @@SESSION.gtid_domain_id=8; > +SET @@SESSION.server_id=10*8 + 1; > +INSERT INTO t SELECT 1+MAX(a) FROM t; > +FLUSH BINARY LOGS DELETE_DOMAIN_ID = (1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 0); > +ERROR HY000: Could not delete gtid domain. Reason: binlog files may contain gtids from the domain ('8') being deleted. Make sure to first purge those files. > +FLUSH BINARY LOGS; > +PURGE BINARY LOGS TO 'master-bin.000006'; > FLUSH BINARY LOGS DELETE_DOMAIN_ID = (1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 0); > Warnings: > Warning 1076 The gtid domain being deleted ('0') is not in the current binlog state > Gtid_list of the current binlog does not contain 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 0: > -show binlog events in 'master-bin.000006' limit 1,1; > +show binlog events in 'master-bin.000007' limit 1,1; > Log_name Pos Event_type Server_id End_log_pos Info > -master-bin.000006 # Gtid_list 1 # [] > +master-bin.000007 # Gtid_list 1 # [] > SET @@SESSION.gtid_domain_id=1;; > SET @@SESSION.server_id=1; > SET @@SESSION.gtid_seq_no=1; > @@ -75,7 +83,7 @@ INSERT INTO t SET a=1; > SELECT @gtid_binlog_state_saved "as original state", @@GLOBAL.gtid_binlog_state as "out of order for 11 domain state"; > as original state out of order for 11 domain state > 1-1-1,1-2-2,11-11-11 1-1-1,1-2-2,11-11-1 > -PURGE BINARY LOGS TO 'master-bin.000007'; > +PURGE BINARY LOGS TO 'master-bin.000008'; > the following command succeeds with warnings > FLUSH BINARY LOGS DELETE_DOMAIN_ID = (1); > Warnings: > diff --git a/mysql-test/suite/binlog/t/binlog_flush_binlogs_delete_domain.test b/mysql-test/suite/binlog/t/binlog_flush_binlogs_delete_domain.test > index 8311f4bd800..1643ecff72d 100644 > --- a/mysql-test/suite/binlog/t/binlog_flush_binlogs_delete_domain.test > +++ b/mysql-test/suite/binlog/t/binlog_flush_binlogs_delete_domain.test > @@ -21,7 +21,6 @@ FLUSH BINARY LOGS DELETE_DOMAIN_ID = (); > --echo but with a warning > --let $binlog_pre_flush=query_get_value(SHOW MASTER STATUS, Position, 1) > FLUSH BINARY LOGS DELETE_DOMAIN_ID = (99); > ---let $binlog_start=$binlog_pre_flush > --source include/show_binary_logs.inc > > # Log one event in a specified domain and try to delete the domain > @@ -62,6 +61,8 @@ FLUSH BINARY LOGS DELETE_DOMAIN_ID = (1); > # expected overrun of the static buffers of underlying dynamic arrays is doing. > --let $domain_cnt=17 > --let $server_in_domain_cnt=3 > +--let $err_domain_id=`SELECT FLOOR($domain_cnt/2)` > +--let $err_server_id=`SELECT FLOOR($server_in_domain_cnt/2)` > --let $domain_list= > --disable_query_log > while ($domain_cnt) > @@ -86,6 +87,16 @@ while ($domain_cnt) > --error ER_BINLOG_CANT_DELETE_GTID_DOMAIN > --eval FLUSH BINARY LOGS DELETE_DOMAIN_ID = ($domain_list) > > +--echo MDEV-31140: Missing error from DELETE_DOMAIN_ID when gtid_binlog_state partially matches GTID_LIST. > +FLUSH BINARY LOGS; > +--let $purge_to_binlog= query_get_value(SHOW MASTER STATUS, File, 1) > +--eval PURGE BINARY LOGS TO '$purge_to_binlog' > +--eval SET @@SESSION.gtid_domain_id=$err_domain_id > +--eval SET @@SESSION.server_id=10*$err_domain_id + $err_server_id > +eval INSERT INTO t SELECT 1+MAX(a) FROM t; > +--error ER_BINLOG_CANT_DELETE_GTID_DOMAIN > +--eval FLUSH BINARY LOGS DELETE_DOMAIN_ID = ($domain_list) > + > # Now satisfy the safety condtion to purge log files containing $domain list > FLUSH BINARY LOGS; > --let $purge_to_binlog= query_get_value(SHOW MASTER STATUS, File, 1) > diff --git a/sql/rpl_gtid.cc b/sql/rpl_gtid.cc > index c4e5c75b10a..7b67a83b3dd 100644 > --- a/sql/rpl_gtid.cc > +++ b/sql/rpl_gtid.cc > @@ -2209,18 +2209,16 @@ rpl_binlog_state::drop_domain(DYNAMIC_ARRAY *ids, > > /* > For each domain_id from ids > - when no such domain in binlog state > - warn && continue > - For each domain.server's last gtid > - when not locate the last gtid in glev.list > - error out binlog state can't change > - otherwise continue > + If the domain is already absent from the binlog state > + Warn && continue > + If any GTID with that domain in binlog state is missing from glev.list > + Error out binlog state can't change > */ > for (ulong i= 0; i < ids->elements; i++) > { > rpl_binlog_state::element *elem= NULL; > uint32 *ptr_domain_id; > - bool not_match; > + bool all_found; > > ptr_domain_id= (uint32*) dynamic_array_ptr(ids, i); > elem= (rpl_binlog_state::element *) > @@ -2235,14 +2233,18 @@ rpl_binlog_state::drop_domain(DYNAMIC_ARRAY *ids, > continue; > } > > - for (not_match= true, k= 0; k < elem->hash.records; k++) > + all_found= true; > + for (k= 0; k < elem->hash.records && all_found; k++) > { > rpl_gtid *d_gtid= (rpl_gtid *)my_hash_element(&elem->hash, k); > - for (ulong l= 0; l < glev->count && not_match; l++) > - not_match= !(*d_gtid == glev->list[l]); > + bool match_found= false; > + for (ulong l= 0; l < glev->count && !match_found; l++) > + match_found= match_found || (*d_gtid == glev->list[l]); > + if (!match_found) > + all_found= false; > } > > - if (not_match) > + if (!all_found) > { > sprintf(errbuf, "binlog files may contain gtids from the domain ('%u') " > "being deleted. Make sure to first purge those files",

1 0

Re: [Maria-developers] MDEV-30946 Index usage for DATE(datetime_column) = const does not work for DELETE and UPDATE
by Sergey Petrunia 12 Apr '23

12 Apr '23

Hi Oleg, Why is the call to Item::date_conds_transformer done after prune_partitions()? This causes partition pruning not to be able to use the conditions: --source include/have_partition.inc --source include/have_sequence.inc create table t1 ( a datetime, key(a) ) partition by range(year(a)) ( partition p0 values less than (2022), partition p1 values less than (MAXVALUE) ); insert into t1 select date_add('2020-01-01', interval seq day) from seq_1_to_1000; explain partitions select * from t1 where year(a) = 2020; explain partitions delete from t1 where year(a) = 2020; drop table t1; Gives: explain partitions select * from t1 where year(a) = 2020; id select_type table partitions type possible_keys key key_len ref rows Extra 1 SIMPLE t1 p0 range a a 6 NULL 348 Using where; Using index explain partitions delete from t1 where year(a) = 2020; id select_type table partitions type possible_keys key key_len ref rows Extra 1 SIMPLE t1 p0,p1 ALL a NULL NULL NULL 1000 Using where Note that SELECT uses one partition while DELETE uses two. Please fix this. BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://petrunia.net

1 0

[Maria-developers] Participation in MariaDB Community's Open Source Project
by 和映泉 09 Apr '23

09 Apr '23

Dear MariaDB Development Team, I am writing to express my interest in participating in your open source project. I am an experienced programmer with knowledge of multiple programming languages and technologies, and I have some understanding of MariaDB products and source code. I believe I can make positive contributions to your project and gain more learning and growth opportunities from it. Here is a summary of my programming experience and skills: - I have 6 years of experience in software development and have participated in commercial projects. - I am proficient in multiple programming languages such as Java, Python, and Scala, and have knowledge of SQL and database technologies. - I am familiar with Linux systems and command-line tools, and can use version control tools such as Git. - I have some understanding and practical experience in software testing, code quality, and performance optimization. In addition, I have also used MySQL products and have some understanding of their functions and features. I believe this will help me better understand and contribute to your open source project. Thank you for taking the time to read my email. I look forward to your reply and hope to have the opportunity to contribute to the MariaDB community. Best regards, Yingquan He

2 1

[Maria-developers] CI failing on almost all branches - time to adopt protected branches?
by Otto Kekäläinen 03 Apr '23

03 Apr '23

Hello! Has there recently been any discussions about enabling the Github 'protected branch' feature? Review of current mainlines shows that ALL but 10.4 are broken and most of them on different tests. I am afraid that quality and test fixes will never catch up if people are allowed to push code that makes testable things fail. Using protected branches would prevent people from being able to push commits on mainline that fail in CI. https://github.com/MariaDB/server/tree/10.3 / 4c4939b NOT OK: - amd64-debian-10-debug-embedded fails on sql_sequence.kill - amd64-fedora-36 fails on main.func_math https://github.com/MariaDB/server/tree/10.4 / 1767390 OK https://github.com/MariaDB/server/tree/10.5 / ac5a534 NOT OK: - amd64-ubuntu-2004-msan fails on main.selectivity_innodb - amd64-windows fails to compile https://github.com/MariaDB/server/tree/10.6 / dc1d621 NOT OK: - buildbot/amd64-ubuntu-2204-msan fails on innodb.gap_locks https://github.com/MariaDB/server/tree/10.7 / bc656c4 NOT OK: - buildbot/amd64-debian-10 fails to start MTR https://github.com/MariaDB/server/tree/10.8 / dd2fe81 NOT OK: - amd64-ubuntu-2204-msan fails on MTR timeout (crash?) https://github.com/MariaDB/server/tree/10.9 / 55e78eb NOT OK: - buildbot/amd64-ubuntu-2204-msan fails on mariabackup.incremental_page_compressed innodb.innodb_page_compressed https://github.com/MariaDB/server/tree/10.10 / 191821f NOT OK: - amd64-ubuntu-2004-debug fails on binlog.binlog_truncate_multi_engine - amd64-ubuntu-2204-debug-ps fails on main.bad_startup_options https://github.com/MariaDB/server/tree/10.11 / d84a282 NOT OK: - buildbot/amd64-ubuntu-2204-msan fails on mariabackup.incremental_page_compressed innodb.innodb_page_compressed https://github.com/MariaDB/server/tree/11.0 / 8e55d7e NOT OK: - buildbot/amd64-ubuntu-2004-msan fails on innodb.table_definition_cache_debug main.lock_sync - continuous-integration/appveyor/branch fails on main.innodb_ext_key

2 1

[Maria-developers] How to get started with GSOC contribution
by Komala Elusuri 25 Mar '23

25 Mar '23

Respected sir, I am Komala, a computer science undergraduate from India. I am new to open source contributions but I am well aware of python, C, C++, Java,JavaScript and Databases. I did certifications on Google cloud computing foundation with Kubernetes and Python projects from Infosys springboard. I have experience as an intern from Smart Knower. I would love to contribute to your organisation but could you please tell me how to get started? Hoping to hear from you soon. Regards Komala

1 0

Re: [Maria-developers] 940d028521f: MDEV-30164 System variable for default collations
by Sergei Golubchik 21 Mar '23

21 Mar '23

Hi, Alexander, I'm sorry it took a while. I'm still thinking about the whole thing, it's a rather big change for a really fringe functionality. But I failed to come up with something better so far. Code-wise the patch is mostly fine. See few small comments below, and one slightly larger comment re. replication. On Mar 07, Alexander Barkov wrote: > revision-id: 940d028521f (mariadb-10.11.1-4-g940d028521f) > parent(s): 71dedd0ebcd > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2022-12-14 20:00:22 +0400 > message: > > MDEV-30164 System variable for default collations > > This patch adds a way to override default collations > (or "character set collations") for desired character sets. > > diff --git a/sql/lex_charset.h b/sql/lex_charset.h > --- a/sql/lex_charset.h > +++ b/sql/lex_charset.h > @@ -544,6 +699,20 @@ struct Lex_exact_charset_extended_collation_attrs_st > { > return m_ci; > } > + CHARSET_INFO *charset_info(const Charset_collation_map_st &map) const > + { > + switch (m_type) > + { > + case Lex_exact_charset_extended_collation_attrs_st::TYPE_CHARACTER_SET: > + return map.get_collation_for_charset(m_ci); > + case TYPE_EMPTY: Lex_exact_charset_extended_collation_attrs_st::TYPE_EMPTY (or all case labels without a struct name) > + case Lex_exact_charset_extended_collation_attrs_st::TYPE_CHARACTER_SET_COLLATE_EXACT: > + case Lex_exact_charset_extended_collation_attrs_st::TYPE_COLLATE_CONTEXTUALLY_TYPED: COLLATE DEFAULT is TYPE_COLLATE_CONTEXTUALLY_TYPED. shouldn't it use the map too? > + case Lex_exact_charset_extended_collation_attrs_st::TYPE_COLLATE_EXACT: > + break; > + } > + return m_ci; > + } > Type type() const > { > return m_type; > diff --git a/sql/log_event.h b/sql/log_event.h > --- a/sql/log_event.h > +++ b/sql/log_event.h > @@ -2147,6 +2153,8 @@ class Query_log_event: public Log_event > bool sql_mode_inited; > bool charset_inited; > > + LEX_CSTRING character_set_collations_str; better LEX_CUSTRING, don't you think? It's not even a _str. > + > uint32 flags2; > sql_mode_t sql_mode; > ulong auto_increment_increment, auto_increment_offset; > diff --git a/sql/log_event_server.cc b/sql/log_event_server.cc > --- a/sql/log_event_server.cc > +++ b/sql/log_event_server.cc > @@ -1194,6 +1194,14 @@ bool Query_log_event::write() > int2store(start+2, auto_increment_offset); > start+= 4; > } > + > + if (thd && thd->variables.character_set_collations.count()) > + { > + *start++= Q_COLLATIONS_SESSION; > + size_t len= thd->variables.character_set_collations.to_binary((char*)start); > + start+= len; > + } Perhaps, detect if it's needed? A cheap way of doing it would be to extend your Elem_st with a query_id. And every time you find_elem_by_charset, you set this elem's query_id to thd->query_id. And here you write only elements with the current query id. If any. Another approach would be to have a bitmap, like uchar used_default_coll_mapping; and in find_elem_by_charset() you set the bit, like used_default_coll_mapping |= 1 << i; and then, again, print affected collations, if any. Most often used_default_coll_mapping will likely be zero one more question. In, say, 10.10->11.1 replication master and slave will have different default collations, but thd->variables.character_set_collations will not reflect that. How do you plan to solve it? > + > if (charset_inited) > { > *start++= Q_CHARSET_CODE; Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 3

Re: [Maria-developers] MDEV-28152: Features for sequence
by Oleksandr Byelkin 21 Mar '23

21 Mar '23

Hi, Yuchen! As for first work for the server itself I find it is very good, but please read my notes below (many of them just code style ones) JFYI if you fix something in the wrong or old code style, you should fix it, but please do not fix the style only things in the code you are not going to change. It is one of the not yet written ( :( ) rules of our fix code guide. As well as code style defined only by old memory and examples in the code. > diff --git a/mysql-test/main/information_schema.result b/mysql-test/main/information_schema.result > index e46014e44b9..ce111c58796 100644 > --- a/mysql-test/main/information_schema.result > +++ b/mysql-test/main/information_schema.result > @@ -81,6 +81,7 @@ REFERENTIAL_CONSTRAINTS > ROUTINES > SCHEMATA > SCHEMA_PRIVILEGES > +SEQUENCES > SESSION_STATUS > SESSION_VARIABLES > SPATIAL_REF_SYS > diff --git a/mysql-test/main/information_schema_all_engines.result b/mysql-test/main/information_schema_all_engines.result > index 23a853e363c..316871995c1 100644 > --- a/mysql-test/main/information_schema_all_engines.result > +++ b/mysql-test/main/information_schema_all_engines.result > @@ -52,6 +52,7 @@ REFERENTIAL_CONSTRAINTS > ROUTINES > SCHEMATA > SCHEMA_PRIVILEGES > +SEQUENCES > SESSION_STATUS > SESSION_VARIABLES > SPATIAL_REF_SYS > @@ -133,6 +134,7 @@ REFERENTIAL_CONSTRAINTS CONSTRAINT_SCHEMA > ROUTINES ROUTINE_SCHEMA > SCHEMATA SCHEMA_NAME > SCHEMA_PRIVILEGES TABLE_SCHEMA > +SEQUENCES SEQUENCE_SCHEMA > SESSION_STATUS VARIABLE_NAME > SESSION_VARIABLES VARIABLE_NAME > SPATIAL_REF_SYS SRID > @@ -214,6 +216,7 @@ REFERENTIAL_CONSTRAINTS CONSTRAINT_SCHEMA > ROUTINES ROUTINE_SCHEMA > SCHEMATA SCHEMA_NAME > SCHEMA_PRIVILEGES TABLE_SCHEMA > +SEQUENCES SEQUENCE_SCHEMA > SESSION_STATUS VARIABLE_NAME > SESSION_VARIABLES VARIABLE_NAME > SPATIAL_REF_SYS SRID > @@ -299,6 +302,7 @@ REFERENTIAL_CONSTRAINTS information_schema.REFERENTIAL_CONSTRAINTS 1 > ROUTINES information_schema.ROUTINES 1 > SCHEMATA information_schema.SCHEMATA 1 > SCHEMA_PRIVILEGES information_schema.SCHEMA_PRIVILEGES 1 > +SEQUENCES information_schema.SEQUENCES 1 > SESSION_STATUS information_schema.SESSION_STATUS 1 > SESSION_VARIABLES information_schema.SESSION_VARIABLES 1 > SPATIAL_REF_SYS information_schema.SPATIAL_REF_SYS 1 > @@ -369,6 +373,7 @@ Database: information_schema > | ROUTINES | > | SCHEMATA | > | SCHEMA_PRIVILEGES | > +| SEQUENCES | > | SESSION_STATUS | > | SESSION_VARIABLES | > | SPATIAL_REF_SYS | > @@ -440,6 +445,7 @@ Database: INFORMATION_SCHEMA > | ROUTINES | > | SCHEMATA | > | SCHEMA_PRIVILEGES | > +| SEQUENCES | > | SESSION_STATUS | > | SESSION_VARIABLES | > | SPATIAL_REF_SYS | > @@ -463,5 +469,5 @@ Wildcard: inf_rmation_schema > | information_schema | > SELECT table_schema, count(*) FROM information_schema.TABLES WHERE table_schema IN ('mysql', 'INFORMATION_SCHEMA', 'test', 'mysqltest') GROUP BY TABLE_SCHEMA; > table_schema count(*) > -information_schema 66 > +information_schema 67 > mysql 31 > diff --git a/mysql-test/main/mysqldump.result b/mysql-test/main/mysqldump.result > index 44ed94e43c9..41607ea3068 100644 > --- a/mysql-test/main/mysqldump.result > +++ b/mysql-test/main/mysqldump.result > @@ -6369,16 +6369,16 @@ NEXTVAL(d.s1) NEXTVAL(d.s2) NEXTVAL(d.s3) NEXTVAL(d.s4) > # Show create before dump > show create sequence d.s1; > Table Create Table > -s1 CREATE SEQUENCE `s1` start with 100 minvalue 100 maxvalue 1100 increment by 10 cache 1000 cycle ENGINE=MyISAM > +s1 CREATE SEQUENCE `s1` as bigint start with 100 minvalue 100 maxvalue 1100 increment by 10 cache 1000 cycle ENGINE=MyISAM > show create sequence d.s2; > Table Create Table > -s2 CREATE SEQUENCE `s2` start with 200 minvalue 200 maxvalue 1200 increment by 20 cache 1000 cycle ENGINE=MyISAM > +s2 CREATE SEQUENCE `s2` as bigint start with 200 minvalue 200 maxvalue 1200 increment by 20 cache 1000 cycle ENGINE=MyISAM > show create sequence d.s3; > Table Create Table > -s3 CREATE SEQUENCE `s3` start with 300 minvalue 300 maxvalue 1300 increment by 30 cache 1000 cycle ENGINE=MyISAM > +s3 CREATE SEQUENCE `s3` as bigint start with 300 minvalue 300 maxvalue 1300 increment by 30 cache 1000 cycle ENGINE=MyISAM > show create sequence d.s4; > Table Create Table > -s4 CREATE SEQUENCE `s4` start with 400 minvalue 400 maxvalue 1400 increment by 40 cache 1000 cycle ENGINE=MyISAM > +s4 CREATE SEQUENCE `s4` as bigint start with 400 minvalue 400 maxvalue 1400 increment by 40 cache 1000 cycle ENGINE=MyISAM > # Dump sequence without `--no-data` > # Restore from mysqldump > SETVAL(`s1`, 1101, 0) > @@ -6392,16 +6392,16 @@ SETVAL(`s4`, 1401, 0) > # Show create after restore > show create sequence d.s1; > Table Create Table > -s1 CREATE SEQUENCE `s1` start with 100 minvalue 100 maxvalue 1100 increment by 10 cache 1000 cycle ENGINE=MyISAM > +s1 CREATE SEQUENCE `s1` as bigint start with 100 minvalue 100 maxvalue 1100 increment by 10 cache 1000 cycle ENGINE=MyISAM > show create sequence d.s2; > Table Create Table > -s2 CREATE SEQUENCE `s2` start with 200 minvalue 200 maxvalue 1200 increment by 20 cache 1000 cycle ENGINE=MyISAM > +s2 CREATE SEQUENCE `s2` as bigint start with 200 minvalue 200 maxvalue 1200 increment by 20 cache 1000 cycle ENGINE=MyISAM > show create sequence d.s3; > Table Create Table > -s3 CREATE SEQUENCE `s3` start with 300 minvalue 300 maxvalue 1300 increment by 30 cache 1000 cycle ENGINE=MyISAM > +s3 CREATE SEQUENCE `s3` as bigint start with 300 minvalue 300 maxvalue 1300 increment by 30 cache 1000 cycle ENGINE=MyISAM > show create sequence d.s4; > Table Create Table > -s4 CREATE SEQUENCE `s4` start with 400 minvalue 400 maxvalue 1400 increment by 40 cache 1000 cycle ENGINE=MyISAM > +s4 CREATE SEQUENCE `s4` as bigint start with 400 minvalue 400 maxvalue 1400 increment by 40 cache 1000 cycle ENGINE=MyISAM > SELECT NEXTVAL(d.s1),NEXTVAL(d.s2),NEXTVAL(d.s3), NEXTVAL(d.s4); > NEXTVAL(d.s1) NEXTVAL(d.s2) NEXTVAL(d.s3) NEXTVAL(d.s4) > 100 200 300 400 > @@ -6418,16 +6418,16 @@ SETVAL(`s4`, 1401, 0) > # Show create after restore `--no-data` > show create sequence d.s1; > Table Create Table > -s1 CREATE SEQUENCE `s1` start with 100 minvalue 100 maxvalue 1100 increment by 10 cache 1000 cycle ENGINE=MyISAM > +s1 CREATE SEQUENCE `s1` as bigint start with 100 minvalue 100 maxvalue 1100 increment by 10 cache 1000 cycle ENGINE=MyISAM > show create sequence d.s2; > Table Create Table > -s2 CREATE SEQUENCE `s2` start with 200 minvalue 200 maxvalue 1200 increment by 20 cache 1000 cycle ENGINE=MyISAM > +s2 CREATE SEQUENCE `s2` as bigint start with 200 minvalue 200 maxvalue 1200 increment by 20 cache 1000 cycle ENGINE=MyISAM > show create sequence d.s3; > Table Create Table > -s3 CREATE SEQUENCE `s3` start with 300 minvalue 300 maxvalue 1300 increment by 30 cache 1000 cycle ENGINE=MyISAM > +s3 CREATE SEQUENCE `s3` as bigint start with 300 minvalue 300 maxvalue 1300 increment by 30 cache 1000 cycle ENGINE=MyISAM > show create sequence d.s4; > Table Create Table > -s4 CREATE SEQUENCE `s4` start with 400 minvalue 400 maxvalue 1400 increment by 40 cache 1000 cycle ENGINE=MyISAM > +s4 CREATE SEQUENCE `s4` as bigint start with 400 minvalue 400 maxvalue 1400 increment by 40 cache 1000 cycle ENGINE=MyISAM > SELECT NEXTVAL(d.s1),NEXTVAL(d.s2),NEXTVAL(d.s3), NEXTVAL(d.s4); > NEXTVAL(d.s1) NEXTVAL(d.s2) NEXTVAL(d.s3) NEXTVAL(d.s4) > 100 200 300 400 > @@ -6443,7 +6443,7 @@ SETVAL(`s4`, 1401, 0) > 1401 > show create sequence d2.s1; > Table Create Table > -s1 CREATE SEQUENCE `s1` start with 100 minvalue 100 maxvalue 1100 increment by 10 cache 1000 cycle ENGINE=MyISAM > +s1 CREATE SEQUENCE `s1` as bigint start with 100 minvalue 100 maxvalue 1100 increment by 10 cache 1000 cycle ENGINE=MyISAM > drop sequence d.s1, d.s2, d.s3, d.s4; > drop database d; > drop database d2; > diff --git a/mysql-test/suite/funcs_1/r/is_columns_is.result b/mysql-test/suite/funcs_1/r/is_columns_is.result > index c88a3a9ac8d..03eece0b4ae 100644 > --- a/mysql-test/suite/funcs_1/r/is_columns_is.result > +++ b/mysql-test/suite/funcs_1/r/is_columns_is.result > @@ -339,6 +339,21 @@ def information_schema SCHEMA_PRIVILEGES IS_GRANTABLE 5 NULL NO varchar 3 9 NULL > def information_schema SCHEMA_PRIVILEGES PRIVILEGE_TYPE 4 NULL NO varchar 64 192 NULL NULL NULL utf8mb3 utf8mb3_general_ci varchar(64) select NEVER NULL > def information_schema SCHEMA_PRIVILEGES TABLE_CATALOG 2 NULL NO varchar 512 1536 NULL NULL NULL utf8mb3 utf8mb3_general_ci varchar(512) select NEVER NULL > def information_schema SCHEMA_PRIVILEGES TABLE_SCHEMA 3 NULL NO varchar 64 192 NULL NULL NULL utf8mb3 utf8mb3_general_ci varchar(64) select NEVER NULL > +def information_schema SEQUENCES CYCLE_OPTION 12 NULL NO bigint NULL NULL 19 0 NULL NULL NULL bigint(21) select NEVER NULL As I remember CYCLE_OPTION is boolean or YES/NO, why does it become bigint ? > +def information_schema SEQUENCES DATA_TYPE 4 NULL NO varchar 64 192 NULL NULL NULL utf8mb3 utf8mb3_general_ci varchar(64) select NEVER NULL > +def information_schema SEQUENCES DECLARED_DATA_TYPE 13 NULL YES int NULL NULL 10 0 NULL NULL NULL int(21) select NEVER NULL > +def information_schema SEQUENCES DECLARED_NUMERIC_PRECISION 14 NULL YES int NULL NULL 10 0 NULL NULL NULL int(21) select NEVER NULL > +def information_schema SEQUENCES DECLARED_NUMERIC_SCALE 15 NULL YES int NULL NULL 10 0 NULL NULL NULL int(21) select NEVER NULL We do not need DECLARED_* it is only needed for feature T322 “Declared data type attributes” and we do not have it. > +def information_schema SEQUENCES INCREMENT 11 NULL NO bigint NULL NULL 19 0 NULL NULL NULL bigint(21) select NEVER NULL > +def information_schema SEQUENCES MAXIMUM_VALUE 10 NULL NO decimal NULL NULL 21 0 NULL NULL NULL decimal(21,0) select NEVER NULL > +def information_schema SEQUENCES MINIMUM_VALUE 9 NULL NO decimal NULL NULL 21 0 NULL NULL NULL decimal(21,0) select NEVER NULL good decision to make it decimal > +def information_schema SEQUENCES NUMERIC_PRECISION 5 NULL NO int NULL NULL 10 0 NULL NULL NULL int(21) select NEVER NULL > +def information_schema SEQUENCES NUMERIC_PRECISION_RADIX 6 NULL YES int NULL NULL 10 0 NULL NULL NULL int(21) select NEVER NULL I have not found tests of NUMERIC_* output for all supported by us types. > +def information_schema SEQUENCES NUMERIC_SCALE 7 NULL YES int NULL NULL 10 0 NULL NULL NULL int(21) select NEVER NULL > +def information_schema SEQUENCES SEQUENCE_CATALOG 1 NULL NO varchar 512 1536 NULL NULL NULL utf8mb3 utf8mb3_general_ci varchar(512) select NEVER NULL > +def information_schema SEQUENCES SEQUENCE_NAME 3 NULL NO varchar 64 192 NULL NULL NULL utf8mb3 utf8mb3_general_ci varchar(64) select NEVER NULL > +def information_schema SEQUENCES SEQUENCE_SCHEMA 2 NULL NO varchar 64 192 NULL NULL NULL utf8mb3 utf8mb3_general_ci varchar(64) select NEVER NULL > +def information_schema SEQUENCES START_VALUE 8 NULL NO decimal NULL NULL 21 0 NULL NULL NULL decimal(21,0) select NEVER NULL I do not see CACHED reflected here, please add it. > def information_schema SESSION_STATUS VARIABLE_NAME 1 NULL NO varchar 64 192 NULL NULL NULL utf8mb3 utf8mb3_general_ci varchar(64) select NEVER NULL > def information_schema SESSION_STATUS VARIABLE_VALUE 2 NULL NO varchar 2048 6144 NULL NULL NULL utf8mb3 utf8mb3_general_ci varchar(2048) select NEVER NULL > def information_schema SESSION_VARIABLES VARIABLE_NAME 1 NULL NO varchar 64 192 NULL NULL NULL utf8mb3 utf8mb3_general_ci varchar(64) select NEVER NULL > @@ -890,6 +905,21 @@ NULL information_schema ROUTINES LAST_ALTERED datetime NULL NULL NULL NULL datet > 3.0000 information_schema SCHEMA_PRIVILEGES TABLE_SCHEMA varchar 64 192 utf8mb3 utf8mb3_general_ci varchar(64) > 3.0000 information_schema SCHEMA_PRIVILEGES PRIVILEGE_TYPE varchar 64 192 utf8mb3 utf8mb3_general_ci varchar(64) > 3.0000 information_schema SCHEMA_PRIVILEGES IS_GRANTABLE varchar 3 9 utf8mb3 utf8mb3_general_ci varchar(3) > +3.0000 information_schema SEQUENCES SEQUENCE_CATALOG varchar 512 1536 utf8mb3 utf8mb3_general_ci varchar(512) > +3.0000 information_schema SEQUENCES SEQUENCE_SCHEMA varchar 64 192 utf8mb3 utf8mb3_general_ci varchar(64) > +3.0000 information_schema SEQUENCES SEQUENCE_NAME varchar 64 192 utf8mb3 utf8mb3_general_ci varchar(64) > +3.0000 information_schema SEQUENCES DATA_TYPE varchar 64 192 utf8mb3 utf8mb3_general_ci varchar(64) > +NULL information_schema SEQUENCES NUMERIC_PRECISION int NULL NULL NULL NULL int(21) > +NULL information_schema SEQUENCES NUMERIC_PRECISION_RADIX int NULL NULL NULL NULL int(21) > +NULL information_schema SEQUENCES NUMERIC_SCALE int NULL NULL NULL NULL int(21) > +NULL information_schema SEQUENCES START_VALUE decimal NULL NULL NULL NULL decimal(21,0) > +NULL information_schema SEQUENCES MINIMUM_VALUE decimal NULL NULL NULL NULL decimal(21,0) > +NULL information_schema SEQUENCES MAXIMUM_VALUE decimal NULL NULL NULL NULL decimal(21,0) > +NULL information_schema SEQUENCES INCREMENT bigint NULL NULL NULL NULL bigint(21) > +NULL information_schema SEQUENCES CYCLE_OPTION bigint NULL NULL NULL NULL bigint(21) > +NULL information_schema SEQUENCES DECLARED_DATA_TYPE int NULL NULL NULL NULL int(21) > +NULL information_schema SEQUENCES DECLARED_NUMERIC_PRECISION int NULL NULL NULL NULL int(21) > +NULL information_schema SEQUENCES DECLARED_NUMERIC_SCALE int NULL NULL NULL NULL int(21) > 3.0000 information_schema SESSION_STATUS VARIABLE_NAME varchar 64 192 utf8mb3 utf8mb3_general_ci varchar(64) > 3.0000 information_schema SESSION_STATUS VARIABLE_VALUE varchar 2048 6144 utf8mb3 utf8mb3_general_ci varchar(2048) > 3.0000 information_schema SESSION_VARIABLES VARIABLE_NAME varchar 64 192 utf8mb3 utf8mb3_general_ci varchar(64) > diff --git a/mysql-test/suite/funcs_1/r/is_columns_is_embedded.result b/mysql-test/suite/funcs_1/r/is_columns_is_embedded.result > index bb12a0c38df..87cc440cdbc 100644 > --- a/mysql-test/suite/funcs_1/r/is_columns_is_embedded.result > +++ b/mysql-test/suite/funcs_1/r/is_columns_is_embedded.result > @@ -339,6 +339,21 @@ def information_schema SCHEMA_PRIVILEGES IS_GRANTABLE 5 NULL NO varchar 3 9 NULL > def information_schema SCHEMA_PRIVILEGES PRIVILEGE_TYPE 4 NULL NO varchar 64 192 NULL NULL NULL utf8mb3 utf8mb3_general_ci varchar(64) NEVER NULL > def information_schema SCHEMA_PRIVILEGES TABLE_CATALOG 2 NULL NO varchar 512 1536 NULL NULL NULL utf8mb3 utf8mb3_general_ci varchar(512) NEVER NULL > def information_schema SCHEMA_PRIVILEGES TABLE_SCHEMA 3 NULL NO varchar 64 192 NULL NULL NULL utf8mb3 utf8mb3_general_ci varchar(64) NEVER NULL > +def information_schema SEQUENCES CYCLE_OPTION 12 NULL NO bigint NULL NULL 19 0 NULL NULL NULL bigint(21) NEVER NULL > +def information_schema SEQUENCES DATA_TYPE 4 NULL NO varchar 64 192 NULL NULL NULL utf8mb3 utf8mb3_general_ci varchar(64) NEVER NULL > +def information_schema SEQUENCES DECLARED_DATA_TYPE 13 NULL YES int NULL NULL 10 0 NULL NULL NULL int(21) NEVER NULL > +def information_schema SEQUENCES DECLARED_NUMERIC_PRECISION 14 NULL YES int NULL NULL 10 0 NULL NULL NULL int(21) NEVER NULL > +def information_schema SEQUENCES DECLARED_NUMERIC_SCALE 15 NULL YES int NULL NULL 10 0 NULL NULL NULL int(21) NEVER NULL > +def information_schema SEQUENCES INCREMENT 11 NULL NO bigint NULL NULL 19 0 NULL NULL NULL bigint(21) NEVER NULL > +def information_schema SEQUENCES MAXIMUM_VALUE 10 NULL NO decimal NULL NULL 21 0 NULL NULL NULL decimal(21,0) NEVER NULL > +def information_schema SEQUENCES MINIMUM_VALUE 9 NULL NO decimal NULL NULL 21 0 NULL NULL NULL decimal(21,0) NEVER NULL > +def information_schema SEQUENCES NUMERIC_PRECISION 5 NULL NO int NULL NULL 10 0 NULL NULL NULL int(21) NEVER NULL > +def information_schema SEQUENCES NUMERIC_PRECISION_RADIX 6 NULL YES int NULL NULL 10 0 NULL NULL NULL int(21) NEVER NULL > +def information_schema SEQUENCES NUMERIC_SCALE 7 NULL YES int NULL NULL 10 0 NULL NULL NULL int(21) NEVER NULL > +def information_schema SEQUENCES SEQUENCE_CATALOG 1 NULL NO varchar 512 1536 NULL NULL NULL utf8mb3 utf8mb3_general_ci varchar(512) NEVER NULL > +def information_schema SEQUENCES SEQUENCE_NAME 3 NULL NO varchar 64 192 NULL NULL NULL utf8mb3 utf8mb3_general_ci varchar(64) NEVER NULL > +def information_schema SEQUENCES SEQUENCE_SCHEMA 2 NULL NO varchar 64 192 NULL NULL NULL utf8mb3 utf8mb3_general_ci varchar(64) NEVER NULL > +def information_schema SEQUENCES START_VALUE 8 NULL NO decimal NULL NULL 21 0 NULL NULL NULL decimal(21,0) NEVER NULL > def information_schema SESSION_STATUS VARIABLE_NAME 1 NULL NO varchar 64 192 NULL NULL NULL utf8mb3 utf8mb3_general_ci varchar(64) NEVER NULL > def information_schema SESSION_STATUS VARIABLE_VALUE 2 NULL NO varchar 2048 6144 NULL NULL NULL utf8mb3 utf8mb3_general_ci varchar(2048) NEVER NULL > def information_schema SESSION_VARIABLES VARIABLE_NAME 1 NULL NO varchar 64 192 NULL NULL NULL utf8mb3 utf8mb3_general_ci varchar(64) NEVER NULL > @@ -890,6 +905,21 @@ NULL information_schema ROUTINES LAST_ALTERED datetime NULL NULL NULL NULL datet > 3.0000 information_schema SCHEMA_PRIVILEGES TABLE_SCHEMA varchar 64 192 utf8mb3 utf8mb3_general_ci varchar(64) > 3.0000 information_schema SCHEMA_PRIVILEGES PRIVILEGE_TYPE varchar 64 192 utf8mb3 utf8mb3_general_ci varchar(64) > 3.0000 information_schema SCHEMA_PRIVILEGES IS_GRANTABLE varchar 3 9 utf8mb3 utf8mb3_general_ci varchar(3) > +3.0000 information_schema SEQUENCES SEQUENCE_CATALOG varchar 512 1536 utf8mb3 utf8mb3_general_ci varchar(512) > +3.0000 information_schema SEQUENCES SEQUENCE_SCHEMA varchar 64 192 utf8mb3 utf8mb3_general_ci varchar(64) > +3.0000 information_schema SEQUENCES SEQUENCE_NAME varchar 64 192 utf8mb3 utf8mb3_general_ci varchar(64) > +3.0000 information_schema SEQUENCES DATA_TYPE varchar 64 192 utf8mb3 utf8mb3_general_ci varchar(64) > +NULL information_schema SEQUENCES NUMERIC_PRECISION int NULL NULL NULL NULL int(21) > +NULL information_schema SEQUENCES NUMERIC_PRECISION_RADIX int NULL NULL NULL NULL int(21) > +NULL information_schema SEQUENCES NUMERIC_SCALE int NULL NULL NULL NULL int(21) > +NULL information_schema SEQUENCES START_VALUE decimal NULL NULL NULL NULL decimal(21,0) > +NULL information_schema SEQUENCES MINIMUM_VALUE decimal NULL NULL NULL NULL decimal(21,0) > +NULL information_schema SEQUENCES MAXIMUM_VALUE decimal NULL NULL NULL NULL decimal(21,0) > +NULL information_schema SEQUENCES INCREMENT bigint NULL NULL NULL NULL bigint(21) > +NULL information_schema SEQUENCES CYCLE_OPTION bigint NULL NULL NULL NULL bigint(21) > +NULL information_schema SEQUENCES DECLARED_DATA_TYPE int NULL NULL NULL NULL int(21) > +NULL information_schema SEQUENCES DECLARED_NUMERIC_PRECISION int NULL NULL NULL NULL int(21) > +NULL information_schema SEQUENCES DECLARED_NUMERIC_SCALE int NULL NULL NULL NULL int(21) > 3.0000 information_schema SESSION_STATUS VARIABLE_NAME varchar 64 192 utf8mb3 utf8mb3_general_ci varchar(64) > 3.0000 information_schema SESSION_STATUS VARIABLE_VALUE varchar 2048 6144 utf8mb3 utf8mb3_general_ci varchar(2048) > 3.0000 information_schema SESSION_VARIABLES VARIABLE_NAME varchar 64 192 utf8mb3 utf8mb3_general_ci varchar(64) > diff --git a/mysql-test/suite/funcs_1/r/is_tables_is.result b/mysql-test/suite/funcs_1/r/is_tables_is.result > index c18f733c86f..8c81a79b605 100644 > --- a/mysql-test/suite/funcs_1/r/is_tables_is.result > +++ b/mysql-test/suite/funcs_1/r/is_tables_is.result > @@ -739,6 +739,31 @@ user_comment > Separator ----------------------------------------------------- > TABLE_CATALOG def > TABLE_SCHEMA information_schema > +TABLE_NAME SEQUENCES > +TABLE_TYPE SYSTEM VIEW > +ENGINE MEMORY > +VERSION 11 > +ROW_FORMAT Fixed > +TABLE_ROWS #TBLR# > +AVG_ROW_LENGTH #ARL# > +DATA_LENGTH #DL# > +MAX_DATA_LENGTH #MDL# > +INDEX_LENGTH #IL# > +DATA_FREE #DF# > +AUTO_INCREMENT NULL > +CREATE_TIME #CRT# > +UPDATE_TIME #UT# > +CHECK_TIME #CT# > +TABLE_COLLATION utf8mb3_general_ci > +CHECKSUM NULL > +CREATE_OPTIONS #CO# > +TABLE_COMMENT #TC# > +MAX_INDEX_LENGTH #MIL# > +TEMPORARY Y > +user_comment > +Separator ----------------------------------------------------- > +TABLE_CATALOG def > +TABLE_SCHEMA information_schema > TABLE_NAME SESSION_STATUS > TABLE_TYPE SYSTEM VIEW > ENGINE MEMORY > @@ -1855,6 +1880,31 @@ user_comment > Separator ----------------------------------------------------- > TABLE_CATALOG def > TABLE_SCHEMA information_schema > +TABLE_NAME SEQUENCES > +TABLE_TYPE SYSTEM VIEW > +ENGINE MEMORY > +VERSION 11 > +ROW_FORMAT Fixed > +TABLE_ROWS #TBLR# > +AVG_ROW_LENGTH #ARL# > +DATA_LENGTH #DL# > +MAX_DATA_LENGTH #MDL# > +INDEX_LENGTH #IL# > +DATA_FREE #DF# > +AUTO_INCREMENT NULL > +CREATE_TIME #CRT# > +UPDATE_TIME #UT# > +CHECK_TIME #CT# > +TABLE_COLLATION utf8mb3_general_ci > +CHECKSUM NULL > +CREATE_OPTIONS #CO# > +TABLE_COMMENT #TC# > +MAX_INDEX_LENGTH #MIL# > +TEMPORARY Y > +user_comment > +Separator ----------------------------------------------------- > +TABLE_CATALOG def > +TABLE_SCHEMA information_schema > TABLE_NAME SESSION_STATUS > TABLE_TYPE SYSTEM VIEW > ENGINE MEMORY > diff --git a/mysql-test/suite/funcs_1/r/is_tables_is_embedded.result b/mysql-test/suite/funcs_1/r/is_tables_is_embedded.result > index c18f733c86f..8c81a79b605 100644 > --- a/mysql-test/suite/funcs_1/r/is_tables_is_embedded.result > +++ b/mysql-test/suite/funcs_1/r/is_tables_is_embedded.result > @@ -739,6 +739,31 @@ user_comment > Separator ----------------------------------------------------- > TABLE_CATALOG def > TABLE_SCHEMA information_schema > +TABLE_NAME SEQUENCES > +TABLE_TYPE SYSTEM VIEW > +ENGINE MEMORY > +VERSION 11 > +ROW_FORMAT Fixed > +TABLE_ROWS #TBLR# > +AVG_ROW_LENGTH #ARL# > +DATA_LENGTH #DL# > +MAX_DATA_LENGTH #MDL# > +INDEX_LENGTH #IL# > +DATA_FREE #DF# > +AUTO_INCREMENT NULL > +CREATE_TIME #CRT# > +UPDATE_TIME #UT# > +CHECK_TIME #CT# > +TABLE_COLLATION utf8mb3_general_ci > +CHECKSUM NULL > +CREATE_OPTIONS #CO# > +TABLE_COMMENT #TC# > +MAX_INDEX_LENGTH #MIL# > +TEMPORARY Y > +user_comment > +Separator ----------------------------------------------------- > +TABLE_CATALOG def > +TABLE_SCHEMA information_schema > TABLE_NAME SESSION_STATUS > TABLE_TYPE SYSTEM VIEW > ENGINE MEMORY > @@ -1855,6 +1880,31 @@ user_comment > Separator ----------------------------------------------------- > TABLE_CATALOG def > TABLE_SCHEMA information_schema > +TABLE_NAME SEQUENCES > +TABLE_TYPE SYSTEM VIEW > +ENGINE MEMORY > +VERSION 11 > +ROW_FORMAT Fixed > +TABLE_ROWS #TBLR# > +AVG_ROW_LENGTH #ARL# > +DATA_LENGTH #DL# > +MAX_DATA_LENGTH #MDL# > +INDEX_LENGTH #IL# > +DATA_FREE #DF# > +AUTO_INCREMENT NULL > +CREATE_TIME #CRT# > +UPDATE_TIME #UT# > +CHECK_TIME #CT# > +TABLE_COLLATION utf8mb3_general_ci > +CHECKSUM NULL > +CREATE_OPTIONS #CO# > +TABLE_COMMENT #TC# > +MAX_INDEX_LENGTH #MIL# > +TEMPORARY Y > +user_comment > +Separator ----------------------------------------------------- > +TABLE_CATALOG def > +TABLE_SCHEMA information_schema > TABLE_NAME SESSION_STATUS > TABLE_TYPE SYSTEM VIEW > ENGINE MEMORY > diff --git a/mysql-test/suite/sql_sequence/alter.result b/mysql-test/suite/sql_sequence/alter.result > index 60b708b8289..e6cd599ec40 100644 > --- a/mysql-test/suite/sql_sequence/alter.result > +++ b/mysql-test/suite/sql_sequence/alter.result > @@ -14,7 +14,7 @@ next value for t1 > alter sequence t1 start=50; > show create sequence t1; > Table Create Table > -t1 CREATE SEQUENCE `t1` start with 50 minvalue 1 maxvalue 9223372036854775806 increment by 1 nocache nocycle ENGINE=MyISAM > +t1 CREATE SEQUENCE `t1` as bigint start with 50 minvalue 1 maxvalue 9223372036854775806 increment by 1 nocache nocycle ENGINE=MyISAM > select * from t1; > next_not_cached_value minimum_value maximum_value start_value increment cache_size cycle_option cycle_count > 2 1 9223372036854775806 50 1 0 0 0 > @@ -24,7 +24,7 @@ next value for t1 > alter sequence t1 minvalue=-100; > show create sequence t1; > Table Create Table > -t1 CREATE SEQUENCE `t1` start with 50 minvalue -100 maxvalue 9223372036854775806 increment by 1 nocache nocycle ENGINE=MyISAM > +t1 CREATE SEQUENCE `t1` as bigint start with 50 minvalue -100 maxvalue 9223372036854775806 increment by 1 nocache nocycle ENGINE=MyISAM > select * from t1; > next_not_cached_value minimum_value maximum_value start_value increment cache_size cycle_option cycle_count > 3 -100 9223372036854775806 50 1 0 0 0 > @@ -33,14 +33,14 @@ ERROR HY000: Sequence 'test.t1' has out of range value for options > alter sequence t1 minvalue=100 start=100 restart=100; > show create sequence t1; > Table Create Table > -t1 CREATE SEQUENCE `t1` start with 100 minvalue 100 maxvalue 9223372036854775806 increment by 1 nocache nocycle ENGINE=MyISAM > +t1 CREATE SEQUENCE `t1` as bigint start with 100 minvalue 100 maxvalue 9223372036854775806 increment by 1 nocache nocycle ENGINE=MyISAM > select * from t1; > next_not_cached_value minimum_value maximum_value start_value increment cache_size cycle_option cycle_count > 100 100 9223372036854775806 100 1 0 0 0 > alter sequence t1 maxvalue=500; > show create sequence t1; > Table Create Table > -t1 CREATE SEQUENCE `t1` start with 100 minvalue 100 maxvalue 500 increment by 1 nocache nocycle ENGINE=MyISAM > +t1 CREATE SEQUENCE `t1` as bigint start with 100 minvalue 100 maxvalue 500 increment by 1 nocache nocycle ENGINE=MyISAM > select * from t1; > next_not_cached_value minimum_value maximum_value start_value increment cache_size cycle_option cycle_count > 100 100 500 100 1 0 0 0 > @@ -49,20 +49,20 @@ CREATE SEQUENCE t1 engine=myisam; > alter sequence t1 nocache; > show create sequence t1; > Table Create Table > -t1 CREATE SEQUENCE `t1` start with 1 minvalue 1 maxvalue 9223372036854775806 increment by 1 nocache nocycle ENGINE=MyISAM > +t1 CREATE SEQUENCE `t1` as bigint start with 1 minvalue 1 maxvalue 9223372036854775806 increment by 1 nocache nocycle ENGINE=MyISAM > alter sequence t1 cache=100; > flush tables; > show create sequence t1; > Table Create Table > -t1 CREATE SEQUENCE `t1` start with 1 minvalue 1 maxvalue 9223372036854775806 increment by 1 cache 100 nocycle ENGINE=MyISAM > +t1 CREATE SEQUENCE `t1` as bigint start with 1 minvalue 1 maxvalue 9223372036854775806 increment by 1 cache 100 nocycle ENGINE=MyISAM > alter sequence t1 nocache; > show create sequence t1; > Table Create Table > -t1 CREATE SEQUENCE `t1` start with 1 minvalue 1 maxvalue 9223372036854775806 increment by 1 nocache nocycle ENGINE=MyISAM > +t1 CREATE SEQUENCE `t1` as bigint start with 1 minvalue 1 maxvalue 9223372036854775806 increment by 1 nocache nocycle ENGINE=MyISAM > flush tables; > show create sequence t1; > Table Create Table > -t1 CREATE SEQUENCE `t1` start with 1 minvalue 1 maxvalue 9223372036854775806 increment by 1 nocache nocycle ENGINE=MyISAM > +t1 CREATE SEQUENCE `t1` as bigint start with 1 minvalue 1 maxvalue 9223372036854775806 increment by 1 nocache nocycle ENGINE=MyISAM > select * from t1; > next_not_cached_value minimum_value maximum_value start_value increment cache_size cycle_option cycle_count > 1 1 9223372036854775806 1 1 0 0 0 > @@ -83,19 +83,19 @@ CREATE SEQUENCE t1 maxvalue=100 engine=myisam; > alter sequence t1 no maxvalue; > show create sequence t1; > Table Create Table > -t1 CREATE SEQUENCE `t1` start with 1 minvalue 1 maxvalue 9223372036854775806 increment by 1 cache 1000 nocycle ENGINE=MyISAM > +t1 CREATE SEQUENCE `t1` as bigint start with 1 minvalue 1 maxvalue 9223372036854775806 increment by 1 cache 1000 nocycle ENGINE=MyISAM > select * from t1; > next_not_cached_value minimum_value maximum_value start_value increment cache_size cycle_option cycle_count > 1 1 9223372036854775806 1 1 1000 0 0 > alter sequence t1 cycle; > show create sequence t1; > Table Create Table > -t1 CREATE SEQUENCE `t1` start with 1 minvalue 1 maxvalue 9223372036854775806 increment by 1 cache 1000 cycle ENGINE=MyISAM > +t1 CREATE SEQUENCE `t1` as bigint start with 1 minvalue 1 maxvalue 9223372036854775806 increment by 1 cache 1000 cycle ENGINE=MyISAM > alter sequence t1 nocycle; > alter sequence t1 start=15 restart minvalue=10 maxvalue=20 cycle; > show create sequence t1; > Table Create Table > -t1 CREATE SEQUENCE `t1` start with 15 minvalue 10 maxvalue 20 increment by 1 cache 1000 cycle ENGINE=MyISAM > +t1 CREATE SEQUENCE `t1` as bigint start with 15 minvalue 10 maxvalue 20 increment by 1 cache 1000 cycle ENGINE=MyISAM > select * from t1; > next_not_cached_value minimum_value maximum_value start_value increment cache_size cycle_option cycle_count > 15 10 20 15 1 1000 1 0 > @@ -129,7 +129,7 @@ CREATE SEQUENCE t1 maxvalue=100; > alter sequence t1 increment=-2 start with 50 minvalue=-100; > show create sequence t1; > Table Create Table > -t1 CREATE SEQUENCE `t1` start with 50 minvalue -100 maxvalue 100 increment by -2 cache 1000 nocycle ENGINE=MyISAM > +t1 CREATE SEQUENCE `t1` as bigint start with 50 minvalue -100 maxvalue 100 increment by -2 cache 1000 nocycle ENGINE=MyISAM > select * from t1; > next_not_cached_value minimum_value maximum_value start_value increment cache_size cycle_option cycle_count > 1 -100 100 50 -2 1000 0 0 > @@ -159,7 +159,7 @@ next value for t1 > alter sequence t1 start=100; > show create sequence t1; > Table Create Table > -t1 CREATE SEQUENCE `t1` start with 100 minvalue 1 maxvalue 9223372036854775806 increment by 1 cache 10 nocycle ENGINE=InnoDB > +t1 CREATE SEQUENCE `t1` as bigint start with 100 minvalue 1 maxvalue 9223372036854775806 increment by 1 cache 10 nocycle ENGINE=InnoDB > select * from t1; > next_not_cached_value minimum_value maximum_value start_value increment cache_size cycle_option cycle_count > 11 1 9223372036854775806 100 1 10 0 0 > @@ -185,15 +185,15 @@ next value for t1 > alter table t1 comment="foo"; > show create sequence t1; > Table Create Table > -t1 CREATE SEQUENCE `t1` start with 1 minvalue 1 maxvalue 9223372036854775806 increment by 1 cache 1000 nocycle ENGINE=InnoDB COMMENT='foo' > +t1 CREATE SEQUENCE `t1` as bigint start with 1 minvalue 1 maxvalue 9223372036854775806 increment by 1 cache 1000 nocycle ENGINE=InnoDB COMMENT='foo' > alter table t1 engine=myisam; > show create sequence t1; > Table Create Table > -t1 CREATE SEQUENCE `t1` start with 1 minvalue 1 maxvalue 9223372036854775806 increment by 1 cache 1000 nocycle ENGINE=MyISAM COMMENT='foo' > +t1 CREATE SEQUENCE `t1` as bigint start with 1 minvalue 1 maxvalue 9223372036854775806 increment by 1 cache 1000 nocycle ENGINE=MyISAM COMMENT='foo' > alter table t1 engine=innodb; > show create sequence t1; > Table Create Table > -t1 CREATE SEQUENCE `t1` start with 1 minvalue 1 maxvalue 9223372036854775806 increment by 1 cache 1000 nocycle ENGINE=InnoDB COMMENT='foo' > +t1 CREATE SEQUENCE `t1` as bigint start with 1 minvalue 1 maxvalue 9223372036854775806 increment by 1 cache 1000 nocycle ENGINE=InnoDB COMMENT='foo' > select * from t1; > next_not_cached_value minimum_value maximum_value start_value increment cache_size cycle_option cycle_count > 3001 1 9223372036854775806 1 1 1000 0 0 > @@ -248,14 +248,186 @@ SELECT NEXTVAL(s); > NEXTVAL(s) > 1 > DROP SEQUENCE s; > +# > +# MDEV-28152 Features for sequence > +# > create sequence s maxvalue 12345; > show create sequence s; > Table Create Table > -s CREATE SEQUENCE `s` start with 1 minvalue 1 maxvalue 12345 increment by 1 cache 1000 nocycle ENGINE=MyISAM > +s CREATE SEQUENCE `s` as bigint start with 1 minvalue 1 maxvalue 12345 increment by 1 cache 1000 nocycle ENGINE=MyISAM > alter sequence s maxvalue 123456789012345678901234; > Warnings: > Note 1292 Truncated incorrect INTEGER value: 'MAXVALUE' > show create sequence s; > Table Create Table > -s CREATE SEQUENCE `s` start with 1 minvalue 1 maxvalue 9223372036854775806 increment by 1 cache 1000 nocycle ENGINE=MyISAM > +s CREATE SEQUENCE `s` as bigint start with 1 minvalue 1 maxvalue 9223372036854775806 increment by 1 cache 1000 nocycle ENGINE=MyISAM > +drop sequence s; > +create sequence s as tinyint; > +show create sequence s; > +Table Create Table > +s CREATE SEQUENCE `s` as tinyint start with 1 minvalue 1 maxvalue 126 increment by 1 cache 1000 nocycle ENGINE=MyISAM > +show create table s; > +Table Create Table > +s CREATE TABLE `s` ( > + `next_not_cached_value` tinyint(5) NOT NULL, > + `minimum_value` tinyint(5) NOT NULL, > + `maximum_value` tinyint(5) NOT NULL, > + `start_value` tinyint(5) NOT NULL COMMENT 'start value when sequences is created or value if RESTART is used', > + `increment` bigint(21) NOT NULL COMMENT 'increment value', > + `cache_size` bigint(21) unsigned NOT NULL, > + `cycle_option` tinyint(1) unsigned NOT NULL COMMENT '0 if no cycles are allowed, 1 if the sequence should begin a new cycle when maximum_value is passed', > + `cycle_count` bigint(21) NOT NULL COMMENT 'How many cycles have been done' > +) ENGINE=MyISAM SEQUENCE=1 > +alter sequence s as int; > +show create sequence s; > +Table Create Table > +s CREATE SEQUENCE `s` as int start with 1 minvalue 1 maxvalue 126 increment by 1 cache 1000 nocycle ENGINE=MyISAM Here (and for each type) this should be tested: 1. check maxvalue cycling and reaching maximum in case of not cycling which was inherited (126 in the case) 2. alter maxvalue according the new type 2.1 make it higher then possible (get a error or fixing the value with a warning) 2.2 make it maximum allowed and test it works 2.3. make it lower (may be around middle range) and test if it works > +show create table s; > +Table Create Table > +s CREATE TABLE `s` ( > + `next_not_cached_value` int(12) NOT NULL, > + `minimum_value` int(12) NOT NULL, > + `maximum_value` int(12) NOT NULL, > + `start_value` int(12) NOT NULL COMMENT 'start value when sequences is created or value if RESTART is used', > + `increment` bigint(21) NOT NULL COMMENT 'increment value', > + `cache_size` bigint(21) unsigned NOT NULL, > + `cycle_option` tinyint(1) unsigned NOT NULL COMMENT '0 if no cycles are allowed, 1 if the sequence should begin a new cycle when maximum_value is passed', > + `cycle_count` bigint(21) NOT NULL COMMENT 'How many cycles have been done' > +) ENGINE=MyISAM SEQUENCE=1 > +alter sequence s maxvalue 12345; > +show create sequence s; > +Table Create Table > +s CREATE SEQUENCE `s` as int start with 1 minvalue 1 maxvalue 12345 increment by 1 cache 1000 nocycle ENGINE=MyISAM > +show create table s; > +Table Create Table > +s CREATE TABLE `s` ( > + `next_not_cached_value` int(12) NOT NULL, > + `minimum_value` int(12) NOT NULL, > + `maximum_value` int(12) NOT NULL, > + `start_value` int(12) NOT NULL COMMENT 'start value when sequences is created or value if RESTART is used', > + `increment` bigint(21) NOT NULL COMMENT 'increment value', > + `cache_size` bigint(21) unsigned NOT NULL, > + `cycle_option` tinyint(1) unsigned NOT NULL COMMENT '0 if no cycles are allowed, 1 if the sequence should begin a new cycle when maximum_value is passed', > + `cycle_count` bigint(21) NOT NULL COMMENT 'How many cycles have been done' > +) ENGINE=MyISAM SEQUENCE=1 > +drop sequence s; > +create sequence s; > +show create sequence s; > +Table Create Table > +s CREATE SEQUENCE `s` as bigint start with 1 minvalue 1 maxvalue 9223372036854775806 increment by 1 cache 1000 nocycle ENGINE=MyISAM > +show create table s; > +Table Create Table > +s CREATE TABLE `s` ( > + `next_not_cached_value` bigint(21) NOT NULL, > + `minimum_value` bigint(21) NOT NULL, > + `maximum_value` bigint(21) NOT NULL, > + `start_value` bigint(21) NOT NULL COMMENT 'start value when sequences is created or value if RESTART is used', > + `increment` bigint(21) NOT NULL COMMENT 'increment value', > + `cache_size` bigint(21) unsigned NOT NULL, > + `cycle_option` tinyint(1) unsigned NOT NULL COMMENT '0 if no cycles are allowed, 1 if the sequence should begin a new cycle when maximum_value is passed', > + `cycle_count` bigint(21) NOT NULL COMMENT 'How many cycles have been done' > +) ENGINE=MyISAM SEQUENCE=1 > +alter sequence s maxvalue 123; > +show create sequence s; > +Table Create Table > +s CREATE SEQUENCE `s` as bigint start with 1 minvalue 1 maxvalue 123 increment by 1 cache 1000 nocycle ENGINE=MyISAM > +show create table s; > +Table Create Table > +s CREATE TABLE `s` ( > + `next_not_cached_value` bigint(21) NOT NULL, > + `minimum_value` bigint(21) NOT NULL, > + `maximum_value` bigint(21) NOT NULL, > + `start_value` bigint(21) NOT NULL COMMENT 'start value when sequences is created or value if RESTART is used', > + `increment` bigint(21) NOT NULL COMMENT 'increment value', > + `cache_size` bigint(21) unsigned NOT NULL, > + `cycle_option` tinyint(1) unsigned NOT NULL COMMENT '0 if no cycles are allowed, 1 if the sequence should begin a new cycle when maximum_value is passed', > + `cycle_count` bigint(21) NOT NULL COMMENT 'How many cycles have been done' > +) ENGINE=MyISAM SEQUENCE=1 > +alter sequence s as tinyint; > +show create sequence s; > +Table Create Table > +s CREATE SEQUENCE `s` as tinyint start with 1 minvalue 1 maxvalue 123 increment by 1 cache 1000 nocycle ENGINE=MyISAM > +show create table s; > +Table Create Table > +s CREATE TABLE `s` ( > + `next_not_cached_value` tinyint(5) NOT NULL, > + `minimum_value` tinyint(5) NOT NULL, > + `maximum_value` tinyint(5) NOT NULL, > + `start_value` tinyint(5) NOT NULL COMMENT 'start value when sequences is created or value if RESTART is used', > + `increment` bigint(21) NOT NULL COMMENT 'increment value', > + `cache_size` bigint(21) unsigned NOT NULL, > + `cycle_option` tinyint(1) unsigned NOT NULL COMMENT '0 if no cycles are allowed, 1 if the sequence should begin a new cycle when maximum_value is passed', > + `cycle_count` bigint(21) NOT NULL COMMENT 'How many cycles have been done' > +) ENGINE=MyISAM SEQUENCE=1 > +drop sequence s; > +create sequence s as int; > +show create sequence s; > +Table Create Table > +s CREATE SEQUENCE `s` as int start with 1 minvalue 1 maxvalue 2147483646 increment by 1 cache 1000 nocycle ENGINE=MyISAM > +show create table s; > +Table Create Table > +s CREATE TABLE `s` ( > + `next_not_cached_value` int(12) NOT NULL, > + `minimum_value` int(12) NOT NULL, > + `maximum_value` int(12) NOT NULL, > + `start_value` int(12) NOT NULL COMMENT 'start value when sequences is created or value if RESTART is used', > + `increment` bigint(21) NOT NULL COMMENT 'increment value', > + `cache_size` bigint(21) unsigned NOT NULL, > + `cycle_option` tinyint(1) unsigned NOT NULL COMMENT '0 if no cycles are allowed, 1 if the sequence should begin a new cycle when maximum_value is passed', > + `cycle_count` bigint(21) NOT NULL COMMENT 'How many cycles have been done' > +) ENGINE=MyISAM SEQUENCE=1 > +alter sequence s as tinyint; > +ERROR 22003: Out of range value for column 'maximum_value' at row 1 > +drop sequence s; Here it would be nice (before that drop) to change maximum_value and then repeat operation to be sure it is allowed. alter sequence s maxvalue 126; alter sequence s as tinyint; > +create sequence s as tinyint; > +show create sequence s; > +Table Create Table > +s CREATE SEQUENCE `s` as tinyint start with 1 minvalue 1 maxvalue 126 increment by 1 cache 1000 nocycle ENGINE=MyISAM > +show create table s; > +Table Create Table > +s CREATE TABLE `s` ( > + `next_not_cached_value` tinyint(5) NOT NULL, > + `minimum_value` tinyint(5) NOT NULL, > + `maximum_value` tinyint(5) NOT NULL, > + `start_value` tinyint(5) NOT NULL COMMENT 'start value when sequences is created or value if RESTART is used', > + `increment` bigint(21) NOT NULL COMMENT 'increment value', > + `cache_size` bigint(21) unsigned NOT NULL, > + `cycle_option` tinyint(1) unsigned NOT NULL COMMENT '0 if no cycles are allowed, 1 if the sequence should begin a new cycle when maximum_value is passed', > + `cycle_count` bigint(21) NOT NULL COMMENT 'How many cycles have been done' > +) ENGINE=MyISAM SEQUENCE=1 > +alter sequence s as int maxvalue 123; > +ERROR 42000: This version of MariaDB doesn't yet support 'ALTER SEQUENCE with both AS <type> and something else.' > +drop sequence s; > +create sequence s as int; > +show create sequence s; > +Table Create Table > +s CREATE SEQUENCE `s` as int start with 1 minvalue 1 maxvalue 2147483646 increment by 1 cache 1000 nocycle ENGINE=MyISAM > +show create table s; > +Table Create Table > +s CREATE TABLE `s` ( > + `next_not_cached_value` int(12) NOT NULL, > + `minimum_value` int(12) NOT NULL, > + `maximum_value` int(12) NOT NULL, > + `start_value` int(12) NOT NULL COMMENT 'start value when sequences is created or value if RESTART is used', > + `increment` bigint(21) NOT NULL COMMENT 'increment value', > + `cache_size` bigint(21) unsigned NOT NULL, > + `cycle_option` tinyint(1) unsigned NOT NULL COMMENT '0 if no cycles are allowed, 1 if the sequence should begin a new cycle when maximum_value is passed', > + `cycle_count` bigint(21) NOT NULL COMMENT 'How many cycles have been done' > +) ENGINE=MyISAM SEQUENCE=1 > +alter sequence s as int unsigned; > +show create sequence s; > +Table Create Table > +s CREATE SEQUENCE `s` as int unsigned start with 1 minvalue 1 maxvalue 2147483646 increment by 1 cache 1000 nocycle ENGINE=MyISAM > +show create table s; > +Table Create Table > +s CREATE TABLE `s` ( > + `next_not_cached_value` int(12) unsigned NOT NULL, > + `minimum_value` int(12) unsigned NOT NULL, > + `maximum_value` int(12) unsigned NOT NULL, > + `start_value` int(12) unsigned NOT NULL COMMENT 'start value when sequences is created or value if RESTART is used', > + `increment` bigint(21) NOT NULL COMMENT 'increment value', > + `cache_size` bigint(21) unsigned NOT NULL, > + `cycle_option` tinyint(1) unsigned NOT NULL COMMENT '0 if no cycles are allowed, 1 if the sequence should begin a new cycle when maximum_value is passed', > + `cycle_count` bigint(21) NOT NULL COMMENT 'How many cycles have been done' > +) ENGINE=MyISAM SEQUENCE=1 > drop sequence s; > diff --git a/mysql-test/suite/sql_sequence/alter.test b/mysql-test/suite/sql_sequence/alter.test > index 3afad38a9e5..4949075f492 100644 > --- a/mysql-test/suite/sql_sequence/alter.test > +++ b/mysql-test/suite/sql_sequence/alter.test > @@ -162,12 +162,63 @@ ALTER TABLE s ORDER BY cache_size; > SELECT NEXTVAL(s); > DROP SEQUENCE s; > > -# > -# MDEV-28152 Features for sequence > -# > > +--echo # > +--echo # MDEV-28152 Features for sequence > +--echo # > + > +# truncation in alter sequence > create sequence s maxvalue 12345; > show create sequence s; > alter sequence s maxvalue 123456789012345678901234; > show create sequence s; > drop sequence s; > + > +# alter first from a narrower type to a wider type, then maxvalue > +create sequence s as tinyint; > +show create sequence s; > +show create table s; > +alter sequence s as int; > +show create sequence s; > +show create table s; > +alter sequence s maxvalue 12345; > +show create sequence s; > +show create table s; > +drop sequence s; > + > +# alter first maxvalue then from a wider type to a narrower type > +create sequence s; > +show create sequence s; > +show create table s; > +alter sequence s maxvalue 123; > +show create sequence s; > +show create table s; > +alter sequence s as tinyint; > +show create sequence s; > +show create table s; > +drop sequence s; > + > +# from a wider type to a narrower type with out of range values > +create sequence s as int; > +show create sequence s; > +show create table s; > +--error ER_WARN_DATA_OUT_OF_RANGE > +alter sequence s as tinyint; > +drop sequence s; > + > +# cannot alter both value type and something else yet. > +create sequence s as tinyint; > +show create sequence s; > +show create table s; > +--error ER_NOT_SUPPORTED_YET > +alter sequence s as int maxvalue 123; > +drop sequence s; > + > +# from signed to unsigned > +create sequence s as int; > +show create sequence s; > +show create table s; > +alter sequence s as int unsigned; > +show create sequence s; > +show create table s; > +drop sequence s; > diff --git a/mysql-test/suite/sql_sequence/alter_notembedded.result b/mysql-test/suite/sql_sequence/alter_notembedded.result > index f3e1f5f18cd..c2ea6633d1d 100644 > --- a/mysql-test/suite/sql_sequence/alter_notembedded.result > +++ b/mysql-test/suite/sql_sequence/alter_notembedded.result > @@ -12,7 +12,7 @@ select nextval(s1); > ERROR 42000: INSERT command denied to user 'normal_1'@'localhost' for table `s_db`.`s1` > show create sequence s1; > Table Create Table > -s1 CREATE SEQUENCE `s1` start with 1 minvalue 1 maxvalue 9223372036854775806 increment by 1 cache 1000 nocycle ENGINE=MyISAM > +s1 CREATE SEQUENCE `s1` as bigint start with 1 minvalue 1 maxvalue 9223372036854775806 increment by 1 cache 1000 nocycle ENGINE=MyISAM > alter sequence s1 restart 50; > ERROR 42000: ALTER command denied to user 'normal_1'@'localhost' for table `s_db`.`s1` > connection default; > diff --git a/mysql-test/suite/sql_sequence/aria.result b/mysql-test/suite/sql_sequence/aria.result > index cfc7d946772..1e42be58240 100644 > --- a/mysql-test/suite/sql_sequence/aria.result > +++ b/mysql-test/suite/sql_sequence/aria.result > @@ -2,7 +2,7 @@ set @@default_storage_engine="aria"; > CREATE SEQUENCE t1 cache=10; > show create sequence t1; > Table Create Table > -t1 CREATE SEQUENCE `t1` start with 1 minvalue 1 maxvalue 9223372036854775806 increment by 1 cache 10 nocycle ENGINE=Aria > +t1 CREATE SEQUENCE `t1` as bigint start with 1 minvalue 1 maxvalue 9223372036854775806 increment by 1 cache 10 nocycle ENGINE=Aria > select NEXT VALUE for t1,seq from seq_1_to_20; > NEXT VALUE for t1 seq > 1 1 > diff --git a/mysql-test/suite/sql_sequence/concurrent_create.result b/mysql-test/suite/sql_sequence/concurrent_create.result > index 2473abef37d..e28c98e46be 100644 > --- a/mysql-test/suite/sql_sequence/concurrent_create.result > +++ b/mysql-test/suite/sql_sequence/concurrent_create.result > @@ -22,6 +22,20 @@ select * from s2; > next_not_cached_value minimum_value maximum_value start_value increment cache_size cycle_option cycle_count > 1 1 9223372036854775806 1 1 1000 0 0 > DROP SEQUENCE s1, s2; > +# > +# MDEV-28152 Features for sequence > +# > +CREATE SEQUENCE s1 as mediumint unsigned ENGINE=InnoDB; > +PREPARE stmt FROM "CREATE TABLE s2 LIKE s1"; > +execute stmt; > +drop table s2; > +execute stmt; > +drop table s2; > +execute stmt; > +select * from s2; > +next_not_cached_value minimum_value maximum_value start_value increment cache_size cycle_option cycle_count > +1 1 16777214 1 1 1000 0 0 > +DROP SEQUENCE s1, s2; > CREATE SEQUENCE s1 ENGINE=InnoDB; > connect con1,localhost,root,,test; > CREATE TABLE s2 LIKE s1;; > diff --git a/mysql-test/suite/sql_sequence/concurrent_create.test b/mysql-test/suite/sql_sequence/concurrent_create.test > index b27a6d3bdb9..b4def8d7ce5 100644 > --- a/mysql-test/suite/sql_sequence/concurrent_create.test > +++ b/mysql-test/suite/sql_sequence/concurrent_create.test > @@ -38,6 +38,20 @@ execute stmt; > select * from s2; > DROP SEQUENCE s1, s2; > > +--echo # > +--echo # MDEV-28152 Features for sequence > +--echo # > + > +CREATE SEQUENCE s1 as mediumint unsigned ENGINE=InnoDB; > +PREPARE stmt FROM "CREATE TABLE s2 LIKE s1"; > +execute stmt; > +drop table s2; > +execute stmt; > +drop table s2; > +execute stmt; > +select * from s2; > +DROP SEQUENCE s1, s2; You also better check second execution of CREATE SEQUENCE with the type in SP/PS (probably not in this test file) > + > # > # MDEV-15117 Server crashes in in open_and_process_table or ASAN > # heap-use-after-free in is_temporary_table upon creating/flushing sequences > diff --git a/mysql-test/suite/sql_sequence/create.result b/mysql-test/suite/sql_sequence/create.result > index e6a382ec3bf..f2d376774a7 100644 > --- a/mysql-test/suite/sql_sequence/create.result > +++ b/mysql-test/suite/sql_sequence/create.result > @@ -4,7 +4,7 @@ Note 1051 Unknown table 'test.t1' > create or replace sequence t1 engine=myisam; > show create sequence t1; > Table Create Table > -t1 CREATE SEQUENCE `t1` start with 1 minvalue 1 maxvalue 9223372036854775806 increment by 1 cache 1000 nocycle ENGINE=MyISAM > +t1 CREATE SEQUENCE `t1` as bigint start with 1 minvalue 1 maxvalue 9223372036854775806 increment by 1 cache 1000 nocycle ENGINE=MyISAM > show create table t1; > Table Create Table > t1 CREATE TABLE `t1` ( > @@ -23,7 +23,7 @@ next_not_cached_value minimum_value maximum_value start_value increment cache_si > create or replace sequence t1 engine=innodb; > show create sequence t1; > Table Create Table > -t1 CREATE SEQUENCE `t1` start with 1 minvalue 1 maxvalue 9223372036854775806 increment by 1 cache 1000 nocycle ENGINE=InnoDB > +t1 CREATE SEQUENCE `t1` as bigint start with 1 minvalue 1 maxvalue 9223372036854775806 increment by 1 cache 1000 nocycle ENGINE=InnoDB > show create table t1; > Table Create Table > t1 CREATE TABLE `t1` ( > @@ -42,7 +42,7 @@ next_not_cached_value minimum_value maximum_value start_value increment cache_si > create or replace sequence t1 engine=maria; > show create sequence t1; > Table Create Table > -t1 CREATE SEQUENCE `t1` start with 1 minvalue 1 maxvalue 9223372036854775806 increment by 1 cache 1000 nocycle ENGINE=Aria > +t1 CREATE SEQUENCE `t1` as bigint start with 1 minvalue 1 maxvalue 9223372036854775806 increment by 1 cache 1000 nocycle ENGINE=Aria > show create table t1; > Table Create Table > t1 CREATE TABLE `t1` ( > @@ -65,59 +65,59 @@ ERROR 42S02: Table 'test.t1' doesn't exist > create or replace sequence t1 start with 10; > show create sequence t1; > Table Create Table > -t1 CREATE SEQUENCE `t1` start with 10 minvalue 1 maxvalue 9223372036854775806 increment by 1 cache 1000 nocycle ENGINE=MyISAM > +t1 CREATE SEQUENCE `t1` as bigint start with 10 minvalue 1 maxvalue 9223372036854775806 increment by 1 cache 1000 nocycle ENGINE=MyISAM > select * from t1; > next_not_cached_value minimum_value maximum_value start_value increment cache_size cycle_option cycle_count > 10 1 9223372036854775806 10 1 1000 0 0 > create or replace sequence t1 minvalue=11; > show create sequence t1; > Table Create Table > -t1 CREATE SEQUENCE `t1` start with 11 minvalue 11 maxvalue 9223372036854775806 increment by 1 cache 1000 nocycle ENGINE=MyISAM > +t1 CREATE SEQUENCE `t1` as bigint start with 11 minvalue 11 maxvalue 9223372036854775806 increment by 1 cache 1000 nocycle ENGINE=MyISAM > select * from t1; > next_not_cached_value minimum_value maximum_value start_value increment cache_size cycle_option cycle_count > 11 11 9223372036854775806 11 1 1000 0 0 > create or replace sequence t1 maxvalue=13 increment by -1; > show create sequence t1; > Table Create Table > -t1 CREATE SEQUENCE `t1` start with 13 minvalue -9223372036854775807 maxvalue 13 increment by -1 cache 1000 nocycle ENGINE=MyISAM > +t1 CREATE SEQUENCE `t1` as bigint start with 13 minvalue -9223372036854775807 maxvalue 13 increment by -1 cache 1000 nocycle ENGINE=MyISAM > select * from t1; > next_not_cached_value minimum_value maximum_value start_value increment cache_size cycle_option cycle_count > 13 -9223372036854775807 13 13 -1 1000 0 0 > create or replace sequence t1 increment by -1 cache 100; > show create sequence t1; > Table Create Table > -t1 CREATE SEQUENCE `t1` start with -1 minvalue -9223372036854775807 maxvalue -1 increment by -1 cache 100 nocycle ENGINE=MyISAM > +t1 CREATE SEQUENCE `t1` as bigint start with -1 minvalue -9223372036854775807 maxvalue -1 increment by -1 cache 100 nocycle ENGINE=MyISAM > select * from t1; > next_not_cached_value minimum_value maximum_value start_value increment cache_size cycle_option cycle_count > -1 -9223372036854775807 -1 -1 -1 100 0 0 > create or replace sequence t1 cycle; > show create sequence t1; > Table Create Table > -t1 CREATE SEQUENCE `t1` start with 1 minvalue 1 maxvalue 9223372036854775806 increment by 1 cache 1000 cycle ENGINE=MyISAM > +t1 CREATE SEQUENCE `t1` as bigint start with 1 minvalue 1 maxvalue 9223372036854775806 increment by 1 cache 1000 cycle ENGINE=MyISAM > select * from t1; > next_not_cached_value minimum_value maximum_value start_value increment cache_size cycle_option cycle_count > 1 1 9223372036854775806 1 1 1000 1 0 > create or replace sequence t1 nocycle; > show create sequence t1; > Table Create Table > -t1 CREATE SEQUENCE `t1` start with 1 minvalue 1 maxvalue 9223372036854775806 increment by 1 cache 1000 nocycle ENGINE=MyISAM > +t1 CREATE SEQUENCE `t1` as bigint start with 1 minvalue 1 maxvalue 9223372036854775806 increment by 1 cache 1000 nocycle ENGINE=MyISAM > select * from t1; > next_not_cached_value minimum_value maximum_value start_value increment cache_size cycle_option cycle_count > 1 1 9223372036854775806 1 1 1000 0 0 > show create sequence t1; > Table Create Table > -t1 CREATE SEQUENCE `t1` start with 1 minvalue 1 maxvalue 9223372036854775806 increment by 1 cache 1000 nocycle ENGINE=MyISAM > +t1 CREATE SEQUENCE `t1` as bigint start with 1 minvalue 1 maxvalue 9223372036854775806 increment by 1 cache 1000 nocycle ENGINE=MyISAM > create or replace sequence t1 cycle minvalue= 14; > show create sequence t1; > Table Create Table > -t1 CREATE SEQUENCE `t1` start with 14 minvalue 14 maxvalue 9223372036854775806 increment by 1 cache 1000 cycle ENGINE=MyISAM > +t1 CREATE SEQUENCE `t1` as bigint start with 14 minvalue 14 maxvalue 9223372036854775806 increment by 1 cache 1000 cycle ENGINE=MyISAM > select * from t1; > next_not_cached_value minimum_value maximum_value start_value increment cache_size cycle_option cycle_count > 14 14 9223372036854775806 14 1 1000 1 0 > create or replace sequence t1 cycle increment by -1; > show create sequence t1; > Table Create Table > -t1 CREATE SEQUENCE `t1` start with -1 minvalue -9223372036854775807 maxvalue -1 increment by -1 cache 1000 cycle ENGINE=MyISAM > +t1 CREATE SEQUENCE `t1` as bigint start with -1 minvalue -9223372036854775807 maxvalue -1 increment by -1 cache 1000 cycle ENGINE=MyISAM > drop sequence t1; > create sequence if not exists t1; > create sequence if not exists t1 start with 10; > @@ -128,81 +128,21 @@ next_not_cached_value minimum_value maximum_value start_value increment cache_si > 1 1 9223372036854775806 1 1 1000 0 0 > show create sequence t1; > Table Create Table > -t1 CREATE SEQUENCE `t1` start with 1 minvalue 1 maxvalue 9223372036854775806 increment by 1 cache 1000 nocycle ENGINE=MyISAM > +t1 CREATE SEQUENCE `t1` as bigint start with 1 minvalue 1 maxvalue 9223372036854775806 increment by 1 cache 1000 nocycle ENGINE=MyISAM For backward compatibility (it is present in mariadbdump) and reducing sizeof the patch we can not print "as bigint" if it is bigint, because it is default. > create or replace sequence t1 start with 10 minvalue=10 maxvalue=11 nocache cycle; > show create sequence t1; > Table Create Table > -t1 CREATE SEQUENCE `t1` start with 10 minvalue 10 maxvalue 11 increment by 1 nocache cycle ENGINE=MyISAM > +t1 CREATE SEQUENCE `t1` as bigint start with 10 minvalue 10 maxvalue 11 increment by 1 nocache cycle ENGINE=MyISAM > select * from t1; > next_not_cached_value minimum_value maximum_value start_value increment cache_size cycle_option cycle_count > 10 10 11 10 1 0 1 0 > create or replace sequence t1 start with 10 minvalue=-10 maxvalue=11 cache=10 cycle increment by 10; > show create sequence t1; > Table Create Table > -t1 CREATE SEQUENCE `t1` start with 10 minvalue -10 maxvalue 11 increment by 10 cache 10 cycle ENGINE=MyISAM > +t1 CREATE SEQUENCE `t1` as bigint start with 10 minvalue -10 maxvalue 11 increment by 10 cache 10 cycle ENGINE=MyISAM > select * from t1; > next_not_cached_value minimum_value maximum_value start_value increment cache_size cycle_option cycle_count > 10 -10 11 10 10 10 1 0 > -create or replace sequence t1 minvalue -999999999999999999999; > -Warnings: > -Note 1292 Truncated incorrect INTEGER value: 'MINVALUE' > -show create sequence t1; > -Table Create Table > -t1 CREATE SEQUENCE `t1` start with -9223372036854775807 minvalue -9223372036854775807 maxvalue 9223372036854775806 increment by 1 cache 1000 nocycle ENGINE=MyISAM > -create or replace sequence t1 minvalue -9223372036854775808; > -Warnings: > -Note 1292 Truncated incorrect INTEGER value: 'MINVALUE' > -show create sequence t1; > -Table Create Table > -t1 CREATE SEQUENCE `t1` start with -9223372036854775807 minvalue -9223372036854775807 maxvalue 9223372036854775806 increment by 1 cache 1000 nocycle ENGINE=MyISAM > -create or replace sequence t1 minvalue -9223372036854775807; > -show create sequence t1; > -Table Create Table > -t1 CREATE SEQUENCE `t1` start with -9223372036854775807 minvalue -9223372036854775807 maxvalue 9223372036854775806 increment by 1 cache 1000 nocycle ENGINE=MyISAM > -create or replace sequence t1 minvalue 9223372036854775805; > -show create sequence t1; > -Table Create Table > -t1 CREATE SEQUENCE `t1` start with 9223372036854775805 minvalue 9223372036854775805 maxvalue 9223372036854775806 increment by 1 cache 1000 nocycle ENGINE=MyISAM > -create or replace sequence t1 minvalue 9223372036854775806; > -ERROR HY000: Sequence 'test.t1' has out of range value for options > -create or replace sequence t1 minvalue 9223372036854775807; > -ERROR HY000: Sequence 'test.t1' has out of range value for options > -create or replace sequence t1 minvalue 9223372036854775808; > -ERROR HY000: Sequence 'test.t1' has out of range value for options > -create or replace sequence t1 minvalue 9999999999999999999999; > -ERROR HY000: Sequence 'test.t1' has out of range value for options > -create or replace sequence t1 maxvalue -999999999999999999999 increment by -1; > -ERROR HY000: Sequence 'test.t1' has out of range value for options > -create or replace sequence t1 maxvalue -9223372036854775808 increment by -1; > -ERROR HY000: Sequence 'test.t1' has out of range value for options > -create or replace sequence t1 maxvalue -9223372036854775807 increment by -1; > -ERROR HY000: Sequence 'test.t1' has out of range value for options > -create or replace sequence t1 maxvalue -9223372036854775806 increment by -1; > -show create sequence t1; > -Table Create Table > -t1 CREATE SEQUENCE `t1` start with -9223372036854775806 minvalue -9223372036854775807 maxvalue -9223372036854775806 increment by -1 cache 1000 nocycle ENGINE=MyISAM > -create or replace sequence t1 maxvalue 9223372036854775806; > -show create sequence t1; > -Table Create Table > -t1 CREATE SEQUENCE `t1` start with 1 minvalue 1 maxvalue 9223372036854775806 increment by 1 cache 1000 nocycle ENGINE=MyISAM > -create or replace sequence t1 maxvalue 9223372036854775807; > -Warnings: > -Note 1292 Truncated incorrect INTEGER value: 'MAXVALUE' > -show create sequence t1; > -Table Create Table > -t1 CREATE SEQUENCE `t1` start with 1 minvalue 1 maxvalue 9223372036854775806 increment by 1 cache 1000 nocycle ENGINE=MyISAM > -create or replace sequence t1 maxvalue 9223372036854775808; > -Warnings: > -Note 1292 Truncated incorrect INTEGER value: 'MAXVALUE' > -show create sequence t1; > -Table Create Table > -t1 CREATE SEQUENCE `t1` start with 1 minvalue 1 maxvalue 9223372036854775806 increment by 1 cache 1000 nocycle ENGINE=MyISAM > -create or replace sequence t1 maxvalue 9999999999999999999999; > -Warnings: > -Note 1292 Truncated incorrect INTEGER value: 'MAXVALUE' > -show create sequence t1; > -Table Create Table > -t1 CREATE SEQUENCE `t1` start with 1 minvalue 1 maxvalue 9223372036854775806 increment by 1 cache 1000 nocycle ENGINE=MyISAM Why was it deleted? > create or replace sequence t1 start with 10 NO MAXVALUE NO MINVALUE; > create or replace sequence t1 start with 10 maxvalue 10; > create or replace sequence t1 start with 10 minvalue 10; > @@ -213,7 +153,7 @@ drop sequence if exists t1; > create sequence t1 increment by 0; > show create sequence t1; > Table Create Table > -t1 CREATE SEQUENCE `t1` start with 1 minvalue 1 maxvalue 9223372036854775806 increment by 0 cache 1000 nocycle ENGINE=MyISAM > +t1 CREATE SEQUENCE `t1` as bigint start with 1 minvalue 1 maxvalue 9223372036854775806 increment by 0 cache 1000 nocycle ENGINE=MyISAM > select * from t1; > next_not_cached_value minimum_value maximum_value start_value increment cache_size cycle_option cycle_count > 1 1 9223372036854775806 1 0 1000 0 0 > @@ -293,7 +233,7 @@ flush tables; > create or replace sequence t1 comment= "test 1"; > show create sequence t1; > Table Create Table > -t1 CREATE SEQUENCE `t1` start with 1 minvalue 1 maxvalue 9223372036854775806 increment by 1 cache 1000 nocycle ENGINE=MyISAM COMMENT='test 1' > +t1 CREATE SEQUENCE `t1` as bigint start with 1 minvalue 1 maxvalue 9223372036854775806 increment by 1 cache 1000 nocycle ENGINE=MyISAM COMMENT='test 1' > show create table t1; > Table Create Table > t1 CREATE TABLE `t1` ( > @@ -309,7 +249,7 @@ t1 CREATE TABLE `t1` ( > create or replace sequence t1 comment= "test 2" min_rows=1 max_rows=2; > show create sequence t1; > Table Create Table > -t1 CREATE SEQUENCE `t1` start with 1 minvalue 1 maxvalue 9223372036854775806 increment by 1 cache 1000 nocycle ENGINE=MyISAM COMMENT='test 2' > +t1 CREATE SEQUENCE `t1` as bigint start with 1 minvalue 1 maxvalue 9223372036854775806 increment by 1 cache 1000 nocycle ENGINE=MyISAM COMMENT='test 2' > show create table t1; > Table Create Table > t1 CREATE TABLE `t1` ( > @@ -338,7 +278,7 @@ CREATE TABLE t1 ( > ) sequence=1; > show create sequence t1; > Table Create Table > -t1 CREATE SEQUENCE `t1` start with 1 minvalue 1 maxvalue 9223372036854775806 increment by 1 cache 1000 nocycle ENGINE=MyISAM > +t1 CREATE SEQUENCE `t1` as bigint start with 1 minvalue 1 maxvalue 9223372036854775806 increment by 1 cache 1000 nocycle ENGINE=MyISAM > show create table t1; > Table Create Table > t1 CREATE TABLE `t1` ( > @@ -373,7 +313,7 @@ CREATE OR REPLACE TABLE t1 ( > `cycle_option` tinyint(1) unsigned NOT NULL, > `cycle_count` bigint(21) NOT NULL > ) sequence=1; > -ERROR HY000: Sequence 'test.t1' table structure is invalid (next_not_cached_value) > +ERROR HY000: Sequence 'test.t1' table structure is invalid (minimum_value) > CREATE OR REPLACE TABLE t1 ( > `next_not_cached_value` bigint(21) NOT NULL, > `minimum_value` bigint(21) NOT NULL, > @@ -607,7 +547,7 @@ t CREATE TABLE `t` ( > ) ENGINE=MyISAM SEQUENCE=1 > show create sequence t; > Table Create Table > -t CREATE SEQUENCE `t` start with 1 minvalue 1 maxvalue 9223372036854775806 increment by 1 cache 1000 nocycle ENGINE=MyISAM > +t CREATE SEQUENCE `t` as bigint start with 1 minvalue 1 maxvalue 9223372036854775806 increment by 1 cache 1000 nocycle ENGINE=MyISAM > drop tables t, s; > # > # MDEV-13714 SEQUENCE option fix > @@ -772,3 +712,439 @@ CREATE TRIGGER s1 BEFORE UPDATE ON seq1 FOR EACH ROW SET @a= 5; > ERROR HY000: Trigger's 'seq1' is a view, temporary table or sequence > DROP SEQUENCE seq1; > # End of 10.4 test > +###### > +# MDEV-28152 Features for sequence > +###### > +# ----- > +# Truncating out-of-bound numbers for minvalue and maxvalue > +# ----- > +create or replace sequence t1 minvalue -999999999999999999999; > +Warnings: > +Note 1292 Truncated incorrect INTEGER value: 'MINVALUE' > +show create sequence t1; > +Table Create Table > +t1 CREATE SEQUENCE `t1` as bigint start with -9223372036854775807 minvalue -9223372036854775807 maxvalue 9223372036854775806 increment by 1 cache 1000 nocycle ENGINE=MyISAM > +create or replace sequence t1 minvalue -9223372036854775808; > +Warnings: > +Note 1292 Truncated incorrect INTEGER value: 'MINVALUE' > +show create sequence t1; > +Table Create Table > +t1 CREATE SEQUENCE `t1` as bigint start with -9223372036854775807 minvalue -9223372036854775807 maxvalue 9223372036854775806 increment by 1 cache 1000 nocycle ENGINE=MyISAM > +create or replace sequence t1 minvalue -9223372036854775807; > +show create sequence t1; > +Table Create Table > +t1 CREATE SEQUENCE `t1` as bigint start with -9223372036854775807 minvalue -9223372036854775807 maxvalue 9223372036854775806 increment by 1 cache 1000 nocycle ENGINE=MyISAM > +create or replace sequence t1 minvalue 9223372036854775805; > +show create sequence t1; > +Table Create Table > +t1 CREATE SEQUENCE `t1` as bigint start with 9223372036854775805 minvalue 9223372036854775805 maxvalue 9223372036854775806 increment by 1 cache 1000 nocycle ENGINE=MyISAM > +create or replace sequence t1 minvalue 9223372036854775806; > +ERROR HY000: Sequence 'test.t1' has out of range value for options > +create or replace sequence t1 minvalue 9223372036854775807; > +ERROR HY000: Sequence 'test.t1' has out of range value for options > +create or replace sequence t1 minvalue 9223372036854775808; > +ERROR HY000: Sequence 'test.t1' has out of range value for options > +create or replace sequence t1 minvalue 9999999999999999999999; > +ERROR HY000: Sequence 'test.t1' has out of range value for options > +create or replace sequence t1 maxvalue -999999999999999999999 increment by -1; > +ERROR HY000: Sequence 'test.t1' has out of range value for options > +create or replace sequence t1 maxvalue -9223372036854775808 increment by -1; > +ERROR HY000: Sequence 'test.t1' has out of range value for options > +create or replace sequence t1 maxvalue -9223372036854775807 increment by -1; > +ERROR HY000: Sequence 'test.t1' has out of range value for options > +create or replace sequence t1 maxvalue -9223372036854775806 increment by -1; > +show create sequence t1; > +Table Create Table > +t1 CREATE SEQUENCE `t1` as bigint start with -9223372036854775806 minvalue -9223372036854775807 maxvalue -9223372036854775806 increment by -1 cache 1000 nocycle ENGINE=MyISAM > +create or replace sequence t1 maxvalue 9223372036854775806; > +show create sequence t1; > +Table Create Table > +t1 CREATE SEQUENCE `t1` as bigint start with 1 minvalue 1 maxvalue 9223372036854775806 increment by 1 cache 1000 nocycle ENGINE=MyISAM > +create or replace sequence t1 maxvalue 9223372036854775807; > +Warnings: > +Note 1292 Truncated incorrect INTEGER value: 'MAXVALUE' > +show create sequence t1; > +Table Create Table > +t1 CREATE SEQUENCE `t1` as bigint start with 1 minvalue 1 maxvalue 9223372036854775806 increment by 1 cache 1000 nocycle ENGINE=MyISAM > +create or replace sequence t1 maxvalue 9223372036854775808; > +Warnings: > +Note 1292 Truncated incorrect INTEGER value: 'MAXVALUE' > +show create sequence t1; > +Table Create Table > +t1 CREATE SEQUENCE `t1` as bigint start with 1 minvalue 1 maxvalue 9223372036854775806 increment by 1 cache 1000 nocycle ENGINE=MyISAM > +create or replace sequence t1 maxvalue 9999999999999999999999; > +Warnings: > +Note 1292 Truncated incorrect INTEGER value: 'MAXVALUE' > +show create sequence t1; > +Table Create Table > +t1 CREATE SEQUENCE `t1` as bigint start with 1 minvalue 1 maxvalue 9223372036854775806 increment by 1 cache 1000 nocycle ENGINE=MyISAM > +# ----- > +# Create with value types > +# ----- > +create or replace sequence t1 as tinyint; > +show create sequence t1; > +Table Create Table > +t1 CREATE SEQUENCE `t1` as tinyint start with 1 minvalue 1 maxvalue 126 increment by 1 cache 1000 nocycle ENGINE=MyISAM > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `next_not_cached_value` tinyint(5) NOT NULL, > + `minimum_value` tinyint(5) NOT NULL, > + `maximum_value` tinyint(5) NOT NULL, > + `start_value` tinyint(5) NOT NULL COMMENT 'start value when sequences is created or value if RESTART is used', > + `increment` bigint(21) NOT NULL COMMENT 'increment value', > + `cache_size` bigint(21) unsigned NOT NULL, > + `cycle_option` tinyint(1) unsigned NOT NULL COMMENT '0 if no cycles are allowed, 1 if the sequence should begin a new cycle when maximum_value is passed', > + `cycle_count` bigint(21) NOT NULL COMMENT 'How many cycles have been done' > +) ENGINE=MyISAM SEQUENCE=1 > +create or replace sequence t1 as smallint; > +show create sequence t1; > +Table Create Table > +t1 CREATE SEQUENCE `t1` as smallint start with 1 minvalue 1 maxvalue 32766 increment by 1 cache 1000 nocycle ENGINE=MyISAM > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `next_not_cached_value` smallint(7) NOT NULL, > + `minimum_value` smallint(7) NOT NULL, > + `maximum_value` smallint(7) NOT NULL, > + `start_value` smallint(7) NOT NULL COMMENT 'start value when sequences is created or value if RESTART is used', > + `increment` bigint(21) NOT NULL COMMENT 'increment value', > + `cache_size` bigint(21) unsigned NOT NULL, > + `cycle_option` tinyint(1) unsigned NOT NULL COMMENT '0 if no cycles are allowed, 1 if the sequence should begin a new cycle when maximum_value is passed', > + `cycle_count` bigint(21) NOT NULL COMMENT 'How many cycles have been done' > +) ENGINE=MyISAM SEQUENCE=1 > +create or replace sequence t1 as mediumint; > +show create sequence t1; > +Table Create Table > +t1 CREATE SEQUENCE `t1` as mediumint start with 1 minvalue 1 maxvalue 8388606 increment by 1 cache 1000 nocycle ENGINE=MyISAM > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `next_not_cached_value` mediumint(10) NOT NULL, > + `minimum_value` mediumint(10) NOT NULL, > + `maximum_value` mediumint(10) NOT NULL, > + `start_value` mediumint(10) NOT NULL COMMENT 'start value when sequences is created or value if RESTART is used', > + `increment` bigint(21) NOT NULL COMMENT 'increment value', > + `cache_size` bigint(21) unsigned NOT NULL, > + `cycle_option` tinyint(1) unsigned NOT NULL COMMENT '0 if no cycles are allowed, 1 if the sequence should begin a new cycle when maximum_value is passed', > + `cycle_count` bigint(21) NOT NULL COMMENT 'How many cycles have been done' > +) ENGINE=MyISAM SEQUENCE=1 > +create or replace sequence t1 as int; > +show create sequence t1; > +Table Create Table > +t1 CREATE SEQUENCE `t1` as int start with 1 minvalue 1 maxvalue 2147483646 increment by 1 cache 1000 nocycle ENGINE=MyISAM > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `next_not_cached_value` int(12) NOT NULL, > + `minimum_value` int(12) NOT NULL, > + `maximum_value` int(12) NOT NULL, > + `start_value` int(12) NOT NULL COMMENT 'start value when sequences is created or value if RESTART is used', > + `increment` bigint(21) NOT NULL COMMENT 'increment value', > + `cache_size` bigint(21) unsigned NOT NULL, > + `cycle_option` tinyint(1) unsigned NOT NULL COMMENT '0 if no cycles are allowed, 1 if the sequence should begin a new cycle when maximum_value is passed', > + `cycle_count` bigint(21) NOT NULL COMMENT 'How many cycles have been done' > +) ENGINE=MyISAM SEQUENCE=1 > +create or replace sequence t1 as bigint; > +show create sequence t1; > +Table Create Table > +t1 CREATE SEQUENCE `t1` as bigint start with 1 minvalue 1 maxvalue 9223372036854775806 increment by 1 cache 1000 nocycle ENGINE=MyISAM > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `next_not_cached_value` bigint(21) NOT NULL, > + `minimum_value` bigint(21) NOT NULL, > + `maximum_value` bigint(21) NOT NULL, > + `start_value` bigint(21) NOT NULL COMMENT 'start value when sequences is created or value if RESTART is used', > + `increment` bigint(21) NOT NULL COMMENT 'increment value', > + `cache_size` bigint(21) unsigned NOT NULL, > + `cycle_option` tinyint(1) unsigned NOT NULL COMMENT '0 if no cycles are allowed, 1 if the sequence should begin a new cycle when maximum_value is passed', > + `cycle_count` bigint(21) NOT NULL COMMENT 'How many cycles have been done' > +) ENGINE=MyISAM SEQUENCE=1 > +create or replace sequence t1 as tinyint unsigned; > +show create sequence t1; > +Table Create Table > +t1 CREATE SEQUENCE `t1` as tiny unsigned start with 1 minvalue 1 maxvalue 254 increment by 1 cache 1000 nocycle ENGINE=MyISAM You better also check the minimum possible value for each signed and unsigned type (I see default 1 in most cases here, it would be nice to see legal minimums checked (illegal as I can see tested later)). > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `next_not_cached_value` tinyint(5) unsigned NOT NULL, > + `minimum_value` tinyint(5) unsigned NOT NULL, > + `maximum_value` tinyint(5) unsigned NOT NULL, > + `start_value` tinyint(5) unsigned NOT NULL COMMENT 'start value when sequences is created or value if RESTART is used', > + `increment` bigint(21) NOT NULL COMMENT 'increment value', > + `cache_size` bigint(21) unsigned NOT NULL, > + `cycle_option` tinyint(1) unsigned NOT NULL COMMENT '0 if no cycles are allowed, 1 if the sequence should begin a new cycle when maximum_value is passed', > + `cycle_count` bigint(21) NOT NULL COMMENT 'How many cycles have been done' > +) ENGINE=MyISAM SEQUENCE=1 > +create or replace sequence t1 as smallint unsigned; > +show create sequence t1; > +Table Create Table > +t1 CREATE SEQUENCE `t1` as smallint unsigned start with 1 minvalue 1 maxvalue 65534 increment by 1 cache 1000 nocycle ENGINE=MyISAM > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `next_not_cached_value` smallint(7) unsigned NOT NULL, > + `minimum_value` smallint(7) unsigned NOT NULL, > + `maximum_value` smallint(7) unsigned NOT NULL, > + `start_value` smallint(7) unsigned NOT NULL COMMENT 'start value when sequences is created or value if RESTART is used', > + `increment` bigint(21) NOT NULL COMMENT 'increment value', > + `cache_size` bigint(21) unsigned NOT NULL, > + `cycle_option` tinyint(1) unsigned NOT NULL COMMENT '0 if no cycles are allowed, 1 if the sequence should begin a new cycle when maximum_value is passed', > + `cycle_count` bigint(21) NOT NULL COMMENT 'How many cycles have been done' > +) ENGINE=MyISAM SEQUENCE=1 > +create or replace sequence t1 as mediumint unsigned; > +show create sequence t1; > +Table Create Table > +t1 CREATE SEQUENCE `t1` as mediumint unsigned start with 1 minvalue 1 maxvalue 16777214 increment by 1 cache 1000 nocycle ENGINE=MyISAM > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `next_not_cached_value` mediumint(10) unsigned NOT NULL, > + `minimum_value` mediumint(10) unsigned NOT NULL, > + `maximum_value` mediumint(10) unsigned NOT NULL, > + `start_value` mediumint(10) unsigned NOT NULL COMMENT 'start value when sequences is created or value if RESTART is used', > + `increment` bigint(21) NOT NULL COMMENT 'increment value', > + `cache_size` bigint(21) unsigned NOT NULL, > + `cycle_option` tinyint(1) unsigned NOT NULL COMMENT '0 if no cycles are allowed, 1 if the sequence should begin a new cycle when maximum_value is passed', > + `cycle_count` bigint(21) NOT NULL COMMENT 'How many cycles have been done' > +) ENGINE=MyISAM SEQUENCE=1 > +create or replace sequence t1 as int unsigned; > +show create sequence t1; > +Table Create Table > +t1 CREATE SEQUENCE `t1` as int unsigned start with 1 minvalue 1 maxvalue 4294967294 increment by 1 cache 1000 nocycle ENGINE=MyISAM > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `next_not_cached_value` int(12) unsigned NOT NULL, > + `minimum_value` int(12) unsigned NOT NULL, > + `maximum_value` int(12) unsigned NOT NULL, > + `start_value` int(12) unsigned NOT NULL COMMENT 'start value when sequences is created or value if RESTART is used', > + `increment` bigint(21) NOT NULL COMMENT 'increment value', > + `cache_size` bigint(21) unsigned NOT NULL, > + `cycle_option` tinyint(1) unsigned NOT NULL COMMENT '0 if no cycles are allowed, 1 if the sequence should begin a new cycle when maximum_value is passed', > + `cycle_count` bigint(21) NOT NULL COMMENT 'How many cycles have been done' > +) ENGINE=MyISAM SEQUENCE=1 > +create or replace sequence t1 as bigint unsigned; > +show create sequence t1; > +Table Create Table > +t1 CREATE SEQUENCE `t1` as bigint unsigned start with 1 minvalue 1 maxvalue 18446744073709551614 increment by 1 cache 1000 nocycle ENGINE=MyISAM > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `next_not_cached_value` bigint(21) unsigned NOT NULL, > + `minimum_value` bigint(21) unsigned NOT NULL, > + `maximum_value` bigint(21) unsigned NOT NULL, > + `start_value` bigint(21) unsigned NOT NULL COMMENT 'start value when sequences is created or value if RESTART is used', > + `increment` bigint(21) NOT NULL COMMENT 'increment value', > + `cache_size` bigint(21) unsigned NOT NULL, > + `cycle_option` tinyint(1) unsigned NOT NULL COMMENT '0 if no cycles are allowed, 1 if the sequence should begin a new cycle when maximum_value is passed', > + `cycle_count` bigint(21) NOT NULL COMMENT 'How many cycles have been done' > +) ENGINE=MyISAM SEQUENCE=1 > +create or replace sequence t1 as tinyint zerofill; > +ERROR HY000: Incorrect value 'ZEROFILL' for option 'AS' > +create or replace sequence t1 as mediumint unsigned increment by -1; > +ERROR HY000: Sequence 'test.t1' has out of range value for options > +create or replace sequence t1 as bigint unsigned start with 12345678901234567890; > +show create sequence t1; > +Table Create Table > +t1 CREATE SEQUENCE `t1` as bigint unsigned start with 12345678901234567890 minvalue 1 maxvalue 18446744073709551614 increment by 1 cache 1000 nocycle ENGINE=MyISAM > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `next_not_cached_value` bigint(21) unsigned NOT NULL, > + `minimum_value` bigint(21) unsigned NOT NULL, > + `maximum_value` bigint(21) unsigned NOT NULL, > + `start_value` bigint(21) unsigned NOT NULL COMMENT 'start value when sequences is created or value if RESTART is used', > + `increment` bigint(21) NOT NULL COMMENT 'increment value', > + `cache_size` bigint(21) unsigned NOT NULL, > + `cycle_option` tinyint(1) unsigned NOT NULL COMMENT '0 if no cycles are allowed, 1 if the sequence should begin a new cycle when maximum_value is passed', > + `cycle_count` bigint(21) NOT NULL COMMENT 'How many cycles have been done' > +) ENGINE=MyISAM SEQUENCE=1 > +# ----- > +# value types + truncating > +# ----- > +create or replace sequence t1 minvalue -23 maxvalue 9999 as tinyint unsigned; > +Warnings: > +Note 1292 Truncated incorrect INTEGER value: 'MINVALUE' > +Note 1292 Truncated incorrect INTEGER value: 'MAXVALUE' > +show create sequence t1; > +Table Create Table > +t1 CREATE SEQUENCE `t1` as tiny unsigned start with 1 minvalue 1 maxvalue 254 increment by 1 cache 1000 nocycle ENGINE=MyISAM > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `next_not_cached_value` tinyint(5) unsigned NOT NULL, > + `minimum_value` tinyint(5) unsigned NOT NULL, > + `maximum_value` tinyint(5) unsigned NOT NULL, > + `start_value` tinyint(5) unsigned NOT NULL COMMENT 'start value when sequences is created or value if RESTART is used', > + `increment` bigint(21) NOT NULL COMMENT 'increment value', > + `cache_size` bigint(21) unsigned NOT NULL, > + `cycle_option` tinyint(1) unsigned NOT NULL COMMENT '0 if no cycles are allowed, 1 if the sequence should begin a new cycle when maximum_value is passed', > + `cycle_count` bigint(21) NOT NULL COMMENT 'How many cycles have been done' > +) ENGINE=MyISAM SEQUENCE=1 > +create or replace sequence t1 minvalue -32768 maxvalue 32767 as smallint; > +Warnings: > +Note 1292 Truncated incorrect INTEGER value: 'MINVALUE' > +Note 1292 Truncated incorrect INTEGER value: 'MAXVALUE' > +show create sequence t1; > +Table Create Table > +t1 CREATE SEQUENCE `t1` as smallint start with -32767 minvalue -32767 maxvalue 32766 increment by 1 cache 1000 nocycle ENGINE=MyISAM > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `next_not_cached_value` smallint(7) NOT NULL, > + `minimum_value` smallint(7) NOT NULL, > + `maximum_value` smallint(7) NOT NULL, > + `start_value` smallint(7) NOT NULL COMMENT 'start value when sequences is created or value if RESTART is used', > + `increment` bigint(21) NOT NULL COMMENT 'increment value', > + `cache_size` bigint(21) unsigned NOT NULL, > + `cycle_option` tinyint(1) unsigned NOT NULL COMMENT '0 if no cycles are allowed, 1 if the sequence should begin a new cycle when maximum_value is passed', > + `cycle_count` bigint(21) NOT NULL COMMENT 'How many cycles have been done' > +) ENGINE=MyISAM SEQUENCE=1 > +create or replace sequence t1 minvalue 0 maxvalue 65535 as smallint unsigned; > +Warnings: > +Note 1292 Truncated incorrect INTEGER value: 'MINVALUE' > +Note 1292 Truncated incorrect INTEGER value: 'MAXVALUE' > +show create sequence t1; > +Table Create Table > +t1 CREATE SEQUENCE `t1` as smallint unsigned start with 1 minvalue 1 maxvalue 65534 increment by 1 cache 1000 nocycle ENGINE=MyISAM > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `next_not_cached_value` smallint(7) unsigned NOT NULL, > + `minimum_value` smallint(7) unsigned NOT NULL, > + `maximum_value` smallint(7) unsigned NOT NULL, > + `start_value` smallint(7) unsigned NOT NULL COMMENT 'start value when sequences is created or value if RESTART is used', > + `increment` bigint(21) NOT NULL COMMENT 'increment value', > + `cache_size` bigint(21) unsigned NOT NULL, > + `cycle_option` tinyint(1) unsigned NOT NULL COMMENT '0 if no cycles are allowed, 1 if the sequence should begin a new cycle when maximum_value is passed', > + `cycle_count` bigint(21) NOT NULL COMMENT 'How many cycles have been done' > +) ENGINE=MyISAM SEQUENCE=1 > +create or replace sequence t1 minvalue -12345678901234 as mediumint unsigned maxvalue 12345678901234; > +Warnings: > +Note 1292 Truncated incorrect INTEGER value: 'MINVALUE' > +Note 1292 Truncated incorrect INTEGER value: 'MAXVALUE' > +show create sequence t1; > +Table Create Table > +t1 CREATE SEQUENCE `t1` as mediumint unsigned start with 1 minvalue 1 maxvalue 16777214 increment by 1 cache 1000 nocycle ENGINE=MyISAM > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `next_not_cached_value` mediumint(10) unsigned NOT NULL, > + `minimum_value` mediumint(10) unsigned NOT NULL, > + `maximum_value` mediumint(10) unsigned NOT NULL, > + `start_value` mediumint(10) unsigned NOT NULL COMMENT 'start value when sequences is created or value if RESTART is used', > + `increment` bigint(21) NOT NULL COMMENT 'increment value', > + `cache_size` bigint(21) unsigned NOT NULL, > + `cycle_option` tinyint(1) unsigned NOT NULL COMMENT '0 if no cycles are allowed, 1 if the sequence should begin a new cycle when maximum_value is passed', > + `cycle_count` bigint(21) NOT NULL COMMENT 'How many cycles have been done' > +) ENGINE=MyISAM SEQUENCE=1 > +create or replace sequence t1 as bigint unsigned minvalue -12345678901234 maxvalue 12345678901234; > +Warnings: > +Note 1292 Truncated incorrect INTEGER value: 'MINVALUE' > +show create sequence t1; > +Table Create Table > +t1 CREATE SEQUENCE `t1` as bigint unsigned start with 1 minvalue 1 maxvalue 12345678901234 increment by 1 cache 1000 nocycle ENGINE=MyISAM > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `next_not_cached_value` bigint(21) unsigned NOT NULL, > + `minimum_value` bigint(21) unsigned NOT NULL, > + `maximum_value` bigint(21) unsigned NOT NULL, > + `start_value` bigint(21) unsigned NOT NULL COMMENT 'start value when sequences is created or value if RESTART is used', > + `increment` bigint(21) NOT NULL COMMENT 'increment value', > + `cache_size` bigint(21) unsigned NOT NULL, > + `cycle_option` tinyint(1) unsigned NOT NULL COMMENT '0 if no cycles are allowed, 1 if the sequence should begin a new cycle when maximum_value is passed', > + `cycle_count` bigint(21) NOT NULL COMMENT 'How many cycles have been done' > +) ENGINE=MyISAM SEQUENCE=1 > +# ----- > +# indistinguishable values during parsing if we did not pass back Longlong_hybrid from the parser. > +# ----- > +create or replace sequence t1 as bigint maxvalue -1 increment by -1; > +show create sequence t1; > +Table Create Table > +t1 CREATE SEQUENCE `t1` as bigint start with -1 minvalue -9223372036854775807 maxvalue -1 increment by -1 cache 1000 nocycle ENGINE=MyISAM > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `next_not_cached_value` bigint(21) NOT NULL, > + `minimum_value` bigint(21) NOT NULL, > + `maximum_value` bigint(21) NOT NULL, > + `start_value` bigint(21) NOT NULL COMMENT 'start value when sequences is created or value if RESTART is used', > + `increment` bigint(21) NOT NULL COMMENT 'increment value', > + `cache_size` bigint(21) unsigned NOT NULL, > + `cycle_option` tinyint(1) unsigned NOT NULL COMMENT '0 if no cycles are allowed, 1 if the sequence should begin a new cycle when maximum_value is passed', > + `cycle_count` bigint(21) NOT NULL COMMENT 'How many cycles have been done' > +) ENGINE=MyISAM SEQUENCE=1 > +create or replace sequence t1 as bigint maxvalue 18446744073709551615; > +Warnings: > +Note 1292 Truncated incorrect INTEGER value: 'MAXVALUE' > +show create sequence t1; > +Table Create Table > +t1 CREATE SEQUENCE `t1` as bigint start with 1 minvalue 1 maxvalue 9223372036854775806 increment by 1 cache 1000 nocycle ENGINE=MyISAM > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `next_not_cached_value` bigint(21) NOT NULL, > + `minimum_value` bigint(21) NOT NULL, > + `maximum_value` bigint(21) NOT NULL, > + `start_value` bigint(21) NOT NULL COMMENT 'start value when sequences is created or value if RESTART is used', > + `increment` bigint(21) NOT NULL COMMENT 'increment value', > + `cache_size` bigint(21) unsigned NOT NULL, > + `cycle_option` tinyint(1) unsigned NOT NULL COMMENT '0 if no cycles are allowed, 1 if the sequence should begin a new cycle when maximum_value is passed', > + `cycle_count` bigint(21) NOT NULL COMMENT 'How many cycles have been done' > +) ENGINE=MyISAM SEQUENCE=1 > +create or replace sequence t1 as bigint unsigned maxvalue -1; > +ERROR HY000: Sequence 'test.t1' has out of range value for options > +create or replace sequence t1 as bigint unsigned maxvalue 18446744073709551615; > +Warnings: > +Note 1292 Truncated incorrect INTEGER value: 'MAXVALUE' > +show create sequence t1; > +Table Create Table > +t1 CREATE SEQUENCE `t1` as bigint unsigned start with 1 minvalue 1 maxvalue 18446744073709551614 increment by 1 cache 1000 nocycle ENGINE=MyISAM > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `next_not_cached_value` bigint(21) unsigned NOT NULL, > + `minimum_value` bigint(21) unsigned NOT NULL, > + `maximum_value` bigint(21) unsigned NOT NULL, > + `start_value` bigint(21) unsigned NOT NULL COMMENT 'start value when sequences is created or value if RESTART is used', > + `increment` bigint(21) NOT NULL COMMENT 'increment value', > + `cache_size` bigint(21) unsigned NOT NULL, > + `cycle_option` tinyint(1) unsigned NOT NULL COMMENT '0 if no cycles are allowed, 1 if the sequence should begin a new cycle when maximum_value is passed', > + `cycle_count` bigint(21) NOT NULL COMMENT 'How many cycles have been done' > +) ENGINE=MyISAM SEQUENCE=1 > +# ----- > +# value types + out of range start > +# ----- > +create or replace sequence t1 start with -123456789012345678901 as tinyint unsigned; > +ERROR 42000: You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near '123456789012345678901 as tinyint unsigned' at line 1 > +create or replace sequence t1 start with -1 as tinyint unsigned; > +ERROR HY000: Sequence 'test.t1' has out of range value for options > +create or replace sequence t1 start with 0 as tinyint unsigned; > +ERROR HY000: Sequence 'test.t1' has out of range value for options > +show create sequence t1; > +Table Create Table > +t1 CREATE SEQUENCE `t1` as bigint unsigned start with 1 minvalue 1 maxvalue 18446744073709551614 increment by 1 cache 1000 nocycle ENGINE=MyISAM > +create or replace sequence t1 start with 1 as tinyint unsigned; > +show create sequence t1; > +Table Create Table > +t1 CREATE SEQUENCE `t1` as tiny unsigned start with 1 minvalue 1 maxvalue 254 increment by 1 cache 1000 nocycle ENGINE=MyISAM > +create or replace sequence t1 start with 254 as tinyint unsigned; > +create or replace sequence t1 start with 255 as tinyint unsigned; > +ERROR HY000: Sequence 'test.t1' has out of range value for options > +create or replace sequence t1 start with 256 as tinyint unsigned; > +ERROR HY000: Sequence 'test.t1' has out of range value for options > +create or replace sequence t1 start with 123456789012345678901 as tinyint unsigned; > +ERROR 42000: You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near '123456789012345678901 as tinyint unsigned' at line 1 > +drop sequence t1; > +create sequence s1 as tinyint unsigned increment by 23; > +create sequence s2 start with 42 minvalue -9223372036854775807; > +create sequence s3 as bigint unsigned start with 12345678901234567890 cycle; > +select * from information_schema.sequences; > +SEQUENCE_CATALOG SEQUENCE_SCHEMA SEQUENCE_NAME DATA_TYPE NUMERIC_PRECISION NUMERIC_PRECISION_RADIX NUMERIC_SCALE START_VALUE MINIMUM_VALUE MAXIMUM_VALUE INCREMENT CYCLE_OPTION DECLARED_DATA_TYPE DECLARED_NUMERIC_PRECISION DECLARED_NUMERIC_SCALE > +def test s3 bigint unsigned 64 2 0 12345678901234567890 1 18446744073709551614 1 1 NULL NULL NULL > +def test s2 bigint 64 2 0 42 -9223372036854775807 9223372036854775806 1 0 NULL NULL NULL > +def test s1 tiny unsigned 8 2 0 1 1 254 23 0 NULL NULL NULL > +drop sequence s1, s2, s3; > diff --git a/mysql-test/suite/sql_sequence/create.test b/mysql-test/suite/sql_sequence/create.test > index 54e181483d6..388b2fd3fe9 100644 > --- a/mysql-test/suite/sql_sequence/create.test > +++ b/mysql-test/suite/sql_sequence/create.test > @@ -71,41 +71,6 @@ create or replace sequence t1 start with 10 minvalue=-10 maxvalue=11 cache=10 cy > show create sequence t1; > select * from t1; > > -# Truncating out-of-bound numbers for minvalue and maxvalue > -create or replace sequence t1 minvalue -999999999999999999999; > -show create sequence t1; > -create or replace sequence t1 minvalue -9223372036854775808; > -show create sequence t1; > -create or replace sequence t1 minvalue -9223372036854775807; > -show create sequence t1; > -create or replace sequence t1 minvalue 9223372036854775805; > -show create sequence t1; > ---error ER_SEQUENCE_INVALID_DATA > -create or replace sequence t1 minvalue 9223372036854775806; > ---error ER_SEQUENCE_INVALID_DATA > -create or replace sequence t1 minvalue 9223372036854775807; > ---error ER_SEQUENCE_INVALID_DATA > -create or replace sequence t1 minvalue 9223372036854775808; > ---error ER_SEQUENCE_INVALID_DATA > -create or replace sequence t1 minvalue 9999999999999999999999; > - > ---error ER_SEQUENCE_INVALID_DATA > -create or replace sequence t1 maxvalue -999999999999999999999 increment by -1; > ---error ER_SEQUENCE_INVALID_DATA > -create or replace sequence t1 maxvalue -9223372036854775808 increment by -1; > ---error ER_SEQUENCE_INVALID_DATA > -create or replace sequence t1 maxvalue -9223372036854775807 increment by -1; > -create or replace sequence t1 maxvalue -9223372036854775806 increment by -1; > -show create sequence t1; > -create or replace sequence t1 maxvalue 9223372036854775806; > -show create sequence t1; > -create or replace sequence t1 maxvalue 9223372036854775807; > -show create sequence t1; > -create or replace sequence t1 maxvalue 9223372036854775808; > -show create sequence t1; > -create or replace sequence t1 maxvalue 9999999999999999999999; > -show create sequence t1; > - > # NO MINVALUE, NO MAXVALUE > create or replace sequence t1 start with 10 NO MAXVALUE NO MINVALUE; > > @@ -582,3 +547,160 @@ CREATE TRIGGER s1 BEFORE UPDATE ON seq1 FOR EACH ROW SET @a= 5; > DROP SEQUENCE seq1; > > --echo # End of 10.4 test > + > +--echo ###### > +--echo # MDEV-28152 Features for sequence > +--echo ###### > + > +--echo # ----- > +--echo # Truncating out-of-bound numbers for minvalue and maxvalue > +--echo # ----- > +--disable_ps_protocol > +create or replace sequence t1 minvalue -999999999999999999999; > +show create sequence t1; > +create or replace sequence t1 minvalue -9223372036854775808; > +show create sequence t1; > +create or replace sequence t1 minvalue -9223372036854775807; > +show create sequence t1; > +create or replace sequence t1 minvalue 9223372036854775805; > +show create sequence t1; > +--error ER_SEQUENCE_INVALID_DATA > +create or replace sequence t1 minvalue 9223372036854775806; > +--error ER_SEQUENCE_INVALID_DATA > +create or replace sequence t1 minvalue 9223372036854775807; > +--error ER_SEQUENCE_INVALID_DATA > +create or replace sequence t1 minvalue 9223372036854775808; > +--error ER_SEQUENCE_INVALID_DATA > +create or replace sequence t1 minvalue 9999999999999999999999; > +--error ER_SEQUENCE_INVALID_DATA > +create or replace sequence t1 maxvalue -999999999999999999999 increment by -1; > +--error ER_SEQUENCE_INVALID_DATA > +create or replace sequence t1 maxvalue -9223372036854775808 increment by -1; > +--error ER_SEQUENCE_INVALID_DATA > +create or replace sequence t1 maxvalue -9223372036854775807 increment by -1; > +create or replace sequence t1 maxvalue -9223372036854775806 increment by -1; > +show create sequence t1; > +create or replace sequence t1 maxvalue 9223372036854775806; > +show create sequence t1; > +create or replace sequence t1 maxvalue 9223372036854775807; > +show create sequence t1; > +create or replace sequence t1 maxvalue 9223372036854775808; > +show create sequence t1; > +create or replace sequence t1 maxvalue 9999999999999999999999; > +show create sequence t1; > +--enable_ps_protocol > + > +--echo # ----- > +--echo # Create with value types > +--echo # ----- > +create or replace sequence t1 as tinyint; > +show create sequence t1; > +show create table t1; > +create or replace sequence t1 as smallint; > +show create sequence t1; > +show create table t1; > +create or replace sequence t1 as mediumint; > +show create sequence t1; > +show create table t1; > +create or replace sequence t1 as int; > +show create sequence t1; > +show create table t1; > +create or replace sequence t1 as bigint; > +show create sequence t1; > +show create table t1; > +create or replace sequence t1 as tinyint unsigned; > +show create sequence t1; > +show create table t1; > +create or replace sequence t1 as smallint unsigned; > +show create sequence t1; > +show create table t1; > +create or replace sequence t1 as mediumint unsigned; > +show create sequence t1; > +show create table t1; > +create or replace sequence t1 as int unsigned; > +show create sequence t1; > +show create table t1; > +create or replace sequence t1 as bigint unsigned; > +show create sequence t1; > +show create table t1; > +#zerofill is not supported > +--error ER_BAD_OPTION_VALUE > +create or replace sequence t1 as tinyint zerofill; > +#an unsigned sequence has to have positive increment > +--error ER_SEQUENCE_INVALID_DATA > +create or replace sequence t1 as mediumint unsigned increment by -1; > +#start with a number between longlong_max and ulonglong_max > +create or replace sequence t1 as bigint unsigned start with 12345678901234567890; > +show create sequence t1; > +show create table t1; > + > +--echo # ----- > +--echo # value types + truncating > +--echo # ----- > +--disable_ps_protocol > +create or replace sequence t1 minvalue -23 maxvalue 9999 as tinyint unsigned; > +show create sequence t1; > +show create table t1; > +create or replace sequence t1 minvalue -32768 maxvalue 32767 as smallint; > +show create sequence t1; > +show create table t1; > +create or replace sequence t1 minvalue 0 maxvalue 65535 as smallint unsigned; > +show create sequence t1; > +show create table t1; > +create or replace sequence t1 minvalue -12345678901234 as mediumint unsigned maxvalue 12345678901234; > +show create sequence t1; > +show create table t1; > +create or replace sequence t1 as bigint unsigned minvalue -12345678901234 maxvalue 12345678901234; > +show create sequence t1; > +show create table t1; > +--enable_ps_protocol > + > +--echo # ----- > +--echo # indistinguishable values during parsing if we did not pass back Longlong_hybrid from the parser. > +--echo # ----- > +#signed, -1: no truncation. Note that we need a negative increment because this is a nagative sequence > +create or replace sequence t1 as bigint maxvalue -1 increment by -1; > +show create sequence t1; > +show create table t1; > +--disable_ps_protocol > +#signed, ulonglong_max: turncating to longlong_max-1 > +create or replace sequence t1 as bigint maxvalue 18446744073709551615; > +show create sequence t1; > +show create table t1; > +#unsigned, -1: truncation and invalid data (max_value truncated to 1 which is equal to min_value) > +--error ER_SEQUENCE_INVALID_DATA > +create or replace sequence t1 as bigint unsigned maxvalue -1; > +#unsigned, ulonglong_max: truncating to ulonglong_max-1 > +create or replace sequence t1 as bigint unsigned maxvalue 18446744073709551615; > +show create sequence t1; > +show create table t1; > +--enable_ps_protocol > + > +--echo # ----- > +--echo # value types + out of range start > +--echo # ----- > +--error ER_PARSE_ERROR > +create or replace sequence t1 start with -123456789012345678901 as tinyint unsigned; > +--error ER_SEQUENCE_INVALID_DATA > +create or replace sequence t1 start with -1 as tinyint unsigned; > +--error ER_SEQUENCE_INVALID_DATA > +create or replace sequence t1 start with 0 as tinyint unsigned; > +show create sequence t1; > +create or replace sequence t1 start with 1 as tinyint unsigned; > +show create sequence t1; > +create or replace sequence t1 start with 254 as tinyint unsigned; > +--error ER_SEQUENCE_INVALID_DATA > +create or replace sequence t1 start with 255 as tinyint unsigned; > +--error ER_SEQUENCE_INVALID_DATA > +create or replace sequence t1 start with 256 as tinyint unsigned; > +--error ER_PARSE_ERROR > +create or replace sequence t1 start with 123456789012345678901 as tinyint unsigned; > + > +drop sequence t1; > + > +# information_schema.sequences > +create sequence s1 as tinyint unsigned increment by 23; > +create sequence s2 start with 42 minvalue -9223372036854775807; > +create sequence s3 as bigint unsigned start with 12345678901234567890 cycle; > +select * from information_schema.sequences; > +drop sequence s1, s2, s3; > diff --git a/mysql-test/suite/sql_sequence/mysqldump.result b/mysql-test/suite/sql_sequence/mysqldump.result > index 0199bb7162f..3316fdac5ae 100644 > --- a/mysql-test/suite/sql_sequence/mysqldump.result > +++ b/mysql-test/suite/sql_sequence/mysqldump.result > @@ -3,9 +3,9 @@ CREATE TABLE t1(a INT, KEY (a)) KEY_BLOCK_SIZE=1024; > insert into t1 values (1),(2); > CREATE SEQUENCE x1 engine=innodb; > # dump whole database > -CREATE SEQUENCE `a1` start with 1 minvalue 1 maxvalue 9223372036854775806 increment by 1 cache 1000 nocycle ENGINE=Aria; > +CREATE SEQUENCE `a1` as bigint start with 1 minvalue 1 maxvalue 9223372036854775806 increment by 1 cache 1000 nocycle ENGINE=Aria; > SELECT SETVAL(`a1`, 1, 0); > -CREATE SEQUENCE `x1` start with 1 minvalue 1 maxvalue 9223372036854775806 increment by 1 cache 1000 nocycle ENGINE=InnoDB; > +CREATE SEQUENCE `x1` as bigint start with 1 minvalue 1 maxvalue 9223372036854775806 increment by 1 cache 1000 nocycle ENGINE=InnoDB; > SELECT SETVAL(`x1`, 1, 0); > /*!40101 SET @saved_cs_client = @@character_set_client */; > /*!40101 SET character_set_client = utf8 */; > @@ -18,9 +18,9 @@ INSERT INTO `t1` VALUES > (1), > (2); > # dump by tables order 1 > -CREATE SEQUENCE `a1` start with 1 minvalue 1 maxvalue 9223372036854775806 increment by 1 cache 1000 nocycle ENGINE=Aria; > +CREATE SEQUENCE `a1` as bigint start with 1 minvalue 1 maxvalue 9223372036854775806 increment by 1 cache 1000 nocycle ENGINE=Aria; > SELECT SETVAL(`a1`, 1, 0); > -CREATE SEQUENCE `x1` start with 1 minvalue 1 maxvalue 9223372036854775806 increment by 1 cache 1000 nocycle ENGINE=InnoDB; > +CREATE SEQUENCE `x1` as bigint start with 1 minvalue 1 maxvalue 9223372036854775806 increment by 1 cache 1000 nocycle ENGINE=InnoDB; > SELECT SETVAL(`x1`, 1, 0); > /*!40101 SET @saved_cs_client = @@character_set_client */; > /*!40101 SET character_set_client = utf8 */; > @@ -33,9 +33,9 @@ INSERT INTO `t1` VALUES > (1), > (2); > # dump by tables order 2 > -CREATE SEQUENCE `a1` start with 1 minvalue 1 maxvalue 9223372036854775806 increment by 1 cache 1000 nocycle ENGINE=Aria; > +CREATE SEQUENCE `a1` as bigint start with 1 minvalue 1 maxvalue 9223372036854775806 increment by 1 cache 1000 nocycle ENGINE=Aria; > SELECT SETVAL(`a1`, 1, 0); > -CREATE SEQUENCE `x1` start with 1 minvalue 1 maxvalue 9223372036854775806 increment by 1 cache 1000 nocycle ENGINE=InnoDB; > +CREATE SEQUENCE `x1` as bigint start with 1 minvalue 1 maxvalue 9223372036854775806 increment by 1 cache 1000 nocycle ENGINE=InnoDB; > SELECT SETVAL(`x1`, 1, 0); > /*!40101 SET @saved_cs_client = @@character_set_client */; > /*!40101 SET character_set_client = utf8 */; > @@ -59,9 +59,9 @@ INSERT INTO `t1` VALUES > (1), > (2); > # dump by tables only sequences > -CREATE SEQUENCE `a1` start with 1 minvalue 1 maxvalue 9223372036854775806 increment by 1 cache 1000 nocycle ENGINE=Aria; > +CREATE SEQUENCE `a1` as bigint start with 1 minvalue 1 maxvalue 9223372036854775806 increment by 1 cache 1000 nocycle ENGINE=Aria; > SELECT SETVAL(`a1`, 1, 0); > -CREATE SEQUENCE `x1` start with 1 minvalue 1 maxvalue 9223372036854775806 increment by 1 cache 1000 nocycle ENGINE=InnoDB; > +CREATE SEQUENCE `x1` as bigint start with 1 minvalue 1 maxvalue 9223372036854775806 increment by 1 cache 1000 nocycle ENGINE=InnoDB; > SELECT SETVAL(`x1`, 1, 0); > # end of dumps > DROP TABLE a1,t1,x1; > diff --git a/mysql-test/suite/sql_sequence/next.result b/mysql-test/suite/sql_sequence/next.result > index 9d55921006b..76f42143305 100644 > --- a/mysql-test/suite/sql_sequence/next.result > +++ b/mysql-test/suite/sql_sequence/next.result > @@ -548,3 +548,92 @@ SELECT SETVAL (v,0); > ERROR 42S02: 'test.v' is not a SEQUENCE > UNLOCK TABLES; > DROP VIEW v; > +# > +# MDEV-28152 Features for sequence > +# > +create or replace sequence t1 as tinyint unsigned minvalue 1 maxvalue 2; > +show create sequence t1; > +Table Create Table > +t1 CREATE SEQUENCE `t1` as tiny unsigned start with 1 minvalue 1 maxvalue 2 increment by 1 cache 1000 nocycle ENGINE=MyISAM > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `next_not_cached_value` tinyint(5) unsigned NOT NULL, > + `minimum_value` tinyint(5) unsigned NOT NULL, > + `maximum_value` tinyint(5) unsigned NOT NULL, > + `start_value` tinyint(5) unsigned NOT NULL COMMENT 'start value when sequences is created or value if RESTART is used', > + `increment` bigint(21) NOT NULL COMMENT 'increment value', > + `cache_size` bigint(21) unsigned NOT NULL, > + `cycle_option` tinyint(1) unsigned NOT NULL COMMENT '0 if no cycles are allowed, 1 if the sequence should begin a new cycle when maximum_value is passed', > + `cycle_count` bigint(21) NOT NULL COMMENT 'How many cycles have been done' > +) ENGINE=MyISAM SEQUENCE=1 > +select * from t1; > +next_not_cached_value minimum_value maximum_value start_value increment cache_size cycle_option cycle_count > +1 1 2 1 1 1000 0 0 > +select next value for t1; > +next value for t1 > +1 > +select next value for t1; > +next value for t1 > +2 > +select next value for t1; > +ERROR HY000: Sequence 'test.t1' has run out > +create or replace sequence t1 as tinyint unsigned minvalue 1 maxvalue 2 cycle; > +show create sequence t1; > +Table Create Table > +t1 CREATE SEQUENCE `t1` as tiny unsigned start with 1 minvalue 1 maxvalue 2 increment by 1 cache 1000 cycle ENGINE=MyISAM > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `next_not_cached_value` tinyint(5) unsigned NOT NULL, > + `minimum_value` tinyint(5) unsigned NOT NULL, > + `maximum_value` tinyint(5) unsigned NOT NULL, > + `start_value` tinyint(5) unsigned NOT NULL COMMENT 'start value when sequences is created or value if RESTART is used', > + `increment` bigint(21) NOT NULL COMMENT 'increment value', > + `cache_size` bigint(21) unsigned NOT NULL, > + `cycle_option` tinyint(1) unsigned NOT NULL COMMENT '0 if no cycles are allowed, 1 if the sequence should begin a new cycle when maximum_value is passed', > + `cycle_count` bigint(21) NOT NULL COMMENT 'How many cycles have been done' > +) ENGINE=MyISAM SEQUENCE=1 > +select * from t1; > +next_not_cached_value minimum_value maximum_value start_value increment cache_size cycle_option cycle_count > +1 1 2 1 1 1000 1 0 > +select next value for t1; > +next value for t1 > +1 > +select next value for t1; > +next value for t1 > +2 > +select next value for t1; > +next value for t1 > +1 > +create or replace sequence t1 minvalue -23 maxvalue 99999 as tinyint; > +Warnings: > +Note 1292 Truncated incorrect INTEGER value: 'MAXVALUE' > +show create sequence t1; > +Table Create Table > +t1 CREATE SEQUENCE `t1` as tinyint start with -23 minvalue -23 maxvalue 126 increment by 1 cache 1000 nocycle ENGINE=MyISAM > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `next_not_cached_value` tinyint(5) NOT NULL, > + `minimum_value` tinyint(5) NOT NULL, > + `maximum_value` tinyint(5) NOT NULL, > + `start_value` tinyint(5) NOT NULL COMMENT 'start value when sequences is created or value if RESTART is used', > + `increment` bigint(21) NOT NULL COMMENT 'increment value', > + `cache_size` bigint(21) unsigned NOT NULL, > + `cycle_option` tinyint(1) unsigned NOT NULL COMMENT '0 if no cycles are allowed, 1 if the sequence should begin a new cycle when maximum_value is passed', > + `cycle_count` bigint(21) NOT NULL COMMENT 'How many cycles have been done' > +) ENGINE=MyISAM SEQUENCE=1 > +select * from t1; > +next_not_cached_value minimum_value maximum_value start_value increment cache_size cycle_option cycle_count > +-23 -23 126 -23 1 1000 0 0 > +select next value for t1; > +next value for t1 > +-23 > +select next value for t1; > +next value for t1 > +-22 > +select next value for t1; > +next value for t1 > +-21 > +drop sequence t1; > diff --git a/mysql-test/suite/sql_sequence/next.test b/mysql-test/suite/sql_sequence/next.test > index a80f9fad561..f2054e5f116 100644 > --- a/mysql-test/suite/sql_sequence/next.test > +++ b/mysql-test/suite/sql_sequence/next.test > @@ -297,3 +297,36 @@ SELECT SETVAL (v,0); > > UNLOCK TABLES; > DROP VIEW v; > + > +--echo # > +--echo # MDEV-28152 Features for sequence > +--echo # > + > +create or replace sequence t1 as tinyint unsigned minvalue 1 maxvalue 2; > +show create sequence t1; > +show create table t1; > +select * from t1; > +select next value for t1; > +select next value for t1; > +--error ER_SEQUENCE_RUN_OUT > +select next value for t1; > + > +create or replace sequence t1 as tinyint unsigned minvalue 1 maxvalue 2 cycle; > +show create sequence t1; > +show create table t1; > +select * from t1; > +select next value for t1; > +select next value for t1; > +select next value for t1; > + > +--disable_ps_protocol > +create or replace sequence t1 minvalue -23 maxvalue 99999 as tinyint; > +--enable_ps_protocol > +show create sequence t1; > +show create table t1; > +select * from t1; > +select next value for t1; > +select next value for t1; > +select next value for t1; > + > +drop sequence t1; It is also interesting to see rolling over maximum value. > diff --git a/mysql-test/suite/sql_sequence/replication.result b/mysql-test/suite/sql_sequence/replication.result > index 94b1c72b9e0..2cfc246f2bf 100644 > --- a/mysql-test/suite/sql_sequence/replication.result > +++ b/mysql-test/suite/sql_sequence/replication.result > @@ -149,7 +149,7 @@ CREATE TABLE `s2` ( > ) ENGINE=myisam DEFAULT CHARSET=latin1 sequence=1; > show create sequence s2; > Table Create Table > -s2 CREATE SEQUENCE `s2` start with 1 minvalue 1 maxvalue 9223372036854775806 increment by 1 cache 1000 nocycle ENGINE=MyISAM > +s2 CREATE SEQUENCE `s2` as bigint start with 1 minvalue 1 maxvalue 9223372036854775806 increment by 1 cache 1000 nocycle ENGINE=MyISAM > drop sequence s2; > ########################################### > select sequence syntax test > @@ -188,7 +188,7 @@ alter table s2 rename to s2_1; > rename table s2_1 to s2_2; > show create sequence s2_2; > Table Create Table > -s2_2 CREATE SEQUENCE `s2_2` start with 1 minvalue 1 maxvalue 9223372036854775806 increment by 1 cache 1000 nocycle ENGINE=MyISAM > +s2_2 CREATE SEQUENCE `s2_2` as bigint start with 1 minvalue 1 maxvalue 9223372036854775806 increment by 1 cache 1000 nocycle ENGINE=MyISAM > select * from s2_2; > next_not_cached_value minimum_value maximum_value start_value increment cache_size cycle_option cycle_count > 1 1 9223372036854775806 1 1 1000 0 0 > diff --git a/mysql-test/suite/sql_sequence/slave_nextval.result b/mysql-test/suite/sql_sequence/slave_nextval.result > index bfbc472e117..36efa85d920 100644 > --- a/mysql-test/suite/sql_sequence/slave_nextval.result > +++ b/mysql-test/suite/sql_sequence/slave_nextval.result > @@ -4,7 +4,7 @@ CREATE SEQUENCE s; > INSERT INTO s VALUES (1,1,4,1,1,1,0,0); > show create sequence s; > Table Create Table > -s CREATE SEQUENCE `s` start with 1 minvalue 1 maxvalue 4 increment by 1 cache 1 nocycle ENGINE=MyISAM > +s CREATE SEQUENCE `s` as bigint start with 1 minvalue 1 maxvalue 4 increment by 1 cache 1 nocycle ENGINE=MyISAM > SELECT NEXTVAL(s); > NEXTVAL(s) > 1 > @@ -38,7 +38,7 @@ CREATE SEQUENCE s; > INSERT INTO s VALUES (1,1,3,1,1,1,1,0); > show create sequence s; > Table Create Table > -s CREATE SEQUENCE `s` start with 1 minvalue 1 maxvalue 3 increment by 1 cache 1 cycle ENGINE=MyISAM > +s CREATE SEQUENCE `s` as bigint start with 1 minvalue 1 maxvalue 3 increment by 1 cache 1 cycle ENGINE=MyISAM > SELECT NEXTVAL(s); > NEXTVAL(s) > 1 > diff --git a/sql/handler.h b/sql/handler.h > index 6b05da2ca98..f573dc835e9 100644 > --- a/sql/handler.h > +++ b/sql/handler.h > @@ -1055,6 +1055,7 @@ enum enum_schema_tables > SCH_PROCEDURES, > SCH_SCHEMATA, > SCH_SCHEMA_PRIVILEGES, > + SCH_SEQUENCES, > SCH_SESSION_STATUS, > SCH_SESSION_VARIABLES, > SCH_STATISTICS, > diff --git a/sql/sql_sequence.cc b/sql/sql_sequence.cc > index 11be2e1a691..c5754dc9670 100644 > --- a/sql/sql_sequence.cc > +++ b/sql/sql_sequence.cc > @@ -30,15 +30,6 @@ > #include "wsrep_mysqld.h" > #endif > > -struct Field_definition > -{ > - const char *field_name; > - uint length; > - const Type_handler *type_handler; > - LEX_CSTRING comment; > - ulong flags; > -}; > - > /* > Structure for all SEQUENCE tables > > @@ -48,30 +39,87 @@ struct Field_definition > a column named NEXTVAL. > */ > > +#define MAX_AUTO_INCREMENT_VALUE 65535 > + > +Sequence_row_definition sequence_structure(const Type_handler* handler) > +{ > + // We don't really care about src because it is unused in max_display_length_for_field(). > + const Conv_source src(handler, 0, system_charset_info); > + const uint32 len= handler->max_display_length_for_field(src) + 1; > + const LEX_CSTRING empty= {STRING_WITH_LEN("")}; > + const uint flag_unsigned= handler->is_unsigned() ? UNSIGNED_FLAG : 0; > #define FL (NOT_NULL_FLAG | NO_DEFAULT_VALUE_FLAG) > +#define FLV (NOT_NULL_FLAG | NO_DEFAULT_VALUE_FLAG | flag_unsigned) FL was a bad name (probably meant flag) and so you added even worse FLV (what it mean?) Please make both probably not long, but at least a bit meaningful. > + return {{{"next_not_cached_value", len, handler, empty, FLV}, > + {"minimum_value", len, handler, empty, FLV}, > + {"maximum_value", len, handler, empty, FLV}, > + {"start_value", len, handler, > + {STRING_WITH_LEN("start value when sequences is created or value " > + "if RESTART is used")}, FLV}, > + {"increment", 21, &type_handler_slonglong, > + {STRING_WITH_LEN("increment value")}, FL}, > + {"cache_size", 21, &type_handler_ulonglong, empty, > + FL | UNSIGNED_FLAG}, > + {"cycle_option", 1, &type_handler_utiny, > + {STRING_WITH_LEN("0 if no cycles are allowed, 1 if the sequence " "should begin a new cycle when maximum_value is " "passed")}, FL | UNSIGNED_FLAG}, > + {"cycle_count", 21, &type_handler_slonglong, > + {STRING_WITH_LEN("How many cycles have been done")}, FL}, > + {NULL, 0, &type_handler_slonglong, {STRING_WITH_LEN("")}, 0}}}; > +#undef FLV > +#undef FL > +} > + > +bool sequence_definition::is_allowed_value_type(enum_field_types type) > +{ > + switch (type) > + { > + case MYSQL_TYPE_TINY: > + case MYSQL_TYPE_SHORT: > + case MYSQL_TYPE_LONG: > + case MYSQL_TYPE_INT24: > + case MYSQL_TYPE_LONGLONG: > + return true; > + default: > + return false; > + } > +} > > -static Field_definition sequence_structure[]= > +Type_handler const *sequence_definition::value_type_handler() > { > - {"next_not_cached_value", 21, &type_handler_slonglong, > - {STRING_WITH_LEN("")}, FL}, > - {"minimum_value", 21, &type_handler_slonglong, {STRING_WITH_LEN("")}, FL}, > - {"maximum_value", 21, &type_handler_slonglong, {STRING_WITH_LEN("")}, FL}, > - {"start_value", 21, &type_handler_slonglong, {STRING_WITH_LEN("start value when sequences is created or value if RESTART is used")}, FL}, > - {"increment", 21, &type_handler_slonglong, > - {STRING_WITH_LEN("increment value")}, FL}, > - {"cache_size", 21, &type_handler_ulonglong, {STRING_WITH_LEN("")}, > - FL | UNSIGNED_FLAG}, > - {"cycle_option", 1, &type_handler_utiny, {STRING_WITH_LEN("0 if no cycles are allowed, 1 if the sequence should begin a new cycle when maximum_value is passed")}, > - FL | UNSIGNED_FLAG }, > - {"cycle_count", 21, &type_handler_slonglong, > - {STRING_WITH_LEN("How many cycles have been done")}, FL}, > - {NULL, 0, &type_handler_slonglong, {STRING_WITH_LEN("")}, 0} > -}; > + const Type_handler *handler= Type_handler::get_handler_by_field_type(value_type); > + return is_unsigned ? handler->type_handler_unsigned() : handler; > +} > > -#undef FL > +longlong sequence_definition::value_type_max() > +{ > + // value_type != MYSQL_TYPE_LONGLONG to avoid undefined behaviour > + // https://stackoverflow.com/questions/9429156/by-left-shifting-can-a-number-b… We use // only for one line coments please use /* */ above (please fx it in all your cases) > + return is_unsigned && value_type != MYSQL_TYPE_LONGLONG ? > + ~(~0ULL << 8 * value_type_handler()->calc_pack_length(0)) : > + ~value_type_min(); > +} > > +longlong sequence_definition::value_type_min() { > + return is_unsigned ? 0 : > + ~0ULL << (8 * value_type_handler()->calc_pack_length(0) - 1); > +} > > -#define MAX_AUTO_INCREMENT_VALUE 65535 > +/* > + Truncate `original` to `result`. > + If `original` is greater than value_type_max(), truncate down to value_type_max() > + If `original` is less than value_type_min(), truncate up to value_type_min() > +*/ Please use your standard description of all functions/methods /** One-line description Here longer multiline description of the function and maybe how it works @param paramater1 parameter1 description @param param2 param2 description @note Some notes about function and or implementation can be multiline. @return description what it return */ it also can be instead of @return returning value description @retval TRUE error @retval FALSE everything OK > +longlong sequence_definition::truncate_value(const Longlong_hybrid& original) > +{ > + if (is_unsigned) > + return original.to_ulonglong(value_type_max()); > + else if (original.is_unsigned_outside_of_signed_range()) > + return value_type_max(); > + else > + return original.value() > value_type_max() ? value_type_max() > + : original.value() < value_type_min() ? value_type_min() > + : original.value(); > +} > > /* > Check whether sequence values are valid. > @@ -82,49 +130,66 @@ static Field_definition sequence_structure[]= > true invalid > */ > > +// from_parser: whether to check foo_from_parser or foo, where foo in > +// {min_value, max_value, ...} Above method comment is in old style. You want to fix it (add a note), do just rewrite it please in the new style /** (see my example in this review) > bool sequence_definition::check_and_adjust(THD *thd, bool set_reserved_until) > { > longlong max_increment; > - DBUG_ENTER("sequence_definition::check"); > + DBUG_ENTER("sequence_definition::check_and_adjust"); > > if (!(real_increment= increment)) > real_increment= global_system_variables.auto_increment_increment; > > /* > - If min_value is not set, set it to LONGLONG_MIN or 1, depending on > + If min_value is not set, set it to value_type_min()+1 or 1, depending on > real_increment > */ > - if (!(used_fields & seq_field_used_min_value)) > - min_value= real_increment < 0 ? LONGLONG_MIN+1 : 1; > + if (!(used_fields & seq_field_specified_min_value)) > + min_value= real_increment < 0 ? value_type_min()+1 : 1; > + else > + { > + min_value= truncate_value(min_value_from_parser); > + if ((is_unsigned && (ulonglong) min_value <= (ulonglong) value_type_min()) || above line is too long (at least should fit in 80 better in 72 character) > + (!is_unsigned && min_value <= value_type_min())) > + { > + push_warning_printf( > + thd, Sql_condition::WARN_LEVEL_NOTE, ER_TRUNCATED_WRONG_VALUE, > + ER_THD(thd, ER_TRUNCATED_WRONG_VALUE), "INTEGER", "MINVALUE"); > + min_value= value_type_min() + 1; > + } > + } > > /* > - If max_value is not set, set it to LONGLONG_MAX or -1, depending on > + If max_value is not set, set it to value_type_max()-1 or -1, depending on > real_increment > */ > - if (!(used_fields & seq_field_used_max_value)) > - max_value= real_increment < 0 ? -1 : LONGLONG_MAX-1; > - > - if (max_value == LONGLONG_MAX) > - { > - push_warning_printf(thd, Sql_condition::WARN_LEVEL_NOTE, > - ER_TRUNCATED_WRONG_VALUE, > - ER_THD(thd, ER_TRUNCATED_WRONG_VALUE), > - "INTEGER", "MAXVALUE"); > - max_value= LONGLONG_MAX - 1; > - } > - if (min_value == LONGLONG_MIN) > + if (!(used_fields & seq_field_specified_max_value)) > + max_value= real_increment < 0 ? -1 : value_type_max()-1; > + else > { > - push_warning_printf( > - thd, Sql_condition::WARN_LEVEL_NOTE, ER_TRUNCATED_WRONG_VALUE, > - ER_THD(thd, ER_TRUNCATED_WRONG_VALUE), "INTEGER", "MINVALUE"); > - min_value= LONGLONG_MIN + 1; > + max_value= truncate_value(max_value_from_parser); > + if ((is_unsigned && (ulonglong) max_value >= (ulonglong) value_type_max()) || > + (!is_unsigned && max_value >= value_type_max())) > + { > + push_warning_printf( > + thd, Sql_condition::WARN_LEVEL_NOTE, ER_TRUNCATED_WRONG_VALUE, > + ER_THD(thd, ER_TRUNCATED_WRONG_VALUE), "INTEGER", "MAXVALUE"); > + max_value= value_type_max() - 1; > + } > } > > if (!(used_fields & seq_field_used_start)) > { > /* Use min_value or max_value for start depending on real_increment */ > start= real_increment < 0 ? max_value : min_value; > - } > + } else > + // If the supplied start value is out of range for the value type, > + // instead of immediately reporting error, we truncate it to > + // value_type_min or value_type_max depending on which side it is > + // one. Whenever such truncation happens, the condition that > + // max_value >= start >= min_value will be violated, and the error > + // will be reported then. We use // only for one line coments please use /* */ above (please fx it in all your cases) > + start= truncate_value(start_from_parser); Do we check that start value fit in [minvalue, maxvalue] range? > > if (set_reserved_until) > reserved_until= start; > @@ -136,12 +201,23 @@ bool sequence_definition::check_and_adjust(THD *thd, bool set_reserved_until) > llabs(real_increment) : > MAX_AUTO_INCREMENT_VALUE); > > - if (max_value >= start && > - max_value > min_value && > + // Common case for error, signed or unsigned. > + if (!is_allowed_value_type(value_type) || cache < 0) > + DBUG_RETURN(TRUE); > + > + // TODO: check for cache < (ULONGLONG_MAX - max_increment) / max_increment Why above is in TODO, IMHO it is not so difficult to make (maybe I do not see something?) > + if (is_unsigned && (ulonglong) max_value >= (ulonglong) start && > + (ulonglong) max_value > (ulonglong) min_value && > + (ulonglong) start >= (ulonglong) min_value && > + // Just like the case in signed, where a positive sequence > + // cannot have a negatvie increment, an unsigned sequence is > + // positive, so the increment has to be positive Why we can not count backward for unsigned sequence? Taking into account that increment probably should not exceed half of allowed range we always can store it as bigint (or decimal if we should support sequences with 1 value in them). We use // only for one line coments please use /* */ above (please fx it in all your cases) > + (real_increment > 0 && (ulonglong) reserved_until >= (ulonglong) min_value)) > + DBUG_RETURN(FALSE); > + > + if (!is_unsigned && max_value >= start && max_value > min_value && > start >= min_value && > - max_value != LONGLONG_MAX && > - min_value != LONGLONG_MIN && > - cache >= 0 && cache < (LONGLONG_MAX - max_increment) / max_increment && > + cache < (LONGLONG_MAX - max_increment) / max_increment && > ((real_increment > 0 && reserved_until >= min_value) || > (real_increment < 0 && reserved_until <= max_value))) > DBUG_RETURN(FALSE); > @@ -165,6 +241,11 @@ void sequence_definition::read_fields(TABLE *table) > cache= table->field[5]->val_int(); > cycle= table->field[6]->val_int(); > round= table->field[7]->val_int(); > + value_type= table->field[0]->type(); > + is_unsigned= table->field[0]->is_unsigned(); > + min_value_from_parser= Longlong_hybrid(min_value, is_unsigned); > + max_value_from_parser= Longlong_hybrid(max_value, is_unsigned); > + start_from_parser= Longlong_hybrid(start, is_unsigned); > dbug_tmp_restore_column_map(&table->read_set, old_map); > used_fields= ~(uint) 0; > print_dbug(); > @@ -181,10 +262,10 @@ void sequence_definition::store_fields(TABLE *table) > > /* zero possible delete markers & null bits */ > memcpy(table->record[0], table->s->default_values, table->s->null_bytes); > - table->field[0]->store(reserved_until, 0); > - table->field[1]->store(min_value, 0); > - table->field[2]->store(max_value, 0); > - table->field[3]->store(start, 0); > + table->field[0]->store(reserved_until, is_unsigned); > + table->field[1]->store(min_value, is_unsigned); > + table->field[2]->store(max_value, is_unsigned); > + table->field[3]->store(start, is_unsigned); > table->field[4]->store(increment, 0); > table->field[5]->store(cache, 0); > table->field[6]->store((longlong) cycle != 0, 0); > @@ -210,10 +291,17 @@ bool check_sequence_fields(LEX *lex, List<Create_field> *fields) > uint field_count; > uint field_no; > const char *reason; > + Sequence_row_definition row_structure; > DBUG_ENTER("check_sequence_fields"); > > field_count= fields->elements; > - if (field_count != array_elements(sequence_structure)-1) > + if (!field_count) > + { > + reason= "Wrong number of columns"; > + goto err; > + } > + row_structure= sequence_structure(fields->head()->type_handler()); > + if (field_count != array_elements(row_structure.fields)-1) > { > reason= "Wrong number of columns"; We do not have only english speaking users, it was done bad before and now we have chance to fix it and make it correct via localised errors. > goto err; > @@ -236,7 +324,7 @@ bool check_sequence_fields(LEX *lex, List<Create_field> *fields) > > for (field_no= 0; (field= it++); field_no++) > { > - Field_definition *field_def= &sequence_structure[field_no]; > + const Sequence_field_definition *field_def= &row_structure.fields[field_no]; > if (my_strcasecmp(system_charset_info, field_def->field_name, > field->field_name.str) || > field->flags != field_def->flags || > @@ -265,12 +353,13 @@ bool check_sequence_fields(LEX *lex, List<Create_field> *fields) > true Failure (out of memory) > */ > > -bool prepare_sequence_fields(THD *thd, List<Create_field> *fields) > +bool sequence_definition::prepare_sequence_fields(List<Create_field> *fields, bool alter) too long line > { > - Field_definition *field_info; > DBUG_ENTER("prepare_sequence_fields"); > + const Sequence_row_definition row_def= sequence_structure(value_type_handler()); We put DBUG_ENTER after variables definition (C code heritage) > > - for (field_info= sequence_structure; field_info->field_name ; field_info++) > + for (const Sequence_field_definition *field_info= row_def.fields; > + field_info->field_name; field_info++) > { > Create_field *new_field; > LEX_CSTRING field_name= {field_info->field_name, > @@ -285,6 +374,8 @@ bool prepare_sequence_fields(THD *thd, List<Create_field> *fields) > new_field->char_length= field_info->length; > new_field->comment= field_info->comment; > new_field->flags= field_info->flags; > + if (alter) > + new_field->change = field_name; > if (unlikely(fields->push_back(new_field))) > DBUG_RETURN(TRUE); /* purify inspected */ > } > @@ -313,20 +404,18 @@ bool sequence_insert(THD *thd, LEX *lex, TABLE_LIST *org_table_list) > Reprepare_observer *save_reprepare_observer; > sequence_definition *seq= lex->create_info.seq_create_info; > bool temporary_table= org_table_list->table != 0; > + /* > + seq is 0 if sequence was created with CREATE TABLE instead of > + CREATE SEQUENCE > + */ > + bool create_new= !seq; > Open_tables_backup open_tables_backup; > Query_tables_list query_tables_list_backup; > TABLE_LIST table_list; // For sequence table > DBUG_ENTER("sequence_insert"); > > - /* > - seq is 0 if sequence was created with CREATE TABLE instead of > - CREATE SEQUENCE > - */ > - if (!seq) > - { > - if (!(seq= new (thd->mem_root) sequence_definition)) > - DBUG_RETURN(TRUE); > - } > + if (create_new && !(seq= new (thd->mem_root) sequence_definition)) > + DBUG_RETURN(TRUE); > > #ifdef WITH_WSREP > if (WSREP_ON && seq->cache != 0) > @@ -387,7 +476,15 @@ bool sequence_insert(THD *thd, LEX *lex, TABLE_LIST *org_table_list) > else > table= org_table_list->table; > > - seq->reserved_until= seq->start; > + if (create_new) > + { > + seq->value_type= (*table->s->field)->type(); > + seq->is_unsigned= (*table->s->field)->is_unsigned(); > + // fixme: why do we need true here? because it is other copy so we reset it to the start, right? > + if (seq->check_and_adjust(thd, true)) > + DBUG_RETURN(TRUE); > + } > + > error= seq->write_initial_sequence(table); > { > uint save_unsafe_rollback_flags= > @@ -432,9 +529,9 @@ SEQUENCE::~SEQUENCE() > A sequence table can have many readers (trough normal SELECT's). > > We mark that we have a write lock in the table object so that > - ha_sequence::ha_write() can check if we have a lock. If already locked, then > + ha_sequence::write_row() can check if we have a lock. If already locked, then > ha_write() knows that we are running a sequence operation. If not, then > - ha_write() knows that it's an INSERT. > + ha_write() knows that it's an INSERT statement. > */ > > void SEQUENCE::write_lock(TABLE *table) > @@ -734,10 +831,9 @@ longlong SEQUENCE::next_value(TABLE *table, bool second_round, int *error) > write_lock(table); > > res_value= next_free_value; > - next_free_value= increment_value(next_free_value); > + next_free_value= increment_value(next_free_value, real_increment); > > - if ((real_increment > 0 && res_value < reserved_until) || > - (real_increment < 0 && res_value > reserved_until)) > + if (within_bounds(res_value, reserved_until, reserved_until, real_increment > 0)) > { > write_unlock(table); > DBUG_RETURN(res_value); > @@ -754,30 +850,10 @@ longlong SEQUENCE::next_value(TABLE *table, bool second_round, int *error) > overflow > */ > add_to= cache ? real_increment * cache : real_increment; > - out_of_values= 0; > > - if (real_increment > 0) > - { > - if (reserved_until > max_value - add_to || > - reserved_until + add_to > max_value) > - { > - reserved_until= max_value + 1; > - out_of_values= res_value >= reserved_until; > - } > - else > - reserved_until+= add_to; > - } > - else > - { > - if (reserved_until + add_to < min_value || > - reserved_until < min_value - add_to) > - { > - reserved_until= min_value - 1; > - out_of_values= res_value <= reserved_until; > - } > - else > - reserved_until+= add_to; > - } > + // TODO: consider extracting this refactoring to a separate earlier commit. You have send it on review, so I suppose you take decision about above? > + reserved_until= increment_value(reserved_until, add_to); > + out_of_values= !within_bounds(res_value, max_value + 1, min_value - 1, add_to > 0); > if (out_of_values) > { > if (!cycle || second_round) > @@ -866,7 +942,7 @@ int SEQUENCE::set_value(TABLE *table, longlong next_val, ulonglong next_round, > > write_lock(table); > if (is_used) > - next_val= increment_value(next_val); > + next_val= increment_value(next_val, real_increment); > > if (round > next_round) > goto end; // error = -1 > @@ -953,6 +1029,35 @@ bool Sql_cmd_alter_sequence::execute(THD *thd) > first_table)) > DBUG_RETURN(TRUE); > #endif /* WITH_WSREP */ > + > + if (new_seq->used_fields & seq_field_used_as) > + { > + // This shouldn't happen as it should have been prevented during > + // parsing. If something should not happened you better put DEBUG_ASSERT to notice in debug version if something goes wrong. Chenge comment stile if you want to leave it > + if (new_seq->used_fields - seq_field_used_as) > + DBUG_RETURN(TRUE); > + > + first_table->lock_type= TL_READ_NO_INSERT; > + first_table->mdl_request.set_type(MDL_SHARED_NO_WRITE); > + Alter_info alter_info; > + alter_info.flags= ALTER_CHANGE_COLUMN; > + if (new_seq->prepare_sequence_fields(&alter_info.create_list, true)) > + DBUG_RETURN(TRUE); > + Table_specification_st create_info; > + create_info.init(); > + create_info.alter_info= &alter_info; > + if (if_exists()) > + thd->push_internal_handler(&no_such_table_handler); > + error= mysql_alter_table(thd, &null_clex_str, &null_clex_str, &create_info, first_table, &alter_info, 0, (ORDER *) 0, 0, 0); > + if (if_exists()) > + { > + trapped_errors= no_such_table_handler.safely_trapped_errors(); > + thd->pop_internal_handler(); > + } > + // Do we need to store the sequence value in table share, like below? I do not understand comment above. (what it is about?) > + DBUG_RETURN(error); > + } > + > if (if_exists()) > thd->push_internal_handler(&no_such_table_handler); > error= open_and_lock_tables(thd, first_table, FALSE, 0); > @@ -989,22 +1094,30 @@ bool Sql_cmd_alter_sequence::execute(THD *thd) > if (!(new_seq->used_fields & seq_field_used_increment)) > new_seq->increment= seq->increment; > if (!(new_seq->used_fields & seq_field_used_min_value)) > - new_seq->min_value= seq->min_value; > + new_seq->min_value_from_parser= seq->min_value_from_parser; > if (!(new_seq->used_fields & seq_field_used_max_value)) > - new_seq->max_value= seq->max_value; > + new_seq->max_value_from_parser= seq->max_value_from_parser; > if (!(new_seq->used_fields & seq_field_used_start)) > - new_seq->start= seq->start; > + new_seq->start_from_parser= seq->start_from_parser; > if (!(new_seq->used_fields & seq_field_used_cache)) > new_seq->cache= seq->cache; > if (!(new_seq->used_fields & seq_field_used_cycle)) > new_seq->cycle= seq->cycle; > + if (!(new_seq->used_fields & seq_field_used_as)) > + { > + new_seq->value_type= seq->value_type; > + new_seq->is_unsigned= seq->is_unsigned; > + } > > /* If we should restart from a new value */ > if (new_seq->used_fields & seq_field_used_restart) > { > if (!(new_seq->used_fields & seq_field_used_restart_value)) > - new_seq->restart= new_seq->start; > - new_seq->reserved_until= new_seq->restart; > + new_seq->restart_from_parser= new_seq->start_from_parser; > + // Similar to start, we just need to truncate reserved_until and > + // the errors will be reported in check_and_adjust if truncation > + // happens on the wrong end. We use // only for one line coments please use /* */ above (please fx it in all your cases) > + new_seq->reserved_until= new_seq->truncate_value(new_seq->restart_from_parser); > } > > /* Let check_and_adjust think all fields are used */ > diff --git a/sql/sql_sequence.h b/sql/sql_sequence.h > index e2968cc20ae..0de92a90639 100644 > --- a/sql/sql_sequence.h > +++ b/sql/sql_sequence.h > @@ -25,12 +25,35 @@ > #define seq_field_used_cycle 32 > #define seq_field_used_restart 64 > #define seq_field_used_restart_value 128 > +#define seq_field_used_as 256 > +#define seq_field_specified_min_value 512 > +#define seq_field_specified_max_value 1024 > > /* Field position in sequence table for some fields we refer to directly */ > #define NEXT_FIELD_NO 0 > #define MIN_VALUE_FIELD_NO 1 > #define ROUND_FIELD_NO 7 > > +#include "mysql_com.h" > +#include "sql_type_int.h" > + > +class Create_field; > +class Type_handler; > + > +struct Sequence_field_definition > +{ > + const char *field_name; > + uint length; > + const Type_handler *type_handler; > + LEX_CSTRING comment; > + ulong flags; > +}; > + > +struct Sequence_row_definition > +{ > + Sequence_field_definition fields[9]; > +}; > + > /** > sequence_definition is used when defining a sequence as part of create > */ > @@ -39,20 +62,37 @@ class sequence_definition :public Sql_alloc > { > public: > sequence_definition(): > - min_value(1), max_value(LONGLONG_MAX-1), start(1), increment(1), > - cache(1000), round(0), restart(0), cycle(0), used_fields(0) > + min_value_from_parser(1, false), > + max_value_from_parser(LONGLONG_MAX-1, false), start_from_parser(1, false), > + increment(1), cache(1000), round(0), restart_from_parser(0, false), cycle(0), used_fields(0), > + // We use value type and is_unsigned instead of a handler because > + // Type_handler is incomplete, which we cannot initialise here > + // with &type_handler_slonglong. We use // only for one line coments please use /* */ above (please fx it in all your cases) > + value_type(MYSQL_TYPE_LONGLONG), is_unsigned(false) > {} > longlong reserved_until; > longlong min_value; > longlong max_value; > longlong start; > + Longlong_hybrid min_value_from_parser; > + Longlong_hybrid max_value_from_parser; > + Longlong_hybrid start_from_parser; > longlong increment; > longlong cache; > ulonglong round; > + // TODO: allow unsigned in restart > longlong restart; // alter sequence restart value > + Longlong_hybrid restart_from_parser; > bool cycle; > uint used_fields; // Which fields where used in CREATE > - > + enum_field_types value_type; // value type of the sequence > + bool is_unsigned; > + > + Type_handler const *value_type_handler(); > + // max value for the value type, e.g. 32767 for smallint. > + longlong value_type_max(); > + // min value for the value type, e.g. -32768 for smallint. > + longlong value_type_min(); > bool check_and_adjust(THD *thd, bool set_reserved_until); > void store_fields(TABLE *table); > void read_fields(TABLE *table); > @@ -60,12 +100,16 @@ class sequence_definition :public Sql_alloc > int write(TABLE *table, bool all_fields); > /* This must be called after sequence data has been updated */ > void adjust_values(longlong next_value); > + longlong truncate_value(const Longlong_hybrid& original); > inline void print_dbug() > { > DBUG_PRINT("sequence", ("reserved: %lld start: %lld increment: %lld min_value: %lld max_value: %lld cache: %lld round: %lld", > reserved_until, start, increment, min_value, > max_value, cache, round)); > } > + static bool is_allowed_value_type(enum_field_types type); > + bool prepare_sequence_fields(List<Create_field> *fields, bool alter); > + > protected: > /* > The following values are the values from sequence_definition > @@ -107,24 +151,54 @@ class SEQUENCE :public sequence_definition > longlong next_value(TABLE *table, bool second_round, int *error); > int set_value(TABLE *table, longlong next_value, ulonglong round_arg, > bool is_used); > - longlong increment_value(longlong value) > + bool within_bounds(const longlong value, const longlong upper, const longlong lower, bool increasing) > + { > + return > + (is_unsigned && increasing && (ulonglong) value < (ulonglong) upper) || > + (is_unsigned && !increasing && (ulonglong) value > (ulonglong) lower) || > + (!is_unsigned && increasing && value < upper) || > + (!is_unsigned && !increasing && value > lower); > + } > + > + longlong increment_value(longlong value, const longlong increment) > { > - if (real_increment > 0) > + if (is_unsigned) > { > - if (value > max_value - real_increment || > - value + real_increment > max_value) > - value= max_value + 1; > + if (increment > 0) > + { > + // in case value + increment overflows > + if ((ulonglong) value > (ulonglong) max_value - (ulonglong) increment || > + // in case max_value - increment underflows > + (ulonglong) value + (ulonglong) increment > (ulonglong) max_value) > + value= max_value + 1; > + else > + value+= increment; > + } > else > - value+= real_increment; > - } > - else > - { > - if (value + real_increment < min_value || > - value < min_value - real_increment) > - value= min_value - 1; > + { > + if ((ulonglong) value - (ulonglong) (-increment) < (ulonglong) min_value || > + (ulonglong) value < (ulonglong) min_value + (ulonglong) (- increment)) > + value= min_value - 1; > + else > + value+= increment; > + } > + } else > + if (increment > 0) > + { > + if (value > max_value - increment || > + value + increment > max_value) > + value= max_value + 1; > + else > + value+= increment; > + } > else > - value+= real_increment; > - } > + { > + if (value + increment < min_value || > + value < min_value - increment) > + value= min_value - 1; > + else > + value+= increment; > + } > return value; > } > > @@ -159,9 +233,6 @@ class SEQUENCE_LAST_VALUE > uchar table_version[MY_UUID_SIZE]; > }; > > - > -class Create_field; > -extern bool prepare_sequence_fields(THD *thd, List<Create_field> *fields); > extern bool check_sequence_fields(LEX *lex, List<Create_field> *fields); > extern bool sequence_insert(THD *thd, LEX *lex, TABLE_LIST *table_list); > #endif /* SQL_SEQUENCE_INCLUDED */ > diff --git a/sql/sql_show.cc b/sql/sql_show.cc > index 1fd31bd4947..12e4f655515 100644 > --- a/sql/sql_show.cc > +++ b/sql/sql_show.cc > @@ -2694,12 +2694,29 @@ static int show_create_sequence(THD *thd, TABLE_LIST *table_list, > > packet->append(STRING_WITH_LEN("CREATE SEQUENCE ")); > append_identifier(thd, packet, &alias); > + /* Do not show " as <type>" in oracle mode as it is not supported: In case of multiline comment /* should be on separate line > + https://docs.oracle.com/en/database/oracle/oracle-database/21/sqlrf/CREATE-… you also probably should not show it if it is default (BIGINT), see other comment about it in the beginning. BTW add test of making sequence with/without oracle mode and dumping and restoring (mysqldump) them for all types. You can find examples of the tests by "ls mysql-test/main/*dump*.test". > + */ > + if (!(sql_mode & MODE_ORACLE)) > + { > + packet->append(STRING_WITH_LEN(" as ")); > + packet->append(seq->value_type_handler()->name().lex_cstring()); > + } > packet->append(STRING_WITH_LEN(" start with ")); > - packet->append_longlong(seq->start); > + if (seq->is_unsigned) > + packet->append_ulonglong(seq->start); > + else > + packet->append_longlong(seq->start); > packet->append(STRING_WITH_LEN(" minvalue ")); > - packet->append_longlong(seq->min_value); > + if (seq->is_unsigned) > + packet->append_ulonglong(seq->min_value); > + else > + packet->append_longlong(seq->min_value); > packet->append(STRING_WITH_LEN(" maxvalue ")); > - packet->append_longlong(seq->max_value); > + if (seq->is_unsigned) > + packet->append_ulonglong(seq->max_value); > + else > + packet->append_longlong(seq->max_value); > packet->append(STRING_WITH_LEN(" increment by ")); > packet->append_longlong(seq->increment); > if (seq->cache) > @@ -5512,6 +5529,36 @@ int fill_schema_schemata(THD *thd, TABLE_LIST *tables, COND *cond) > DBUG_RETURN(0); > } > > +static int get_schema_sequence_record(THD *thd, TABLE_LIST *tables, > + TABLE *table, bool res, > + const LEX_CSTRING *db_name, > + const LEX_CSTRING *table_name) > +{ > + DBUG_ENTER("get_sequence_record"); > + CHARSET_INFO *cs= system_charset_info; > + restore_record(table, s->default_values); > + sequence_definition *seq= tables->table->s->sequence; > + if (tables->table->s->table_type == TABLE_TYPE_SEQUENCE) > + { > + const Type_handler *handler= seq->value_type_handler(); > + table->field[0]->store(STRING_WITH_LEN("def"), cs); > + table->field[1]->store(db_name->str, db_name->length, cs); > + table->field[2]->store(table_name->str, table_name->length, cs); > + table->field[3]->store(handler->name().lex_cstring(), cs); > + table->field[4]->store(8 * handler->calc_pack_length(0)); > + table->field[5]->store(2); > + table->field[5]->set_notnull(); > + table->field[6]->store(0); > + table->field[6]->set_notnull(); > + table->field[7]->store(seq->start, seq->is_unsigned); > + table->field[8]->store(seq->min_value, seq->is_unsigned); > + table->field[9]->store(seq->max_value, seq->is_unsigned); > + table->field[10]->store(seq->increment, 0); > + table->field[11]->store(seq->cycle); > + DBUG_RETURN(schema_table_store_record(thd, table)); > + } > + DBUG_RETURN(0); > +} > > static int get_schema_tables_record(THD *thd, TABLE_LIST *tables, > TABLE *table, bool res, > @@ -9419,6 +9466,29 @@ ST_FIELD_INFO proc_fields_info[]= > }; > > > +ST_FIELD_INFO sequence_fields_info[]= > +{ > + Column("SEQUENCE_CATALOG", Catalog(), NOT_NULL, OPEN_FRM_ONLY), > + Column("SEQUENCE_SCHEMA", Name(), NOT_NULL, OPEN_FRM_ONLY), > + Column("SEQUENCE_NAME", Name(), NOT_NULL, "Table", OPEN_FRM_ONLY), > + Column("DATA_TYPE", Name(), NOT_NULL), > + Column("NUMERIC_PRECISION", SLong(21), NOT_NULL), > + Column("NUMERIC_PRECISION_RADIX", SLong(21), NULLABLE), > + Column("NUMERIC_SCALE", SLong(21), NULLABLE), > + // Decimal types for these values to incorporate possibly unsigned > + // longlongs. We use // only for one line coments please use /* */ above (please fx it in all your cases) > + Column("START_VALUE", Decimal(2100), NOT_NULL), > + Column("MINIMUM_VALUE", Decimal(2100), NOT_NULL), > + Column("MAXIMUM_VALUE", Decimal(2100), NOT_NULL), > + Column("INCREMENT", SLonglong(21), NOT_NULL), > + Column("CYCLE_OPTION", SLonglong(21), NOT_NULL), > + Column("DECLARED_DATA_TYPE", SLong(21), NULLABLE), > + Column("DECLARED_NUMERIC_PRECISION", SLong(21), NULLABLE), > + Column("DECLARED_NUMERIC_SCALE", SLong(21), NULLABLE), > + CEnd() > +}; > + > + > ST_FIELD_INFO stat_fields_info[]= > { > Column("TABLE_CATALOG", Catalog(), NOT_NULL, OPEN_FRM_ONLY), > @@ -9985,6 +10055,8 @@ ST_SCHEMA_TABLE schema_tables[]= > fill_schema_schemata, make_schemata_old_format, 0, 1, -1, 0, 0}, > {"SCHEMA_PRIVILEGES", Show::schema_privileges_fields_info, 0, > fill_schema_schema_privileges, 0, 0, -1, -1, 0, 0}, > + {"SEQUENCES", Show::sequence_fields_info, 0, > + get_all_tables, make_old_format, get_schema_sequence_record, 1, 2, 0, 0}, > {"SESSION_STATUS", Show::variables_fields_info, 0, > fill_status, make_old_format, 0, 0, -1, 0, 0}, > {"SESSION_VARIABLES", Show::variables_fields_info, 0, > diff --git a/sql/sql_yacc.yy b/sql/sql_yacc.yy > index e3298a4a6c1..27ce8bcdbfc 100644 > --- a/sql/sql_yacc.yy > +++ b/sql/sql_yacc.yy > @@ -200,6 +200,9 @@ void _CONCAT_UNDERSCORED(turn_parser_debug_on,yyparse)() > ulonglong ulonglong_number; > longlong longlong_number; > uint sp_instr_addr; > + // Longlong_hybrid does not have a default constructor, hence the > + // default value below. We use // only for one line coments please use /* */ above (please fx it in all your cases) > + Longlong_hybrid longlong_hybrid_number= Longlong_hybrid(0, false); > > /* structs */ > LEX_CSTRING lex_str; > @@ -1466,7 +1469,10 @@ bool my_yyoverflow(short **a, YYSTYPE **b, size_t *yystacksize); > ulonglong_num real_ulonglong_num > > %type <longlong_number> > - sequence_value_num sequence_truncated_value_num > + sequence_value_num > + > +%type <longlong_hybrid_number> > + sequence_value_hybrid_num sequence_truncated_value_hybrid_num > > %type <choice> choice > > @@ -2429,8 +2435,8 @@ create: > } > > /* No fields specified, generate them */ > - if (unlikely(prepare_sequence_fields(thd, > - &lex->alter_info.create_list))) > + if (unlikely(lex->create_info.seq_create_info->prepare_sequence_fields( > + &lex->alter_info.create_list, false))) > MYSQL_YYABORT; > > /* CREATE SEQUENCE always creates a sequence */ > @@ -2605,13 +2611,25 @@ sequence_defs: > ; > > sequence_def: > - MINVALUE_SYM opt_equal sequence_truncated_value_num > + AS int_type field_options > + { > + if (unlikely(Lex->create_info.seq_create_info->used_fields & > + seq_field_used_as)) > + my_yyabort_error((ER_DUP_ARGUMENT, MYF(0), "AS")); > + if ($3 & ZEROFILL_FLAG) > + my_yyabort_error((ER_BAD_OPTION_VALUE, MYF(0), "ZEROFILL", "AS")); > + Lex->create_info.seq_create_info->value_type = $2->field_type(); > + Lex->create_info.seq_create_info->is_unsigned = $3 & UNSIGNED_FLAG ? true : false; > + Lex->create_info.seq_create_info->used_fields|= seq_field_used_as; > + } > + | MINVALUE_SYM opt_equal sequence_truncated_value_hybrid_num > { > if (unlikely(Lex->create_info.seq_create_info->used_fields & > seq_field_used_min_value)) > my_yyabort_error((ER_DUP_ARGUMENT, MYF(0), "MINVALUE")); > - Lex->create_info.seq_create_info->min_value= $3; > + Lex->create_info.seq_create_info->min_value_from_parser= $3; > Lex->create_info.seq_create_info->used_fields|= seq_field_used_min_value; > + Lex->create_info.seq_create_info->used_fields|= seq_field_specified_min_value; > } > | NO_SYM MINVALUE_SYM > { > @@ -2625,13 +2643,14 @@ sequence_def: > my_yyabort_error((ER_DUP_ARGUMENT, MYF(0), "MINVALUE")); > Lex->create_info.seq_create_info->used_fields|= seq_field_used_min_value; > } > - | MAXVALUE_SYM opt_equal sequence_truncated_value_num > + | MAXVALUE_SYM opt_equal sequence_truncated_value_hybrid_num > { > if (unlikely(Lex->create_info.seq_create_info->used_fields & > seq_field_used_max_value)) > my_yyabort_error((ER_DUP_ARGUMENT, MYF(0), "MAXVALUE")); > - Lex->create_info.seq_create_info->max_value= $3; > + Lex->create_info.seq_create_info->max_value_from_parser= $3; > Lex->create_info.seq_create_info->used_fields|= seq_field_used_max_value; > + Lex->create_info.seq_create_info->used_fields|= seq_field_specified_max_value; > } > | NO_SYM MAXVALUE_SYM > { > @@ -2645,12 +2664,12 @@ sequence_def: > my_yyabort_error((ER_DUP_ARGUMENT, MYF(0), "MAXVALUE")); > Lex->create_info.seq_create_info->used_fields|= seq_field_used_max_value; > } > - | START_SYM opt_with sequence_value_num > + | START_SYM opt_with sequence_value_hybrid_num > { > if (unlikely(Lex->create_info.seq_create_info->used_fields & > seq_field_used_start)) > my_yyabort_error((ER_DUP_ARGUMENT, MYF(0), "START")); > - Lex->create_info.seq_create_info->start= $3; > + Lex->create_info.seq_create_info->start_from_parser= $3; > Lex->create_info.seq_create_info->used_fields|= seq_field_used_start; > } > | INCREMENT_SYM opt_by sequence_value_num > @@ -2705,7 +2724,7 @@ sequence_def: > my_yyabort_error((ER_DUP_ARGUMENT, MYF(0), "RESTART")); > Lex->create_info.seq_create_info->used_fields|= seq_field_used_restart; > } > - | RESTART_SYM opt_with sequence_value_num > + | RESTART_SYM opt_with sequence_value_hybrid_num > { > if (unlikely(Lex->sql_command != SQLCOM_ALTER_SEQUENCE)) > { > @@ -2715,7 +2734,7 @@ sequence_def: > if (unlikely(Lex->create_info.seq_create_info->used_fields & > seq_field_used_restart)) > my_yyabort_error((ER_DUP_ARGUMENT, MYF(0), "RESTART")); > - Lex->create_info.seq_create_info->restart= $3; > + Lex->create_info.seq_create_info->restart_from_parser= $3; > Lex->create_info.seq_create_info->used_fields|= seq_field_used_restart | seq_field_used_restart_value; > } > ; > @@ -7122,6 +7141,8 @@ alter: > { > /* Create a generic ALTER SEQUENCE statment. */ > Lex->m_sql_cmd= new (thd->mem_root) Sql_cmd_alter_sequence($3); > + if ((Lex->create_info.seq_create_info->used_fields & seq_field_used_as) && (Lex->create_info.seq_create_info->used_fields - seq_field_used_as)) Above lie is to long > + my_yyabort_error((ER_NOT_SUPPORTED_YET, MYF(0), "ALTER SEQUENCE with both AS <type> and something else.")); Above lie is to long > if (unlikely(Lex->m_sql_cmd == NULL)) > MYSQL_YYABORT; > } stmt_end {} > @@ -12563,6 +12584,7 @@ real_ulong_num: > | dec_num_error { MYSQL_YYABORT; } > ; > > +// For simple sequence metadata values that are signed and do not need truncation Above lie is to long > sequence_value_num: > opt_plus NUM { int error; $$= (longlong) my_strtoll10($2.str, (char**) 0, &error); } > | opt_plus LONG_NUM { int error; $$= (longlong) my_strtoll10($2.str, (char**) 0, &error); } > @@ -12579,15 +12601,74 @@ sequence_value_num: > } > ; > > -sequence_truncated_value_num: > - opt_plus NUM { int error; $$= (longlong) my_strtoll10($2.str, (char**) 0, &error); } > - | opt_plus LONG_NUM { int error; $$= (longlong) my_strtoll10($2.str, (char**) 0, &error); } > - | opt_plus ULONGLONG_NUM { $$= LONGLONG_MAX; } > - | opt_plus DECIMAL_NUM { $$= LONGLONG_MAX; } > - | '-' NUM { int error; $$= -(longlong) my_strtoll10($2.str, (char**) 0, &error); } > - | '-' LONG_NUM { int error; $$= -(longlong) my_strtoll10($2.str, (char**) 0, &error); } > - | '-' ULONGLONG_NUM { $$= LONGLONG_MIN; } > - | '-' DECIMAL_NUM { $$= LONGLONG_MIN; } > +// For sequence metadata values that may be unsigned but do not need truncation (start, restart) Above lie is to long (it looks like you have them a lot, so please fix all lines to fit in 80 (better 72 character). > +sequence_value_hybrid_num: > + opt_plus NUM > + { > + int error; > + $$= Longlong_hybrid(my_strtoll10($2.str, (char**) 0, &error), false); > + } > + | opt_plus LONG_NUM > + { > + int error; > + $$= Longlong_hybrid(my_strtoll10($2.str, (char**) 0, &error), false); > + } > + | opt_plus ULONGLONG_NUM > + { > + int error; > + $$= Longlong_hybrid(my_strtoll10($2.str, (char**) 0, &error), true); > + } > + | '-' NUM > + { > + int error; > + $$= Longlong_hybrid(- my_strtoll10($2.str, (char**) 0, &error), false); > + } > + | '-' LONG_NUM > + { > + int error; > + $$= Longlong_hybrid(- my_strtoll10($2.str, (char**) 0, &error), false); > + } > + | '-' ULONGLONG_NUM > + { > + int error; > + const ulonglong abs= my_strtoll10($2.str, (char**) 0, &error); > + if (abs == 1 + (ulonglong) LONGLONG_MAX) > + $$= Longlong_hybrid(LONGLONG_MIN, false); > + else > + thd->parse_error(ER_DATA_OUT_OF_RANGE); > + } > + ; > + > +// For sequence metadata values that may be unsigned and need truncation (maxvalue, minvalue) > +sequence_truncated_value_hybrid_num: > + opt_plus NUM > + { > + int error; > + $$= Longlong_hybrid(my_strtoll10($2.str, (char**) 0, &error), false); > + } > + | opt_plus LONG_NUM > + { > + int error; > + $$= Longlong_hybrid(my_strtoll10($2.str, (char**) 0, &error), false); > + } > + | opt_plus ULONGLONG_NUM > + { > + int error; > + $$= Longlong_hybrid(my_strtoll10($2.str, (char**) 0, &error), true); > + } > + | opt_plus DECIMAL_NUM { $$= Longlong_hybrid(ULONGLONG_MAX, true); } > + | '-' NUM > + { > + int error; > + $$= Longlong_hybrid(- my_strtoll10($2.str, (char**) 0, &error), false); > + } > + | '-' LONG_NUM > + { > + int error; > + $$= Longlong_hybrid(- my_strtoll10($2.str, (char**) 0, &error), false); > + } > + | '-' ULONGLONG_NUM { $$= Longlong_hybrid(LONGLONG_MIN, false); } > + | '-' DECIMAL_NUM { $$= Longlong_hybrid(LONGLONG_MIN, false); } > ; > > ulonglong_num:

3 6

Re: [Maria-developers] 8c1ad2a9fe9: MDEV-30633 DATETIME to TIMESTAMP conversion to return maximum timestamp on overflow
by Sergei Golubchik 08 Mar '23

08 Mar '23

Hi, Alexander, please add tests with fractional seconds. remember, that in UTC the max timestamp(2) is 2038-01-19 03:14:07.99 and max timestamp(6) is 2038-01-19 03:14:07.999999 On Mar 08, Alexander Barkov wrote: > revision-id: 8c1ad2a9fe9 (mariadb-10.11.1-48-g8c1ad2a9fe9) > parent(s): ce4a289f1c3 > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2023-02-13 17:25:18 +0400 > message: > > MDEV-30633 DATETIME to TIMESTAMP conversion to return maximum timestamp on overflow a bit more verbose description would be nice > diff --git a/mysql-test/main/events_bugs.result b/mysql-test/main/events_bugs.result > --- a/mysql-test/main/events_bugs.result > +++ b/mysql-test/main/events_bugs.result > @@ -35,12 +35,34 @@ SET NAMES latin1; > set @a=3; > CREATE PROCEDURE p_16 () CREATE EVENT e_16 ON SCHEDULE EVERY @a SECOND DO SET @a=5; > ERROR HY000: Recursion of EVENT DDL statements is forbidden when body is present > +SET time_zone='+00:00'; > +SET timestamp=UNIX_TIMESTAMP('2023-02-13 00:00:00'); > create event e_55 on schedule at 99990101000000 do drop table t; > -ERROR HY000: Incorrect AT value: '99990101000000' Hmm, why did you not keep it an error? > +Warnings: > +Warning 1292 Truncated incorrect timestamp value: '9999-01-01 00:00:00' > +Warning 1105 Event scheduler is switched off, use SET GLOBAL event_scheduler=ON to enable it. > diff --git a/mysql-test/main/func_time.result b/mysql-test/main/func_time.result > --- a/mysql-test/main/func_time.result > +++ b/mysql-test/main/func_time.result > @@ -599,19 +599,25 @@ Warnings: > Warning 1292 Truncated incorrect unixtime value: '2147483648' > select unix_timestamp('2039-01-20 01:00:00'); > unix_timestamp('2039-01-20 01:00:00') > -NULL > +2147483647 > +Warnings: > +Warning 1292 Truncated incorrect timestamp value: '2039-01-20 01:00:00' it's consistent with MariaDB [test]> select cast(1e30 as int); +---------------------+ | cast(1e30 as int) | +---------------------+ | 9223372036854775807 | +---------------------+ 1 row in set, 1 warning (0.000 sec) Note (Code 1916): Got overflow when converting '1e30' to SIGNED BIGINT. Value truncated good. Except that it's Note vs Warning. Want to unify this somehow? > select unix_timestamp('1968-01-20 01:00:00'); > unix_timestamp('1968-01-20 01:00:00') > NULL isn't it strange? That overflow is capped to max value, while underflow is NULL? > diff --git a/mysql-test/main/select.result b/mysql-test/main/select.result > --- a/mysql-test/main/select.result > +++ b/mysql-test/main/select.result > @@ -3766,11 +3766,15 @@ AND t1.ts BETWEEN "2006-01-01" AND "2006-12-31"; > id select_type table type possible_keys key key_len ref rows Extra > 1 SIMPLE t2 const PRIMARY PRIMARY 4 const 1 > 1 SIMPLE t1 range ts ts 4 NULL 2 Using index condition; Using where > +Warnings: > +Warning 1292 Truncated incorrect timestamp value: '2999-12-31 00:00:00' Looks questionable. The query compares timestamp with a datetime. I think in this case timestamp should be casted to a datetime (so, no warning), not vice versa. > SELECT * FROM t1 LEFT JOIN t2 ON (t1.a=t2.a) WHERE t1.a=30 > AND t1.ts BETWEEN t2.dt1 AND t2.dt2 > AND t1.ts BETWEEN "2006-01-01" AND "2006-12-31"; > a ts a dt1 dt2 > 30 2006-01-03 23:00:00 30 2006-01-01 00:00:00 2999-12-31 00:00:00 > +Warnings: > +Warning 1292 Truncated incorrect timestamp value: '2999-12-31 00:00:00' > DROP TABLE t1,t2; > create table t1 (a bigint unsigned); > insert into t1 values > diff --git a/sql/item_timefunc.cc b/sql/item_timefunc.cc > --- a/sql/item_timefunc.cc > +++ b/sql/item_timefunc.cc > @@ -2793,11 +2793,17 @@ bool Item_func_convert_tz::get_date(THD *thd, MYSQL_TIME *ltime, > return true; > > { > - uint not_used; > - my_time_tmp= from_tz->TIME_to_gmt_sec(ltime, &not_used); > + uint error_code; > + my_time_tmp= from_tz->TIME_to_gmt_sec(ltime, &error_code); > ulong sec_part= ltime->second_part; > - /* my_time_tmp is guaranteed to be in the allowed range */ > - if (my_time_tmp) > + /* > + my_time_tmp is guaranteed to be in the allowed range. > + Don't perform the conversion in case the source DATETIME was above > + TIMESTAMP_MAX_VALUE (and was truncated to TIMESTAMP_MAX_VALUE). why not? > + */ > + if (my_time_tmp && > + (my_time_tmp != TIMESTAMP_MAX_VALUE || > + error_code != ER_WARN_DATA_OUT_OF_RANGE)) why `my_time_tmp != TIMESTAMP_MAX_VALUE` ? shouldn't error_code check be sufficient? > to_tz->gmt_sec_to_TIME(ltime, my_time_tmp); > /* we rely on the fact that no timezone conversion can change sec_part */ > ltime->second_part= sec_part; Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 6e28a576913: move alloca() definition from all *.h files to one place
by Sergei Golubchik 10 Feb '23

10 Feb '23

Hi, Julius, this is the combined diff of both commits: git diff e1ce867e5ab0c^^ e1ce867e5ab0c On Feb 10, Julius Goryavsky wrote: > diff --git a/include/CMakeLists.txt b/include/CMakeLists.txt > index a7b98a11050..7d87c4a42b3 100644 > --- a/include/CMakeLists.txt > +++ b/include/CMakeLists.txt > @@ -104,6 +105,7 @@ ENDMACRO() > > INSTALL_COMPAT_HEADER(my_global.h "") > INSTALL_COMPAT_HEADER(my_config.h "") > +INSTALL_COMPAT_HEADER(my_alloca.h "") I wouldn't use INSTALL_COMPAT_HEADER for my_alloca.h > INSTALL_COMPAT_HEADER(my_sys.h "") > INSTALL_COMPAT_HEADER(mysql_version.h " > #include <mariadb_version.h> > diff --git a/include/my_alloca.h b/include/my_alloca.h > new file mode 100644 > index 00000000000..fec0320195e > --- /dev/null > +++ b/include/my_alloca.h > @@ -0,0 +1,37 @@ > +/* Copyright (c) 2023, MariaDB Corporation. > + > + This program is free software; you can redistribute it and/or modify > + it under the terms of the GNU General Public License as published by > + the Free Software Foundation; version 2 of the License. > + > + This program is distributed in the hope that it will be useful, > + but WITHOUT ANY WARRANTY; without even the implied warranty of > + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > + GNU General Public License for more details. > + > + You should have received a copy of the GNU General Public License > + along with this program; if not, write to the Free Software > + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1335 USA */ > + > +#ifndef MY_ALLOCA_INCLUDED > +#define MY_ALLOCA_INCLUDED > + > +#ifdef _WIN32 > +#include <malloc.h> /*for alloca*/ > +#ifndef alloca When can alloca be defined? > +#define alloca _alloca > +#endif > +#else > +#include <stdlib.h> why stdlib.h? > +#ifdef HAVE_ALLOCA_H > +#include <alloca.h> > +#endif > +#endif > + > +#if defined(HAVE_ALLOCA) > +#if defined(__GNUC__) && !defined(HAVE_ALLOCA_H) && !defined(alloca) add a comment describing your MinGW use case when this #if is needed > +#define alloca __builtin_alloca > +#endif /* GNUC */ > +#endif > + > +#endif /* MY_ALLOCA_INCLUDED */ > diff --git a/include/mysql/service_encryption.h b/include/mysql/service_encryption.h > index 69d205a27e8..cab1418e1d6 100644 > --- a/include/mysql/service_encryption.h > +++ b/include/mysql/service_encryption.h > @@ -24,21 +24,19 @@ > *provider* (encryption plugin). > */ > > +#ifndef MYSQL_ABI_CHECK > +#include <my_alloca.h> > +#endif > + > #ifdef __cplusplus > extern "C" { > #endif > > #ifndef MYSQL_ABI_CHECK why not to put #include <my_alloca.h> here? > #ifdef _WIN32 > -#include <malloc.h> > #ifndef __cplusplus > #define inline __inline > #endif > -#else > -#include <stdlib.h> I suspect stdlib.h belongs here, not in my_alloca.h > -#ifdef HAVE_ALLOCA_H > -#include <alloca.h> > -#endif > #endif > #endif Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] ceb75e9bd9e: MDEV-30203: Move mysql symlinks to different package (fixes)
by Sergei Golubchik 10 Feb '23

10 Feb '23

Hi, Daniel, On Feb 09, Daniel Black wrote: > revision-id: ceb75e9bd9e (mariadb-10.11.1-152-gceb75e9bd9e) > parent(s): 526fce28b90 > author: Daniel Black > committer: Daniel Black > timestamp: 2023-02-09 12:20:57 +1100 > message: > > MDEV-30203: Move mysql symlinks to different package (fixes) > > Fixes MDEV-30571 MariaDB-client does not pull MariaDB-compat anymore, conflicts with RHEL packages > > Fixes MDEV-30574 MariaDB-server conflicts with MariaDB-common from previous versions > > MariaDB-Server now depends on MariaDB-common 11.0+. > > MariaDB-{Client,Server} append to the Requirements rather than overriding them. > > MariaDB-Server package now depends on the MariaDB-client 11.0+. > > Because of moves between packages MariaDB-Client conflicts with > MariaDB-server < 11.0.0. > > diff --git a/cmake/cpack_rpm.cmake b/cmake/cpack_rpm.cmake > index 638f6b2dabb..e88a5493eac 100644 > --- a/cmake/cpack_rpm.cmake > +++ b/cmake/cpack_rpm.cmake > @@ -189,7 +189,7 @@ FOREACH(SYM_COMPONENT Server Client) > SET(CPACK_RPM_${SYM}_PACKAGE_ARCHITECTURE "noarch") > SET(CPACK_RPM_${SYM}_USER_FILELIST ${ignored}) > STRING(TOLOWER ${SYM_COMPONENT} SYM_COMPONENT_LOWER) > - SET(CPACK_RPM_${SYM}_PACKAGE_REQUIRES "MariaDB-${SYM_COMPONENT_LOWER} >= 11.0.0") > + SETA(CPACK_RPM_${SYM}_PACKAGE_REQUIRES "MariaDB-${SYM_COMPONENT_LOWER} >= 11.0.0") why? > SETA(CPACK_RPM_${SYM_COMPONENT_LOWER}_PACKAGE_RECOMMENDS "MariaDB-${SYM}") > ENDFOREACH() > > @@ -205,10 +205,10 @@ SETA(CPACK_RPM_client_PACKAGE_PROVIDES > "mysql-client" > "mytop") > SETA(CPACK_RPM_client_PACKAGE_CONFLICTS > - "MariaDB-server < 10.6.0") > + "MariaDB-server < 11.0.0") because of resolve-stack-dump, ok > SETA(CPACK_RPM_common_PACKAGE_CONFLICTS > - "MariaDB-server < 10.6.1") > + "MariaDB-server < 11.0.0") why? > SETA(CPACK_RPM_devel_PACKAGE_OBSOLETES > "MySQL-devel") Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

[Maria-developers] MDEV-30218: On what should be passed as 'rows' to filtering code.
by Sergey Petrunia 01 Feb '23

01 Feb '23

Hello Igor, So I'm looking at the bb-10.4-release-igor tree with your latest patch: commit c4fa6c3c4eaacefd9bf50b5c88c2c72474a15bc5 (HEAD -> bb-10.4-release-igor, origin/bb-10.4-release-igor) Author: Igor Babaev <igor(a)askmonty.org> Date: Tue Jan 31 13:14:53 2023 -0800 MDEV-30218 Incorrect optimization for rowid_filtering Correction over the last patch for this MDEV. I modify the code to add a printout: diff --git a/sql/sql_select.cc b/sql/sql_select.cc index 4bdfb659513..2cd4495bc03 100644 --- a/sql/sql_select.cc +++ b/sql/sql_select.cc @@ -7965,6 +7965,8 @@ best_access_path(JOIN *join, table->best_range_rowid_filter_for_partial_join(start_key->key, rows, access_cost_factor); + trace_access_idx.add("ROWS_FOR_FILTER", rows); + trace_access_idx.add("REAL_ROWS_FOR_FILTER", tmp); if (filter) { tmp-= filter->get_adjusted_gain(rows) - filter->get_cmp_gain(rows); then, I'm trying this example: create table t1 (a int, b int); insert into t1 select seq, seq from seq_1_to_10000; create table t2 (a int, key(a)); insert into t2 select seq from seq_1_to_10000; insert into t2 select * from t2; insert into t2 select * from t2; analyze table t2 persistent for all; create table t3 ( pk int primary key, a int, b int, unique key(a), key(b) ); insert into t3 select seq, seq, seq from seq_1_to_100000; set optimizer_trace=1; explain select straight_join * from t1, t2, t3 where t2.a=t1.a and t3.a=t1.b and t3.b < 5000; +------+-------------+-------+---------------+---------------+------+---------+---------+--------+---------------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +------+-------------+-------+---------------+---------------+------+---------+---------+--------+---------------------------------+ | 1 | SIMPLE | t1 | ALL | NULL | NULL | NULL | NULL | 10157 | Using where | | 1 | SIMPLE | t2 | ref | a | a | 5 | j1.t1.a | 4 | Using index | | 1 | SIMPLE | t3 | eq_ref|filter | a,b | a|b | 5|5 | j1.t1.b | 1 (5%) | Using where; Using rowid filter | +------+-------------+-------+---------------+---------------+------+---------+---------+--------+---------------------------------+ good so far. select * from information_schema.optimizer_trace; shows this (note the added printouts in caps): { "plan_prefix": ["t1", "t2"], "table": "t3", "best_access_path": { "considered_access_paths": [ { "access_type": "eq_ref", "index": "a", "ROWS_FOR_FILTER": 40628, "REAL_ROWS_FOR_FILTER": 10157, "rowid_filter_key": "b", "rows": 1, "cost": 9498, "chosen": true }, Let's run the query and check what really happens: ANALYZE FORMAT=JSON select straight_join * from t1, t2, t3 where t2.a=t1.a and t3.a=t1.b and t3.b < 5000; shows: ... "table": { "table_name": "t3", "access_type": "eq_ref", "key": "a", ... "rowid_filter": { ... "r_lookups": 10000, ... }, "r_loops": 40000, ... So, ROWS_FOR_FILTER=40K REAL_ROWS_FOR_FILTER=10K which of the numbers is a closer match here for the observed value of r_lookups= 10K ? BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://petrunia.net

1 0

[Maria-developers] Unable to reset my mariadb root password.
by sanaboyina sarvani 01 Feb '23

01 Feb '23

Hello, I have just installed mariadb in my macos system. After installation it asks for the root password which I have not set yet. Tried to reset the root password but that did not work. So, how to know my mariadb root password? Is it possible to reset the password? Thank you. Regards

1 0

[Maria-developers] Optimizer call followup: negative gains from rowid filters
by Sergey Petrunia 31 Jan '23

31 Jan '23

Hello, Just taking notes from the optimizer call: Check out this branch: 10.4-mdev30218-print-negative-gain (git log -3 output below) It has: - 10.4-release contents - then mdev30218_bb-10.4-release.diff patch provided by Igor - then a debug patch by me. I run: ./mtr --mem --parallel=5 --force main/rowid_filter*test grep -rn filter_gain var/ I get this: negative gains from filters: var/4/log/mysqld.1.err:52:filter_gain:-0.0569276, query=select 1 var/4/log/mysqld.1.err:58:filter_gain:-0.0569276, query=explain extended select 1 var/4/log/mysqld.1.err:64:filter_gain:-0.250816, query=SELECT * FROM t1 HAVING (7, 9) IN (SELECT t2.i1, t2.i2 FROM t2 WHERE t2.i1 = 3) var/4/log/mysqld.1.err:65:filter_gain:-0.250816, query=EXPLAIN EXTENDED var/4/log/mysqld.1.err:67:filter_gain:-10.207, query=explain var/4/log/mysqld.1.err:69:filter_gain:-10.207, query=SELECT * FROM t1 WHERE a > 0 AND b=0 var/4/log/mysqld.1.err:70:filter_gain:-1.17502, query=select count(*) from t1 where a between 21 and 30 and b=2 var/4/log/mysqld.1.err:71:filter_gain:-1.17502, query=explain extended select count(*) from t1 where a between 21 and 30 and b=2 var/4/log/mysqld.1.err:72:filter_gain:-1.17502, query=select * from t1 where a between 21 and 30 and b=2 var/4/log/mysqld.1.err:73:filter_gain:-2.18259, query=SELECT * FROM t1 var/4/log/mysqld.1.err:77:filter_gain:-2.18259, query=EXPLAIN EXTENDED SELECT * FROM t1 var/4/log/mysqld.1.err:81:filter_gain:-0.151308, query=set statement optimizer_switch='rowid_filter=on' for explain extended select t.id, fi.* var/4/log/mysqld.1.err:90:filter_gain:-0.151308, query=set statement optimizer_switch='rowid_filter=on' for select t.id, fi.* var/4/log/mysqld.1.err:99:filter_gain:-0.151308, query=set statement optimizer_switch='rowid_filter=on' for explain extended select t.id, fi.* var/4/log/mysqld.1.err:108:filter_gain:-0.151308, query=set statement optimizer_switch='rowid_filter=on' for select t.id, fi.* var/4/log/mysqld.1.err:117:filter_gain:-0.151308, query=set statement optimizer_switch='rowid_filter=on' for analyze format=json select t.id, fi.* var/3/log/mysqld.1.err:51:filter_gain:-0.0569276, query=select 1 var/3/log/mysqld.1.err:57:filter_gain:-0.0569276, query=explain extended select 1 var/3/log/mysqld.1.err:63:filter_gain:-0.250816, query=SELECT * FROM t1 HAVING (7, 9) IN (SELECT t2.i1, t2.i2 FROM t2 WHERE t2.i1 = 3) var/3/log/mysqld.1.err:64:filter_gain:-0.250816, query=EXPLAIN EXTENDED var/3/log/mysqld.1.err:66:filter_gain:-12.7213, query=explain var/3/log/mysqld.1.err:68:filter_gain:-12.7213, query=SELECT * FROM t1 WHERE a > 0 AND b=0 === git log -3 output === commit 82d5977b9fd6be5f5152d6831cd744fc739a2f24 (HEAD -> 10.4-mdev30218-print-negative-gain, origin/10.4-mdev30218-print-negative-gain, bb-10.4-release) Author: Sergei Petrunia <sergey(a)mariadb.com> Date: Tue Jan 31 20:11:17 2023 +0300 Debug patch: print filter_gain if it is less than 0. commit 47a40d90627dcf3c870cecd33c345c2e224986af Author: Sergei Petrunia <sergey(a)mariadb.com> Date: Tue Jan 31 20:07:02 2023 +0300 Apply mdev30218_bb-10.4-release.diff commit c8f2e9a5c0ac5905f28b050b7df5a9ffd914b7e7 (origin/bb-10.4-release) Author: Oleksandr Byelkin <sanja(a)mariadb.com> Date: Mon Jan 30 11:49:42 2023 +0100 Fix number of rows passing in case of EQ_REF BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://petrunia.net

1 0

Re: [Maria-developers] dcd69cbd19a: MDEV-29447 MDEV-26285 Refactor spider_db_mbase_util::open_item_func
by Sergei Golubchik 30 Jan '23

30 Jan '23

Hi, Yuchen, Thanks! There are two kinds of comments below, you'll see which is which. I realize that this patch mainly just moves parts of spider code around. Some of my comments apply to the new code that didn't exist before and appeared only in this patch. It's what needs to be fixed (or discussed, if you disagree) before pushing. Other comments apply to the old spider code that this patch moved. If you'd like you can say "I'm just moving spider code around" and ignore all comments of that kind. /Sergei On Jan 22, Yuchen Pei wrote: > commit dcd69cbd19a > Author: Yuchen Pei <yuchen.pei(a)mariadb.com> > Date: Tue Jan 3 16:24:04 2023 +1100 > > MDEV-29447 MDEV-26285 Refactor spider_db_mbase_util::open_item_func > > Porting commit 3836098c29ef1b7ff9d5fbde99b690eab73a0df1 (MDEV-26285) Because 3836098c29ef will not be, apparently, pushed anywhere, you'd better put a self-containing comment here. May be use the comment of 3836098c29ef? Note that you shouldn't push your 10.4-11.0 versions of the patch directly, only push the 10.3 one (when ok) and let it propagate to 11.0 through merges. Your 10.4-11.0 patches are very useful, though, they will serve as guidelines for whoever will merge this change up. > to current versions 10.3+ to fix a problem (MDEV-29447) where field > items that are arguments of a func item may be used before created / > initialised. > > Signed-off-by: Yuchen Pei <yuchen.pei(a)mariadb.com> > > diff --git a/storage/spider/spd_db_mysql.cc b/storage/spider/spd_db_mysql.cc > index e942d1d9063..a886a4e6e99 100644 > --- a/storage/spider/spd_db_mysql.cc > +++ b/storage/spider/spd_db_mysql.cc > @@ -4035,6 +4042,170 @@ int spider_db_mbase_util::open_item_func( > uint alias_length, > bool use_fields, > spider_fields *fields > +) { > + DBUG_ENTER("spider_db_mbase_util::check_item_func"); incorrect function name in DBUG_ENTER > + > + int error = check_item_func(item_func, spider, alias, > + alias_length, use_fields, fields); > + if (error) > + DBUG_RETURN(error); > + if (!str) > + DBUG_RETURN(0); > + > + DBUG_RETURN(print_item_func(item_func, spider, str, alias, > + alias_length, use_fields, fields)); > +} > + > +namespace { > + bool item_func_is_timestampdiff( We generally use static in these cases. Why would an anonymous namespace be better? > + const char *func_name, > + int func_name_length > + ) { > + return func_name_length == 13 && > + !strncasecmp("timestampdiff", func_name, func_name_length); We build with rtti nowadays, so you can use dynamic_cast instead. and you won't need func_name and func_name_length. Ideally you won't need a check for timestampdiff at all > + } > + > + bool not_func_should_be_skipped( > + Item_func *item_func > + ){ > + DBUG_ENTER("not_func_should_be_skipped"); add DBUG_ASSERT(item_func->functype() == Item_func::NOT_FUNC); > + Item **item_list = item_func->arguments(); > + > + if (item_list[0]->type() == Item::COND_ITEM) > + { > + DBUG_PRINT("info",("spider item_list[0] is COND_ITEM")); > + Item_cond *item_cond = (Item_cond *) item_list[0]; > + if (item_cond->functype() == Item_func::COND_AND_FUNC) > + { > + DBUG_PRINT("info",("spider item_cond is COND_AND_FUNC")); > + List_iterator_fast<Item> lif(*(item_cond->argument_list())); > + bool has_expr_cache_item = FALSE; > + bool has_isnotnull_func = FALSE; > + bool has_other_item = FALSE; > + while(Item *item = lif++) > + { > + if ( > + item->type() == Item::EXPR_CACHE_ITEM > + ) { > + DBUG_PRINT("info",("spider EXPR_CACHE_ITEM")); > + has_expr_cache_item = TRUE; > + } else > + if ( > + item->type() == Item::FUNC_ITEM && > + ((Item_func *) item)->functype() == Item_func::ISNOTNULL_FUNC > + ) { > + DBUG_PRINT("info",("spider ISNOTNULL_FUNC")); > + has_isnotnull_func = TRUE; > + } else { > + DBUG_PRINT("info",("spider has other item")); > + DBUG_PRINT("info",("spider COND type=%d", item->type())); > + has_other_item = TRUE; > + } > + } > + if (has_expr_cache_item && has_isnotnull_func && !has_other_item) So, "not func should be skipped" if it's NOT (expr1 AND expr2 AND ...) and all expr's are either IS NOT NULL or a cached subquery. Why? IS NULL, IS NOT FALSE, non-cached subqueries, everything else is fine, IS NOT NULL if it is followed by `AND 2>1` is fine, OR is fine. Doesn't make much sense to me. > + { > + DBUG_PRINT("info",("spider NOT EXISTS skip")); > + DBUG_RETURN(TRUE); > + } > + } > + } > + DBUG_RETURN(FALSE); > + } > +} > + > +/** > + Check if the given item_func and its arguments can be pushed down to > + a data node. This function is recursive because we need to also check > + the arguments of the item_func. > + > + @return 0: success, otherwise: error > + */ > +int spider_db_mbase_util::check_item_func( > + Item_func *item_func, > + ha_spider *spider, > + const char *alias, > + uint alias_length, > + bool use_fields, > + spider_fields *fields > +) { > + DBUG_ENTER("spider_db_mbase_util::check_item_func"); > + > + Item_func::Functype func_type = item_func->functype(); > + DBUG_PRINT("info",("spider functype = %d", func_type)); > + > + const char *func_name = (char*) item_func->func_name(); > + int func_name_length = strlen(func_name); > + DBUG_PRINT("info",("spider func_name = %s", func_name)); > + > + /* The blacklist of the functions that cannot be pushed down */ > + if ( > + func_type == Item_func::TRIG_COND_FUNC || > + func_type == Item_func::CASE_SEARCHED_FUNC || > + func_type == Item_func::CASE_SIMPLE_FUNC Does CASE need to be skipped at all? > + ) { > + DBUG_RETURN(ER_SPIDER_COND_SKIP_NUM); > + } > + > + if (func_type == Item_func::NOT_FUNC) > + { > + /* Why the following check is necessary? */ indeed > + if(not_func_should_be_skipped(item_func)) > + DBUG_RETURN(ER_SPIDER_COND_SKIP_NUM); > + } > + > + if (func_type == Item_func::UDF_FUNC) old code used switch here. I think it looks more natural than a sequence of if's. A compiler might optimize them internally into a switch, but I'd better do it explicitly. > + { > + int use_pushdown_udf = spider_param_use_pushdown_udf( > + spider->trx->thd, spider->share->use_pushdown_udf); > + if (!use_pushdown_udf) { > + DBUG_RETURN(ER_SPIDER_COND_SKIP_NUM); > + } > + } > + > + if (func_type == Item_func::FT_FUNC) { > + if (spider_db_check_ft_idx(item_func, spider) == MAX_KEY) > + DBUG_RETURN(ER_SPIDER_COND_SKIP_NUM); > + } > + > + if (func_type == Item_func::UNKNOWN_FUNC) > + { > + if (item_func_is_timestampdiff(func_name, func_name_length)) { it doesn't seem that it needs to be skipped at all > + DBUG_RETURN(ER_SPIDER_COND_SKIP_NUM); > + } > + } > + /* End of the blacklist */ > + > + /* Check the arguments recursively */ > + if (uint item_count = item_func->argument_count()) > + { > + Item **item_list = item_func->arguments(); > + for (uint roop_count = 0; roop_count < item_count; roop_count++) > + { > + Item *item = item_list[roop_count]; > + if (int error_num = spider_db_print_item_type(item, NULL, spider, NULL, > + alias, alias_length, dbton_id, use_fields, fields)) it'd be more logical to invoke check_item_func() here instead of spider_db_print_item_type with str==NULL > + DBUG_RETURN(error_num); > + } > + } > + > + DBUG_RETURN(0); > +} > + > +/** > + The function print the string corresponding to the given item_func to str. > + The function is assumed to be called only when the check by the function > + check_item_func() has passed. > + > + @return 0: success, otherwise: error > + */ > +int spider_db_mbase_util::print_item_func( > + Item_func *item_func, > + ha_spider *spider, > + spider_string *str, > + const char *alias, > + uint alias_length, > + bool use_fields, > + spider_fields *fields > ) { > int error_num; > Item *item, **item_list = item_func->arguments(); > @@ -4049,13 +4220,14 @@ int spider_db_mbase_util::open_item_func( > last_str_length = SPIDER_SQL_NULL_CHAR_LEN; > int use_pushdown_udf; > bool merge_func = FALSE; > - DBUG_ENTER("spider_db_mbase_util::open_item_func"); > - if (str) > - { > + DBUG_ENTER("spider_db_mbase_util::print_item_func"); > + DBUG_ASSERT(!check_item_func(item_func, spider, alias, alias_length, > + use_fields, fields) && str); Make it two asserts please. One for !check_item_func(...) and the other DBUG_ASSERT(str); Generally, always try to split assert into as many separate asserts as possible, don't do assert(A && B); So that when it fails it'd provide as much information as possible about what exactly went wrong. > + > if (str->reserve(SPIDER_SQL_OPEN_PAREN_LEN)) > DBUG_RETURN(HA_ERR_OUT_OF_MEM); > str->q_append(SPIDER_SQL_OPEN_PAREN_STR, SPIDER_SQL_OPEN_PAREN_LEN); > - } > + > DBUG_PRINT("info",("spider functype = %d", item_func->functype())); > switch (item_func->functype()) > { > @@ -4374,94 +4523,12 @@ int spider_db_mbase_util::open_item_func( > { > if (!strncasecmp("utc_timestamp", func_name, func_name_length)) pretty much all that code can use dynamic_cast nowadays > { > - if (str) > str->length(str->length() - SPIDER_SQL_OPEN_PAREN_LEN); > DBUG_RETURN(spider_db_open_item_string(item_func, NULL, spider, str, > alias, alias_length, dbton_id, use_fields, fields)); > } else if (!strncasecmp("timestampdiff", func_name, func_name_length)) > { > -#ifdef ITEM_FUNC_TIMESTAMPDIFF_ARE_PUBLIC but it _is_ public, so you can push down Item_func_timestamp_diff all right > - Item_func_timestamp_diff *item_func_timestamp_diff = > - (Item_func_timestamp_diff *) item_func; > - if (str) > - { > - const char *interval_str; > - uint interval_len; > - switch (item_func_timestamp_diff->int_type) > - { > - case INTERVAL_YEAR: > - interval_str = SPIDER_SQL_YEAR_STR; > - interval_len = SPIDER_SQL_YEAR_LEN; > - break; > - case INTERVAL_QUARTER: > - interval_str = SPIDER_SQL_QUARTER_STR; > - interval_len = SPIDER_SQL_QUARTER_LEN; > - break; > - case INTERVAL_MONTH: > - interval_str = SPIDER_SQL_MONTH_STR; > - interval_len = SPIDER_SQL_MONTH_LEN; > - break; > - case INTERVAL_WEEK: > - interval_str = SPIDER_SQL_WEEK_STR; > - interval_len = SPIDER_SQL_WEEK_LEN; > - break; > - case INTERVAL_DAY: > - interval_str = SPIDER_SQL_DAY_STR; > - interval_len = SPIDER_SQL_DAY_LEN; > - break; > - case INTERVAL_HOUR: > - interval_str = SPIDER_SQL_HOUR_STR; > - interval_len = SPIDER_SQL_HOUR_LEN; > - break; > - case INTERVAL_MINUTE: > - interval_str = SPIDER_SQL_MINUTE_STR; > - interval_len = SPIDER_SQL_MINUTE_LEN; > - break; > - case INTERVAL_SECOND: > - interval_str = SPIDER_SQL_SECOND_STR; > - interval_len = SPIDER_SQL_SECOND_LEN; > - break; > - case INTERVAL_MICROSECOND: > - interval_str = SPIDER_SQL_MICROSECOND_STR; > - interval_len = SPIDER_SQL_MICROSECOND_LEN; > - break; > - default: > - interval_str = ""; > - interval_len = 0; > - break; > - } > - str->length(str->length() - SPIDER_SQL_OPEN_PAREN_LEN); > - if (str->reserve(func_name_length + SPIDER_SQL_OPEN_PAREN_LEN + > - interval_len + SPIDER_SQL_COMMA_LEN)) > - DBUG_RETURN(HA_ERR_OUT_OF_MEM); > - str->q_append(func_name, func_name_length); > - str->q_append(SPIDER_SQL_OPEN_PAREN_STR, SPIDER_SQL_OPEN_PAREN_LEN); > - str->q_append(interval_str, interval_len); > - str->q_append(SPIDER_SQL_COMMA_STR, SPIDER_SQL_COMMA_LEN); > - } > - if ((error_num = spider_db_print_item_type(item_list[0], NULL, spider, > - str, alias, alias_length, dbton_id, use_fields, fields))) > - DBUG_RETURN(error_num); > - if (str) > - { > - if (str->reserve(SPIDER_SQL_COMMA_LEN)) > - DBUG_RETURN(HA_ERR_OUT_OF_MEM); > - str->q_append(SPIDER_SQL_COMMA_STR, SPIDER_SQL_COMMA_LEN); > - } > - if ((error_num = spider_db_print_item_type(item_list[1], NULL, spider, > - str, alias, alias_length, dbton_id, use_fields, fields))) > - DBUG_RETURN(error_num); > - if (str) > - { > - if (str->reserve(SPIDER_SQL_CLOSE_PAREN_LEN)) > - DBUG_RETURN(HA_ERR_OUT_OF_MEM); > - str->q_append(SPIDER_SQL_CLOSE_PAREN_STR, > - SPIDER_SQL_CLOSE_PAREN_LEN); > - } > - DBUG_RETURN(0); > -#else > DBUG_RETURN(ER_SPIDER_COND_SKIP_NUM); > -#endif > } > } else if (func_name_length == 14) > { > @@ -5038,84 +4996,17 @@ int spider_db_mbase_util::open_item_func( > func_name = (char*)item_func->func_name(); > func_name_length = strlen(func_name); > } > - } > break; > case Item_func::CASE_SEARCHED_FUNC: > case Item_func::CASE_SIMPLE_FUNC: > -#ifdef ITEM_FUNC_CASE_PARAMS_ARE_PUBLIC Item_func_case looks somewhat different today, but I suspect it can be pushed too > - Item_func_case *item_func_case = (Item_func_case *) item_func; > - if (str) > - { > - if (str->reserve(SPIDER_SQL_CASE_LEN)) > - DBUG_RETURN(HA_ERR_OUT_OF_MEM); > - str->q_append(SPIDER_SQL_CASE_STR, SPIDER_SQL_CASE_LEN); > - } > - if (item_func_case->first_expr_num != -1) > - { > - if ((error_num = spider_db_print_item_type( > - item_list[item_func_case->first_expr_num], NULL, spider, str, > - alias, alias_length, dbton_id, use_fields, fields))) > - DBUG_RETURN(error_num); > - } > - for (roop_count = 0; roop_count < item_func_case->ncases; > - roop_count += 2) > - { > - if (str) > - { > - if (str->reserve(SPIDER_SQL_WHEN_LEN)) > - DBUG_RETURN(HA_ERR_OUT_OF_MEM); > - str->q_append(SPIDER_SQL_WHEN_STR, SPIDER_SQL_WHEN_LEN); > - } > - if ((error_num = spider_db_print_item_type( > - item_list[roop_count], NULL, spider, str, > - alias, alias_length, dbton_id, use_fields, fields))) > - DBUG_RETURN(error_num); > - if (str) > - { > - if (str->reserve(SPIDER_SQL_THEN_LEN)) > - DBUG_RETURN(HA_ERR_OUT_OF_MEM); > - str->q_append(SPIDER_SQL_THEN_STR, SPIDER_SQL_THEN_LEN); > - } > - if ((error_num = spider_db_print_item_type( > - item_list[roop_count + 1], NULL, spider, str, > - alias, alias_length, dbton_id, use_fields, fields))) > - DBUG_RETURN(error_num); > - } > - if (item_func_case->else_expr_num != -1) > - { > - if (str) > - { > - if (str->reserve(SPIDER_SQL_ELSE_LEN)) > - DBUG_RETURN(HA_ERR_OUT_OF_MEM); > - str->q_append(SPIDER_SQL_ELSE_STR, SPIDER_SQL_ELSE_LEN); > - } > - if ((error_num = spider_db_print_item_type( > - item_list[item_func_case->else_expr_num], NULL, spider, str, > - alias, alias_length, dbton_id, use_fields, fields))) > - DBUG_RETURN(error_num); > - } > - if (str) > - { > - if (str->reserve(SPIDER_SQL_END_LEN + SPIDER_SQL_CLOSE_PAREN_LEN)) > - DBUG_RETURN(HA_ERR_OUT_OF_MEM); > - str->q_append(SPIDER_SQL_END_STR, SPIDER_SQL_END_LEN); > - str->q_append(SPIDER_SQL_CLOSE_PAREN_STR, > - SPIDER_SQL_CLOSE_PAREN_LEN); > - } > - DBUG_RETURN(0); > -#else > DBUG_RETURN(ER_SPIDER_COND_SKIP_NUM); > -#endif > case Item_func::JSON_EXTRACT_FUNC: > func_name = (char*) item_func->func_name(); > func_name_length = strlen(func_name); > - if (str) > - { > if (str->reserve(func_name_length + SPIDER_SQL_OPEN_PAREN_LEN)) > DBUG_RETURN(HA_ERR_OUT_OF_MEM); > str->q_append(func_name, func_name_length); > str->q_append(SPIDER_SQL_OPEN_PAREN_STR, SPIDER_SQL_OPEN_PAREN_LEN); > - } > func_name = SPIDER_SQL_COMMA_STR; > func_name_length = SPIDER_SQL_COMMA_LEN; > separator_str = SPIDER_SQL_COMMA_STR; Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

[Maria-developers] To getting started for contribution at your GitHub repo
by Omkar shivaji Devgirikar 29 Jan '23

29 Jan '23

Respect sir, I Omkar Devgirikar , I am a student and as a beginner i want to start contributing in your server repo with C++ language . Can you please guide me with your project like resources for this project, how get started and some good first issues. Please look at my mail seriously and guide me please. I hope you will look at it. Thank You

2 2

Re: [Maria-developers] 06ce67c644b: MDEV-27653 long uniques don't work with unicode collations
by Sergei Golubchik 16 Jan '23

16 Jan '23

Hi, Alexander, On Jan 15, Alexander Barkov wrote: > revision-id: 06ce67c644b (mariadb-10.4.27-41-g06ce67c644b) > parent(s): 6cb84346e1b > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2023-01-10 18:27:16 +0400 > message: > > MDEV-27653 long uniques don't work with unicode collations > > diff --git a/sql/sql_admin.cc b/sql/sql_admin.cc > index 213d77f8237..5dd19c877af 100644 > --- a/sql/sql_admin.cc > +++ b/sql/sql_admin.cc > @@ -772,11 +772,14 @@ static bool mysql_admin_table(THD* thd, TABLE_LIST* tables, > int check_for_upgrade= file->ha_check_for_upgrade(check_opt); > > if (check_old_types == HA_ADMIN_NEEDS_ALTER || > - check_for_upgrade == HA_ADMIN_NEEDS_ALTER) > + check_old_types == HA_ADMIN_NEEDS_UPGRADE || > + check_for_upgrade == HA_ADMIN_NEEDS_ALTER || > + check_for_upgrade == HA_ADMIN_NEEDS_UPGRADE) eh. So old code was returning HA_ADMIN_NEEDS_ALTER actually quite intentionally. REPAIR TABLE was automatically switching to ALTER if these checks were returning HA_ADMIN_NEEDS_ALTER. So it didn't matter if the message confusingly said "Please use REPAIR", because it would've been ALTER internally anyway. Meaning, I suspect, that the old code was fine and you didn't need to change `case HA_ADMIN_NEEDS_ALTER` and didn't need to replace `return HA_ADMIN_NEEDS_UPGRADE` with HA_ADMIN_NEEDS_ALTER. All you needed to do was to return HA_ADMIN_NEEDS_ALTER from the check_long_hash_compatibility() method and the rest would've likely worked automatically. Or at least with much smaller changes. > { > /* We use extra_open_options to be able to open crashed tables */ > thd->open_options|= extra_open_options; > - result_code= admin_recreate_table(thd, table); > + result_code= admin_recreate_table(thd, table) ? HA_ADMIN_FAILED : > + HA_ADMIN_OK; good catch > thd->open_options&= ~extra_open_options; > goto send_result; > } Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] 0bef50e50b5: MDEV-20501: Assertion `maybe_null || !null_value' failed in Item_func_round::date_op
by Sergei Golubchik 13 Jan '23

13 Jan '23

Hi, Sergei, On Jan 13, Sergei Petrunia wrote: > revision-id: 0bef50e50b5 (mariadb-10.4.27-33-g0bef50e50b5) > parent(s): 5db970fc760 > author: Sergei Petrunia > committer: Sergei Petrunia > timestamp: 2023-01-04 16:50:12 +0300 > message: > > MDEV-20501: Assertion `maybe_null || !null_value' failed in Item_func_round::date_op > > When the optimizer finds a constant (or system) table $TBL which is empty > or has no matching row, it would set table->null_row=true. This is done > even for tables with table->maybe_null==0. > > Then, it would proceed to perform query optimization phases (what for?) > which will attempt to evaluate Item expressions referring to $TBL. > > Eventually some Item expression will get the value of $TBL.not_null_field, > get SQL NULL and fail an assertion. > > Fixed by not performing any query optimization steps after we've got > constant/empty tables with no matching rows. > > Test result changes contain a lot of changes like > - ... Impossible WHERE noticed after reading const tables > + ... no matching row in const table > > as well as other changes caused by slightly-different processing of > the special case of empty constant tables. > > diff --git a/sql/sql_select.cc b/sql/sql_select.cc > index 6a441c5047b..57ffd58f8b4 100644 > --- a/sql/sql_select.cc > +++ b/sql/sql_select.cc > @@ -2382,6 +2382,9 @@ int JOIN::optimize_stage2() > if (subq_exit_fl) > goto setup_subq_exit; > > + if (zero_result_cause) > + goto setup_subq_exit; In many other cases it's, like zero_result_cause= "No matching min/max row"; subq_exit_fl= true; that is, subq_exit_fl is set whenever zero_result_cause is set. Meaning, perhaps you should do the same in your new code below and then you wouldn't need those lines you've added above? > if (unlikely(thd->check_killed())) > DBUG_RETURN(1); > > @@ -5392,7 +5398,32 @@ make_join_statistics(JOIN *join, List<TABLE_LIST> &tables_list, > } > } > } while (ref_changed); > - > + > + /* > + ConstRowNotFoundShortcut-1: > + Some constant/system tables have mo matching rows. This means that > + the join operation output will be empty. > + Short-cut further optimization steps. > + Note that some query plan steps will still be performed to handle > + implicit grouping, join result setup, etc. > + See also: ConstRowNotFoundShortcut-{2,3}. > + > + Do not do this if we're optimizing for some UNION's fake_select_lex. > + We might be running UglySubqueryReoptimization (grep for name) and > + in this case constant tables are not reliable. > + */ > + if ((join->const_table_map & ~found_const_table_map) && > + !(join->select_lex->master_unit() && > + join->select_lex->master_unit()->fake_select_lex == join->select_lex)) why wouldn't you check for `describe` here? "reoptimization" hack is only needed for EXPLAIN. > + { > + join->zero_result_cause= "no matching row in const table"; > + join->table_count=0; > + join->const_tables= 0; > + join->join_tab= NULL; > + > + DBUG_RETURN(0); > + } > + > join->sort_by_table= get_sort_by_table(join->order, join->group_list, > join->select_lex->leaf_tables, > join->const_table_map); Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 5dcde8f6523: MDEV-27653 long uniques don't work with unicode collations
by Sergei Golubchik 06 Jan '23

06 Jan '23

Hi, Alexander, Looks good. A couple of comments, see below On Jan 06, Alexander Barkov wrote: > revision-id: 5dcde8f6523 (mariadb-10.4.26-64-g5dcde8f6523) > parent(s): ce443c85547 > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2022-10-28 15:37:44 +0400 > message: > > MDEV-27653 long uniques don't work with unicode collations > > diff --git a/mysql-test/main/ctype_utf8.test b/mysql-test/main/ctype_utf8.test > index cc61c2ae0fe..e2d4e4ab906 100644 > --- a/mysql-test/main/ctype_utf8.test > +++ b/mysql-test/main/ctype_utf8.test > @@ -2310,3 +2310,133 @@ VALUES (_latin1 0xDF) UNION VALUES(_utf8'a' COLLATE utf8_bin); > --echo # > --echo # End of 10.3 tests > --echo # > + > + > +--echo # > +--echo # Start of 10.4 tests > +--echo # > + > +--echo # > +--echo # MDEV-27653 long uniques don't work with unicode collations > +--echo # > + > +SET NAMES utf8mb3; > + > +# CHAR > + > +CREATE TABLE t1 ( > + a CHAR(30) COLLATE utf8mb3_general_ci, > + UNIQUE KEY(a) USING HASH > +); > +SHOW CREATE TABLE t1; > +INSERT INTO t1 VALUES ('a'); > +--error ER_DUP_ENTRY > +INSERT INTO t1 VALUES ('ä'); > +SELECT * FROM t1; > +DROP TABLE t1; > + > +CREATE TABLE t1 ( > + a CHAR(30) COLLATE utf8mb3_general_ci, > + UNIQUE KEY(a(10)) USING HASH > +); > +SHOW CREATE TABLE t1; > +INSERT INTO t1 VALUES ('a'); > +--error ER_DUP_ENTRY > +INSERT INTO t1 VALUES ('ä'); > +SELECT * FROM t1; > +DROP TABLE t1; > + > + > +# VARCHAR > + > +CREATE TABLE t1 ( > + a VARCHAR(30) COLLATE utf8mb3_general_ci, > + UNIQUE KEY(a) USING HASH > +); > +SHOW CREATE TABLE t1; > +INSERT INTO t1 VALUES ('a'); > +--error ER_DUP_ENTRY > +INSERT INTO t1 VALUES ('ä'); > +SELECT * FROM t1; > +DROP TABLE t1; > + > +CREATE TABLE t1 ( > + a VARCHAR(30) COLLATE utf8mb3_general_ci, > + UNIQUE KEY(a(10)) USING HASH > +); > +SHOW CREATE TABLE t1; > +INSERT INTO t1 VALUES ('a'); > +--error ER_DUP_ENTRY > +INSERT INTO t1 VALUES ('ä'); > +SELECT * FROM t1; > +DROP TABLE t1; > + > + > +# TEXT > + > +CREATE TABLE t1 (a TEXT COLLATE utf8mb3_general_ci UNIQUE); > +SHOW CREATE TABLE t1; > +INSERT INTO t1 VALUES ('a'); > +--error ER_DUP_ENTRY > +INSERT INTO t1 VALUES ('ä'); > +SELECT * FROM t1; > +DROP TABLE t1; > + > +CREATE TABLE t1 ( > + a LONGTEXT COLLATE utf8mb3_general_ci, > + UNIQUE KEY(a(10)) USING HASH > +); > +SHOW CREATE TABLE t1; > +INSERT INTO t1 VALUES ('a'); > +--error ER_DUP_ENTRY > +INSERT INTO t1 VALUES ('ä'); > +SELECT * FROM t1; > +DROP TABLE t1; > + > + > +# Testing upgrade: > +# Prior to MDEV-27653, the UNIQUE HASH function errorneously > +# took into account string octet length. > +# Old tables should still open and work, but with a wrong results. > + > +copy_file std_data/mysql_upgrade/mdev27653_100422_text.frm $MYSQLD_DATADIR/test/t1.frm; > +copy_file std_data/mysql_upgrade/mdev27653_100422_text.MYD $MYSQLD_DATADIR/test/t1.MYD; > +copy_file std_data/mysql_upgrade/mdev27653_100422_text.MYI $MYSQLD_DATADIR/test/t1.MYI; > +SHOW CREATE TABLE t1; > +SELECT a, OCTET_LENGTH(a) FROM t1 ORDER BY BINARY a; > +CHECK TABLE t1; > + > +# There is already a one byte value 'a' in the table > +--error ER_DUP_ENTRY > +INSERT INTO t1 VALUES ('A'); > + > +# There is already a two-byte value 'ä' in the table > +--error ER_DUP_ENTRY > +INSERT INTO t1 VALUES ('Ä'); > + > +# There were no three-byte values in the table so far. > +# The below value violates UNIQUE, but it gets inserted. > +# This is wrong but expected for a pre-MDEV-27653 table. > +INSERT INTO t1 VALUES ('Ấ'); > +SELECT a, OCTET_LENGTH(a) FROM t1 ORDER BY BINARY a; > +CHECK TABLE t1; > + > +# ALTER FORCE fails: it tries to rebuild the table > +# with a correct UNIQUE HASH function, but there are duplicates! > +--error ER_DUP_ENTRY > +ALTER TABLE t1 FORCE; please, try ALTER IGNORE TABLE too > + > +# Let's remove all duplicate values, so only the one-byte 'a' stays. > +# ALTER..FORCE should work after that. > +DELETE FROM t1 WHERE OCTET_LENGTH(a)>1; > +ALTER TABLE t1 FORCE; > + > +# Make sure that 'a' and 'ä' cannot co-exists any more, > +# because the table was recreated with a correct UNIQUE HASH function. > +--error ER_DUP_ENTRY > +INSERT INTO t1 VALUES ('ä'); > +DROP TABLE t1; > + > +--echo # > +--echo # End of 10.4 tests > +--echo # > diff --git a/sql/table.cc b/sql/table.cc > index b9260853381..97f71284005 100644 > --- a/sql/table.cc > +++ b/sql/table.cc > @@ -1267,7 +1286,11 @@ bool parse_vcol_defs(THD *thd, MEM_ROOT *mem_root, TABLE *table, > list_item= new (mem_root) Item_field(thd, keypart->field); > field_list->push_back(list_item, mem_root); > } > - Item_func_hash *hash_item= new(mem_root)Item_func_hash(thd, *field_list); > + > + Item_func_hash *hash_item= make_unique_hash_func(thd, mem_root, > + table->s->mysql_version, > + field_list); would be good to fix CHECK ... FOR UPGRADE too. > + > Virtual_column_info *v= new (mem_root) Virtual_column_info(); > field->vcol_info= v; > field->vcol_info->expr= hash_item; Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 92ff948d021: MDEV-29231 View returns wrong value with SQL_MODE 'NO_BACKSLASH_ESCAPES'
by Sergei Golubchik 06 Jan '23

06 Jan '23

Hi, Oleksandr, On Jan 06, Oleksandr Byelkin wrote: > revision-id: 92ff948d021 (mariadb-10.3.37-53-g92ff948d021) > parent(s): 180b2bcd538 > author: Oleksandr Byelkin > committer: Oleksandr Byelkin > timestamp: 2022-12-05 20:42:03 +0100 > message: > > MDEV-29231 View returns wrong value with SQL_MODE 'NO_BACKSLASH_ESCAPES' > > 1. make view printing sql_mode independent > 1.1. Print always current escape character > 1.2. Allow escape character usage independent of sql_mode > 2. Add workaround for parsing "like 3 in (0,1) escape 3" problem > (alwaysuse parances in case of printing ESCAPE) > > diff --git a/mysql-test/main/ctype_cp1250_ch.result b/mysql-test/main/ctype_cp1250_ch.result > index b0aa4cff382..88175ae5013 100644 > --- a/mysql-test/main/ctype_cp1250_ch.result > +++ b/mysql-test/main/ctype_cp1250_ch.result > @@ -129,7 +129,7 @@ EXPLAIN EXTENDED SELECT * FROM t1 WHERE CONCAT(c1)='a' AND CONCAT(c1) LIKE 'a '; > id select_type table type possible_keys key key_len ref rows filtered Extra > 1 SIMPLE t1 ALL NULL NULL NULL NULL 2 100.00 Using where > Warnings: > -Note 1003 select `test`.`t1`.`c1` AS `c1` from `test`.`t1` where concat(`test`.`t1`.`c1`) = 'a' and concat(`test`.`t1`.`c1`) like 'a ' > +Note 1003 select `test`.`t1`.`c1` AS `c1` from `test`.`t1` where concat(`test`.`t1`.`c1`) = 'a' and concat(`test`.`t1`.`c1`) like ('a ') escape '\\' I don't think there's a need to print the escape clause here, it only makes the output less readable. > diff --git a/mysql-test/main/default.result b/mysql-test/main/default.result > index c9ac96367aa..5c72b894bf6 100644 > --- a/mysql-test/main/default.result > +++ b/mysql-test/main/default.result > @@ -1936,7 +1936,7 @@ CREATE OR REPLACE TABLE t1 ( col INT DEFAULT ( 1 LIKE ( NOW() BETWEEN '2000-01-0 > SHOW CREATE TABLE t1; > Table Create Table > t1 CREATE TABLE `t1` ( > - `col` int(11) DEFAULT (1 like (current_timestamp() between '2000-01-01' and '2012-12-12')) > + `col` int(11) DEFAULT (1 like (current_timestamp() between '2000-01-01' and '2012-12-12') escape '\\') So, there're basically two approaches to a fix and we're inconsistently using both for different items. 1. one can always print expressions in the environment-independent way - create longer less readable text - not all items support it, new syntax might be needed (cannot emulate NO_BACKSLASH_ESCAPES with the ESCAPE clause) 2. one can set the environment (sql_mode) to the same hard-coded value before parsing views and vcols - one cannot dump the view/table definition in one sql_mode and load in another + this isn't normally a problem, as mysqldump also uses a fixed sql_mode - some item properties (what are they?) might be only set at fix_fields, fixing sql_mode during parsing won't help there Could we try to agree to one approach and use it consistently for all items from now on? > diff --git a/mysql-test/main/parser.result b/mysql-test/main/parser.result > index 0bb4e82c8b8..79ec9f539eb 100644 > --- a/mysql-test/main/parser.result > +++ b/mysql-test/main/parser.result > @@ -1325,11 +1325,11 @@ select 1 between 2 in (3,4) and 5 AS `1 between (2 in (3,4)) and 5` > create or replace view v1 as select 1 between (2 like 3) and 4; > Select view_definition from information_schema.views where table_schema='test' and table_name='v1'; > view_definition > -select 1 between 2 like 3 and 4 AS `1 between (2 like 3) and 4` > +select 1 between 2 like (3) escape '\\' and 4 AS `1 between (2 like 3) and 4` > create or replace view v1 as select 1 not between (2 like 3) and 4; > Select view_definition from information_schema.views where table_schema='test' and table_name='v1'; > view_definition > -select 1 not between 2 like 3 and 4 AS `1 not between (2 like 3) and 4` > +select 1 not between 2 like (3) escape '\\' and 4 AS `1 not between (2 like 3) and 4` > drop view v1; > # > # Start of 10.2 tests > diff --git a/mysql-test/main/precedence.result b/mysql-test/main/precedence.result > index fc6579651b4..ce26ca2acf2 100644 > --- a/mysql-test/main/precedence.result > +++ b/mysql-test/main/precedence.result > @@ -113,7 +113,7 @@ NOT 2 != 3 NOT (2 != 3) (NOT 2) != 3 > create or replace view v1 as select NOT 2 LIKE 3, NOT (2 LIKE 3), (NOT 2) LIKE 3; > Select view_definition from information_schema.views where table_schema='test' and table_name='v1'; > view_definition > -select 2 not like 3 AS `NOT 2 LIKE 3`,2 not like 3 AS `NOT (2 LIKE 3)`,!2 like 3 AS `(NOT 2) LIKE 3` > +select 2 not like (3) escape '\\' AS `NOT 2 LIKE 3`,2 not like (3) escape '\\' AS `NOT (2 LIKE 3)`,!2 like (3) escape '\\' AS `(NOT 2) LIKE 3` > select NOT 2 LIKE 3, NOT (2 LIKE 3), (NOT 2) LIKE 3 union select * from v1; > NOT 2 LIKE 3 NOT (2 LIKE 3) (NOT 2) LIKE 3 > 1 1 0 > diff --git a/mysql-test/main/precedence_bugs.result b/mysql-test/main/precedence_bugs.result > index 723ab823b48..9654e3c31db 100644 > --- a/mysql-test/main/precedence_bugs.result > +++ b/mysql-test/main/precedence_bugs.result > @@ -54,7 +54,7 @@ drop table t1; > create view v1 as select 1 like (now() between '2000-01-01' and '2012-12-12' ); > show create view v1; > View v1 > -Create View CREATE ALGORITHM=UNDEFINED DEFINER=`root`@`localhost` SQL SECURITY DEFINER VIEW `v1` AS select 1 like (current_timestamp() between '2000-01-01' and '2012-12-12') AS `1 like (now() between '2000-01-01' and '2012-12-12' )` > +Create View CREATE ALGORITHM=UNDEFINED DEFINER=`root`@`localhost` SQL SECURITY DEFINER VIEW `v1` AS select 1 like (current_timestamp() between '2000-01-01' and '2012-12-12') escape '\\' AS `1 like (now() between '2000-01-01' and '2012-12-12' )` > character_set_client latin1 > collation_connection latin1_swedish_ci > drop view v1; if you'll always print ESCAPE clause, it'll make many tests in parser.test, precedence.test and precedence_bugs.test meaningless. Don't just update result files, please, remove tests that no longer make sense. Or rewrite them, when possible (but in precedence.test it's not). > diff --git a/sql/item_cmpfunc.cc b/sql/item_cmpfunc.cc > index b7b0c981c2d..702689a7496 100644 > --- a/sql/item_cmpfunc.cc > +++ b/sql/item_cmpfunc.cc > @@ -5262,10 +5262,15 @@ void Item_func_like::print(String *str, enum_query_type query_type) > str->append(STRING_WITH_LEN(" not ")); > str->append(func_name()); > str->append(' '); > - if (escape_used_in_parsing) > + if (escape_used_in_parsing || (query_type | QT_VIEW_INTERNAL)) > { > - args[1]->print_parenthesised(str, query_type, precedence()); > - str->append(STRING_WITH_LEN(" escape ")); > + /* > + Explicit parencies is workaround against parser bug which üprevent > + patsing "like 3 in (0,1) escape 3" correctly > + */ > + str->append('('); > + args[1]->print(str, query_type); > + str->append(STRING_WITH_LEN(") escape ")); no, this is wrong. Don't just force parentheses there. Instead, fix the precedence, so that the parentheses would be printed automatically when needed. > escape_item->print_parenthesised(str, query_type, higher_precedence()); > } > else > @@ -5390,10 +5395,7 @@ bool fix_escape_item(THD *thd, Item *escape_item, String *tmp_str, > if (escape_str) > { > const char *escape_str_ptr= escape_str->ptr(); > - if (escape_used_in_parsing && ( > - (((thd->variables.sql_mode & MODE_NO_BACKSLASH_ESCAPES) && > - escape_str->numchars() != 1) || > - escape_str->numchars() > 1))) > + if (escape_used_in_parsing && escape_str->numchars() > 1) I still think that ESCAPE '' doesn't mean "no escape character". I tried and in my tests it didn't. > { > my_error(ER_WRONG_ARGUMENTS,MYF(0),"ESCAPE"); > return TRUE; Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 6ff49e48e4f: MDEV-26161: fix of view protocol
by Sergei Golubchik 06 Jan '23

06 Jan '23

Hi, Oleksandr, On Jan 06, Oleksandr Byelkin wrote: > revision-id: 6ff49e48e4f (mariadb-10.3.37-54-g6ff49e48e4f) > parent(s): 92ff948d021 > author: Oleksandr Byelkin > committer: Oleksandr Byelkin > timestamp: 2022-12-06 10:22:06 +0100 > message: > > MDEV-26161: fix of view protocol could you please amend the commit comment to explain why the statement produces a different error in a view protocol? > diff --git a/mysql-test/main/gis.result b/mysql-test/main/gis.result > index 358be520b06..37613bc71a0 100644 > --- a/mysql-test/main/gis.result > +++ b/mysql-test/main/gis.result > @@ -4980,11 +4980,11 @@ ERROR HY000: Illegal parameter data type geometry for operation 'is_used_lock' > # MDEV-26161 crash in Gis_point::calculate_haversine > # > select st_distance_sphere(x'01030000000400000004000000000000', multipoint(point(124,204)), 10); > -ERROR 22003: Cannot get geometry object from data you send to the GEOMETRY field > +Got one of the listed errors > select st_distance_sphere(x'010300000004000000040000', multipoint(point(124,204)), 10); > -ERROR 22003: Cannot get geometry object from data you send to the GEOMETRY field > +Got one of the listed errors > select st_distance_sphere(x'010300000001000000040000', multipoint(point(124,204)), 10); > -ERROR 22003: Cannot get geometry object from data you send to the GEOMETRY field > +Got one of the listed errors > # > # End of 10.3 tests > # > diff --git a/mysql-test/main/gis.test b/mysql-test/main/gis.test > index 716fab9bfeb..07b207dcdc9 100644 > --- a/mysql-test/main/gis.test > +++ b/mysql-test/main/gis.test > @@ -3093,11 +3093,11 @@ SELECT IS_USED_LOCK(POINT(1,1)); > --echo # > --echo # MDEV-26161 crash in Gis_point::calculate_haversine > --echo # > ---error ER_CANT_CREATE_GEOMETRY_OBJECT > +--error ER_CANT_CREATE_GEOMETRY_OBJECT,ER_INTERNAL_ERROR > select st_distance_sphere(x'01030000000400000004000000000000', multipoint(point(124,204)), 10); > ---error ER_CANT_CREATE_GEOMETRY_OBJECT > +--error ER_CANT_CREATE_GEOMETRY_OBJECT,ER_INTERNAL_ERROR > select st_distance_sphere(x'010300000004000000040000', multipoint(point(124,204)), 10); > ---error ER_CANT_CREATE_GEOMETRY_OBJECT > +--error ER_CANT_CREATE_GEOMETRY_OBJECT,ER_INTERNAL_ERROR > select st_distance_sphere(x'010300000001000000040000', multipoint(point(124,204)), 10); > > --echo # > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] MDEV-28602 Wrong result with outer join, merged derived table and view
by Sergey Petrunia 04 Jan '23

04 Jan '23

Hi Rex, Please find below input for the commit. (I've actually committed an alternative fix now, but before doing that I wrote all the trivial comments below so I'm sending these anyway in order to show what is expected of a patch. I'm also including the rationale for creating an alternative patch) > commit 75af3195482d3da277825663d1150c0dfc55420a > Author: Rex <rex.johnston(a)mariadb.com> > Date: Fri Dec 23 05:28:59 2022 +1200 > > MDEV-28602 Wrong result with outer join, merged derived table and view > > const item reference being mishandled in outer to inner join on > filling in null values. > First, fixVersion. It is currently set to 10.11, but affectsVersion starts from 10.3 I think the bug and the fix are applicable for all versions. Second: commit comment. This is not descriptive enough: > const item reference being mishandled in outer to inner join on > filling in null values. Something like this would be better: <commit-comment> A LEFT JOIN with a constant as a column of the inner table produced wrong query result if the optimizer had to write the inner table column into a temp table: SELECT ... FROM (SELECT /*non-mergeable select*/ FROM t1 LEFT JOIN (SELECT 'Y' as Val) t2 ON ...) Fixed this by: 1. Making Item_ref::save_in_field() call check_null_ref() to see if the referred table has a NULL-complemented row. 2. In order to do #1, moved null_ref_table() and check_null_ref() from Item_direct_view_ref to Item_ref. </commit-comment> Note: it starts with a siccint problem description, which is followed by a description of the fix. > diff --git a/sql/sql_join_cache.cc b/sql/sql_join_cache.cc > index 620c52a3f40..65bedc3d337 100644 > --- a/sql/sql_join_cache.cc > +++ b/sql/sql_join_cache.cc > @@ -2622,7 +2622,7 @@ enum_nested_loop_state JOIN_CACHE::join_null_complements(bool skip_last) > get_record(); > /* The outer row is complemented by nulls for each inner table */ > restore_record(join_tab->table, s->default_values); > - mark_as_null_row(join_tab->table); > + mark_as_null_row(join_tab->table); In the future please make sure patches do not contain unrelated meaningless changes like this one. > rc= generate_full_extensions(get_curr_rec()); > if (rc != NESTED_LOOP_OK && rc != NESTED_LOOP_NO_MORE_ROWS) > goto finish; > diff --git a/mysql-test/main/merge.test b/mysql-test/main/merge.test > index 0485f3ed1c3..05b05287e1b 100644 > --- a/mysql-test/main/merge.test > +++ b/mysql-test/main/merge.test Why is the patch in the merge.test ? merge.test at the top says clearly "Test of MERGE TABLES". That is, this is a test for tables with ENGINE=MERGE, a feature not related to this bug. The right test files for this bug would be main/join_outer.test and main/derived.test, I would prefer the first one. > @@ -2886,3 +2886,50 @@ drop table tm, t; > --echo # > --echo # End of 10.8 tests > --echo # > + > +--echo # > +--echo # MDEV-28602 Wrong result with outer join, merged derived table and view > +--echo # > + > +drop table if exists t1, t2; ^^^ This is not needed as there is a DROP TABLE right above > +create table t1 ( > + Election int(10) unsigned NOT NULL > +); > + > +insert into t1 (Election) values (1); > + +create table t2 ( + VoteID int(10), + ElectionID int(10), + UserID int(10) +); + +insert into t2 (ElectionID, UserID) values (2, 30), (3, 30); +# INSERT INTO t2 (ElectionID, UserID) VALUES (1, 30); ^^ No need to have commented-out lines in the commit. +drop view if exists v1; ^^^ same as above: this is not needed. +create view v1 as select * from t1 + left join ( select 'Y' AS Voted, ElectionID from t2 ) AS T + on T.ElectionID = t1.Election +limit 9; +# limit X causes merge algorithm select as opposed to temp table +select * from v1; ... > diff --git a/sql/item.h b/sql/item.h > index 2d598546b91..8684477b558 100644 > --- a/sql/item.h > +++ b/sql/item.h > @@ -5531,18 +5531,42 @@ class Item_sp > > class Item_ref :public Item_ident > { > +public: > + > +#define NO_NULL_TABLE (reinterpret_cast<TABLE *>(0x1)) > + > + void set_null_ref_table() > + { > + null_ref_table= NO_NULL_TABLE; > + } > + > + bool check_null_ref() This works, but looks like a change that is 1. Too big 2. Looks like a partial implementation I'm looking at methods of Item_direct_view_ref: - save_val() - save_org_in_field() - save_in_result_field() - val_real() and other val_XXX() and they all follow the same pattern: if (check_null_ref()) produce SQL NULL; else Call Item_direct_ref::same_method(); This suggests we could add Item_direct_view_ref::save_in_field() and use the same approach. Conversely, if we move check_null_ref() to be in Item_ref, we will end up with: - check_null_ref() is in Item_ref - methods that make use of it are in Item_direct_view_ref() - methods of Item_ref() do not make check_null_ref() check, except for save_in_field() which does it. I think the former looks more logical than the latter. I've made the patch for it and I'll ask Sanja (the Item_ref/views expert) to check it. BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://petrunia.net

1 0

Re: [Maria-developers] 6a8268b7893: MDEV-28915: mysql_upgrade fails due to old_mode="", with "Cannot load from mysql.proc. The table is probably corrupted"
by Sergei Golubchik 28 Dec '22

28 Dec '22

Hi, Rucha, On Dec 28, Rucha Deodhar wrote: > revision-id: 6a8268b7893 (mariadb-10.6.11-62-g6a8268b7893) > parent(s): 4ca5a0ec98d > author: Rucha Deodhar > committer: Rucha Deodhar > timestamp: 2022-12-15 17:45:25 +0530 > message: > > MDEV-28915: mysql_upgrade fails due to old_mode="", with "Cannot load from > mysql.proc. The table is probably corrupted" > > Analysis: When mysql_upgrade runs statements for upgrade, characterset is > converted to utf8mb4 because server starts with old_mode that interprets > utf8 to utf8mb4, but mysql.proc table has "utf8mb3" as hardcoded, so > it crashes with corrupted table. > > Fix: Changed Table_check_intact::check() definition to allow both > utf8mb3 and utf8mb4 by checking prefix and changing the upgrade scripts > to explicitly use utf8mb3 > > diff --git a/mysql-test/main/mysql_upgrade-28915.opt b/mysql-test/main/mysql_upgrade-28915.opt > new file mode 100644 > index 00000000000..0be5567a64d > --- /dev/null > +++ b/mysql-test/main/mysql_upgrade-28915.opt > @@ -0,0 +1 @@ > +--old-mode=NO_DUP_KEY_WARNINGS_WITH_IGNORE it's a bit confusing, try to use --old-mode= that is, empty old_mode, otherwise one starts thinking what this bug has to do with NO_DUP_KEY_WARNINGS_WITH_IGNORE. > diff --git a/mysql-test/main/mysql_upgrade-28915.test b/mysql-test/main/mysql_upgrade-28915.test > new file mode 100644 > index 00000000000..61b1fc263cb > --- /dev/null > +++ b/mysql-test/main/mysql_upgrade-28915.test > @@ -0,0 +1,78 @@ > +--source include/mysql_upgrade_preparation.inc > + > +let $MYSQLD_DATADIR= `select @@datadir`; > + > +--echo # > +--echo # Beginning of 10.6 test > +--echo # > +--echo # MDEV-28915: mysql_upgrade fails due to old_mode="", with "Cannot load > +--echo # from mysql.proc. The table is probably corrupted" > + > +--echo # Show that tables are created with utf8mb3 even without UTF8_IS_UTF8MB3 (see the .opt file) I don't think you've shown it. mtr does not recreate system tables for every test. You need to force it to re-bootstrap with one of the options that do it, for example, --innodb-page-size=16k (search mtr for "rebootstrap" for details). But even that won't help, because it'll only add "rebootstrap" options to the server's command line when bootstrapping. May be a combination of .cnf and .opt can do what you want, I didn't try. > + > +SHOW CREATE TABLE mysql.proc; > +SHOW CREATE TABLE mysql.event; > + > +--exec $MYSQL_UPGRADE --force 2>&1 > + > +--remove_file $MYSQLD_DATADIR/mysql_upgrade_info > + > +SHOW CREATE TABLE mysql.proc; > +SHOW CREATE TABLE mysql.event; > + > +--echo # Emulate that tables were created with utf8mb4 by an older version > + > +ALTER TABLE mysql.proc MODIFY db CHAR(64) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NOT NULL, > + MODIFY name CHAR(64) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NOT NULL, > + MODIFY specific_name CHAR(64) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL, > + MODIFY definer VARCHAR(64) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL, > + MODIFY comment TEXT CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL, > + MODIFY character_set_client CHAR(32) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL, > + MODIFY collation_connection CHAR(32) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL, > + MODIFY db_collation CHAR(32) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL; > + > +ALTER TABLE mysql.event MODIFY db CHAR(64) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL DEFAULT '', > + MODIFY name CHAR(64) CHARACTER SET utf8mb4 NOT NULL DEFAULT '', > + MODIFY definer VARCHAR(384) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL DEFAULT '', > + MODIFY comment CHAR(64) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL DEFAULT '', > + MODIFY character_set_client CHAR(32) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin DEFAULT NULL, > + MODIFY collation_connection CHAR(32) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin DEFAULT NULL, > + MODIFY db_collation CHAR(32) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin DEFAULT NULL; > + > +--echo # mysql_upgrade changes columns from utf8mb4 to utf8mb3 > + > +--exec $MYSQL_UPGRADE --force 2>&1 > +--remove_file $MYSQLD_DATADIR/mysql_upgrade_info > + > +--vertical_results > + > +SHOW CREATE TABLE mysql.proc; > +SHOW CREATE TABLE mysql.event; > + > +CREATE TABLE t1 (id1 INT, val1 VARCHAR(5)); > + > +DELIMITER |; > + > +CREATE PROCEDURE sp1 () > + BEGIN > + SELECT val1 FROM t1; > + END| You've made changes in sp.cc and event_db_repository.cc to ensure that these tables are usable even with utf8mb4 charset. But you don't test it if it works. You create a procedure and an event here, *after* tables were converted back to utf8mb3. you likely want to do it earlier, before mysql_upgrade, just after ALTER TABLE. > + > +DELIMITER ;| > + > +SELECT name, body_utf8, body FROM mysql.proc WHERE name like 'sp1'; > +CALL sp1(); > +SELECT name, body_utf8, body FROM mysql.proc WHERE name like 'sp1'; > + > +SET GLOBAL event_scheduler=ON; > + > +SELECT name, body_utf8, body FROM mysql.event; > +CREATE EVENT ev1 ON SCHEDULE EVERY 1 SECOND DO INSERT INTO t1 VALUES (1, 'abc'); > +SELECT name, body_utf8, body FROM mysql.event; > + > +SET GLOBAL event_scheduler=OFF; > +DROP EVENT ev1; > +DROP PROCEDURE sp1; > +DROP TABLE t1; > + > +--echo # end of 10.6 test Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 4ddf606debf: MDEV-30151 parse error 1=2 not between/in
by Sergei Golubchik 28 Dec '22

28 Dec '22

Hi, Alexander, I'm not sure I understand everything you did there, so questions below On Dec 28, Alexander Barkov wrote: > revision-id: 4ddf606debf (mariadb-10.3.37-59-g4ddf606debf) > parent(s): 8f30973234d > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2022-12-13 10:35:16 +0400 > message: > > MDEV-30151 parse error 1=2 not between/in > > diff --git a/sql/sql_yacc.yy b/sql/sql_yacc.yy > index 1f6485dac6a..ca614ff63e9 100644 > --- a/sql/sql_yacc.yy > +++ b/sql/sql_yacc.yy > @@ -1687,7 +1687,8 @@ bool my_yyoverflow(short **a, YYSTYPE **b, size_t *yystacksize); > > %left PREC_BELOW_NOT > > -%nonassoc NOT_SYM > +/* The precendence of boolean NOT is in fact here. See the comment below. */ > + > %left '=' EQUAL_SYM GE '>' LE '<' NE > %nonassoc IS > %right BETWEEN_SYM > @@ -1699,6 +1700,24 @@ bool my_yyoverflow(short **a, YYSTYPE **b, size_t *yystacksize); > %left '*' '/' '%' DIV_SYM MOD_SYM > %left '^' > %left MYSQL_CONCAT_SYM > +/* > + Boolean negation has a special branch in "expr" starting with NOT_SYM. > + The precedence of logical negation is determined by the grammar itself > + (without using Bison terminal symbol precedence) in this order > + - Boolean factor (i.e. logical AND) > + - Boolean NOT > + - Boolean test (such as '=', IS NULL, IS TRUE) > + > + But we also need a precedence for NOT_SYM in other contexts, > + to shift (without reduce) in these cases: > + predicate <here> NOT IN ... > + predicate <here> NOT BETWEEN ... > + predicate <here> NOT LIKE ... > + predicate <here> NOT REGEXP ... > + If the precedence of NOT_SYM was low, it would reduce immediately > + after scanning "predicate" and then produce a syntax error on "NOT". > +*/ > +%nonassoc NOT_SYM > %nonassoc NEG '~' NOT2_SYM BINARY > %nonassoc COLLATE_SYM > > @@ -1938,6 +1957,7 @@ bool my_yyoverflow(short **a, YYSTYPE **b, size_t *yystacksize); > literal insert_ident order_ident temporal_literal > simple_ident expr sum_expr in_sum_expr > variable variable_aux > + boolean_test boolean_factor > predicate bit_expr parenthesized_expr > table_wild simple_expr column_default_non_parenthesized_expr udf_expr > primary_expr string_factor_expr mysql_concatenation_expr > @@ -9840,79 +9860,87 @@ expr: > MYSQL_YYABORT; > } > } > - | NOT_SYM expr %prec NOT_SYM > + | boolean_factor what does that achieve? > + ; > + > +boolean_factor: > + NOT_SYM boolean_factor > { > $$= negate_expression(thd, $2); > if (unlikely($$ == NULL)) > MYSQL_YYABORT; > } > - | expr IS TRUE_SYM %prec IS > + | boolean_test %prec PREC_BELOW_NOT > + ; > + > +boolean_test: > + boolean_test IS TRUE_SYM %prec IS > { > $$= new (thd->mem_root) Item_func_istrue(thd, $1); > if (unlikely($$ == NULL)) > MYSQL_YYABORT; > } > - | expr IS not TRUE_SYM %prec IS > + | boolean_test IS not TRUE_SYM %prec IS > { > $$= new (thd->mem_root) Item_func_isnottrue(thd, $1); > if (unlikely($$ == NULL)) > MYSQL_YYABORT; > } > - | expr IS FALSE_SYM %prec IS > + | boolean_test IS FALSE_SYM %prec IS > { > $$= new (thd->mem_root) Item_func_isfalse(thd, $1); > if (unlikely($$ == NULL)) > MYSQL_YYABORT; > } > - | expr IS not FALSE_SYM %prec IS > + | boolean_test IS not FALSE_SYM %prec IS > { > $$= new (thd->mem_root) Item_func_isnotfalse(thd, $1); > if (unlikely($$ == NULL)) > MYSQL_YYABORT; > } > - | expr IS UNKNOWN_SYM %prec IS > + | boolean_test IS UNKNOWN_SYM %prec IS > { > $$= new (thd->mem_root) Item_func_isnull(thd, $1); > if (unlikely($$ == NULL)) > MYSQL_YYABORT; > } > - | expr IS not UNKNOWN_SYM %prec IS > + | boolean_test IS not UNKNOWN_SYM %prec IS > { > $$= new (thd->mem_root) Item_func_isnotnull(thd, $1); > if (unlikely($$ == NULL)) > MYSQL_YYABORT; > } > - | expr IS NULL_SYM %prec PREC_BELOW_NOT > + | boolean_test IS NULL_SYM %prec PREC_BELOW_NOT can you try %prec IS here? I suspect that if this affects anything, it'll cause bugs > { > $$= new (thd->mem_root) Item_func_isnull(thd, $1); > if (unlikely($$ == NULL)) > MYSQL_YYABORT; > } > - | expr IS not NULL_SYM %prec IS > + | boolean_test IS not NULL_SYM %prec IS > { > $$= new (thd->mem_root) Item_func_isnotnull(thd, $1); > if (unlikely($$ == NULL)) > MYSQL_YYABORT; > } > - | expr EQUAL_SYM predicate %prec EQUAL_SYM > + | boolean_test EQUAL_SYM predicate %prec EQUAL_SYM > { > $$= new (thd->mem_root) Item_func_equal(thd, $1, $3); > if (unlikely($$ == NULL)) > MYSQL_YYABORT; > } > - | expr comp_op predicate %prec '=' > + | boolean_test comp_op predicate %prec '=' > { > $$= (*$2)(0)->create(thd, $1, $3); > if (unlikely($$ == NULL)) > MYSQL_YYABORT; > } > - | expr comp_op all_or_any '(' subselect ')' %prec '=' > + | boolean_test comp_op all_or_any '(' subselect ')' %prec '=' > { > $$= all_any_subquery_creator(thd, $1, $2, $3, $5); > if (unlikely($$ == NULL)) > MYSQL_YYABORT; > } > - | predicate > + | predicate %prec BETWEEN_SYM > ; > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 4d9f8a3c31e: MDEV-28669 addendum: additional tests and mtr changes
by Sergei Golubchik 27 Dec '22

27 Dec '22

Hi, Julius, On Dec 27, Julius Goryavsky wrote: > commit 4d9f8a3c31e > Author: Julius Goryavsky <julius.goryavsky(a)mariadb.com> > Date: Mon Dec 19 10:29:55 2022 +0100 > > diff --git a/mysql-test/mysql-test-run.pl b/mysql-test/mysql-test-run.pl > index 12afab4d28c..8df8198fc99 100755 > --- a/mysql-test/mysql-test-run.pl > +++ b/mysql-test/mysql-test-run.pl > @@ -313,7 +313,7 @@ my %mysqld_logs; > my $opt_debug_sync_timeout= 300; # Default timeout for WAIT_FOR actions. > my $warn_seconds = 60; > > -my $rebootstrap_re= '--innodb[-_](?:page[-_]size|checksum[-_]algorithm|undo[-_]tablespaces|log[-_]group[-_]home[-_]dir|data[-_]home[-_]dir)|data[-_]file[-_]path|force_rebootstrap'; > +my $rebootstrap_re= '--(?:loose[-_])?(?:innodb[-_](?:page[-_]size|file(?:[-_]format|per[-_]table)|compression[-_](?:default|algorithm)|checksum(?:s|[-_]algorithm)|undo[-_](?:directory|tablespaces)|(?:data|log[-_]group)[-_]home[-_]dir|buffer[-_]pool[-_]filename|data[-_]file[-_]path|default[-_](?:encryption[-_]key[-_]id|page[-_]encryption)|sys[-_]|(?:index|table)[-_]stats)|aria[-_]log[-_](?:dir[-_]path|file[-_]size))|force_rebootstrap'; 1. drop force_rebootstrap please 2. this code was already not very readable, but got fairly ridiculous now. Let's change the approach. E.g. put them all in a hash, like my %rebootstrap_check = map { $_ => 1 } qw( innodb-page-size innodb-file-format ... ); you'll need to normalize an option before looking it up in the hash > > sub testcase_timeout ($) { return $opt_testcase_timeout * 60; } > sub check_timeout ($) { return testcase_timeout($_[0]); } > @@ -2762,7 +2762,29 @@ sub mysql_server_start($) { > return; > } > > - my $datadir= $mysqld->value('datadir'); > + my $extra_opts= get_extra_opts($mysqld, $tinfo); > + > + # The test options can contain the --datadir parameter, but > + # the bootstrap function can only read datadir location from > + # a .cnf file. So we need to parse the options to get the > + # actual location of the data directory, considering both > + # the options and the .cnf file, and then store the resulting > + # value in a "$mysqld" hash - to use later, when we need to > + # know the actual location of the data directory: > + my $datadir=""; > + foreach my $opt ( @$extra_opts ) > + { > + if ($opt =~ /^--datadir=/) { > + $datadir = substr($opt, 10); > + last; strictly speaking, `last` is incorrect, there can be more than one --datadir > + } > + } > + # If datadir is not in the options, then read it from .cnf: > + if (! $datadir) { > + $datadir = $mysqld->value('datadir'); > + } > + $mysqld->{'datadir'} = $datadir; > + > if (not $opt_start_dirty) > { > > @@ -2785,17 +2807,59 @@ sub mysql_server_start($) { > } > } > > + # Run <tname>-master.sh > + if ($mysqld->option('#!run-master-sh') and > + defined $tinfo->{master_sh} and > + run_system('/bin/sh ' . $tinfo->{master_sh}) ) > + { > + $tinfo->{'comment'}= "Failed to execute '$tinfo->{master_sh}'"; > + return 1; > + } > + > + # Run <tname>-slave.sh > + if ($mysqld->option('#!run-slave-sh') and > + defined $tinfo->{slave_sh} and > + run_system('/bin/sh ' . $tinfo->{slave_sh})) > + { > + $tinfo->{'comment'}= "Failed to execute '$tinfo->{slave_sh}'"; > + return 1; > + } > + > my $mysqld_basedir= $mysqld->value('basedir'); > - my $extra_opts= get_extra_opts($mysqld, $tinfo); > > if ( $basedir eq $mysqld_basedir ) > { > if (! $opt_start_dirty) # If dirty, keep possibly grown system db > { > + # Find the name of the current section in the configuration > + # file and its suffix: > + my $section = $mysqld->{name}; > + my $server_name; > + my $suffix = ""; > + if (index($section, '.') != -1) { > + ($server_name, $suffix) = $section =~ /^\s*([^\s.]+)\s*\.\s*([^\s]+)/; > + } > + else { > + $server_name = $section =~ /^\s*([^\s]+)/; > + } Replace all 11 lines above with: my ($suffix) = ($mysqld->{name} =~ /\.(.*)/); > # Some InnoDB options are incompatible with the default bootstrap. > # If they are used, re-bootstrap > my @rebootstrap_opts; > @rebootstrap_opts = grep {/$rebootstrap_re/o} @$extra_opts if $extra_opts; > + # Let's store the additional bootstrap options in > + # the environment variable - they may be used later > + # in the mtr tests - for re-bootstrap or run the > + # recovery, etc: > + my $extra_text = "--datadir=$datadir"; that's rather random. Why do you add only this one option, that doesn't make sense. Better handle any additional requirements in the test. > + if (@rebootstrap_opts) { why do you need an if() ? > + $extra_text .= ' '.join(' ', @rebootstrap_opts); > + } > + if ($suffix) { > + $ENV{'MTR_BOOTSTRAP_OPTS_'.$suffix} = $extra_text; > + } > + else { > + $ENV{'MTR_BOOTSTRAP_OPTS'} = $extra_text; Is that even possible? > + } > if (@rebootstrap_opts) > { > mtr_verbose("Re-bootstrap with @rebootstrap_opts"); > diff --git a/mysql-test/suite/galera/disabled.def b/mysql-test/suite/galera/disabled.def > index 84babda2fa0..18e7c074059 100644 > --- a/mysql-test/suite/galera/disabled.def > +++ b/mysql-test/suite/galera/disabled.def > @@ -28,3 +28,4 @@ query_cache: MDEV-15805 Test failure on galera.query_cache > versioning_trx_id: MDEV-18590: galera.versioning_trx_id: Test failure: mysqltest: Result content mismatch > galera_wsrep_provider_unset_set: wsrep_provider is read-only for security reasons > pxc-421: wsrep_provider is read-only for security reasons > +galera_sst_rsync_innodb_nest : MDEV-29591 nesting innodb subdirectories in datadir causes SST to fail Why do you disable it if the whole commit is about fixing the test? How can I see that the test works? > diff --git a/mysql-test/suite/galera/r/galera_sst_rsync_innodb__innodb_dir,debug.rdiff b/mysql-test/suite/galera/r/galera_sst_rsync_innodb__innodb_dir,debug.rdiff > new file mode 100644 > index 00000000000..027f4860a39 > --- /dev/null > +++ b/mysql-test/suite/galera/r/galera_sst_rsync_innodb__innodb_dir,debug.rdiff > @@ -0,0 +1,114 @@ looks like a wrong file name? > +--- galera_sst_rsync_innodb_nest.result > ++++ galera_sst_rsync_innodb_nest.reject > +@@ -286,3 +286,111 @@ > + DROP TABLE t1; > + COMMIT; Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] f984f49fba7: MDEV-30203 - deb fix piuparts
by Sergei Golubchik 21 Dec '22

21 Dec '22

Hi, Daniel, This is a review of `git diff 12786f0e779f 5f76981eb649`, that is, for commits: 5f76981eb64 rpm: packages that dont match component name 25eaa0aebfa MDEV-30205: [Cmtr: suite look in mariadb-test path 0981201f027 mtr - wsrep use mariadb executables eba0b35035a mtr: maradb names for *embedded and mariadb-client-test 974ca88f0fe deb: fix autotest - exclude for MDEV-30205 b19d75408bb Deb: Breaks/Replaces/COnflicts c60f0341950 MDEV-30203 deb breaks/replaces on compat packages f984f49fba7 MDEV-30203 - deb fix piuparts 310d025ff0d MDEV-30205: fix deb autopkgtest 89f6cf47ee2 rpm: fix packaging of compat/non-compat scripts links 494ed76d15a deb: mysqlhotcopy to mariadb-client-compat d03075f3a1f deb: server-core mysql* -> server-compat 9453812193d wsrep_sst_mariabackup to use mariadb-backup (and message accordingly) a4f843a7a5a Fix uncollected symlink eb9a469bf9a Add package recommended dependencies c135be62274 Remove duplicate entry 84c7cd29c61 Fix Debian packaging for mariabackup 5dbc7ae8bb4 MDEV-30203: compat packages - RPM remove mariabackup link eb4fa5d750f MDEV-30203: debian - make compat packages d4be0b9be73 MDEV-30203: Create mysql symlink packages (RPM) - scripts 441879fd493 MDEV-30203: Create mysql symlink packages (RPM) On Dec 19, Daniel Black wrote: > diff --git a/cmake/cpack_rpm.cmake b/cmake/cpack_rpm.cmake > index 339363b2169..7b48fd64f4c 100644 > --- a/cmake/cpack_rpm.cmake > +++ b/cmake/cpack_rpm.cmake > @@ -85,6 +85,35 @@ SET(CPACK_RPM_server_PACKAGE_DESCRIPTION "${CPACK_RPM_PACKAGE_DESCRIPTION}") > SET(CPACK_RPM_test_PACKAGE_SUMMARY "MariaDB database regression test suite") > SET(CPACK_RPM_test_PACKAGE_DESCRIPTION "${CPACK_RPM_PACKAGE_DESCRIPTION}") > > +# "set/append array" - append a set of strings, separated by a space > +MACRO(SETA var) > + FOREACH(v ${ARGN}) > + SET(${var} "${${var}} ${v}") > + ENDFOREACH() > +ENDMACRO(SETA) > + > +FOREACH(SYM_COMPONENT Server Client Test) > + SET(SYM ${SYM_COMPONENT}_symlinks) > + string(TOLOWER ${SYM} SYM) * why all cmake commands are uppercase, but `string` is lowercase? * SET is redundant, you can do STRING(TOLOWER ${SYM_COMPONENT}_symlinks SYM) * may be, better to use ${SYM_COMPONENT}-symlinks ? Like in MariaDB-server-symlinks ? > + SET(SYMCOMP ${SYM_COMPONENT}Symlinks) > + string(TOUPPER ${SYMCOMP} SYMCOMP_UPPER) > + SET(CPACK_COMPONENT_${SYMCOMP_UPPER}_GROUP "${SYM}") > + SET(CPACK_COMPONENTS_ALL "${CPACK_COMPONENTS_ALL}" "${SYMCOMP}") > + SET(CPACK_RPM_${SYM}_PACKAGE_SUMMARY "MySQL compatible symlinks for MariaDB database ${SYM_COMPONENT} binaries/scripts") > + SET(CPACK_RPM_${SYM}_PACKAGE_DESCRIPTION "${CPACK_RPM_PACKAGE_DESCRIPTION}") > + SET(CPACK_RPM_${SYM}_PACKAGE_ARCHITECTURE "noarch") > + SETA(CPACK_RPM_${SYM_COMPONENT}_PACKAGE_RECOMMENDS "MariaDB-${SYM}") You forgot to say that symlink package DEPENDS on the non-symlink package > +ENDFOREACH() > + > +# TODO change to 11.0 on rebase > +SETA(CPACK_RPM_client_symlinks_PACKAGE_CONFLICTS > + "MariaDB-client < 10.11.2" > + "MariaDB-server < 10.11.2") > +SETA(CPACK_RPM_server_symlinks_PACKAGE_CONFLICTS > + "MariaDB-server < 10.11.2") > +SETA(CPACK_RPM_test_symlinks_PACKAGE_CONFLICTS > + "MariaDB-test < 10.11.2") > + > # libmariadb3 > SET(CPACK_RPM_shared_PACKAGE_SUMMARY "LGPL MariaDB database client library") > SET(CPACK_RPM_shared_PACKAGE_DESCRIPTION "This is LGPL MariaDB client library that can be used to connect to MySQL > diff --git a/cmake/symlinks.cmake b/cmake/symlinks.cmake > index 3f3b4e4a9b5..a4eb6c2b42c 100644 > --- a/cmake/symlinks.cmake > +++ b/cmake/symlinks.cmake > @@ -12,7 +12,6 @@ endmacro() > REGISTER_SYMLINK("mariadb" "mysql") > REGISTER_SYMLINK("mariadb-access" "mysqlaccess") > REGISTER_SYMLINK("mariadb-admin" "mysqladmin") > -REGISTER_SYMLINK("mariadb-backup" "mariabackup") better not. there're scripts that might be using it. This is why I suggested `strncmp(my_progname, "mariadb", 7)` and not `strncmp(my_progname, "maria", 5)` in your PR > REGISTER_SYMLINK("mariadb-binlog" "mysqlbinlog") > REGISTER_SYMLINK("mariadb-check" "mysqlcheck") > REGISTER_SYMLINK("mariadb-client-test-embedded" "mysql_client_test_embedded") > diff --git a/debian/control b/debian/control > index 81418c13656..6b225ca4b3d 100644 > --- a/debian/control > +++ b/debian/control > @@ -548,7 +548,8 @@ Provides: default-mysql-client, > virtual-mysql-client > Recommends: libdbd-mariadb-perl | libdbd-mysql-perl, > libdbi-perl, > - libterm-readkey-perl > + libterm-readkey-perl, > + mariadb-client-compat Please, no. deb and rpm packages are different enough already. don't use different suffixes for them. Choose either -symlink or -compat (or something else) and use it consistently both in deb and in rpm. > Description: MariaDB database client binaries > MariaDB is a fast, stable and true multi-user, multi-threaded SQL database > server. SQL (Structured Query Language) is the most popular database query > @@ -558,6 +559,110 @@ Description: MariaDB database client binaries > This package includes the client binaries and the additional tools > innotop and mariadb-report (mysqlreport). > > +Package: mariadb-client-compat > +Architecture: all > +Depends: mariadb-client > +Multi-Arch: foreign > +Description: MySQL compatibility links to mariadb-client binaries/scripts. > +Conflicts: mariadb-client (<< ${source:Version}), > + mariadb-client-10.0, I wonder, would it work, if you won't set any Conflicts/Breaks, but only set Depends? It'd be a lot simpler that way (if it'll work). > + mariadb-client-10.1, > + mariadb-client-10.2, > + mariadb-client-10.3, > + mariadb-client-10.4, > + mariadb-client-10.5, > + mariadb-client-10.6, > + mariadb-client-10.7, > + mariadb-client-10.8, > + mariadb-client-5.1, > + mariadb-client-5.2, > + mariadb-client-5.3, > + mariadb-client-5.5, > + mariadb-client-core (<< ${source:Version}), > + mariadb-client-core-10.0, > + mariadb-client-core-10.1, > + mariadb-client-core-10.2, > + mariadb-client-core-10.3, > + mariadb-client-core-10.4, > + mariadb-client-core-10.5, > + mariadb-client-core-10.6, > + mariadb-client-core-10.7, > + mariadb-client-core-10.8, > + mariadb-server (<< ${source:Version}), > + mariadb-server-10.0, > + mariadb-server-10.1, > + mariadb-server-10.2, > + mariadb-server-10.3, > + mariadb-server-10.4, > + mariadb-server-10.5, > + mariadb-server-10.6, > + mariadb-server-10.7, > + mariadb-server-10.8, > + mysql-client (<< 5.0.51), > + mysql-client-5.0, > + mysql-client-5.1, > + mysql-client-5.5, > + mysql-client-5.6, > + mysql-client-5.7, > + mysql-client-8.0, > + mysql-client-core-5.0, > + mysql-client-core-5.1, > + mysql-client-core-5.5, > + mysql-client-core-5.6, > + mysql-client-core-5.7, > + mysql-client-core-8.0, > + mysql-server-core-5.7, > + mysql-server-core-8.0, > + percona-server-server-5.6, > + percona-server-server, > + percona-xtradb-cluster-server-5.6, > + percona-xtradb-cluster-server-5.7, > + percona-xtradb-cluster-server > +Breaks: mariadb-client (<< ${source:Version}), > + mariadb-client-10.0, > + mariadb-client-10.1, > + mariadb-client-10.2, > + mariadb-client-10.3, > + mariadb-client-10.4, > + mariadb-client-10.5, > + mariadb-client-10.6, > + mariadb-client-10.7, > + mariadb-client-10.8, > + mariadb-client-5.1, > + mariadb-client-core (<< ${source:Version}), > + mariadb-client-core-10.0, > + mariadb-client-core-10.1, > + mariadb-client-core-10.2, > + mariadb-client-core-10.3, > + mariadb-client-core-10.4, > + mariadb-client-core-10.5, > + mariadb-client-core-10.6, > + mariadb-client-core-10.7, > + mariadb-client-core-10.8, > + mariadb-server (<< ${source:Version}), > + mariadb-server-10.0, > + mariadb-server-10.1, > + mariadb-server-10.2, > + mariadb-server-10.3, > + mariadb-server-10.4, > + mariadb-server-10.5, > + mariadb-server-10.6, > + mariadb-server-10.7, > + mariadb-server-10.8, > + mysql-server-5.5, > + mysql-server-5.6, > + mysql-server-5.7, > + mysql-server-8.0, > + mysql-server-core-5.5, > + mysql-server-core-5.6, > + mysql-server-core-5.7, > + mysql-server-core-8.0, > + percona-server-server-5.6, > + percona-server-server, > + percona-xtradb-cluster-server-5.6, > + percona-xtradb-cluster-server-5.7, > + percona-xtradb-cluster-server > + > Package: mariadb-server-core > Architecture: any > Depends: mariadb-common (>= ${source:Version}), > diff --git a/debian/mariadb-backup.install b/debian/mariadb-backup.install > index e450f8f46a0..a288377ddd0 100644 > --- a/debian/mariadb-backup.install > +++ b/debian/mariadb-backup.install > @@ -1,6 +1,4 @@ > -usr/bin/mariabackup same here. needs mariadb-backup-compat > usr/bin/mariadb-backup > usr/bin/mbstream > -usr/share/man/man1/mariabackup.1 > usr/share/man/man1/mariadb-backup.1 > usr/share/man/man1/mbstream.1 > diff --git a/debian/mariadb-client.links b/debian/mariadb-client.links > index 62e3651daf5..8ccce12c609 100644 > --- a/debian/mariadb-client.links > +++ b/debian/mariadb-client.links > @@ -2,11 +2,6 @@ usr/bin/mariadb-check usr/bin/mariadb-analyze > usr/bin/mariadb-check usr/bin/mariadb-optimize > usr/bin/mariadb-check usr/bin/mariadb-repair > usr/bin/mariadb-check usr/bin/mariadbcheck > -usr/bin/mariadb-check usr/bin/mysqlanalyze > -usr/bin/mariadb-check usr/bin/mysqlcheck > -usr/bin/mariadb-check usr/bin/mysqloptimize > -usr/bin/mariadb-check usr/bin/mysqlrepair > -usr/bin/mariadb-report usr/bin/mysqlreport > usr/share/man/man1/mariadb-check.1.gz usr/share/man/man1/mariadb-analyze.1.gz > usr/share/man/man1/mariadb-check.1.gz usr/share/man/man1/mariadb-optimize.1.gz > usr/share/man/man1/mariadb-check.1.gz usr/share/man/man1/mariadb-repair.1.gz you forgot to move symlinks to man pages > diff --git a/debian/mariadb-server-compat.install b/debian/mariadb-server-compat.install > new file mode 100644 > index 00000000000..36701d8e0c6 > --- /dev/null > +++ b/debian/mariadb-server-compat.install > @@ -0,0 +1,12 @@ > +usr/bin/mysqld_multi > +usr/bin/mysqld_safe > +usr/bin/mysqld_safe_helper > +usr/bin/mysql_install_db > +usr/bin/mysql_upgrade > +usr/sbin/mysqld > +usr/share/man/man1/mysqld_multi.1 > +usr/share/man/man1/mysqld_safe.1 > +usr/share/man/man1/mysqld_safe_helper.1 great. do you do the same for rpms? I didn't see where manpages are moved to -symlinks packages in rpms. > +usr/share/man/man1/mysql_install_db.1 > +usr/share/man/man1/mysql_upgrade.1 > +usr/share/man/man8/mysqld.8 > diff --git a/debian/mariadb-test-compat.install b/debian/mariadb-test-compat.install > new file mode 100644 > index 00000000000..7b498d7d1c8 > --- /dev/null > +++ b/debian/mariadb-test-compat.install > @@ -0,0 +1 @@ > +usr/share/mysql/mysql-test/mysql-test-run.pl You forgot mysql-test-run, mysqltest, mysql_client_test, and their *embedded variants. But, you know, it's probably not worth a new package at all. It's the *test* package (not interesting for users, never used in production, so no immutability requirements). I suggest to keep those symlinks in the mariadb-test package. Your other PR will add a warning. And eventually we drop them. Optional/Recommended/Suggested - sounds like too much toubles for a *test* package. > diff --git a/mysql-test/lib/mtr_cases.pm b/mysql-test/lib/mtr_cases.pm > index fe202279ac7..ad1ac6fdf88 100644 > --- a/mysql-test/lib/mtr_cases.pm > +++ b/mysql-test/lib/mtr_cases.pm > @@ -387,14 +387,14 @@ sub collect_suite_name($$) > else > { > my @dirs = my_find_dir(dirname($::glob_mysql_test_dir), > - ["mysql-test/suite", @plugin_suitedirs ], > + ["mariadb-test/suite", "mysql-test/suite", @plugin_suitedirs ], Eh, no. That is, yes, but not in this set of commits, not in this MDEV > $suitename); > # > # if $suitename contained wildcards, we'll have many suites and > # their overlays here. Let's group them appropriately. > # > for (@dirs) { > - m@^.*/(?:mysql-test/suite|$plugin_suitedir_regex)/(.*)$@o or confess $_; > + m@^.*/(?:mariadb-test/suite|mysql-test/suite|$plugin_suitedir_regex)/(.*)$@o or confess $_; > push @{$suites{$1}}, $_; > } > } Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 3

Re: [Maria-developers] MDEV-14443 test cases only
by Sergei Golubchik 19 Dec '22

19 Dec '22

Hi, Vicențiu, This is a review of _test cases only_. Mainly, to discuss the behavior. Random thoughts here and comments inline in the diff. There are two models for DENY that I can think of: "a hole in grants" and "a mask". The difference lies in where DENY is applied. In the "a hole in grants" model one has a certain set of grants (may be with "holes"), a role has another set of grants, and when one does SET ROLE, one gets a union of those grants. Like, one has SELECT on db1.* and DENY to db1.t1, and some role has SELECT on db1.t1 - by setting this role a user will have full SELECT on db1.* While in the "mask" model, denies are separate entities, when setting a role a union of denies is subtracted from the union of grants. In the example above a user would still be denied access to db1.t1. I personally see the "hole" model as more logical and easier to understand. It also allows to "patch" holes with roles, what the "mask" model cannot do. The "mask" model needs getting used to, the fact that after SET ROLE one can end up having less access than before. It's not clear who can DENY what and to whom. And DENY is very difficult to work around, esp. as one can DENY IGNORE DENIES. But it allows to DENY something to PUBLIC, and the "hole" model cannot really do that. And I think it'd be a popular use case. Ideally, I'd prefer to have some kind of a compromise model that's as simple to understand as a "hole", but allows to deny to public. Unfortunately, I don't see how to do that. You've implemented the "mask" behavior. Okay. It solves an important use case. More comments below: > diff --git a/mysql-test/suite/deny/columns.result b/mysql-test/suite/deny/columns.result > --- /dev/null > +++ b/mysql-test/suite/deny/columns.result > @@ -0,0 +1,126 @@ > +create user foo; > +create database deny_db; > +create table deny_db.t1 (a int, b int, secret int); > +create table deny_db.t2 (a2 int, b2 int, secret2 int); > +insert into deny_db.t2 values (100, 200, 300); > +grant all on *.* to foo; > +revoke ignore denies on *.* from foo; > +grant all on deny_db.* to foo; > +grant all on deny_db.t1 to foo; > +grant all on deny_db.t2 to foo; > +grant select (a, b, secret) on deny_db.t1 to foo; > +grant insert (a, b, secret) on deny_db.t1 to foo; > +grant update (a, b, secret) on deny_db.t1 to foo; > +grant references (a, b, secret) on deny_db.t1 to foo; > +grant select (a2, b2, secret2) on deny_db.t2 to foo; > +grant insert (a2, b2, secret2) on deny_db.t2 to foo; > +grant update (a2, b2, secret2) on deny_db.t2 to foo; > +grant references (a2, b2, secret2) on deny_db.t2 to foo; this is quite confusing, tons of redundant grants on different levels, it makes it very unclear of what causes what. > +# > +# Test select and insert ... select commands. > +# > +deny select (secret) on deny_db.t1 to foo; > +connect con1,localhost,foo,,; > +select * from deny_db.t1; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for column 'secret' in table 't1' case in point. Normally, this should've been SELECT command denied to user 'foo'@'localhost' for table 't1' like in create database db; create table db.t1 (a int, b int, secret int); insert db.t1 values (1,2,100); create user foo@localhost; grant select (a,b) on db.t1 to foo@localhost; connect foo,localhost,foo; select a,b from db.t1; --error ER_TABLEACCESS_DENIED_ERROR select * from db.t1; --error ER_COLUMNACCESS_DENIED_ERROR select a,b,secret from db.t1; that is `select *` is ER_TABLEACCESS_DENIED_ERROR, and if I mention the column by name, then it's ER_COLUMNACCESS_DENIED_ERROR But you have so many redundant overlapping grants, that I really cannot see what error is correct in your test here. > +select secret from deny_db.t1; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for column 'secret' in table 't1' > +# > +# These are different code paths. Check both. > +# > +insert into deny_db.t1 (a, b, secret) values (1, 2, 3); > +insert into deny_db.t1 values (1, 2, 3); > +# > +# Check INSERT SELECT as we need to account for 2 different kinds of > +# privileges. > +# > +insert into deny_db.t1 select * from deny_db.t1; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for column 'secret' in table 't1' > +insert into deny_db.t1 select * from deny_db.t2; > +show full columns from deny_db.t1; > +Field Type Collation Null Key Default Extra Privileges Comment > +a int(11) NULL YES NULL select,insert,update,references > +b int(11) NULL YES NULL select,insert,update,references > +secret int(11) NULL YES NULL insert,update,references do you have a test where a denied column isn't shown by SHOW COLUMNS? that is, where you don't have other privileges on it? > +show full columns from deny_db.t2; > +Field Type Collation Null Key Default Extra Privileges Comment > +a2 int(11) NULL YES NULL select,insert,update,references > +b2 int(11) NULL YES NULL select,insert,update,references > +secret2 int(11) NULL YES NULL select,insert,update,references > +disconnect con1; > +connection default; > +revoke deny select (secret) on deny_db.t1 from foo; > +deny select(secret2) on deny_db.t2 to foo; > +connect con1,localhost,foo,,; > +insert into deny_db.t1 (a, b, secret) values (1, 2, 3); > +insert into deny_db.t1 values (1, 2, 3); > +insert into deny_db.t1 select * from deny_db.t1; > +insert into deny_db.t1 select * from deny_db.t2; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for column 'secret2' in table 't2' > +show full columns from deny_db.t1; > +Field Type Collation Null Key Default Extra Privileges Comment > +a int(11) NULL YES NULL select,insert,update,references > +b int(11) NULL YES NULL select,insert,update,references > +secret int(11) NULL YES NULL select,insert,update,references > +show full columns from deny_db.t2; > +Field Type Collation Null Key Default Extra Privileges Comment > +a2 int(11) NULL YES NULL select,insert,update,references > +b2 int(11) NULL YES NULL select,insert,update,references > +secret2 int(11) NULL YES NULL insert,update,references > +disconnect con1; > +# > +# Test insert commands > +# > +connection default; > +deny insert (secret) on deny_db.t1 to foo; > +connect con1,localhost,foo,,; > +insert into deny_db.t1 (a, b, secret) values (1, 2, 3); > +ERROR 42000: INSERT command denied to user 'foo'@'localhost' for column 'secret' in table 't1' > +insert into deny_db.t1 values (1, 2, 3); > +ERROR 42000: INSERT command denied to user 'foo'@'localhost' for column 'secret' in table 't1' didn't test this one, but likely should behave as select? that is, ER_TABLEACCESS_DENIED_ERROR please, test with GRANT (without DENY) before making any changes > +insert into deny_db.t1 select * from deny_db.t1; > +ERROR 42000: INSERT command denied to user 'foo'@'localhost' for column 'secret' in table 't1' > +insert into deny_db.t1 select * from deny_db.t2; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for column 'secret2' in table 't2' > +show full columns from deny_db.t1; > +Field Type Collation Null Key Default Extra Privileges Comment > +a int(11) NULL YES NULL select,insert,update,references > +b int(11) NULL YES NULL select,insert,update,references > +secret int(11) NULL YES NULL select,update,references > +disconnect con1; > +# > +# Test update commands. > +# > +connection default; > +deny select (secret) on deny_db.t1 to foo; > +deny select (secret2) on deny_db.t2 to foo; > +connect con1, localhost,foo,,; > +update deny_db.t1 set a=111, b=222 where secret=10; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for column 'secret' in table 't1' > +update deny_db.t1 set a=123 where a = (select secret2 from deny_db.t2); > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for column 'secret2' in table 't2' > +show full columns from deny_db.t1; > +Field Type Collation Null Key Default Extra Privileges Comment > +a int(11) NULL YES NULL select,insert,update,references > +b int(11) NULL YES NULL select,insert,update,references > +secret int(11) NULL YES NULL update,references > +show full columns from deny_db.t2; > +Field Type Collation Null Key Default Extra Privileges Comment > +a2 int(11) NULL YES NULL select,insert,update,references > +b2 int(11) NULL YES NULL select,insert,update,references > +secret2 int(11) NULL YES NULL insert,update,references > +disconnect con1; > +connection default; > +deny update (secret) on deny_db.t1 to foo; > +connect con1, localhost,foo,,; > +update deny_db.t1 set a=111, b=222; > +update deny_db.t1 set secret=333, a=111; > +ERROR 42000: UPDATE command denied to user 'foo'@'localhost' for column 'secret' in table 't1' > +update deny_db.t1 set secret=333; > +ERROR 42000: UPDATE command denied to user 'foo'@'localhost' for column 'secret' in table 't1' > +update deny_db.t1 set a=123, secret=333; > +ERROR 42000: UPDATE command denied to user 'foo'@'localhost' for column 'secret' in table 't1' > +disconnect con1; > +connection default; > +drop user foo; > +drop database deny_db; > diff --git a/mysql-test/suite/deny/columns_role.result b/mysql-test/suite/deny/columns_role.result > --- /dev/null > +++ b/mysql-test/suite/deny/columns_role.result > @@ -0,0 +1,57 @@ > +create database some_db; > +create view some_db.v1 as (select 1 as col); > +create user foo; > +create role bar; > +grant select on *.* to foo; > +grant show view on *.* to foo; > +grant bar to foo; > +grant select(col) on some_db.v1 to foo; > +revoke ignore denies on *.* from foo; why foo has ignore denies here? I don't see where it was granted. > +deny select (col) on some_db.v1 to bar; > +connect con1,localhost,foo,,; > +describe some_db.v1; > +Field Type Null Key Default Extra > +col int(1) NO 0 > +set role bar; > +describe some_db.v1; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for table 'v1' this is very questionable. Should SET ROLE ever restrict what one can do? > +show create view some_db.v1; > +View Create View character_set_client collation_connection > +v1 CREATE ALGORITHM=UNDEFINED DEFINER=`root`@`localhost` SQL SECURITY DEFINER VIEW `some_db`.`v1` AS (select 1 AS `col`) latin1 latin1_swedish_ci > +disconnect con1; > +connection default; > +drop user foo; > +drop role bar; > +drop database some_db; > +# > +# Test if only role level denies are active. > +# > +create database some_db; > +create view some_db.v1 as (select 1 as col); > +create user foo; > +create role bar; > +grant bar to foo; > +grant select(col) on some_db.v1 to foo; > +deny select (col) on some_db.v1 to bar; > +connect con1,localhost,foo,,; > +# > +# Due to select(col) privilege on v1 we can see some_db here. > +# > +show databases; > +Database > +information_schema > +some_db > +test > +set role bar; > +# > +# Now the role's denies take effect and we can no longer see some_db. > +# > +show databases; > +Database > +information_schema > +test > +disconnect con1; > +connection default; > +drop user foo; > +drop role bar; > +drop database some_db; > diff --git a/mysql-test/suite/deny/deny_datastructures.result b/mysql-test/suite/deny/deny_datastructures.result > --- /dev/null > +++ b/mysql-test/suite/deny/deny_datastructures.result > @@ -0,0 +1,109 @@ > +create user foo; > +create role bar; > +# > +# Test deny format for global privileges. > +# > +deny select on *.* to foo; > +deny insert on *.* to bar; > +select user, host, JSON_EXTRACT(priv, '$.deny') from mysql.global_priv where user = 'foo'; > +user host JSON_EXTRACT(priv, '$.deny') > +foo % {"global": 1, "version_id": VERSION_ID} > +select user, host, JSON_EXTRACT(priv, '$.deny') from mysql.global_priv where user = 'bar'; > +user host JSON_EXTRACT(priv, '$.deny') > +bar {"global": 2, "version_id": VERSION_ID} > +flush privileges; > +deny select on some_db.* to foo; > +deny show view on some_db.* to foo; > +deny insert on some_other_db.* to foo; > +deny insert on some_other_db.* to bar; > +deny select on some_other_db.some_table to foo; > +deny insert on some_other_db.some_table_second to foo; > +deny update on some_other_db.some_table_third to foo; > +deny select(a, b) on some_other_db.some_table_third to foo; > +deny select, insert(a, b, c) on some_other_db.some_table_fourth to foo; > +select user, host, JSON_DETAILED(JSON_EXTRACT(priv, '$.deny')) from mysql.global_priv where user = 'foo'; > +user host JSON_DETAILED(JSON_EXTRACT(priv, '$.deny')) > +foo % { > + "global": 1, > + "db": > + [ > + > + { > + "name": "`some_db`", > + "access": 4194305 > + }, > + > + { > + "name": "`some_other_db`", > + "access": 2 > + } > + ], > + "table": > + [ > + > + { > + "name": "`some_other_db`.`some_table_second`", you sure you want that? you'd need to parse the name back when reading privileges, why would you need this extra complexity? I'd suggest something like "name" : [ "some_other_db", "some_table_second" ] (one component for db grants, three components for column grants) This way you don't need to parse the identifier to split on dots and remove backticks. You can even use [] for global grants and not split them into global/db/table/column lists, but have simply [ { "name ": [], "access": 1 }, { "name": ["some_db"], "access": 4194305 }, { "name": ["some_other_db"], "access": 2 } { "name": ["some_other_db", "some_table_second"], "access": 2 }, { "name": ["some_other_db", "some_table"], "access": 1 }, { "name": ["some_other_db", "some_table_third"], "access": 4 }, { "name": ["some_other_db", "some_table_fourth"], "access": 1 } { "name": ["some_other_db", "some_table_third", "a"], "access": 1 }, { "name": ["some_other_db", "some_table_fourth", "b"], "access": 2 }, { "name": ["some_other_db", "some_table_fourth", "c"], "access": 2 }, { "name": ["some_other_db", "some_table_third", "b"], "access": 1 }, ] May be it's easier if they're split into global/db/table/column, but there's defenitely no reason to put all identifier chain in one json property, quoted according to SQL rules. > + "access": 2 > + }, > + > + { > + "name": "`some_other_db`.`some_table`", > + "access": 1 > + }, > + > + { > + "name": "`some_other_db`.`some_table_third`", > + "access": 4 > + }, > + > + { > + "name": "`some_other_db`.`some_table_fourth`", > + "access": 1 > + } > + ], > + "column": > + [ > + > + { > + "name": "`some_other_db`.`some_table_third`.`a`", > + "access": 1 > + }, > + > + { > + "name": "`some_other_db`.`some_table_fourth`.`b`", > + "access": 2 > + }, > + > + { > + "name": "`some_other_db`.`some_table_fourth`.`c`", > + "access": 2 > + }, > + > + { > + "name": "`some_other_db`.`some_table_third`.`b`", > + "access": 1 > + }, > + > + { > + "name": "`some_other_db`.`some_table_fourth`.`a`", > + "access": 2 > + } > + ], > + "version_id": VERSION_ID > +} > +select user, host, JSON_DETAILED(JSON_EXTRACT(priv, '$.deny')) from mysql.global_priv where user = 'bar'; > +user host JSON_DETAILED(JSON_EXTRACT(priv, '$.deny')) > +bar { > + "global": 2, > + "db": > + [ > + > + { > + "name": "`some_other_db`", > + "access": 2 > + } > + ], > + "version_id": VERSION_ID > +} > +drop user foo; > +drop role bar; > diff --git a/mysql-test/suite/deny/dml.test b/mysql-test/suite/deny/dml.test > --- /dev/null > +++ b/mysql-test/suite/deny/dml.test > @@ -0,0 +1,241 @@ would be good to have a comment at the top of every test file you add to mention the 1) MDEV and 2) the purpose for that file to exist. E.g. in this case I wouldn't need to waste time guessing how dml.test is different from columns.test, if both have column-level grants. And even the content looks suspiciously the same. > +--source include/not_embedded.inc > + > +create user foo; > + > +create database deny_db; > +create table deny_db.t1 (a int, b int, secret int); > +insert into deny_db.t1 values (1, 2, 3); > + > +create table deny_db.t2 (a int, b int, secret int); > +insert into deny_db.t2 values (10, 20, 30); > + > +grant select (a) on deny_db.t1 to foo; > +grant insert (a) on deny_db.t1 to foo; > +grant update (a) on deny_db.t1 to foo; > +grant delete on deny_db.t1 to foo; > + > +--connect (con1,localhost,foo,,) > +insert into deny_db.t1 (a) values (1), (2), (3); > +disconnect con1; > + > +connection default; > +deny insert on deny_db.* to foo; > + > +--connect (con1,localhost,foo,,) > +--error ER_TABLEACCESS_DENIED_ERROR > +insert into deny_db.t1 (a) values (1), (2), (3); > +update deny_db.t1 set a=a*10; > +delete from deny_db.t1 where a = 20; > +disconnect con1; > + > +connection default; > +select * from deny_db.t1 order by a; > +truncate deny_db.t1; > + > +revoke deny insert on deny_db.* from foo; > + > + > +--connect (con1,localhost,foo,,) > +insert into deny_db.t1 (a) values (1), (2), (3); > +--echo # > +--echo # Test that revoking insert deny allows inserting only into allowed > +--echo # to insert columns. > +--echo # > +insert into deny_db.t1 (a) values (1), (2), (3); > + > +--error ER_COLUMNACCESS_DENIED_ERROR > +insert into deny_db.t1 (a,b) values (1,10), (2, 20), (3, 30); > +disconnect con1; > + > +connection default; > +--echo # > +--echo # Test delete/update command denies. > +--echo # > +deny select on deny_db.* to foo; > + > +--connect (con1,localhost,foo,,) > +--echo # > +--echo # Test not allowed to delete / update because of lack of select > +--echo # rights. > +--echo # > + > +show grants; > +--error ER_COLUMNACCESS_DENIED_ERROR > +update deny_db.t1 set a=a*10 where a = 1; Did you verify that in all cases the error is the same as if the privilege simply wasn't granted in the first place? In this case the error is correct, I've checked. > + > +--error ER_COLUMNACCESS_DENIED_ERROR > +delete from deny_db.t1 where a = 20; > +disconnect con1; > + > + > +connection default; > +--echo # > +--echo # Now deny update and delete directly, but keep select. > +--echo # > +revoke deny select on deny_db.* from foo; > +deny delete, update on deny_db.* to foo; > + > +--connect (con1,localhost,foo,,) > +--echo # > +--echo # Test not allowed to delete / update because of lack of delete/update > +--echo # rights. > +--echo # > + > +--error ER_TABLEACCESS_DENIED_ERROR > +update deny_db.t1 set a=a*10; > + > +--error ER_TABLEACCESS_DENIED_ERROR > +delete from deny_db.t1 where a = 20; > +disconnect con1; > + > +--echo # > +--echo # Test masking table level grants. > +--echo # > + > +connection default; > +grant select on deny_db.t1 to foo; > +grant insert on deny_db.t1 to foo; > +grant update on deny_db.t1 to foo; > +grant delete on deny_db.t1 to foo; > + > +--connect (con1,localhost,foo,,) > +insert into deny_db.t1 (a,b) values (1,10), (2, 20), (3, 30); > +disconnect con1; > + > +connection default; > +deny insert on deny_db.* to foo; > + > +--connect (con1,localhost,foo,,) > +--error ER_TABLEACCESS_DENIED_ERROR > +insert into deny_db.t1 (a,b) values (1,10), (2, 20), (3, 30); > + > +--error ER_TABLEACCESS_DENIED_ERROR > +update deny_db.t1 set a=a*10; > + > +--error ER_TABLEACCESS_DENIED_ERROR > +delete from deny_db.t1 where a = 20; > + > +disconnect con1; > + > +connection default; > +revoke deny update, delete on deny_db.* from foo; > + > +--connect (con1,localhost,foo,,) > +--error ER_TABLEACCESS_DENIED_ERROR > +insert into deny_db.t1 (a,b) values (1,10), (2, 20), (3, 30); > + > +update deny_db.t1 set a=a*10; > +delete from deny_db.t1 where a = 20; > +disconnect con1; > + > +--echo # > +--echo # Test masking database & global level grants. > +--echo # > +connection default; > +grant select, insert, update, delete on *.* to foo; > +grant select, insert, update, delete on deny_db.* to foo; > +grant select, insert, update, delete on deny_db.t1 to foo; > +deny select, insert on deny_db.* to foo; > + > +--connect (con1,localhost,foo,,) > +--error ER_TABLEACCESS_DENIED_ERROR > +insert into deny_db.t1 (a,b) values (1,10), (2, 20), (3, 30); > + > +--error ER_COLUMNACCESS_DENIED_ERROR > +update deny_db.t1 set a=a*10; > + > +--error ER_COLUMNACCESS_DENIED_ERROR > +delete from deny_db.t1 where a = 20; > +disconnect con1; > + > +connection default; > +revoke deny select on deny_db.* from foo; > +deny update, delete on deny_db.* to foo; > + > +--connect (con1,localhost,foo,,) > +--error ER_TABLEACCESS_DENIED_ERROR > +insert into deny_db.t1 (a,b) values (1,10), (2, 20), (3, 30); > + > +--error ER_TABLEACCESS_DENIED_ERROR > +update deny_db.t1 set a=a*10; > + > +--error ER_TABLEACCESS_DENIED_ERROR > +delete from deny_db.t1 where a = 20; > +disconnect con1; > + > +connection default; > +deny select on deny_db.* to foo; > + > +--connect (con1,localhost,foo,,) > +--error ER_TABLEACCESS_DENIED_ERROR > +insert into deny_db.t1 (a,b) values (1,10), (2, 20), (3, 30); > + > +--error ER_TABLEACCESS_DENIED_ERROR > +update deny_db.t1 set a=a*10; > + > +--error ER_TABLEACCESS_DENIED_ERROR > +delete from deny_db.t1 where a = 20; > +disconnect con1; > + > +--echo # > +--echo # Test revoking all denies. > +--echo # > +connection default; > +revoke deny select, insert, update, delete on deny_db.* from foo; by the way, what does REVOKE ALL PRIVILEGES do, does it remove denies? please, add a test. > +truncate deny_db.t1; > + > +--connect (con1,localhost,foo,,) > +insert into deny_db.t1 (a,b) values (1,10), (2, 20), (3, 30); > +update deny_db.t1 set a=a*10; > +select * from deny_db.t1; > +delete from deny_db.t1 where a = 20; > +disconnect con1; > + > +connection default; > + > +--echo # > +--echo # Test masking via table level denies > +--echo # > + > +deny select on deny_db.t1 to foo; > + > +--connect (con1,localhost,foo,,) > +insert into deny_db.t1 (a,b) values (1,10), (2, 20), (3, 30); > +--error ER_COLUMNACCESS_DENIED_ERROR > +update deny_db.t1 set a=a*10; > +--error ER_COLUMNACCESS_DENIED_ERROR > +update deny_db.t1 set a=a*10 where a = 10; > +--echo # > +--echo # Delete works without select. > +--echo # > +delete from deny_db.t1; > +--error ER_COLUMNACCESS_DENIED_ERROR > +delete from deny_db.t1 where a = 20; > + > +--error ER_TABLEACCESS_DENIED_ERROR > +select * from deny_db.t1; > +disconnect con1; > + > +connection default; > +revoke deny select on deny_db.t1 from foo; > +deny insert on deny_db.t1 to foo; > +deny update on deny_db.t1 to foo; > +deny delete on deny_db.t1 to foo; > + > +--connect (con1,localhost,foo,,) > +--error ER_TABLEACCESS_DENIED_ERROR > +insert into deny_db.t1 (a,b) values (1,10), (2, 20), (3, 30); > +--error ER_TABLEACCESS_DENIED_ERROR > +update deny_db.t1 set a=a*10; > +--error ER_TABLEACCESS_DENIED_ERROR > +update deny_db.t1 set a=a*10 where a = 10; > +--error ER_TABLEACCESS_DENIED_ERROR > +delete from deny_db.t1; > +--error ER_TABLEACCESS_DENIED_ERROR > +delete from deny_db.t1 where a = 20; > + > +select * from deny_db.t1; > + > +connection default; > +drop database deny_db; > +drop user foo; > diff --git a/mysql-test/suite/deny/global.result b/mysql-test/suite/deny/global.result > --- /dev/null > +++ b/mysql-test/suite/deny/global.result > @@ -0,0 +1,327 @@ > +create user foo; > +create database deny_db; > +create table deny_db.t1 (a int, b int, secret int); > +insert into deny_db.t1 values (1, 2, 3); > +create table deny_db.t2 (a int, b int, secret int); > +insert into deny_db.t2 values (10, 20, 30); > +# > +# Test global denies. > +# > +show databases; > +Database > +deny_db > +information_schema > +mtr > +mysql > +performance_schema > +sys > +test > +grant select (secret) on deny_db.t1 to foo; > +connect con1,localhost,foo,,; > +select * from deny_db2.t1; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for table 't1' > +show databases; > +Database > +deny_db > +information_schema > +test > +use deny_db; > +# > +# It is still possible to leak column names without denies. > +# > +select * from deny_db.t1; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for column 'a' in table 't1' this is a bug. See my test above, it's ER_TABLEACCESS_DENIED_ERROR. Apparently, it returns ER_TABLEACCESS_DENIED_ERROR if there's at least one granted column before the non-granted column. That is, if you'd have create table deny_db.t1 (secret int, a int, b int); you'd get ER_TABLEACCESS_DENIED_ERROR. please fix it (where applicable, 10.3, perhaps?) > +select a from deny_db.t1; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for column 'a' in table 't1' > +select b from deny_db.t1; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for column 'b' in table 't1' > +select table_name, column_name from information_schema.columns > +where table_schema = 'deny_db'; > +table_name column_name > +t1 secret > +# > +# Invalid field error is returned if table exists. > +# > +select c from deny_db.t1; > +ERROR 42S22: Unknown column 'c' in 'field list' I feel like the whole column-leaking behavior is a bug too. This, likely, should be ER_COLUMNACCESS_DENIED_ERROR too > +# > +# Table access denied if table does exist but is not granted. > +# > +select c from deny_db.t2; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for table 't2' > +# > +# Same error if table does not exist. > +# > +select c from deny_db.t_not_exists; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for table 't_not_exists' > +select secret from deny_db.t1; > +secret > +3 > +# > +# Now apply deny. > +# > +connection default; > +deny select on *.* to foo; I see you reconnect to apply denies. What are the rules here? With grants, table/column grants work right away, while global and current-db grants are cached in the thd. but table/column denies aren't stored in separate tables, that's why the question. What are the rules here? > +disconnect con1; > +connect con1,localhost,foo,,; > +show databases; > +Database > +information_schema > +test > +use information_schema; > +use deny_db; > +ERROR 42000: Access denied for user 'foo'@'%' to database 'deny_db' > +select * from deny_db.t1; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for table 't1' > +select c from deny_db.t1; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for table 't1' > +select c from deny_db.t2; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for table 't2' > +select c from deny_db.t_not_exists; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for table 't_not_exists' > +select secret from deny_db.t1; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for table 't1' > +# > +# Grant table access then test deny. > +# > +connection default; > +grant select on deny_db.t1 to foo; > +connection con1; > +show databases; > +Database > +information_schema > +test > +use information_schema; > +use deny_db; > +ERROR 42000: Access denied for user 'foo'@'%' to database 'deny_db' > +select * from deny_db.t1; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for table 't1' > +select c from deny_db.t1; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for table 't1' > +select c from deny_db.t2; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for table 't2' > +select c from deny_db.t_not_exists; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for table 't_not_exists' > +select secret from deny_db.t1; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for table 't1' > +# > +# Grant database access then test deny. > +# > +connection default; > +grant select on deny_db.* to foo; > +connection con1; > +show databases; > +Database > +information_schema > +test > +use information_schema; > +use deny_db; > +ERROR 42000: Access denied for user 'foo'@'%' to database 'deny_db' > +select * from deny_db.t1; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for table 't1' > +select c from deny_db.t1; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for table 't1' > +select c from deny_db.t2; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for table 't2' > +select c from deny_db.t_not_exists; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for table 't_not_exists' > +select secret from deny_db.t1; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for table 't1' > +# > +# Grant global access then test deny. > +# > +connection default; > +grant select on *.* to foo; > +connection con1; > +show databases; > +Database > +information_schema > +test > +use information_schema; > +use deny_db; > +ERROR 42000: Access denied for user 'foo'@'%' to database 'deny_db' > +select * from deny_db.t1; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for table 't1' > +select c from deny_db.t1; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for table 't1' > +select c from deny_db.t2; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for table 't2' > +select c from deny_db.t_not_exists; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for table 't_not_exists' > +select secret from deny_db.t1; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for table 't1' > +connection default; > +# > +# Test underlying data loading. > +# > +select user, host, JSON_EXTRACT(priv, '$.deny') from mysql.global_priv where user = 'foo'; > +user host JSON_EXTRACT(priv, '$.deny') > +foo % {"global": 1, "version_id": 101100} > +show grants for foo; > +Grants for foo@% > +GRANT SELECT ON *.* TO `foo`@`%` > +DENY SELECT ON *.* TO `foo`@`%` > +GRANT SELECT ON `deny_db`.* TO `foo`@`%` > +GRANT SELECT, SELECT (secret) ON `deny_db`.`t1` TO `foo`@`%` meaning, the user will see what is denied to him? I expect it'll be a questionable item, some users will want denies to be not shown. Although I don't see how they cannot be. > +flush privileges; > +show grants for foo; > +Grants for foo@% > +GRANT SELECT ON *.* TO `foo`@`%` > +DENY SELECT ON *.* TO `foo`@`%` > +GRANT SELECT ON `deny_db`.* TO `foo`@`%` > +GRANT SELECT, SELECT (secret) ON `deny_db`.`t1` TO `foo`@`%` > +connection con1; > +show databases; > +Database > +information_schema > +test > +use information_schema; > +use deny_db; > +ERROR 42000: Access denied for user 'foo'@'%' to database 'deny_db' > +select * from deny_db.t1; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for table 't1' > +select c from deny_db.t1; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for table 't1' > +select c from deny_db.t2; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for table 't2' > +select c from deny_db.t_not_exists; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for table 't_not_exists' > +select secret from deny_db.t1; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for table 't1' > +disconnect con1; > +connection default; > +# > +# Now test insert not being denied. > +# > +grant insert(secret) on deny_db.t1 to foo; > +connect con1,localhost,foo,,; > +show databases; > +Database > +deny_db > +information_schema > +test > +insert into deny_db.t1(secret) values (10000); > +disconnect con1; > +connection default; > +drop database deny_db; > +drop user foo; > +# > +# Test denying all rights globally. please, add tests with DENY TO PUBLIC > +# User should still be able to connect and see information_schema > +# *at least*. > +# > +create user foo; > +create database some_db; > +grant select on *.* to foo; > +show grants for foo; > +Grants for foo@% > +GRANT SELECT ON *.* TO `foo`@`%` > +deny select on *.* to foo; > +show grants for foo; > +Grants for foo@% > +GRANT SELECT ON *.* TO `foo`@`%` > +DENY SELECT ON *.* TO `foo`@`%` > +deny all on *.* to foo; > +show grants for foo; > +Grants for foo@% > +GRANT SELECT ON *.* TO `foo`@`%` > +DENY ALL PRIVILEGES ON *.* TO `foo`@`%` > +connect con1,localhost,foo,,information_schema; > +show databases; > +Database > +information_schema > +disconnect con1; > +connection default; > +drop user foo; > +drop database some_db; > +############################################## > +# Test SELECT command interacting with deny. # > +############################################## > +create user foo; > +create user bar; > +create database some_db; > +create table some_db.t1 (a int, secret int); > +insert into some_db.t1 values (1, 100); > +use some_db; > +create view v1 as (select a from t1); > +create view v2 as (select secret from t1); > +connect con1,localhost,foo,,; > +select table_name, table_type from information_schema.tables where table_schema like 'some_db'; > +table_name table_type > +disconnect con1; > +connection default; > +grant select on *.* to foo; > +grant select on some_db.* to foo; > +grant select on some_db.t1 to foo; > +grant select(a) on some_db.t1 to foo; > +# > +# See what foo sees before denies. > +# > +connect con1,localhost,foo,,; > +select table_name, table_type from information_schema.tables where table_schema like 'some_db'; > +table_name table_type > +t1 BASE TABLE > +v1 VIEW > +v2 VIEW > +disconnect con1; > +connection default; > +show grants for foo; > +Grants for foo@% > +GRANT SELECT ON *.* TO `foo`@`%` > +GRANT SELECT ON `some_db`.* TO `foo`@`%` > +GRANT SELECT, SELECT (a) ON `some_db`.`t1` TO `foo`@`%` > +deny select on *.* to foo; > +show grants for foo; > +Grants for foo@% > +GRANT SELECT ON *.* TO `foo`@`%` > +DENY SELECT ON *.* TO `foo`@`%` > +GRANT SELECT ON `some_db`.* TO `foo`@`%` > +GRANT SELECT, SELECT (a) ON `some_db`.`t1` TO `foo`@`%` > +connect con1,localhost,foo,,; > +show grants; > +Grants for foo@% > +GRANT SELECT ON *.* TO `foo`@`%` > +DENY SELECT ON *.* TO `foo`@`%` > +GRANT SELECT ON `some_db`.* TO `foo`@`%` > +GRANT SELECT, SELECT (a) ON `some_db`.`t1` TO `foo`@`%` > +use some_db; > +ERROR 42000: Access denied for user 'foo'@'%' to database 'some_db' > +select * from some_db.t1; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for table 't1' > +select * from some_db.v1; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for table 'v1' > +select * from some_db.v2; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for table 'v2' > +select table_name, table_type from information_schema.tables where table_schema like "some_db"; > +table_name table_type > +select table_name, column_name from information_schema.columns where table_schema like "some_db"; > +table_name column_name > +disconnect con1; > +connection default; > +grant insert(a) on some_db.t1 to bar; > +grant insert(a) on some_db.t1 to foo; > +connect con1,localhost,foo,,; > +select table_name, column_name from information_schema.columns where table_schema like "some_db"; > +table_name column_name > +t1 a > +disconnect con1; > +connect con2,localhost,bar,,; > +select table_name, column_name from information_schema.columns where table_schema like "some_db"; > +table_name column_name > +t1 a > +disconnect con2; > +connection default; > +deny insert on *.* to foo; > +deny insert on *.* to bar; > +connect con1,localhost,foo,,; > +select table_name, column_name from information_schema.columns where table_schema like "some_db"; > +table_name column_name > +disconnect con1; > +connect con2,localhost,bar,,; > +select table_name, column_name from information_schema.columns where table_schema like "some_db"; > +table_name column_name > +disconnect con2; > +connection default; > +drop user foo; > +drop user bar; > +drop database some_db; > diff --git a/mysql-test/suite/deny/ignore_denies.result b/mysql-test/suite/deny/ignore_denies.result > --- /dev/null > +++ b/mysql-test/suite/deny/ignore_denies.result > @@ -0,0 +1,76 @@ > +create user foo; > +create role bar; > +grant select, ignore denies on *.* to foo; > +deny select on *.* to foo; > +deny select on mysql.* to foo; > +deny select on mysql.global_priv to foo; > +deny select(user) on mysql.global_priv to foo; > +grant bar to foo; > +grant ignore denies on *.* to bar; > +connect con1,localhost,foo,,; > +# > +# The user should still be able to access mysql.global_priv due > +# to the IGNORE DENIES privilege being active. > +# > +select JSON_EXTRACT(priv, '$.deny') from mysql.global_priv where user = 'foo'; > +JSON_EXTRACT(priv, '$.deny') > +{"global": 1, "db": [{"name": "`mysql`", "access": 1}], "table": [{"name": "`mysql`.`global_priv`", "access": 1}], "column": [{"name": "`mysql`.`global_priv`.`user`", "access": 1}], "version_id": VERSION_ID} > +disconnect con1; > +connection default; > +# > +# Now test inheriting IGNORE DENIES from a role. > +# > +revoke ignore denies on *.* from foo; > +connect con1,localhost,foo,,; > +select JSON_EXTRACT(priv, '$.deny') from mysql.global_priv where user = 'foo'; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for table 'global_priv' > +# > +# Bar provides foo with IGNORE DENIES privilege. > +# > +set role bar; > +select JSON_EXTRACT(priv, '$.deny') from mysql.global_priv where user = 'foo'; > +JSON_EXTRACT(priv, '$.deny') > +{"global": 1, "db": [{"name": "`mysql`", "access": 1}], "table": [{"name": "`mysql`.`global_priv`", "access": 1}], "column": [{"name": "`mysql`.`global_priv`.`user`", "access": 1}], "version_id": VERSION_ID} > +disconnect con1; > +connection default; > +# > +# Denying IGNORE DENIES, overrides a user's ability to ignore denies. This is very questionable. One can DENY IGNORE DENIES TO PUBLIC and basically lock the server down for everyone, there will be no way to override that. (this is an observation, not ready to suggest a solution atm) > +# > +deny ignore denies on *.* to foo; > +grant ignore denies on *.* to foo; > +connect con1,localhost,foo,,; > +# > +# Foo can't ignore the SELECT priv deny. > +# > +select JSON_EXTRACT(priv, '$.deny') from mysql.global_priv where user = 'foo'; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for table 'global_priv' > +# > +# And neither if bar role provides the ignore deny priv. > +# > +set role bar; > +select JSON_EXTRACT(priv, '$.deny') from mysql.global_priv where user = 'foo'; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for table 'global_priv' > +disconnect con1; > +connection default; > +# > +# Now test what happens when IGNORE DENIES is denied from a role. > +# > +revoke deny ignore denies on *.* from foo; > +deny ignore denies on *.* to bar; > +connect con1,localhost,foo,,; > +# > +# Foo can ignore the SELECT priv deny when Bar isn't activated > +# > +select JSON_EXTRACT(priv, '$.deny') from mysql.global_priv where user = 'foo'; > +JSON_EXTRACT(priv, '$.deny') > +{"global": 1, "db": [{"name": "`mysql`", "access": 1}], "table": [{"name": "`mysql`.`global_priv`", "access": 1}], "column": [{"name": "`mysql`.`global_priv`.`user`", "access": 1}], "version_id": VERSION_ID} > +# > +# But when Bar is active, denies can not be ignored any more. > +# > +set role bar; > +select JSON_EXTRACT(priv, '$.deny') from mysql.global_priv where user = 'foo'; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for table 'global_priv' > +disconnect con1; > +connection default; > +drop user foo; > +drop role bar; > diff --git a/mysql-test/suite/deny/show_generic.result b/mysql-test/suite/deny/show_generic.result > --- /dev/null > +++ b/mysql-test/suite/deny/show_generic.result > @@ -0,0 +1,197 @@ > +# > +# This test checks that denies take part in SHOW commands properly. > +# > +create user foo; > +create database some_db; > +create table some_db.t1 (id int); > +connect con1,localhost,foo,,; > +show tables from some_db; > +ERROR 42000: Access denied for user 'foo'@'%' to database 'some_db' > +disconnect con1; > +connection default; > +grant insert on some_db.t1 to foo; > +connect con1,localhost,foo,,; > +show tables from some_db; > +Tables_in_some_db > +t1 > +disconnect con1; > +connection default; > +deny insert on some_db.* to foo; > +connect con1,localhost,foo,,; > +show databases; > +Database > +information_schema > +test > +show tables from some_db; > +ERROR 42000: Access denied for user 'foo'@'%' to database 'some_db' > +disconnect con1; > +connection default; > +# > +# Test that dropping a user does not give access to connections that > +# previously had elements denied. > +# > +# We will use two users, foo which will receive denies, bar which will > +# not. This will showcase the difference in behaviour. > +# > +create user bar; > +# > +# This global grant is cached in sctx->master_access on connection. > +# > +grant select on *.* to foo; > +grant select on *.* to bar; > +connect con1,localhost,foo,,; > +show tables from some_db; > +Tables_in_some_db > +t1 > +disconnect con1; > +connection default; > +# > +# Here we add a mask for some_db, it should no longer be visible to foo. > +# > +deny select on some_db.* to foo; > +connect con1,localhost,foo,,; > +use some_db; > +ERROR 42000: Access denied for user 'foo'@'%' to database 'some_db' > +show tables from some_db; > +ERROR 42000: Access denied for user 'foo'@'%' to database 'some_db' > +# > +# bar user has access to the database (no denies present) > +# > +connect con2,localhost,bar,,; > +use some_db; > +show tables from some_db; > +Tables_in_some_db > +t1 > +# > +# Do not disconnect foo & bar, but drop the users. > +# > +connection default; > +drop user foo; > +drop user bar; > +# > +# Our current running connection, because it featured denies previously > +# now doesn't have access. > +# > +connection con1; > +use some_db; > +ERROR 42000: Access denied for user 'foo'@'%' to database 'some_db' > +show tables from some_db; > +ERROR 42000: Access denied for user 'foo'@'%' to database 'some_db' > +disconnect con1; > +# > +# However, bar does not have any denies active. In this case we can keep > +# the previous MariaDB behaviour of using the "cache" within > +# sctx->master_access to grant access. This comment is confusing. As far as I can see both foo and bar preserve the behavior after `drop user`. Which is fine and what I was expected anyway. But comments imply there's some difference in behavior between them? > +# > +connection con2; > +use some_db; > +show tables from some_db; > +Tables_in_some_db > +t1 > +disconnect con2; > +connection default; > +drop database some_db; > +# > +# Test table level denies interacting with show tables > +# > +create database some_db; > +create user foo; > +create table some_db.t1 (a int, b int); > +create table some_db.t2 (a int, b int); > +grant select on *.* to foo; > +deny select on some_db.t1 to foo; > +deny insert on some_db.t2 to foo; > +select user, host, JSON_EXTRACT(priv, '$.deny') from mysql.global_priv > +where user = 'foo'; > +user host JSON_EXTRACT(priv, '$.deny') > +foo % {"table": [{"name": "`some_db`.`t2`", "access": 2}, {"name": "`some_db`.`t1`", "access": 1}], "version_id": VERSION_ID} > +connect con1,localhost,foo,,; > +show tables from some_db; > +Tables_in_some_db > +t2 > +show columns from some_db.t1; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for table 't1' > +show columns from some_db.t2; > +Field Type Null Key Default Extra > +a int(11) YES NULL > +b int(11) YES NULL > +select table_name, column_name, privileges from information_schema.columns > +where table_schema like 'some_db' order by table_name, column_name; > +table_name column_name privileges > +t2 a select > +t2 b select > +disconnect con1; > +connection default; > +grant insert on some_db.* to foo; > +connect con1,localhost,foo,,; > +show tables from some_db; > +Tables_in_some_db > +t1 > +t2 > +# > +# See MDEV-28783, this should not error out when global/db grants exist > +# (except for SELECT priv). > +# > +show columns from some_db.t1; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for table 't1' > +show columns from some_db.t2; > +Field Type Null Key Default Extra > +a int(11) YES NULL > +b int(11) YES NULL > +select table_name, column_name, privileges from information_schema.columns > +where table_schema like 'some_db' order by table_name, column_name; > +table_name column_name privileges > +t1 a insert > +t1 b insert > +t2 a select > +t2 b select > +disconnect con1; > +connection default; > +deny insert on some_db.t1 to foo; > +deny select on some_db.t2 to foo; > +connect con1,localhost,foo,,; > +# > +# some_db should still be visible, but it should show up as empty. > +# > +show databases; > +Database > +information_schema > +mtr > +mysql > +performance_schema > +some_db > +sys > +test > +show tables from some_db; > +Tables_in_some_db > +show columns from some_db.t1; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for table 't1' > +show columns from some_db.t2; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for table 't2' > +select table_name, column_name, privileges from information_schema.columns > +where table_schema like 'some_db' order by table_name, column_name; > +table_name column_name privileges > +disconnect con1; > +connection default; > +grant select(a) on some_db.t1 to foo; > +grant update(a) on some_db.t1 to foo; > +connect con1,localhost,foo,,; > +# > +# Update privilege on the column is not masked, only see a column. > +# > +show tables from some_db; > +Tables_in_some_db > +t1 > +show columns from some_db.t1; > +Field Type Null Key Default Extra > +a int(11) YES NULL > +show columns from some_db.t2; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for table 't2' > +select table_name, column_name, privileges from information_schema.columns > +where table_schema like 'some_db' order by table_name, column_name; > +table_name column_name privileges > +t1 a update > +disconnect con1; > +connection default; > +drop user foo; > +drop database some_db; Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

[Maria-developers] Review input for MDEV-21092, 21095, 29997
by Sergei Petrunia 08 Dec '22

08 Dec '22

Hi Rex, Please find below review input for the collection of patches for these MDEVs. First, please try to have each piece of functionality in its own commit. You can use "git rebase -i" to make your last commits be one commit, as well as re-order them. Then, you can do a carefully considered "git push --force" to overwrite the contents of your branch with the new set of commits. in prune_partitions(): + String parts; + String_list parts_list; + + make_used_partitions_str(thd->mem_root, prune_param.part_info, &parts, parts_list ); + trace_partition_pruning.add("partition_prune", parts.ptr()); This creates a list of used partitions regardless whether we need it or not (and typically trace=off, so we don't'). Please put this code into if (unlikely(thd->trace_started())) { ... } Consider this testcase: create table t2 (a int, b int) partition by hash(a) partitions 10; create table t3 (a int, b int) partition by hash(a) partitions 10; set optimizer_trace=1; explain format=json select * from t2,t3 where t2.a in (2,3,4) and t3.a in (4,5); select * from information_schema.optimizer_trace\G This gives: ... { "partition_prune": "p2,p3,p4" }, { "partition_prune": "p4,p5" }, ... which makes it impossible to see which table is pruning is done for. Let's make the trace be: { "prune_partitions" : { "table": "t2", "used_partitions": "p2,p3,p4" } } { "prune_partitions" : { "table": "t3", "used_partitions": "p4,p5" } } . in item_subselect.cc: + { + OPT_TRACE_TRANSFORM(thd, trace_wrapper, trace_transform, + in_subs->get_select_lex()->select_number, + "EXISTS (SELECT)", "IN (SELECT)"); + trace_transform.add( "upper_not", ( upper_not?true:false ) ); + } coding style: everything inside { } should be indented one level (=2 spaces) right. in sql_select.cc: "attached_conditions_summary": [ { "table": "t1", "attached": "t1.c < 30", "index": "t1.a < 10 and t1.b < 20" } Please change "index" to "index_condition". and "attached" to "attached_condition". Let's not print index_condition if it is null. (Yes, this is somewhat inconsistent with printing "attached":null) in item.cc: + #include "my_json_writer.h" + const char *dbug_print_optrace( ) dbug_print_optrace() is fine. I think it needs a comment like this: /* Debug printout: print the trace trace we're currently writing if any, and return it. */ (Feel free to fix poor English if necessary) Please fix the coding style in the function body as discussed on Slack. Please move #include to the other #includes at the top of the file. Can add a comment, "// For dbug_print_optrace" Would dbug_print_opt_trace() be a better name? + const char *dbug_print_items(Item *item) + { + *big_buf= 0; + + dbug_add_print_items( item ); + return big_buf; + } This function doesn't seem to be needed? For the rest of the functions: as discussed on Slack, we can add them in a separate commit if we decide that they're useful. Could you provide a very short text describing: The entry point (which function is to be called from gdb?). The use case. Is it "to print List<Item>" ? I don't recall seeing this data structure in the server... Is there a lot of need to print Item_cond and then print its children? This looks self-repetitive... I'd like to see a log of an example gdb session...

2 2

[Maria-developers] How to get started with contribution.
by Pinak faldu 29 Nov '22

29 Nov '22

Respected sir/madam, I am Pinak Faldu, a computer science undergraduate student. Currently, I am in my first year of a master's degree at Sinhgad College. I am new to open source contributions, but I am very familiar with ReactJS, MongoDB, MySql, Javascript, and C++. I would like to contribute to your organization, but I don't know where to begin. Could you please tell me how to get started? Hoping to hear from you soon. Regards Pinak

2 1

Re: [Maria-developers] 52f489ebccb: MDEV-29069 follow-up: support partially suitable keys
by Sergei Golubchik 02 Nov '22

02 Nov '22

Hi, Nikita, This review applies to the combined diff e2f8dff^..52f489e On Oct 04, Nikita Malyavin wrote: > diff --git a/sql/log_event.h b/sql/log_event.h > index 91a9eeee10f..8452e665c98 100644 > --- a/sql/log_event.h > +++ b/sql/log_event.h > @@ -5275,8 +5278,11 @@ class Rows_log_event : public Log_event > uchar *m_key; /* Buffer to keep key value during searches */ > KEY *m_key_info; /* Pointer to KEY info for m_key_nr */ > uint m_key_nr; /* Key number */ > + uint m_key_parts_suit; /* A number of key_parts suited to lookup */ that's very confusing name. Took me a while to understand what you meant. Better call it m_usable_key_parts > bool master_had_triggers; /* set after tables opening */ > > + uint key_parts_suit_event(const KEY *key) const; and this could be "get_usable_key_parts()" > + > int find_key(); // Find a best key to use in find_row() > int find_row(rpl_group_info *); > int write_row(rpl_group_info *, const bool); > diff --git a/sql/log_event_server.cc b/sql/log_event_server.cc > index 422d496d5c3..25705d13641 100644 > --- a/sql/log_event_server.cc > +++ b/sql/log_event_server.cc > @@ -6062,7 +6065,7 @@ int Rows_log_event::do_apply_event(rpl_group_info *rgi) > */ > sql_mode_t saved_sql_mode= thd->variables.sql_mode; > if (!is_auto_inc_in_extra_columns()) > - thd->variables.sql_mode= (rpl_data.is_online_alter() ? saved_sql_mode :0) > + thd->variables.sql_mode= (m_online_alter ? saved_sql_mode :0) wouldn't it be more correct to use saved_sql_mode also in normal replication? > | MODE_NO_AUTO_VALUE_ON_ZERO; > > // row processing loop > @@ -7949,14 +7952,35 @@ uint8 Write_rows_log_event::get_trg_event_map() > **************************************************************************/ > > #if defined(HAVE_REPLICATION) > -/* > - Compares table->record[0] and table->record[1] > +/** > + @brief Compares table->record[0] and table->record[1] > + > + @param masrter_columns a number of columns on the source replica, > + 0 if ONLINE ALTER TABLE > > - Returns TRUE if different. > + @returns true if different. > */ > -static bool record_compare(TABLE *table) > +static bool record_compare(TABLE *table, uint master_columns) > { > - bool result= FALSE; > + bool result= false; > + > + /* > + Determine if the optimized check is possible (and we can > + goto record_compare_exit). > + We should ensure that all extra columns (i.e. fieldnr > master_columns) > + have explicit values. If not, we will exclude them from comparison, > + as they can contain non-deterministic values. > + > + master_columns == 0 case is optimization for ONLINE ALTER to check > + all columns fast. > + */ > + bool all_values_set= master_columns == 0 > + && bitmap_is_set_all(&table->has_value_set); > + for (uint i = master_columns; all_values_set && i < table->s->fields; i++) > + { > + all_values_set= bitmap_is_set(&table->has_value_set, i); > + } wait, this doesn't make any sense. You only do the loop if all_values_set is true. But it is true only if bitmap_is_set_all(&table->has_value_set). There is no point in doing bitmap_is_set() for individual bits, when you already know that all bits are set :) shouldn't it simply be bool all_values_set= bitmap_is_set_all(&table->has_value_set); ? but see below, perhaps you don't need all_values_set at all > + > /** > Compare full record only if: > - there are no blob fields (otherwise we would also need > @@ -7969,40 +7993,43 @@ static bool record_compare(TABLE *table) > */ > if ((table->s->blob_fields + > table->s->varchar_fields + > - table->s->null_fields) == 0) > + table->s->null_fields) == 0 > + && all_values_set) > { > result= cmp_record(table,record[1]); > goto record_compare_exit; > } > > /* Compare null bits */ > - if (memcmp(table->null_flags, > - table->null_flags+table->s->rec_buff_length, > - table->s->null_bytes)) > + if (all_values_set && memcmp(table->null_flags, > + table->null_flags+table->s->rec_buff_length, > + table->s->null_bytes)) > { > - result= TRUE; // Diff in NULL value > + result= true; // Diff in NULL value > goto record_compare_exit; > } > > /* Compare fields */ > for (Field **ptr=table->field ; *ptr ; ptr++) > { > + Field *f= *ptr; > if (table->versioned() && (*ptr)->vers_sys_field()) > { > continue; > } > - /** > - We only compare field contents that are not null. > - NULL fields (i.e., their null bits) were compared > - earlier. > - */ > - if (!(*(ptr))->is_null()) > + /* > + We only compare fields that exist on the source (or in ONLINE > + ALTER case, that were in the original table). For reference, > + replica tables can also contain extra fields. > + */ > + if (f->field_index > master_columns && !f->has_explicit_value()) > + continue; same question as in a previous review. Do you mean that when f->field_index < master_columns, it's ok to compare values below even when !f->has_explicit_value() ? > + > + if (f->is_null() != f->is_null(table->s->rec_buff_length) > + || f->cmp_binary_offset(table->s->rec_buff_length)) may be it's better to create a fast copy of this function, like if (bitmap_is_set_all(&table->has_value_set)) { ... old record_compare() code starting from if ((table->s->blob_fields + table->s->varchar_fields + table->s->null_fields) == 0) } else { ... your code, only the loop over fields. } in this case you won't even need all_values_set or master_columns > { > - if ((*ptr)->cmp_binary_offset(table->s->rec_buff_length)) > - { > - result= TRUE; > - goto record_compare_exit; > - } > + result= true; > + goto record_compare_exit; > } > } > > @@ -8010,6 +8037,32 @@ static bool record_compare(TABLE *table) > return result; > } > > +/** > + Newly added fields with non-deterministic defaults (i.e. DEFAULT(RANDOM()), > + CURRENT_TIMESTAMP, AUTO_INCREMENT) should be excluded from key search. > + Basically we exclude all the default-filled fields based on > + has_explicit_value bitmap. > + */ > +uint Rows_log_event::key_parts_suit_event(const KEY *key) const > +{ > + uint master_columns= m_online_alter ? 0 : m_cols.n_bits; get_master_cols() > + uint p= 0; > + for (;p < key->ext_key_parts; p++) > + { > + Field *f= key->key_part[p].field; > + uint non_deterministic_default= f->default_value && > + f->default_value->flags | VCOL_NOT_STRICTLY_DETERMINISTIC; > + > + int next_number_field= f->table->next_number_field ? > + f->table->next_number_field->field_index : -1; eh? 1. why it's *in the loop* ? 2. why not simply || f == f->table->next_number_field or, to be extra safe || f->table->field[f->field_index] == f->table->next_number_field > + > + if (f->field_index >= master_columns I don't think this condition ^^^ is needed. Same as elsewhere, if f->field_index < master_columns but !bitmap_is_set, does it mean the keyseg is ok? I don't think so. > + && !bitmap_is_set(&m_table->has_value_set, f->field_index) > + && (non_deterministic_default || next_number_field == f->field_index)) > + break; > + } > + return p; > +} > > /** > Find the best key to use when locating the row in @c find_row(). > @@ -8024,13 +8077,26 @@ static bool record_compare(TABLE *table) > */ > int Rows_log_event::find_key() > { > - uint i, best_key_nr, last_part; > + uint i, best_key_nr; > KEY *key, *UNINIT_VAR(best_key); > ulong UNINIT_VAR(best_rec_per_key), tmp; > DBUG_ENTER("Rows_log_event::find_key"); > DBUG_ASSERT(m_table); > > + #ifdef DBUG_TRACE #ifndef DBUG_OFF > + auto _ = make_scope_exit([this]() { > + DBUG_EXECUTE_IF("rpl_report_chosen_key", > + push_warning_printf(m_table->in_use, > + Sql_condition::WARN_LEVEL_NOTE, > + ER_UNKNOWN_ERROR, "Key chosen: %d", > + m_key_nr == MAX_KEY ? > + -1 : m_key_nr);); > + }); > + #endif > + > + m_key_nr= MAX_KEY; > best_key_nr= MAX_KEY; > + uint parts_suit= 0; > > /* > Keys are sorted so that any primary key is first, followed by unique keys, > @@ -8041,33 +8107,38 @@ int Rows_log_event::find_key() > { > if (!m_table->s->keys_in_use.is_set(i)) > continue; > + parts_suit= key_parts_suit_event(key); > + if (!parts_suit) > + continue; > /* > We cannot use a unique key with NULL-able columns to uniquely identify > a row (but we can still select it for range scan below if nothing better > is available). > */ > - if ((key->flags & (HA_NOSAME | HA_NULL_PART_KEY)) == HA_NOSAME) > + if ((key->flags & (HA_NOSAME | HA_NULL_PART_KEY)) == HA_NOSAME && > + parts_suit == key->user_defined_key_parts) this doesn't make use of ext key parts. Should be something like if (parts_suit == key->ext_key_parts || (parts_suit >= key->user_defined_key_parts && (key->flags & ....))) > { > best_key_nr= i; > best_key= key; > + m_key_parts_suit= parts_suit; > break; > } > /* > We can only use a non-unique key if it allows range scans (ie. skip > FULLTEXT indexes and such). > */ > - last_part= key->user_defined_key_parts - 1; > DBUG_PRINT("info", ("Index %s rec_per_key[%u]= %lu", > - key->name.str, last_part, key->rec_per_key[last_part])); > - if (!(m_table->file->index_flags(i, last_part, 1) & HA_READ_NEXT)) > + key->name.str, parts_suit, key->rec_per_key[parts_suit])); > + if (!(m_table->file->index_flags(i, parts_suit, 1) & HA_READ_NEXT)) > continue; > > - tmp= key->rec_per_key[last_part]; > + tmp= key->rec_per_key[parts_suit - 1]; > if (best_key_nr == MAX_KEY || (tmp > 0 && tmp < best_rec_per_key)) > { > best_key_nr= i; > best_key= key; > best_rec_per_key= tmp; > + m_key_parts_suit= parts_suit; > } > } > > @@ -8494,12 +8568,20 @@ Delete_rows_log_event::do_before_row_operations(const Slave_reporting_capability > if (get_flags(STMT_END_F)) > status_var_increment(thd->status_var.com_stat[SQLCOM_DELETE]); > > + const uchar *curr_row_end= m_curr_row_end; > + unpack_row(rgi, m_table, m_width, m_curr_row, &m_cols, > + &curr_row_end, &m_master_reclength, m_rows_end); > + > + KEY *pk_info= m_table->key_info + m_table->s->primary_key; > if ((m_table->file->ha_table_flags() & HA_PRIMARY_KEY_REQUIRED_FOR_POSITION) && > - m_table->s->primary_key < MAX_KEY) > + m_table->s->primary_key < MAX_KEY && > + key_parts_suit_event(pk_info) == pk_info->ext_key_parts) > { > /* > We don't need to allocate any memory for m_key since it is not used. > */ > + m_key_nr= m_table->s->primary_key; > + m_key_parts_suit= pk_info->ext_key_parts; > return 0; > } > if (do_invoke_trigger()) already commented in the previous review that if "We don't need to allocate any memory" than you `return 0` early wthout doing do_invoke_trigger(). Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 11

Re: [Maria-developers] b2800c06b22: MDEV-28545 MyISAM reorganize partition corrupt older table format
by Sergei Golubchik 25 Oct '22

25 Oct '22

Hi, Alexander, Ok to push. One minor comment below: On Oct 25, Alexander Barkov wrote: > revision-id: b2800c06b22 (mariadb-10.4.26-45-gb2800c06b22) > parent(s): 34163154077 > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2022-10-25 11:53:39 +0400 > message: > > MDEV-28545 MyISAM reorganize partition corrupt older table format > diff --git a/sql/sql_table.cc b/sql/sql_table.cc > index 6aae927800e..a621825c395 100644 > --- a/sql/sql_table.cc > +++ b/sql/sql_table.cc > @@ -10068,6 +10068,18 @@ do_continue:; > > set_table_default_charset(thd, create_info, alter_ctx.db); > > + > +#ifdef WITH_PARTITION_STORAGE_ENGINE > + if (!fast_alter_partition) > +#endif To avoid ifdef's you can use if (IF_PARTITIONING(!fast_alter_partition, 1)) > + { > + /* > + We cannot alter partitions and change column data types at the same > + time. > + */ > + Create_field::upgrade_data_types(alter_info->create_list); > + } > + > if (create_info->check_fields(thd, alter_info, > table_list->table_name, table_list->db) || > create_info->fix_period_fields(thd, alter_info)) > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 152295d1cb1: rpl: extra DEFAULTs should not be UPDATEd
by Sergei Golubchik 24 Oct '22

24 Oct '22

Hi, Nikita, As we've discussed on slack, this is wrong. The (serialized) behavior should be as if all DML happened before ALTER. Because if some DML would've happened after ALTER, it would be able to see the new table definition and refer to new columns. On Oct 24, Nikita Malyavin wrote: > revision-id: 152295d1cb1 (mariadb-10.6.1-525-g152295d1cb1) > parent(s): 90274a7428e > author: Nikita Malyavin > committer: Nikita Malyavin > timestamp: 2022-10-14 23:17:59 +0300 > message: > > rpl: extra DEFAULTs should not be UPDATEd > > diff --git a/sql/log_event_server.cc b/sql/log_event_server.cc > index 25705d13641..a72005d70ab 100644 > --- a/sql/log_event_server.cc > +++ b/sql/log_event_server.cc > @@ -8832,11 +8832,17 @@ Update_rows_log_event::do_exec_row(rpl_group_info *rgi) > message= thd->wsrep_info; > #endif /* WSREP_PROC_INFO */ > > - /* this also updates m_curr_row_end */ > thd_proc_info(thd, message); > - if (unlikely((error= unpack_current_row(rgi, &m_cols_ai)))) > + Field **default_field= m_table->default_field; > + m_table->default_field= NULL; > + /* this also updates m_curr_row_end */ > + error= unpack_current_row(rgi, &m_cols_ai); > + m_table->default_field= default_field; > + > + if (unlikely(error)) > goto err; > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] MDEV-29624 MDEV-29655 Fix ASAN errors on pushdown of derived table
by Sergey Petrunia 23 Oct '22

23 Oct '22

Hi Oleg, > commit 4cad1e16a85114815075e5706d6a8faf5e8fc2e5 > Author: Oleg Smirnov <olernov(a)gmail.com> > Date: Wed Oct 19 13:26:19 2022 +0400 > > MDEV-29624 MDEV-29655 Fix ASAN errors on pushdown of derived table > > Deallocation of TABLE_LIST::dt_handler and TABLE_LIST::pushdown_derived > was performed in multiple places if code. This not only made the code > more difficult to maintain but also led to memory leaks and > ASAN heap-use-after-free errors. > This commit puts deallocation of TABLE_LIST::dt_handler and > TABLE_LIST::pushdown_derived to the single point - JOIN::cleanup() I think the patch is ok to push except for this part: > diff --git a/sql/sql_select.cc b/sql/sql_select.cc > index a91b4571b21..4b8956132e8 100644 > --- a/sql/sql_select.cc > +++ b/sql/sql_select.cc > @@ -14029,11 +14030,11 @@ void JOIN::cleanup(bool full) > delete pushdown_query; > pushdown_query= 0; > > - if (!join_tab) > + List_iterator<TABLE_LIST> li(*join_list); > + TABLE_LIST *table_ref; > + while ((table_ref= li++)) > { > - List_iterator<TABLE_LIST> li(*join_list); > - TABLE_LIST *table_ref; > - while ((table_ref= li++)) > + if (!join_tab) > { > if (table_ref->table && > table_ref->jtbm_subselect && > @@ -14043,6 +14044,14 @@ void JOIN::cleanup(bool full) > table_ref->table= NULL; > } > } > + if (table_ref->pushdown_derived) > + { > + delete table_ref->pushdown_derived; > + table_ref->pushdown_derived= NULL; > + } > + delete table_ref->dt_handler; > + table_ref->dt_handler= NULL; > + > } > } > /* Restore ref array to original state */ What do you think of moving the cleanup to JOIN_TAB::cleanup()? I did the following and it seems to work: diff --git a/sql/sql_select.cc b/sql/sql_select.cc index a91b4571b21..11cf3e13365 100644 --- a/sql/sql_select.cc +++ b/sql/sql_select.cc @@ -13529,6 +13530,16 @@ void JOIN_TAB::cleanup() select= 0; delete quick; quick= 0; + if (table && table->pos_in_table_list) + { + TABLE_LIST *tl= table->pos_in_table_list; + + delete tl->pushdown_derived; + tl->pushdown_derived= NULL; + + delete tl->dt_handler; + tl->dt_handler= NULL; + } if (rowid_filter) { delete rowid_filter; BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://petrunia.net

1 0

Re: [Maria-developers] 39e20ca7e28: MDEV-29748 ASAN errors or server crash in File_parser::parse upon concurrent view operations
by Sergei Golubchik 21 Oct '22

21 Oct '22

Hi, Oleksandr, Looks good. Just one question below: On Oct 19, Oleksandr Byelkin wrote: > revision-id: 39e20ca7e28 (mariadb-10.3.36-107-g39e20ca7e28) > parent(s): d6707ab11f6 > author: Oleksandr Byelkin > committer: Oleksandr Byelkin > timestamp: 2022-10-18 16:38:09 +0200 > message: > > MDEV-29748 ASAN errors or server crash in File_parser::parse upon concurrent view operations > > Read the version of the view share when we read definition to prevent > simultaniouse access to a view table SHARE (and so its MEM_ROOT) > from different threads. > > diff --git a/sql/sql_view.cc b/sql/sql_view.cc > index 68a8410f559..1e4a740f871 100644 > --- a/sql/sql_view.cc > +++ b/sql/sql_view.cc > @@ -1175,19 +1175,24 @@ static int mysql_register_view(THD *thd, TABLE_LIST *view, > bool mariadb_view_version_get(TABLE_SHARE *share) > { > DBUG_ASSERT(share->is_view); > + DBUG_ASSERT(share->tabledef_version.length == 0); > > if (!(share->tabledef_version.str= > (uchar*) alloc_root(&share->mem_root, > MICROSECOND_TIMESTAMP_BUFFER_SIZE))) > return TRUE; > - share->tabledef_version.length= 0; // safety if the drfinition file is brocken > > DBUG_ASSERT(share->view_def != NULL); > if (share->view_def->parse((uchar *) &share->tabledef_version, NULL, > view_timestamp_parameters, 1, > &file_parser_dummy_hook)) > + { > + // safety if the definition file is brocken > + share->tabledef_version.length= 0; when can view_def->parse() fail? _Only_ if the definition is invalid? Can it succeed for one TABLE and fail for another, both of the same TABLE_SHARE? > return TRUE; > + } > DBUG_ASSERT(share->tabledef_version.length == MICROSECOND_TIMESTAMP_BUFFER_SIZE-1); > + > return FALSE; > } Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 4

Re: [Maria-developers] 99dcf061c8c: MDEV-29481 mariadb-upgrade prints confusing statement
by Sergei Golubchik 21 Oct '22

21 Oct '22

Hi, Alexander, ok to push! On Oct 21, Alexander Barkov wrote: > revision-id: 99dcf061c8c (mariadb-10.4.26-55-g99dcf061c8c) > parent(s): 1be451ca797 > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2022-10-21 14:31:02 +0400 > message: > > MDEV-29481 mariadb-upgrade prints confusing statement > > This is a new version of the patch instead of the reverted: > > MDEV-28727 ALTER TABLE ALGORITHM=NOCOPY does not work after upgrade > > Ignore the difference in key packing flags HA_BINARY_PACK_KEY and HA_PACK_KEY > during ALTER to allow ALGORITHM=INSTANT and ALGORITHM=NOCOPY in more cases. > > If for some reasons (e.g. due to a bug fix such as MDEV-20704) these > cumulative (over all segments) flags in KEY::flags are different for > the old and new table inside compare_keys_but_name(), the difference > in HA_BINARY_PACK_KEY and HA_PACK_KEY in KEY::flags is not really important: > > MyISAM and Aria can handle such cases well: per-segment flags are stored in > MYI and MAI files anyway and they are read during ha_myisam::open() > ha_maria::open() time. So indexes get opened with correct per-segment > flags that were calculated during the table CREATE time, no matter > what the old (CREATE time) and new (ALTER TIME) per-index compression > flags are, and no matter if they are equal or not. > > All other engine ignore key compression flags, so this change > is safe for other engines as well. > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] c83f87ece4a: MDEV-15990 REPLACE on a precise-versioned table returns duplicate key error (ER_DUP_ENTRY)
by Sergei Golubchik 15 Oct '22

15 Oct '22

Hi, Nikita, Sorry, this comes very late. But the bug is still present in 10.10, and the filter sorts by the date precisely to highlight these old bugs. So, here you are. First comment, please, port it to the latest 10.3. I wasn't able to find the repository with the fix, but the patch itself is avaliable from the pull request as https://github.com/MariaDB/server/commit/c83f87ece4a.patch so it can be salvaged. More comments - see below. On Oct 15, Nikita Malyavin wrote: > From: Nikita Malyavin <nikitamalyavin(a)gmail.com> > Date: Fri, 20 Jul 2018 00:24:40 +1000 > Subject: [PATCH] MDEV-15990: REPLACE on a precise-versioned table returns > * handle zero value in `row_start` `trx_id` > * refuse to do more than one versioned insert on the same transaction and same row > * generalize versioned cases with and without triggers > > diff --git a/sql/sql_insert.cc b/sql/sql_insert.cc > index dda469a1426d5..cd2690128de67 100644 > --- a/sql/sql_insert.cc > +++ b/sql/sql_insert.cc > @@ -1928,32 +1928,15 @@ int write_record(THD *thd, TABLE *table,COPY_INFO *info) > For system versioning wa also use path through delete since we would > save nothing through this cheating. > */ > - if (last_uniq_key(table,key_nr) && > + if (last_uniq_key(table,key_nr) && !table->versioned() && ok! > !table->file->referenced_by_foreign_key() && > (!table->triggers || !table->triggers->has_delete_triggers())) > { > - if (table->versioned(VERS_TRX_ID) && table->vers_write) > - { > - bitmap_set_bit(table->write_set, table->vers_start_field()->field_index); > - table->vers_start_field()->store(0, false); > - } > - if (unlikely(error= table->file->ha_update_row(table->record[1], > - table->record[0])) && > - error != HA_ERR_RECORD_IS_THE_SAME) > + error= table->file->ha_update_row(table->record[1], table->record[0]); > + if (unlikely(error && error != HA_ERR_RECORD_IS_THE_SAME)) > goto err; > if (likely(!error)) > - { > info->deleted++; > - if (table->versioned(VERS_TIMESTAMP) && table->vers_write) > - { > - store_record(table, record[2]); > - error= vers_insert_history_row(table); > - restore_record(table, record[2]); > - if (unlikely(error)) > - goto err; > - } > - } > - else > error= 0; // error was HA_ERR_RECORD_IS_THE_SAME > thd->record_first_successful_insert_id_in_cur_stmt(table->file->insert_id_for_cur_row); > /* > @@ -1969,23 +1952,22 @@ int write_record(THD *thd, TABLE *table,COPY_INFO *info) > TRG_ACTION_BEFORE, TRUE)) > goto before_trg_err; > > - if (!table->versioned(VERS_TIMESTAMP)) > + if (table->versioned(VERS_TRX_ID) && table->vers_write) > + { > + bitmap_set_bit(table->write_set, table->vers_start_field()->field_index); this is conceptually wrong, you cannot set table->write_set in the middle of statement execution. that is you can and it might even work for some engines, but it breaks the API. > + table->vers_start_field()->store(0, false); > + } > + > + if (!table->versioned()) > error= table->file->ha_delete_row(table->record[1]); > else > - { > - store_record(table, record[2]); > - restore_record(table, record[1]); > - table->vers_update_end(); why? is the row_end updated somewhere else? > error= table->file->ha_update_row(table->record[1], > table->record[0]); > - restore_record(table, record[2]); > - } > - if (unlikely(error)) > + if (unlikely(error && error != HA_ERR_RECORD_IS_THE_SAME)) > goto err; > - if (!table->versioned(VERS_TIMESTAMP)) > + if (likely(!error)) > info->deleted++; > - else > - info->updated++; > + error= 0; > if (!table->file->has_transactions()) > thd->transaction.stmt.modified_non_trans_table= TRUE; > if (table->triggers && > @@ -1995,6 +1977,17 @@ int write_record(THD *thd, TABLE *table,COPY_INFO *info) > trg_error= 1; > goto ok_or_after_trg_err; > } > + if (table->versioned(VERS_TIMESTAMP) && table->vers_write) > + { > + store_record(table, record[2]); > + error = vers_insert_history_row(table); if you do that, you shouldn't have been doing ha_update_row() above, as far as I can see. What is that update above for? > + restore_record(table, record[2]); > + if (unlikely(error)) > + goto err; > + } > + if (table->versioned()) > + goto after_trg_n_copied_inc; > + > /* Let us attempt do write_row() once more */ > } > } Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 19772f9c277: MDEV-29748 ASAN errors or server crash in File_parser::parse upon concurrent view operations
by Sergei Golubchik 13 Oct '22

13 Oct '22

Hi, Oleksandr, On Oct 13, Oleksandr Byelkin wrote: > revision-id: 19772f9c277 (mariadb-10.3.36-90-g19772f9c277) > parent(s): e05ab0cfc5f > author: Oleksandr Byelkin > committer: Oleksandr Byelkin > timestamp: 2022-10-12 15:59:46 +0200 > message: > > MDEV-29748 ASAN errors or server crash in File_parser::parse upon concurrent view operations > > Prevent simultaniouse access to a view table SHARE (and so its MEM_ROOT) from different threads. If you're writing into share, why not to do it when a share is populated, then you wouldn't need a mutex. And it'd be a logical place to initialize the share anyway. Doing it at some arbitrary point in time later looks strange. I mean, why not to call this mariadb_view_version_get after the line share->view_def= sql_parse_prepare(&pathstr, &share->mem_root, true); where it logically belongs? > + mysql_mutex_lock(&share->LOCK_share); > > if (!(share->tabledef_version.str= > (uchar*) alloc_root(&share->mem_root, > MICROSECOND_TIMESTAMP_BUFFER_SIZE))) Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] b1b38b64598: MDEV-29181 Potential corruption on FK update on a table with vcol index
by Sergei Golubchik 10 Oct '22

10 Oct '22

Hi, Nikita, On Aug 30, Nikita Malyavin wrote: > revision-id: b1b38b64598 (mariadb-10.5.17-4-gb1b38b64598) > parent(s): 3b656ac8c17 > author: Nikita Malyavin > committer: Nikita Malyavin > timestamp: 2022-08-23 00:27:39 +0300 > message: > > MDEV-29181 Potential corruption on FK update on a table with vcol > index > > vc_templ->mysql_table concept is completely broken. This table pointer > persists with a global access and can be overwritten by arbitrary > thread. Like it wasn't enough, it also could become invalid after > eviction from tc cache. this is a bit misleading. I agree it's completely broken, though :) But not "this table pointer", because, of course it persists with a global access and can be overwritten and so on. This is by design, it's protected by mysql_table_query_id. The problem is that table (dict_table_t) argument of innodb_find_table_for_vc() points to a shared data structure, not thread local, as I believed. So vc_templ is shared too. I suggest to rewrite the comment like vc_templ->mysql_table concept is completely broken. table argument of innodb_find_table_for_vc() points to a shared data structure dict_table_t, and thus vc_templ is shared too and cannot be used for a thread-local cache. > This new solution simply does the following: > * Sets up a referenced table list in TABLE instance (sql_base.cc) > * Iterates through it along with dict_table_t::referenced_set > (row_upd_check_references_constraints) > * Passes corresponding prebuilt through a call chain to > row_ins_foreign_check_on_constraint > * Sets up newly created upd_node_t::prebuilt field and uses it > accordingly is cascade update. Is this upd_node_t::prebuilt used anywhere? As a prebuilt, I mean. I couldn't find it (it's a complex patch, so I couldn've missed it). As far as I can see, it's only used to store a pointer to TABLE. So it seems to me than a simpler fix for this bug could be: * remove vc_templ caching (mysql_table and mysql_table_query_id) * store TABLE* in upd_node_t. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

3 6

Re: [Maria-developers] dcfc60ea2ee: MDEV-29481 mariadb-upgrade prints confusing statement
by Sergei Golubchik 05 Oct '22

05 Oct '22

Hi, Alexander, See a couple of questions below: On Oct 04, Alexander Barkov wrote: > revision-id: dcfc60ea2ee (mariadb-10.4.26-28-gdcfc60ea2ee) > parent(s): 6c2c825e503 > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2022-10-03 10:32:42 +0400 > message: > > MDEV-29481 mariadb-upgrade prints confusing statement > > This is a new version of the patch instead of the reverted: > > MDEV-28727 ALTER TABLE ALGORITHM=NOCOPY does not work after upgrade > diff --git a/mysql-test/main/alter_table_debug_aria.test b/mysql-test/main/alter_table_debug_aria.test > new file mode 100644 > index 00000000000..22df57686e1 > --- /dev/null > +++ b/mysql-test/main/alter_table_debug_aria.test > @@ -0,0 +1,36 @@ > +--source include/have_debug.inc > +--source include/have_maria.inc > + > +--echo # > +--echo # Start of 10.4 tests > +--echo # > + > +--echo # > +--echo # MDEV-28727 ALTER TABLE ALGORITHM=NOCOPY does not work after upgrade > +--echo # > + > +CREATE TABLE t1 (a DOUBLE NOT NULL, KEY(a)) ENGINE=ARIA; > +ALTER TABLE t1 MODIFY a DOUBLE NOT NULL DEFAULT 10, ALGORITHM=NOCOPY; > +DROP TABLE t1; > + > +CREATE TABLE t1 (a DOUBLE NOT NULL, KEY(a)) ENGINE=ARIA; > +ALTER TABLE t1 MODIFY a DOUBLE NOT NULL DEFAULT 10, ALGORITHM=INSTANT; > +DROP TABLE t1; > + > +SET SESSION debug_dbug='+d,emulate_pre_mdev_20704'; > +CREATE TABLE t1 (a DOUBLE NOT NULL, KEY(a)) ENGINE=ARIA; Why did you add these debug-only tests with the emulate_pre_mdev_20704 hack, when you have perfect non-debug tests with real pre-MDEV-20704 created tables ? > +SET SESSION debug_dbug=''; > +--error ER_ALTER_OPERATION_NOT_SUPPORTED > +ALTER TABLE t1 MODIFY a DOUBLE NOT NULL DEFAULT 10, ALGORITHM=INSTANT; > +DROP TABLE t1; > + > +SET SESSION debug_dbug='+d,emulate_pre_mdev_20704'; > +CREATE TABLE t1 (a DOUBLE NOT NULL, KEY(a)) ENGINE=ARIA; > +SET SESSION debug_dbug=''; > +--error ER_ALTER_OPERATION_NOT_SUPPORTED > +ALTER TABLE t1 MODIFY a DOUBLE NOT NULL DEFAULT 10, ALGORITHM=NOCOPY; > +DROP TABLE t1; > + > +--echo # > +--echo # End of 10.4 tests > +--echo # > diff --git a/sql/sql_table.cc b/sql/sql_table.cc > index 2436031200a..c192a4553a9 100644 > --- a/sql/sql_table.cc > +++ b/sql/sql_table.cc > @@ -6748,11 +6753,13 @@ Compare_keys compare_keys_but_name(const KEY *table_key, const KEY *new_key, > const KEY *const new_pk, > const KEY *const old_pk) > { > - if (table_key->algorithm != new_key->algorithm) > + ulong key_flag_mask= HA_KEYFLAG_MASK | > + table->file->key_pack_flags_supported(); I don't understand. You said that these KEY->flags do not affect what the engine does. Why to compare them at all? > + > + if ((table_key->flags & key_flag_mask) != (new_key->flags & key_flag_mask)) > return Compare_keys::NotEqual; > > - if ((table_key->flags & HA_KEYFLAG_MASK) != > - (new_key->flags & HA_KEYFLAG_MASK)) > + if (table_key->algorithm != new_key->algorithm) > return Compare_keys::NotEqual; > > if (table_key->user_defined_key_parts != new_key->user_defined_key_parts) Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] e2f8dffc056: MDEV-29069 ER_KEY_NOT_FOUND on online autoinc addition + concurrent DELETE
by Sergei Golubchik 03 Oct '22

03 Oct '22

Hi, Nikita, Few comments below On Aug 27, Nikita Malyavin wrote: > commit e2f8dffc056 > Author: Nikita Malyavin <nikitamalyavin(a)gmail.com> > Date: Mon Jul 25 23:52:24 2022 +0300 > > diff --git a/sql/log_event_server.cc b/sql/log_event_server.cc > index 422d496d5c3..ca17c5c07a8 100644 > --- a/sql/log_event_server.cc > +++ b/sql/log_event_server.cc > @@ -7949,14 +7952,35 @@ uint8 Write_rows_log_event::get_trg_event_map() > **************************************************************************/ > > #if defined(HAVE_REPLICATION) > -/* > - Compares table->record[0] and table->record[1] > +/** > + @brief Compares table->record[0] and table->record[1] > + > + @param masrter_columns a number of columns on the source replica, > + 0 if ONLINE ALTER TABLE > > - Returns TRUE if different. > + @returns true if different. > */ > -static bool record_compare(TABLE *table) > +static bool record_compare(TABLE *table, uint master_columns) > { > - bool result= FALSE; > + bool result= false; > + > + /* > + Determine if the optimized check is possible (and we can > + goto record_compare_exit). > + We should ensure that all extra columns (i.e. fieldnr > master_columns) > + have explicit values. If not, we will exclude them from comparison, > + as they can contain non-deterministic values. > + > + master_columns == 0 case is optimization for ONLINE ALTER to check > + all columns fast. > + */ > + bool all_values_set= master_columns == 0 > + && bitmap_is_set_all(&table->has_value_set); > + for (uint i = master_columns; all_values_set && i < table->s->fields; i++) > + { > + all_values_set= bitmap_is_set(&table->has_value_set, i); > + } it'd be safer and faster to do all_values_set= bitmap_is_set_all(&table->has_value_set); and, please, add replication (not online alter) tests: 1. slave has more columns 2. binlog row format is "minimal", this might trigger it too > + > /** > Compare full record only if: > - there are no blob fields (otherwise we would also need > @@ -7969,47 +7993,68 @@ static bool record_compare(TABLE *table) > */ > if ((table->s->blob_fields + > table->s->varchar_fields + > - table->s->null_fields) == 0) > + table->s->null_fields) == 0 > + && all_values_set) > { > result= cmp_record(table,record[1]); > goto record_compare_exit; > } > > /* Compare null bits */ > - if (memcmp(table->null_flags, > + if (all_values_set && memcmp(table->null_flags, > table->null_flags+table->s->rec_buff_length, > table->s->null_bytes)) > { > - result= TRUE; // Diff in NULL value > + result= true; // Diff in NULL value > goto record_compare_exit; > } > > /* Compare fields */ > for (Field **ptr=table->field ; *ptr ; ptr++) > { > + Field *f= *ptr; > if (table->versioned() && (*ptr)->vers_sys_field()) > { > continue; > } > - /** > - We only compare field contents that are not null. > - NULL fields (i.e., their null bits) were compared > - earlier. > + /* > + We only compare fields that exist on the source (or in ONLINE > + ALTER case, that were in the original table). For reference, > + replica tables can also contain extra fields. > */ > - if (!(*(ptr))->is_null()) > - { > - if ((*ptr)->cmp_binary_offset(table->s->rec_buff_length)) > + if (f->field_index > master_columns && !f->has_explicit_value()) > + continue; why do you check for f->field_index > master_columns? isn't !f->has_explicit_value() the one that decided? in what case will be ok to compare values when f->field_index < master_columns but no explicit value was set? > + > + if (f->is_null() != f->is_null(table->s->rec_buff_length) > + || f->cmp_binary_offset(table->s->rec_buff_length)) > { > - result= TRUE; > + result= true; > goto record_compare_exit; > } > } > - } > > record_compare_exit: > return result; > } > > +/** > + Newly added fields with non-deterministic defaults (i.e. DEFAULT(RANDOM()), > + CURRENT_TIMESTAMP, AUTO_INCREMENT) should be excluded from key search. > + Basically we exclude all the default-filled fields based on > + has_explicit_value bitmap. > + */ > +bool Rows_log_event::key_suits_event(const KEY *key) const > +{ > + uint master_columns= m_online_alter ? 0 : m_cols.n_bits; > + for (uint p= 0; p < key->ext_key_parts; p++) > + { > + uint field_idx= key->key_part[p].fieldnr - 1; > + if (field_idx >= master_columns > + && !bitmap_is_set(&m_table->has_value_set, field_idx)) > + return false; > + } > + return true; > +} > > /** > Find the best key to use when locating the row in @c find_row(). > @@ -8030,6 +8075,18 @@ int Rows_log_event::find_key() > DBUG_ENTER("Rows_log_event::find_key"); > DBUG_ASSERT(m_table); > > + #ifdef DBUG_TRACE > + auto _ = make_scope_exit([this]() { > + DBUG_EXECUTE_IF("rpl_report_chosen_key", > + push_warning_printf(m_table->in_use, > + Sql_condition::WARN_LEVEL_NOTE, > + ER_UNKNOWN_ERROR, "Key chosen: %d", > + m_key_nr == MAX_KEY ? > + -1 : m_key_nr);); > + }); > + #endif > + > + m_key_nr= MAX_KEY; it would've been simpler and shorter to do it the old style, with a goto > best_key_nr= MAX_KEY; > > /* > @@ -8041,6 +8098,8 @@ int Rows_log_event::find_key() > { > if (!m_table->s->keys_in_use.is_set(i)) > continue; > + if (!key_suits_event(key)) > + continue; 1. again, add a replication test, please. when, e.g., slave has more columns and a unique index over new columns. 2. going over all keyparts for every event is rather redundant. you can calculate the set of usable keys once and put it, for example, in m_table->keys_in_use_for_query. > /* > We cannot use a unique key with NULL-able columns to uniquely identify > a row (but we can still select it for range scan below if nothing better > @@ -8221,7 +8280,8 @@ int Rows_log_event::find_row(rpl_group_info *rgi) > DBUG_DUMP("record[0]", table->record[0], table->s->reclength); > > if ((table->file->ha_table_flags() & HA_PRIMARY_KEY_REQUIRED_FOR_POSITION) && > - table->s->primary_key < MAX_KEY) > + table->s->primary_key < MAX_KEY && > + m_key_nr == table->s->primary_key) add a replication test for that: slave has an extra column which is a PK, and HA_PRIMARY_KEY_REQUIRED_FOR_POSITION is set (InnoDB) > { > /* > Use a more efficient method to fetch the record given by > @@ -8494,12 +8554,18 @@ Delete_rows_log_event::do_before_row_operations(const Slave_reporting_capability > if (get_flags(STMT_END_F)) > status_var_increment(thd->status_var.com_stat[SQLCOM_DELETE]); > > + const uchar *curr_row_end= m_curr_row_end; > + unpack_row(rgi, m_table, m_width, m_curr_row, &m_cols, > + &curr_row_end, &m_master_reclength, m_rows_end); > + > if ((m_table->file->ha_table_flags() & HA_PRIMARY_KEY_REQUIRED_FOR_POSITION) && > - m_table->s->primary_key < MAX_KEY) > + m_table->s->primary_key < MAX_KEY > + && key_suits_event(&m_table->key_info[m_table->s->primary_key])) > { > /* > We don't need to allocate any memory for m_key since it is not used. > */ > + m_key_nr= m_table->s->primary_key; > return 0; > } This is very strange if(). 1. it's only for delete, not for update 2. it's before invoking triggers (bug) 3. it's wrong if the master table has a different primary key (bug) I'd suggest to remove it completely. An optimization of avoiding a malloc can be done inside find_key() after the best key is found. > if (do_invoke_trigger()) Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] 5265f7001c6: MDEV-17124: mariadb 10.1.34, views and prepared statements: ERROR 1615 (HY000): Prepared statement needs to be re-prepared
by Sergei Golubchik 28 Sep '22

28 Sep '22

Hi, Oleksandr, Few minor comments, see below On Sep 27, Oleksandr Byelkin wrote: > revision-id: 5265f7001c6 (mariadb-10.3.36-60-g5265f7001c6) > parent(s): 86953d3df0a > author: Oleksandr Byelkin > committer: Oleksandr Byelkin > timestamp: 2022-09-27 10:23:31 +0200 > message: > > MDEV-17124: mariadb 10.1.34, views and prepared statements: ERROR 1615 (HY000): Prepared statement needs to be re-prepared > > The problem is that if table definition cache (TDC) is full of real tables > which are in tables cache, view definition can not stay there so will be > removed by its own underlying tables. > In situation above old mechanism of detection matching definition in PS > and current version always require reprepare and so prevent executing > the PS. > > One work around is to increase TDC, other - improve version check for > views/triggers (which is done here). Now in suspicious cases we check: > - timestamp (microseconds) of the view to be sure that version really > have changed; > - time (microseconds) of creation of a trigger related to time > (microseconds) of statement preparation. > diff --git a/sql/table.cc b/sql/table.cc > index 506195127b2..f289c2052cb 100644 > --- a/sql/table.cc > +++ b/sql/table.cc > @@ -8861,6 +8861,63 @@ bool TABLE_LIST::is_with_table() > return derived && derived->with_element; > } > > + > +/** > + Check if the definition are the same. > + > + If versions do not match it check definitions (with checking and setting > + trigger definition versions (times) > + > + @sa check_and_update_table_version() > +*/ > + > +bool TABLE_LIST::is_the_same_definition(THD* thd, TABLE_SHARE *s) > +{ > + enum enum_table_ref_type tp= s->get_table_ref_type(); > + if (m_table_ref_type == tp) > + { > + bool res= m_table_ref_version == s->get_table_ref_version(); > + > + /* > + If definition is different check content version > + */ if res == true, why do you need to check tabledef_version? > + if (tabledef_version.length && > + tabledef_version.length == s->tabledef_version.length && > + memcmp(tabledef_version.str, s->tabledef_version.str, > + tabledef_version.length) == 0) > + { > + if (table && table->triggers) > + { > + my_hrtime_t hr_stmt_prepare= thd->hr_prepare_time; > + if (hr_stmt_prepare.val) > + for(uint i= 0; i < TRG_EVENT_MAX; i++) > + for (uint j= 0; j < TRG_ACTION_MAX; j++) > + { > + Trigger *tr= > + table->triggers->get_trigger((trg_event_type)i, > + (trg_action_time_type)j); > + if (tr) > + if (hr_stmt_prepare.val <= tr->hr_create_time.val) > + { > + set_tabledef_version(s); > + return FALSE; > + } > + } > + } > + set_table_id(s); > + return TRUE; > + } > + else > + tabledef_version.length= 0; > + return res; > + } > + else > + set_tabledef_version(s); why not set_table_id() ? > + return FALSE; > +} > + > + > uint TABLE_SHARE::actual_n_key_parts(THD *thd) > { > return use_ext_keys && > diff --git a/sql/sql_trigger.h b/sql/sql_trigger.h > index ae3d1738b16..f7a2ccf2abc 100644 > --- a/sql/sql_trigger.h > +++ b/sql/sql_trigger.h > @@ -198,7 +198,7 @@ class Table_triggers_list: public Sql_alloc > */ > List<ulonglong> definition_modes_list; > /** Create times for triggers */ > - List<ulonglong> create_times; > + List<my_hrtime_t> hr_create_times; I suspect it might be UB. You use FILE_OPTIONS_ULLLIST for hr_create_times, perhaps it's safer to use List<ulonglong> here. > > List<LEX_CSTRING> definers_list; > > @@ -328,4 +328,14 @@ bool mysql_create_or_drop_trigger(THD *thd, TABLE_LIST *tables, bool create); > extern const char * const TRG_EXT; > extern const char * const TRN_EXT; > > + > +/** > + Make time compatible with MySQL 5.7 trigger time. > +*/ > + > +inline my_hrtime_t make_hr_time(my_time_t time, ulong time_sec_part) > +{ > + return my_hrtime_t({((ulonglong) time)*1000000 + time_sec_part}); > +} better put it next to hrtime_to_time/etc and, please, always 'static inline' > + > #endif /* SQL_TRIGGER_INCLUDED */ > diff --git a/sql/sql_view.cc b/sql/sql_view.cc > index 17dea643144..177d01a312e 100644 > --- a/sql/sql_view.cc > +++ b/sql/sql_view.cc > @@ -1154,7 +1163,32 @@ static int mysql_register_view(THD *thd, TABLE_LIST *view, > DBUG_RETURN(error); > } > > +/** > + Check is TABLE_LEST and SHARE match > + @param[in] view TABLE_LIST of the view > + @param[in] share Share object of view > + > + @return false on error or misspatch Eh. I don't understad this comment at all. First, I thought you made two typos, s/is/if/ and s/misspatch/mismatch/. But this function doesn't check if something matches, if gets the view version, as the name says, the name's good. But the comment seems to be from is_the_same_definition() > +*/ > + > +bool mariadb_view_version_get(TABLE_SHARE *share) > +{ > + DBUG_ASSERT(share->is_view); > + > + if (!(share->tabledef_version.str= > + (uchar*) alloc_root(&share->mem_root, > + MICROSECOND_TIMESTAMP_BUFFER_SIZE))) > + return TRUE; > + share->tabledef_version.length= 0; // safety if the drfinition file is brocken > > + DBUG_ASSERT(share->view_def != NULL); > + if (share->view_def->parse((uchar *) &share->tabledef_version, NULL, > + view_timestamp_parameters, 1, > + &file_parser_dummy_hook)) > + return TRUE; > + DBUG_ASSERT(share->tabledef_version.length == MICROSECOND_TIMESTAMP_BUFFER_SIZE-1); > + return FALSE; > +} > > /** > read VIEW .frm and create structures Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 3

Re: [Maria-developers] 349283c5e7a: MDEV-17124: mariadb 10.1.34, views and prepared statements: ERROR 1615 (HY000): Prepared statement needs to be re-prepared
by Sergei Golubchik 27 Sep '22

27 Sep '22

Hi, Oleksandr, On Jan 27, Oleksandr Byelkin wrote: > commit 349283c5e7a > Author: Oleksandr Byelkin <sanja(a)mariadb.com> > Date: Wed Apr 17 15:50:59 2019 +0200 > > MDEV-17124: mariadb 10.1.34, views and prepared statements: ERROR 1615 (HY000): Prepared statement needs to be re-prepared > > The problem is that if table definition cache (TDC) is full of real tables > which are in tables cache, view definition can not stay there so will be > removed by its own underlying tables. > In situation above old mechanism of detection matching definition in PS > and current version always require reprepare and so prevent executing > the PS. > > One work around is to increase TDC, other - improve version check for > views/triggers (which is done here). Now in suspicious cases we check: > - timestamp (ms) of the view to be sure that version really have changed; > - time (ms) of creation of a trigger related to time (ms) of statement > preparation. not "ms" but "us" or "µs". microseconds, not millisecons. > diff --git a/mysql-test/r/ps_ddl.result b/mysql-test/r/ps_ddl.result > index e69c6e06193..005c78a0319 100644 > --- a/mysql-test/r/ps_ddl.result > +++ b/mysql-test/r/ps_ddl.result > @@ -831,8 +831,8 @@ a b c > 20 40 100 > 30 60 150 > call p_verify_reprepare_count(1); > -SUCCESS > - > +ERROR > +Expected: 1, actual: 0 oops, forgot to update? > # Check that we properly handle ALTER VIEW statements. > execute stmt; > a b c > @@ -882,9 +882,9 @@ a b c > 10 50 60 > 20 100 120 > 30 150 180 > -call p_verify_reprepare_count(1); > -SUCCESS > - > +call p_verify_reprepare_count(0); > +ERROR > +Expected: 0, actual: 1 oops, updated too much? > execute stmt; > a b c > 10 50 60 > diff --git a/sql/sql_class.h b/sql/sql_class.h > index 3f0fba8fc10..b9bb9c978fb 100644 > --- a/sql/sql_class.h > +++ b/sql/sql_class.h > @@ -1079,6 +1079,7 @@ class Statement: public ilink, public Query_arena > > LEX_STRING name; /* name for named prepared statements */ > LEX *lex; // parse tree descriptor > + ulonglong ms_prepare_time; // time of preparation in microseconds "ms" here too :( Better just say my_hrtime_t prepare_time; > /* > Points to the query associated with this statement. It's const, but > we need to declare it char * because all table handlers are written > diff --git a/sql/sql_prepare.cc b/sql/sql_prepare.cc > index 64e4cd30561..a4a24cc1a80 100644 > --- a/sql/sql_prepare.cc > +++ b/sql/sql_prepare.cc > @@ -4177,6 +4177,9 @@ bool Prepared_statement::prepare(const char *packet, uint packet_len) > Query_arena *old_stmt_arena; > DBUG_ENTER("Prepared_statement::prepare"); > DBUG_ASSERT(m_sql_mode == thd->variables.sql_mode); > + > + // The same format as for triggers to compare > + ms_prepare_time= my_hrtime().val; do it at the end instead? > /* > If this is an SQLCOM_PREPARE, we also increase Com_prepare_sql. > However, it seems handy if com_stmt_prepare is increased always, > @@ -4519,6 +4523,22 @@ Prepared_statement::execute_loop(String *expanded_query, > DBUG_ASSERT(thd->get_stmt_da()->sql_errno() == ER_NEED_REPREPARE); > thd->clear_error(); > > + { > + /* > + Check if we too fast with reprepare: > + we can be so fast that: > + 1) make change of a trigger, > + 2) prepare, > + 3) try to exacute and reprepare > + in 1 microsecond, so we will wait till > + next microsecond before last reprepare > + */ > + while (first_prepared == my_hrtime().val) > + { > + pthread_yield(); > + } > + } I wouldn't bother. A more likely case is when a system time goes back, e.g. if there's an ntpd running and your loop doesn't help there. I suggest not to do anything special in this case. And if an ::execute() will open a trigger or a view with a timestamp >= than ms_prepare_time, it will cause reprepare every time. Which is exactly the behavior before your bugfix - opening of a trigger or a view causes reprepare - and it is not a problem unless one has a very small table definition cache. > + > error= reprepare(); > > if (! error) /* Success */ > diff --git a/sql/parse_file.cc b/sql/parse_file.cc > index 999f53bd681..92e7d82af1c 100644 > --- a/sql/parse_file.cc > +++ b/sql/parse_file.cc > @@ -174,11 +174,11 @@ write_parameter(IO_CACHE *file, uchar* base, File_option *parameter) > { > /* string have to be allocated already */ > LEX_STRING *val_s= (LEX_STRING *)(base + parameter->offset); > - time_t tm= my_time(0); > - > - get_date(val_s->str, GETDATE_DATE_TIME|GETDATE_GMT|GETDATE_FIXEDLENGTH, > - tm); > - val_s->length= PARSE_FILE_TIMESTAMPLENGTH; > + ulonglong tm= my_hrtime().val; add a comment here, "number of microseconds since Epoch, timezone-independent" or something like that > + // Paded to 19 characters for compatibility > + val_s->length= snprintf(val_s->str, MICROSECOND_TIMESTAMP_BUFFER_SIZE, > + "%019lld", tm); > + DBUG_ASSERT(val_s->length == MICROSECOND_TIMESTAMP_BUFFER_SIZE-1); > if (my_b_write(file, (const uchar *)val_s->str, > PARSE_FILE_TIMESTAMPLENGTH)) > DBUG_RETURN(TRUE); > @@ -835,15 +835,15 @@ File_parser::parse(uchar* base, MEM_ROOT *mem_root, > /* string have to be allocated already */ > LEX_STRING *val= (LEX_STRING *)(base + parameter->offset); > /* yyyy-mm-dd HH:MM:SS = 19(PARSE_FILE_TIMESTAMPLENGTH) characters */ a comment is now wrong > - if (ptr[PARSE_FILE_TIMESTAMPLENGTH] != '\n') > + if (ptr[MICROSECOND_TIMESTAMP_BUFFER_SIZE-1] != '\n') > { > my_error(ER_FPARSER_ERROR_IN_PARAMETER, MYF(0), > parameter->name.str, line); > DBUG_RETURN(TRUE); > } > - memcpy(val->str, ptr, PARSE_FILE_TIMESTAMPLENGTH); > - val->str[val->length= PARSE_FILE_TIMESTAMPLENGTH]= '\0'; > - ptr+= (PARSE_FILE_TIMESTAMPLENGTH+1); > + memcpy(val->str, ptr, MICROSECOND_TIMESTAMP_BUFFER_SIZE-1); > + val->str[val->length= MICROSECOND_TIMESTAMP_BUFFER_SIZE-1]= '\0'; > + ptr+= MICROSECOND_TIMESTAMP_BUFFER_SIZE; > break; > } > case FILE_OPTIONS_STRLIST: > diff --git a/sql/table.h b/sql/table.h > index 14d6b787b4e..82508388f84 100644 > --- a/sql/table.h > +++ b/sql/table.h > @@ -2159,7 +2175,7 @@ struct TABLE_LIST > LEX_STRING source; /* source of CREATE VIEW */ > LEX_STRING view_db; /* saved view database */ > LEX_STRING view_name; /* saved view name */ > - LEX_STRING timestamp; /* GMT time stamp of last operation */ > + LEX_STRING ms_timestamp; /* GMT time stamp of last operation */ better keep "timestamp" name. Or "hrtimestamp" if you'd like > st_lex_user definer; /* definer of view */ > ulonglong file_version; /* version of file's field set */ > ulonglong mariadb_version; /* version of server on creation */ > diff --git a/sql/table.cc b/sql/table.cc > index ca6ce02e4f2..24412966b36 100644 > --- a/sql/table.cc > +++ b/sql/table.cc > @@ -8480,6 +8480,53 @@ bool TABLE_LIST::is_with_table() > } > > > +bool TABLE_LIST::is_table_ref_id_equal(THD* thd, TABLE_SHARE *s) > +{ > + enum enum_table_ref_type tp= s->get_table_ref_type(); > + if (m_table_ref_type == tp) > + { > + bool res= m_table_ref_version == s->get_table_ref_version(); > + > + /* > + If definition is different check content version > + */ > + if (tabledef_version.length && > + tabledef_version.length == s->tabledef_version.length && > + memcmp(tabledef_version.str, s->tabledef_version.str, > + tabledef_version.length) == 0) > + { > + if (table && table->triggers) > + { > + > + ulonglong ms_stmt_prepare= thd->ms_prepare_time; > + if (ms_stmt_prepare) > + for(uint i= 0; i < TRG_EVENT_MAX; i++) > + for (uint j= 0; j < TRG_ACTION_MAX; j++) > + { > + Trigger *tr= > + table->triggers->get_trigger((trg_event_type)i, > + (trg_action_time_type)j); > + if (tr) > + if (ms_stmt_prepare <= tr->ms_create_time) > + { > + set_tabledef_version(s); > + return FALSE; > + } > + } > + } > + set_table_id(s); > + return TRUE; > + } > + else > + tabledef_version.length= 0; > + return res; > + } > + else > + set_tabledef_version(s); > + return FALSE; > +} this is now a very confusing function. it doesn't only check is_table_ref_id_equal, it also sets some of them (tabledef or both tabledef and ref_id), and under unclear conditions. Could you please comment it? And you don't check that view's timestamp is in the past. If it's not, it can be changed still within the same microsecond. > + > + > uint TABLE_SHARE::actual_n_key_parts(THD *thd) > { > return use_ext_keys && > diff --git a/sql/sql_trigger.h b/sql/sql_trigger.h > index e2e6e1b7acc..70f246d4e0d 100644 > --- a/sql/sql_trigger.h > +++ b/sql/sql_trigger.h > @@ -112,7 +112,7 @@ class Trigger :public Sql_alloc > GRANT_INFO subject_table_grants; > sql_mode_t sql_mode; > /* Store create time. Can't be mysql_time_t as this holds also sub seconds */ > - ulonglong create_time; > + ulonglong ms_create_time; // Create time timestamp in microseconds my_hrtime_t create_time; > trg_event_type event; > trg_action_time_type action_time; > uint action_order; > @@ -195,7 +195,7 @@ class Table_triggers_list: public Sql_alloc > */ > List<ulonglong> definition_modes_list; > /** Create times for triggers */ > - List<ulonglong> create_times; > + List<ulonglong> ms_create_times; ditto > > List<LEX_STRING> definers_list; > > @@ -331,4 +331,14 @@ bool mysql_create_or_drop_trigger(THD *thd, TABLE_LIST *tables, bool create); > extern const char * const TRG_EXT; > extern const char * const TRN_EXT; > > + > +/** > + Make time compatible with MySQL 5.7 trigger time. > +*/ > + > +inline ulonglong make_ms_time(my_time_t time, ulong time_sec_part) make_hrtime. and better put it together with hrtime_to_time, hrtime_from_time, etc > +{ > + return ((ulonglong) time)*1000000 + time_sec_part; > +} > + > #endif /* SQL_TRIGGER_INCLUDED */ > diff --git a/sql/sql_view.cc b/sql/sql_view.cc > index 020830483c5..3b51f6d2451 100644 > --- a/sql/sql_view.cc > +++ b/sql/sql_view.cc > @@ -1131,7 +1141,31 @@ static int mysql_register_view(THD *thd, TABLE_LIST *view, > DBUG_RETURN(error); > } > > +/** > + Check is TABLE_LEST and SHARE match > + @param[in] view TABLE_LIST of the view > + @param[in] share Share object of view > + > + @return false on error or misspatch > +*/ > > +bool mariadb_view_version_get(TABLE_SHARE *share) > +{ > + DBUG_ASSERT(share->is_view); > + > + if (!(share->tabledef_version.str= > + (uchar*) alloc_root(&share->mem_root, > + MICROSECOND_TIMESTAMP_BUFFER_SIZE))) > + return TRUE; > + > + DBUG_ASSERT(share->view_def != NULL); > + if (share->view_def->parse((uchar *) &share->tabledef_version, NULL, > + view_timestamp_parameters, 1, > + &file_parser_dummy_hook)) > + return TRUE; > + DBUG_ASSERT(share->tabledef_version.length == MICROSECOND_TIMESTAMP_BUFFER_SIZE-1); If you open an old view with YYYY-MM-DD... timestamp, where do you convert it? If you open a view without a timestamp at all, where do you handle it? > + return FALSE; > +} > > /** > read VIEW .frm and create structures > diff --git a/sql/sql_show.cc b/sql/sql_show.cc > index 9483db9eff9..138fa4bc631 100644 > --- a/sql/sql_show.cc > +++ b/sql/sql_show.cc > @@ -6724,13 +6724,15 @@ static bool store_trigger(THD *thd, Trigger *trigger, > table->field[14]->store(STRING_WITH_LEN("OLD"), cs); > table->field[15]->store(STRING_WITH_LEN("NEW"), cs); > > - if (trigger->create_time) > + if (trigger->ms_create_time) > { > + /* timestamp is in microseconds */ > table->field[16]->set_notnull(); > thd->variables.time_zone->gmt_sec_to_TIME(&timestamp, > - (my_time_t)(trigger->create_time/100)); > - /* timestamp is with 6 digits */ > - timestamp.second_part= (trigger->create_time % 100) * 10000; > + (my_time_t) > + (trigger->ms_create_time/ > + 1000000)); > + timestamp.second_part= trigger->ms_create_time % 1000000; hrtime_to_time() and hrtime_sec_part() > ((Field_temporal_with_date*) table->field[16])->store_time_dec(&timestamp, > 2); > } > @@ -9810,12 +9812,14 @@ static bool show_create_trigger_impl(THD *thd, Trigger *trigger) > trigger->db_cl_name.length, > system_charset_info); > > - if (trigger->create_time) > + if (trigger->ms_create_time) > { > MYSQL_TIME timestamp; > thd->variables.time_zone->gmt_sec_to_TIME(&timestamp, > - (my_time_t)(trigger->create_time/100)); > - timestamp.second_part= (trigger->create_time % 100) * 10000; > + (my_time_t) > + (trigger->ms_create_time/ > + 1000000)); > + timestamp.second_part= trigger->ms_create_time % 1000000; ditto > p->store(&timestamp, 2); > } > else Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 5

Re: [Maria-developers] commit 192fea6b104b249bb4764635bb3a52fb9c8a2148 MDEV-27128
by Alexey Botchkov 18 Sep '22

18 Sep '22

Hi, Rucha. I decided i dislike the overall design of the struct json_schema. Seems better to do it with C++ classes using virtual methods. So instead of structures we'd have classes as json_schema_const json_schema_enum json_scnema_string json_schema_number etc. they all inherited from the class json_schema_item { virtual parse() virtual check() } Parsing the schema we can behave like this: json_parse_schema() checks all the keys on the current level looking for 'const' 'enum' and 'typ[e' keywords only. If it gets 'const' or 'enum' it creates the json_schema_const/enum respectively For the "type" it creates the appropriate json_schema_xxx and callst ->parse() with the same JSON object. So the parse would go over it once again looking for the type-specific keywords. That way we don't need the hash_list as the destructor of the json_schema_xxx class would free it's resources. Minor comments. You compare keys like strncmp((const char*)curr_key, "const", key_len) all over the code. But it is not correct. That way 'cons' or 'con' or 'co' or even 'c' - they all will be considered equal to the 'const'. Just add the 'key_cmp()' function that would compare key lengths then do the strncmp. > double val= je->s.cs->strntod((char *) je->value, je->value_len, &end; You do this even if the value_type is not number. Not very safe. Better to avoid doing that here at all. That queriy just crashes the server. Didn't yet found why SET @schema_array= '{ "typ": "arra", "prefixItem": [ { "typ": "objec"}, { "typ": "objec", "propertie": { "number1": {"typ":"numbe"}, "string1": {"typ":"strin"}, "obj1" : {"typ":"objec", "propertie": { "key1" : {"type":"string"} } } }, "required":["obj1"] } ], "items":false }'; SELECT JSON_SCHEMA_VALID(@schema_array, '[{"key1":"value1"}, {"number1":1, "string1":"str1", "obj1":{"key1":"val1"}}]'); Feel free to discuss the design details! Best regards. HF

1 0

Re: [Maria-developers] e0e75de899f: MDEV-26715 Windows/installer - allow passwordless login for root
by Sergei Golubchik 13 Sep '22

13 Sep '22

Hi, Vladislav, Note, despite what the subject says, below is not for one commit, but for the diff over them all. Just a couple of questions and a comment: > diff --git a/libmariadb b/libmariadb > index a6665e65c05..22f7ed12b46 160000 > --- a/libmariadb > +++ b/libmariadb > @@ -1 +1 @@ > -Subproject commit a6665e65c05c00f9df983a0f788a643ca7bc084c > +Subproject commit 22f7ed12b4645a9fc5d66974551359373090c86b are auth_gssapi plugins using only standard Windows libraries? > diff --git a/mysql-test/main/mysql_install_db_win_utf8.test b/mysql-test/main/mysql_install_db_win_utf8.test > index fc67b66d3ca..0dc75af3914 100644 > --- a/mysql-test/main/mysql_install_db_win_utf8.test > +++ b/mysql-test/main/mysql_install_db_win_utf8.test > @@ -1,6 +1,6 @@ > --source include/windows.inc > --source include/check_utf8_cli.inc > - > +use mysql; Why? > # Create database in tmp directory using mysql_install_db.exe, > # and start server from this directory. > let $ddir= $MYSQLTEST_VARDIR/tmp/датадир; > diff --git a/plugin/auth_gssapi/mysql-test/auth_gssapi/suite.pm b/plugin/auth_gssapi/mysql-test/auth_gssapi/suite.pm > index e77ba05cb5c..71a6bf33b7b 100644 > --- a/plugin/auth_gssapi/mysql-test/auth_gssapi/suite.pm > +++ b/plugin/auth_gssapi/mysql-test/auth_gssapi/suite.pm > @@ -1,8 +1,9 @@ > + > package My::Suite::AuthGSSAPI; > > @ISA = qw(My::Suite); > > -return "No AUTH_GSSAPI plugin" unless $ENV{AUTH_GSSAPI_SO}; > +return "No AUTH_GSSAPI plugin" unless ($ENV{AUTH_GSSAPI_SO} or ($^O eq 'MSWin32')); The second condition normally is something like or $::mysqld_variables{gssapi} eq "ON"; > > return "Not run for embedded server" if $::opt_embedded_server; > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] f5a8327d016: MDEV-29458: Role grant commands do not propagate all grants
by Sergei Golubchik 12 Sep '22

12 Sep '22

Hi, Vicentiu, On Sep 12, Vicentiu Ciorbaru wrote: > revision-id: f5a8327d016 (mariadb-10.3.36-26-gf5a8327d016) > parent(s): fa75057b238 > author: Vicențiu Ciorbaru > committer: Vicențiu Ciorbaru > timestamp: 2022-09-05 16:40:10 +0300 > message: > > MDEV-29458: Role grant commands do not propagate all grants > > diff --git a/sql/sql_acl.cc b/sql/sql_acl.cc > index ba457083b75..3806fb31aa6 100644 > --- a/sql/sql_acl.cc > +++ b/sql/sql_acl.cc > @@ -2741,7 +2741,9 @@ static void acl_update_role(const char *rolename, ulong privileges) > { > ACL_ROLE *role= find_acl_role(rolename); > if (role) > + { > role->initial_role_access= role->access= privileges; > + } > } I wish you put this and the tab-to-spaces change in a separate commit. (even better - tab-to-spaces in a separate commit and this hunk - nowhere :) > @@ -9720,6 +9725,14 @@ static int handle_grant_struct(enum enum_acl_lists struct_no, bool drop, > remove_ptr_from_dynarray(&grantee->role_grants, acl_role); > } > > + /* Remove all of the role_grants from this user. */ > + while (acl_role->role_grants.elements) > + delete_dynamic_element(&acl_role->role_grants, > + acl_role->role_grants.elements - 1); this is a weird way of deleting all elements in the array. What's wrong with delete_dynamic(&acl_role->role_grants); ? > + > + /* all grants must be revoked from this role by now. propagate this */ > + propagate_role_grants(acl_role, PRIVS_TO_MERGE::ALL); > + > my_hash_delete(&acl_roles, (uchar*) acl_role); > DBUG_RETURN(1); > } > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 3f9e6c9c111: MDEV-22133 handle_fatal_signal (sig=11) on optimized builds in handle_grant_table instead of ERROR | Buffer overflow (on optimized builds)
by Sergei Golubchik 09 Sep '22

09 Sep '22

Hi, Oleksandr, On Sep 09, Oleksandr Byelkin wrote: > revision-id: 3f9e6c9c111 (mariadb-10.3.35-36-g3f9e6c9c111) > parent(s): 5feb60ce186 > author: Oleksandr Byelkin > committer: Oleksandr Byelkin > timestamp: 2022-06-24 15:37:25 +0200 > message: > > MDEV-22133 handle_fatal_signal (sig=11) on optimized builds in handle_grant_table instead of ERROR | Buffer overflow (on optimized builds) > > Return an error if we can not read a table required for ACL. > > diff --git a/sql/sql_acl.cc b/sql/sql_acl.cc > index 4c68b4505a4..38d2e9c35a1 100644 > --- a/sql/sql_acl.cc > +++ b/sql/sql_acl.cc > @@ -7420,6 +7420,11 @@ static bool grant_load(THD *thd, > while (!p_table->file->ha_index_next(p_table->record[0])); > } > } > + else > + { > + sql_print_error("Missing system table mysql.procs_priv; " > + "please run mysql_upgrade to create it"); > + } I don't think so. The comment earlier says p_table= procs_priv.table(); // this can be NULL and if "this can be NULL" then it's not an error if it is. > > end_unlock_p: > if (p_table) > @@ -10041,6 +10046,8 @@ static int handle_grant_data(THD *thd, Grant_tables& tables, bool drop, > } > > /* Handle stored routines table. */ > + if (tables.procs_priv_table().table_exists()) > + { This is ok, but also, please, check that all accesses to other optional tables are properly protected. See FIRST_OPTIONAL_TABLE define, all tables below it may not exist. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] ed477a6d30c: MDEV-29056 Replica thread reports error on ALTER ONLINE after LOCK WRITE
by Sergei Golubchik 09 Sep '22

09 Sep '22

Hi, Nikita, No, in this case (and commonly for collateral changes) I'd rather prefer to refine the original (878a92e) commit to not cause the MDEV-29056 in the first place. Ignoring LOCK= clause is logically correct and that's what the old code was doing anyway. But perhaps ALTER TABLE still needs to check that ALGORITHM= and LOCK= clauses are compatible? On Aug 27, Nikita Malyavin wrote: > revision-id: ed477a6d30c (mariadb-10.6.1-521-ged477a6d30c) > parent(s): 56206e60315 > author: Nikita Malyavin > committer: Nikita Malyavin > timestamp: 2022-07-26 00:41:00 +0300 > message: > > MDEV-29056 Replica thread reports error on ALTER ONLINE after LOCK WRITE > > Commit 878a92e (MDEV-28943) changes the behavior of ALTER TABLE under > prelocking. It now ignores passed LOCK= value in that case. > > However, LOCK TABLES command is never replicated, so the replica node > remains unaware of it. > > The solution would be to guess on the replica side on the mode used by > master. To make a deduction reliable, master's locked_tables_mode state > is passed for query log events. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] 68ebe0e2f5e: MDEV-29446 Change SHOW CREATE TABLE to display default collation
by Sergei Golubchik 09 Sep '22

09 Sep '22

Hi, Alexander, On Sep 09, Alexander Barkov wrote: > revision-id: 68ebe0e2f5e (mariadb-10.3.36-25-g68ebe0e2f5e) > parent(s): f501f815bc7 > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2022-09-07 08:10:58 +0400 > message: > > MDEV-29446 Change SHOW CREATE TABLE to display default collation > > diff --git a/sql/sql_show.cc b/sql/sql_show.cc > index a019c3c9dbd..cdf0bc01b6a 100644 > --- a/sql/sql_show.cc > +++ b/sql/sql_show.cc > @@ -2253,18 +2247,6 @@ int show_create_table(THD *thd, TABLE_LIST *table_list, String *packet, > { > packet->append(STRING_WITH_LEN(" CHARACTER SET ")); > packet->append(field->charset()->csname); > - } > - /* > - For string types dump collation name only if > - collation is not primary for the given charset > - > - For generated fields don't print the COLLATE clause if > - the collation matches the expression's collation. > - */ > - if (!(field->charset()->state & MY_CS_PRIMARY) && > - (!field->vcol_info || > - field->charset() != field->vcol_info->expr->collation.collation)) why, I think, this second condition could still apply, no need to remove it. > - { > packet->append(STRING_WITH_LEN(" COLLATE ")); > packet->append(field->charset()->name); > } Note that in 10.10 you need to fix json_table code inside a big if() condition: ... (!(m_explicit_cs->state & MY_CS_PRIMARY) && (str->append(STRING_WITH_LEN(" COLLATE ")) || str->append(&m_explicit_cs->coll_name))))) || ... Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

[Maria-developers] MDEV-25080 Allow pushdown of queries involving UNIONs in outer select to foreign engines
by Oleg Smirnov 09 Sep '22

09 Sep '22

Hello! The implementation for MDEV-25080 is ready and pushed to the *preview-10.11-mdev-25080-union-pushdown *branch. The FederatedX storage engine is adjusted accordingly. The example of how it works can be found at *mysql-test/federated/federatedx_create_handlers.test:* --echo # Pushdown of the whole UNION SELECT * from federated.t1 UNION SELECT * from federated.t2; EXPLAIN SELECT * from federated.t1 UNION SELECT * from federated.t2; --echo # Pushdown of a part of the UNION SELECT * from federated.t1 UNION SELECT * from t3; EXPLAIN SELECT * from federated.t1 UNION SELECT * from t3; -- Best regards, ------------------------- *Oleg Smirnov* Sr. Software Engineer MariaDB Server Team

1 0

[Maria-developers] Cosmetic input re: MDEV-25080 Allow pushdown of UNIONs to foreign engines
by Sergey Petrunia 06 Sep '22

06 Sep '22

Hi Oleg, Please find below coding style/etc input re the patch for MDEV-25080. (See the MDEV for other comments, more are coming). > diff --git a/sql/select_handler.cc b/sql/select_handler.cc > index 795ed8eb641..99bd2c0cf7f 100644 > --- a/sql/select_handler.cc > +++ b/sql/select_handler.cc > @@ -114,10 +125,7 @@ bool select_handler::send_data() > bool select_handler::send_eof() > { > DBUG_ENTER("select_handler::send_eof"); > - > - if (select->join->result->send_eof()) > - DBUG_RETURN(true); > - DBUG_RETURN(false); > + DBUG_RETURN(result->send_eof()); > } What is this change for? If it's just the code cleanup and we HAVE to do it (I don't see an urgent reason), please do it in a separate commit. Reasons against putting this change into this commit: When one sees this change as part of a meaningful patch, they start to examine the change to see what difference it makes. After one minute, one realizes the old and new code are the same. The time and attention have already been wasted, though. > diff --git a/sql/sql_explain.cc b/sql/sql_explain.cc > index 7c3918bfd20..6726e1b28f1 100644 > --- a/sql/sql_explain.cc > +++ b/sql/sql_explain.cc > @@ -555,6 +575,12 @@ int Explain_union::print_explain(Explain_query *query, > } > > if (!using_tmp) > + /* > + The union operation may not employ a temporary table, for example, > + for UNION ALL, in that case the results of the query are sent directly > + to the output. So there is no actual UNION operation and we don't need > + to print the line in the EXPLAIN output. > + */ > return 0; Please add { } braces as the if-part takes more than one line now. > diff --git a/sql/sql_lex.cc b/sql/sql_lex.cc > index 9382625a747..437acbb749c 100644 > --- a/sql/sql_lex.cc > +++ b/sql/sql_lex.cc > @@ -5549,9 +5552,13 @@ void st_select_lex::set_explain_type(bool on_the_fly) > type= "MATERIALIZED UNION"; > else > { > - type= is_uncacheable ? "UNCACHEABLE UNION": "UNION"; > + type= is_uncacheable ? "UNCACHEABLE UNION" > + : "UNION"; > Again: please don't just re-format the code > @@ -5561,15 +5568,14 @@ void st_select_lex::set_explain_type(bool on_the_fly) > { > bool uses_cte= false; > for (JOIN_TAB *tab= first_linear_tab(join, WITHOUT_BUSH_ROOTS, > - WITH_CONST_TABLES); > - tab; > - tab= next_linear_tab(join, tab, WITHOUT_BUSH_ROOTS)) > + WITH_CONST_TABLES); > + tab; tab= next_linear_tab(join, tab, WITHOUT_BUSH_ROOTS)) > { and here. > diff --git a/sql/sql_select.cc b/sql/sql_select.cc > index 45fed46e375..2310583074b 100644 > --- a/sql/sql_select.cc > +++ b/sql/sql_select.cc > @@ -1843,6 +1843,8 @@ int JOIN::optimize() > join_optimization_state init_state= optimization_state; > if (select_lex->pushdown_select) > { > + if (optimization_state == JOIN::OPTIMIZATION_DONE) > + return 0; Please add and assert that after this we have optimization_state==JOIN::NOT_OPTIMIZED. > // Do same as JOIN::optimize_inner does: > fields= &select_lex->item_list; > > @@ -4885,7 +4887,6 @@ void JOIN::cleanup_item_list(List<Item> &items) const > DBUG_VOID_RETURN; > } > > - Please put this back, the coding style says to use two empty lines between the functions. > /** > @brief > Look for provision of the select_handler interface by a foreign engine .. > -select_handler *find_select_handler(THD *thd, > - SELECT_LEX* select_lex) > +select_handler *find_select_handler(THD *thd, SELECT_LEX *select_lex) ^ Don't reformat. > { > - if (select_lex->next_select()) > - return 0; > if (select_lex->master_unit()->outer_select()) > return 0; > > @@ -4921,15 +4922,14 @@ select_handler *find_select_handler(THD *thd, > { > tbl= select_lex->join->tables_list; > } > - else if (thd->lex->query_tables && > - thd->lex->query_tables->next_global) > + else if (thd->lex->query_tables && thd->lex->query_tables->next_global) > { > tbl= thd->lex->query_tables->next_global; > } > else > return 0; > > - for (;tbl; tbl= tbl->next_global) > + for (; tbl; tbl= tbl->next_global) > { > if (!tbl->table) > continue; > @@ -4937,12 +4937,14 @@ select_handler *find_select_handler(THD *thd, > if (!ht->create_select) > continue; > select_handler *sh= ht->create_select(thd, select_lex); > - return sh; > + if (sh) > + return sh; > } > return 0; > } Don't reformat. meaningful changes are ok. > > > + Remove the 3rd empty line. > /** > An entry point to single-unit select (a select without UNION). > ... diff --git a/sql/sql_union.cc b/sql/sql_union.cc index e8bdbcba2f2..4cb9f3e025a 100644 --- a/sql/sql_union.cc +++ b/sql/sql_union.cc @@ -1294,6 +1294,52 @@ bool init_item_int(THD* thd, Item_int* &item) return true; } > +select_handler *find_unit_handler(THD *thd, > + SELECT_LEX_UNIT *unit) ^^^ coding style. ... > @@ -2053,6 +2106,7 @@ void st_select_lex_unit::optimize_bag_operation(bool is_outer_distinct) > bag_set_op_optimized= true; > } > > +select_handler *find_select_handler(THD *thd, SELECT_LEX *select_lex); Please don't have function declaration in the middle of a file... > @@ -2272,14 +2350,14 @@ bool st_select_lex_unit::exec() > if (union_result->flush()) > { > thd->lex->current_select= lex_select_save; > - DBUG_RETURN(1); > + return true; Please fix identation. > diff --git a/storage/federatedx/federatedx_pushdown.cc b/storage/federatedx/federatedx_pushdown.cc > index 664f0570238..492055773cb 100644 > --- a/storage/federatedx/federatedx_pushdown.cc > +++ b/storage/federatedx/federatedx_pushdown.cc > @@ -163,13 +163,13 @@ void ha_federatedx_derived_handler::print_error(int, unsigned long) > } > > > -static select_handler* > -create_federatedx_select_handler(THD* thd, SELECT_LEX *sel) > +template<typename T> > +static select_handler *create_federatedx_handler(THD *thd, T *sel_lex) > { > if (!use_pushdown) > return 0; > Do we need to generate that much code? What if there was a function like bool check_pushdown_allowed(select_lex_node *node) which could process both SELECT_LEX and SELECT_LEX_UNIT objects. Then, we would need two one-liner functions: create_select_handler(type S) { if (check_pushdown_allowed(S)) return new ha_federatedx_select_handler(S) } BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://petrunia.net

1 0

Re: [Maria-developers] 489a7fba324: MDEV-29322 ASAN heap-use-after-free in Query_log_event::do_apply_event
by Sergei Golubchik 04 Sep '22

04 Sep '22

Hi, Andrei, I still don't understand this. 1. Why rgi->options_to_bin_log is only set for GTID_EVENT? Is there a guarantee that there always be a GTID_EVENT before a QUERY_EVENT? What if gtids aren't enabled? 2. how do you guarantee that all query events for a previous value of options_written_to_bin_log have been applied before updating it in the rgi? 3. and if you do guarantee it, why is it atomic? No query events = nobody reads options_written_to_bin_log concurrently. And why not to do a simple patch, like the one I've attached? (note, it's an intentionally minimal patch to show the fix, practically I'd accompany it with a small cleanup patch) Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org On Sep 02, Andrei Elkin wrote: > > > Query_log_event cannot just randomly use some > > options_written_to_bin_log, it must use options_written_to_bin_log > > from the Format_description_log_event that came from the binlog file > > where the Query_log_event was read from. > > [ Not to engage you into a separate discussion, > still why 'randomly'? In the quoted description p.1 says > the Query_log_event uses `options_written_to_bin_log` from the > immediately preceding-in-binlog FD. ] > > > If rli->relay_log.description_event_for_exec is not that > > Format_description_log_event, and assuming that the correct > > Format_description_log_event is not stored anywhere, the Query_log_event > > has to remember options_written_to_bin_log itself, > > It's an option, but the chosen one avoids such remembering per each > Query event. Instead ... > > > there's not much we can do about it. > > > > Or it can apply the mask in the constructor and store not flags2 but > > the new value of thd->variables.option_bits. > > > > Or it can extend flags2 to cover all OPTIONS_WRITTEN_TO_BINLOG. > > ... of taking care of individual events the patch basically goes along > the base to use a placeholder that > A. lasts (unlike FD instance) the whole slave session time, and > B. the stored `options_written_to_binlog` is guaranteed to be a copy > of the most recent FD before the Query > > To help you to understand the method, here is a sequence diagram, > where on the top line > the slave SQL thread, its Relay-log-info object, a similar to that Worker's instance, > and a Worker thread. > > > SQL thread RLI/RGI Worker RGI Worker > | . . . > --->+ read FD . . i > | . . . > +--------------> > stores options > . > --->+ read Query > | . > +-------------------------->+ > finds Worker RGI > to store in theree a copy FD.options > | . > +-------------------------------------------->+ > schedules Query Worker | > . | > .<----------------+ > reads options | > | applies > ... Query > > It must be clear the A holds. > To prove B let after Query a second FD^2 is read by the SQL thread and > that FD^2 is from a different version server. > Then the SQL thread always have a wait-for-workers-to-become-idle > condition, so execution would go like this: > > --->+ read Query . ... > | . | > +-------------------------->+ | > finds Worker RGI | > to store in theree a copy FD.options | > | | > --->+ read FD^2 | applying of > | | Query > ... | > ... wait for Worker to complete Query | > ... | > +<============================================| > | > +-------------->. > | stores FD^2 > | options > ... > > So FD^2 can't affect queries from earlier binlogs. > > I am all ears to hear about your feedback. > > Cheers, > > Andrei

2 2

Re: [Maria-developers] 489a7fba324: MDEV-29322 ASAN heap-use-after-free in Query_log_event::do_apply_event
by Sergei Golubchik 01 Sep '22

01 Sep '22

Hi, Andrei, On Sep 01, Andrei wrote: > revision-id: 489a7fba324 (mariadb-10.5.17-10-g489a7fba324) > parent(s): 0324bde8469 > author: Andrei > committer: Andrei > timestamp: 2022-09-01 14:15:11 +0300 > message: > > MDEV-29322 ASAN heap-use-after-free in Query_log_event::do_apply_event > > The ASAN report was reported in the parallel slave execution of a Query > event and implicitly involved (so also parallelly run) Format-Description > event. > The Query actually had dependency on a preceding "old" FD whose > instance got destructed, so to cause the ASAN error. > > The case is fixed with copying of the value of > `options_written_to_bin_log` the sensitive member > of FD event into worker's stable placeholders, so FD change > won't affect Queries dependent on "old" instances. > > The method observes the following properties: > 0. each worker gets its private copy; > 1. the copied value gets a part of the execution context for any > Queries that follow the FD in binlog; > 2. any new FD forces a new copy which does not any affect on > the past (in binlog order) Queries which might be still in > execution. Sorry, but this doesn't make any sense. The value of options_written_to_bin_log depends on the version of the server that has written that binlog file. Query_log_event cannot just randomly use some options_written_to_bin_log, it must use options_written_to_bin_log from the Format_description_log_event that came from the binlog file where the Query_log_event was read from. If rli->relay_log.description_event_for_exec is not that Format_description_log_event, and assuming that the correct Format_description_log_event is not stored anywhere, the Query_log_event has to remember options_written_to_bin_log itself, there's not much we can do about it. Or it can apply the mask in the constructor and store not flags2 but the new value of thd->variables.option_bits. Or it can extend flags2 to cover all OPTIONS_WRITTEN_TO_BINLOG. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 5154e224a97: MDEV-5215 Granted to PUBLIC
by Sergei Golubchik 25 Aug '22

25 Aug '22

Hi, Oleksandr, > commit 5154e224a97 > Author: Oleksandr Byelkin <sanja(a)mariadb.com> > Date: Mon Dec 13 16:15:21 2021 +0100 > > diff --git a/scripts/mysql_system_tables.sql b/scripts/mysql_system_tables.sql > index fd2f1c95dda..a1721913b2c 100644 > --- a/scripts/mysql_system_tables.sql > +++ b/scripts/mysql_system_tables.sql > @@ -47,6 +47,10 @@ INSERT IGNORE INTO global_priv SELECT * FROM tmp_user_sys WHERE 0 <> @need_sys_u > DROP TABLE tmp_user_sys; > > > +-- This special "role" needed for GRAND ... TO PUBLIC > +INSERT IGNORE INTO global_priv (Host,User,Priv) VALUES ('', 'PUBLIC', concat('{"access":0,"version_id":',regexp_replace(regexp_replace(@@version, '\\b\\d\\b', '0\\0'), '\\D', ''),',"is_role":true}')); 1. why here and not in mysql_system_tables_data.sql ? 2. why this version magic? mysql_system_tables_data.sql doesn't do it for other accounts 3. why do you do it at all? I thought the server can create this role as needed > + > + > CREATE DEFINER='mariadb.sys'@'localhost' SQL SECURITY DEFINER VIEW IF NOT EXISTS user AS SELECT > Host, > User, > @@ -95,7 +99,9 @@ CREATE DEFINER='mariadb.sys'@'localhost' SQL SECURITY DEFINER VIEW IF NOT EXISTS > ELT(IFNULL(JSON_VALUE(Priv, '$.is_role'), 0) + 1, 'N', 'Y') AS is_role, > IFNULL(JSON_VALUE(Priv, '$.default_role'), '') AS default_role, > CAST(IFNULL(JSON_VALUE(Priv, '$.max_statement_time'), 0.0) AS DECIMAL(12,6)) AS max_statement_time > - FROM global_priv; > + FROM global_priv > +-- Do not show special role for GRANT TO PUBLIC > + WHERE not (Host = "" and User = "PUBLIC"); Why? I don't think it's wrong to show a new PUBLIC role. > > -- Remember for later if user table already existed > set @had_user_table= @@warning_count != 0; > diff --git a/sql/sql_acl.cc b/sql/sql_acl.cc > index 601cae2e945..aa9bd91ec52 100644 > --- a/sql/sql_acl.cc > +++ b/sql/sql_acl.cc > @@ -316,6 +318,13 @@ static bool show_table_and_column_privileges(THD *, const char *, const char *, > static int show_routine_grants(THD *, const char *, const char *, > const Sp_handler *sph, char *, int); > > +static ACL_ROLE *acl_public= NULL; > + > +inline privilege_t public_access() > +{ > + return (acl_public ? acl_public->access : NO_ACL); > +} Why not to have acl_public always not null? e.g. define it as static ACL_ROLE acl_public; so it's not even a pointer. And don't store it in the dynamic array. Or make it like static ACL_ROLE acl_public_dummy, *acl_public= &acl_public_dummy; then you can keep he real role in the dynamic array, but still acl_public will never be null. but in the standard PUBLIC is not a role at all, so may be it'd be simpler not to store it in the dynamic array > + > class Grant_tables; > class User_table; > class Proxies_priv_table; > @@ -3148,7 +3183,7 @@ bool acl_getroot(Security_context *sctx, const char *user, const char *host, > } > else // Role, not User > { > - ACL_ROLE *acl_role= find_acl_role(user); > + ACL_ROLE *acl_role= find_acl_role(user, false); are there tests for that? > if (acl_role) > { > res= 0; > @@ -3161,6 +3196,26 @@ bool acl_getroot(Security_context *sctx, const char *user, const char *host, > } > } > > + /* > + PUBLIC magic: > + > + Note: for usual user privileges of 3 component merged together: > + 1) user privileges > + 2) set role privileges > + 3) public privileges > + But for this routine (used in Security_context::change_security_context) > + only 2 component merged: > + 1) user OR role privileges we are switching to because if we're switching to a user, there can be no set role? I feel this comment is more confusing, then helping, really, I'd suggest to remove it completely > + 2) public privileges > + */ > + if (acl_public) > + { > + if (ACL_DB *acl_db= acl_db_find(db, public_name.str, "", "", FALSE)) > + sctx->db_access|= acl_db->access; > + > + sctx->master_access|= acl_public->access; > + } > + > mysql_mutex_unlock(&acl_cache->lock); > DBUG_RETURN(res); > } > @@ -3320,20 +3375,22 @@ int acl_setrole(THD *thd, const char *rolename, privilege_t access) > /* merge the privileges */ > Security_context *sctx= thd->security_ctx; > sctx->master_access= access; > - if (thd->db.str) > - sctx->db_access= acl_get(sctx->host, sctx->ip, sctx->user, thd->db.str, FALSE); > - > if (!strcasecmp(rolename, "NONE")) > { > thd->security_ctx->priv_role[0]= 0; > } > else > { > - if (thd->db.str) > - sctx->db_access|= acl_get("", "", rolename, thd->db.str, FALSE); > /* mark the current role */ > strmake_buf(thd->security_ctx->priv_role, rolename); > } > + if (thd->db.str) > + sctx->db_access= acl_get_all3(sctx, thd->db.str, FALSE); > + > + // PUBLIC magic > + if (acl_public) > + sctx->master_access|= acl_public->access; why not to do it in check_user_can_set_role()? Like - *access = acl_user->access | role->access; + *access = acl_user->access | role->access | acl_public->access; > + > return 0; > } > > @@ -3347,9 +3404,13 @@ static uchar* check_get_key(ACL_USER *buff, size_t *length, > > static void acl_update_role(const char *rolename, const privilege_t privileges) > { > - ACL_ROLE *role= find_acl_role(rolename); > + ACL_ROLE *role= find_acl_role(rolename, true); > if (role) > + { > role->initial_role_access= role->access= privileges; > + if (strcasecmp(rolename, public_name.str) == 0) > + acl_public= role; why? I mean, in what case can acl_public != role here? > + } > } > > > @@ -3638,6 +3703,23 @@ privilege_t acl_get(const char *host, const char *ip, > DBUG_RETURN(db_access & host_access); > } > > +/* > + Check if there is access for the host/user, role, public on the database > +*/ > + > +privilege_t acl_get_all3(Security_context *sctx, const char *db, > + bool db_is_patern) > +{ > + privilege_t access= acl_get(sctx->host, sctx->ip, > + sctx->priv_user, db, db_is_patern); > + if (sctx->priv_role[0]) > + access|= acl_get("", "", sctx->priv_role, db, db_is_patern); > + if (acl_public) > + access|= acl_get("", "", public_name.str, db, db_is_patern); how can this be optimized, considering that in most cases and for most db names there will be no grants to role and public? A couple of ideas: maintain counters total_number_of_db_grants_to_roles ad total_number_of_db_grants_to_public. This is rather crude. keep grants to users, to roles, and to public in separate caches. or in different dynamic arrays (won't help much in a default setup) keep propery per ACL_DB or shared between all ACL_DB's of one DB? that'd be rather tricky to maintain > + return access; > +} > + > + > /* > Check if there are any possible matching entries for this host > > @@ -3759,7 +3841,7 @@ static bool add_role_user_mapping(const char *uname, const char *hname, > const char *rname) > { > ACL_USER_BASE *grantee= find_acl_user_base(uname, hname); > - ACL_ROLE *role= find_acl_role(rname); > + ACL_ROLE *role= find_acl_role(rname, true); why? > > if (grantee == NULL || role == NULL) > return 1; > @@ -4247,7 +4329,7 @@ bool is_acl_user(const char *host, const char *user) > if (*host) // User > res= find_user_exact(host, user) != NULL; > else // Role > - res= find_acl_role(user) != NULL; > + res= find_acl_role(user, false) != NULL; this disallows DEFINER PUBLIC, right? I couldn't find a test for that. > > mysql_mutex_unlock(&acl_cache->lock); > return res; > @@ -7537,11 +7620,11 @@ bool mysql_grant_role(THD *thd, List <LEX_USER> &list, bool revoke) > if (user->host.str) > hostname= user->host; > else > - if ((role_as_user= find_acl_role(user->user.str))) > + if ((role_as_user= find_acl_role(user->user.str, false))) Why is it false here? It looks like it disallows GRANT role TO PUBLIC > hostname= empty_clex_str; > else > { > - if (is_invalid_role_name(username.str)) > + if (check_role_name(username.str, false) == ROLE_NAME_INVALID) > { > append_user(thd, &wrong_users, &username, &empty_clex_str); > result= 1; > @@ -8280,19 +8355,14 @@ bool check_grant(THD *thd, privilege_t want_access, TABLE_LIST *tables, > if (any_combination_will_do) > continue; > > - t_ref->grant.grant_table_user= grant_table; // Remember for column test > - t_ref->grant.grant_table_role= grant_table_role; > - t_ref->grant.version= grant_version; > - t_ref->grant.privilege|= grant_table ? grant_table->privs : NO_ACL; > - t_ref->grant.privilege|= grant_table_role ? grant_table_role->privs : NO_ACL; > + t_ref->grant.privilege|= t_ref->grant.aggregate_privs(); > t_ref->grant.want_privilege= ((want_access & COL_ACLS) & ~t_ref->grant.privilege); > > if (!(~t_ref->grant.privilege & want_access)) > continue; > > - if ((want_access&= ~((grant_table ? grant_table->cols : NO_ACL) | > - (grant_table_role ? grant_table_role->cols : NO_ACL) | > - t_ref->grant.privilege))) > + if ((want_access&= ~(t_ref->grant.aggregate_cols() | > + t_ref->grant.privilege))) > { > goto err; // impossible > } 1. why is this if() here, if it's impossible? 2. I'm a bit worried that you set t_ref->grant.version early when it's still possible to goto err. Old code was setting the version very late after all checks have succeeded. > @@ -8335,6 +8405,49 @@ static void check_grant_column_int(GRANT_TABLE *grant_table, const char *name, > } > } > > +inline privilege_t GRANT_INFO::aggregate_privs() > +{ > + return (grant_table_user ? grant_table_user->privs : NO_ACL) | > + (grant_table_role ? grant_table_role->privs : NO_ACL) | > + (grant_public ? grant_public->privs : NO_ACL); > +} > + > +inline privilege_t GRANT_INFO::aggregate_cols() > +{ > + return (grant_table_user ? grant_table_user->cols : NO_ACL) | > + (grant_table_role ? grant_table_role->cols : NO_ACL) | > + (grant_public ? grant_public->cols : NO_ACL); > +} > + > +void GRANT_INFO::refresh(const Security_context *sctx, > + const char *db, const char *table) > +{ > + if (version != grant_version) > + read(sctx, db, table); > +} > + > +void GRANT_INFO::read(const Security_context *sctx, > + const char *db, const char *table) > +{ > +#ifdef EMBEDDED_LIBRARY > + grant_table_user= grant_table_role= grant_public= NULL; > +#else > + grant_table_user= > + table_hash_search(sctx->host, sctx->ip, db, > + sctx->priv_user, > + table, FALSE); /* purecov: inspected */ you don't need to preserve purecov comments when moving the code around > + grant_table_role= > + sctx->priv_role[0] ? table_hash_search("", NULL, db, > + sctx->priv_role, > + table, TRUE) : NULL; > + grant_public= > + acl_public ? table_hash_search("", NULL, db, > + public_name.str, > + table, TRUE) : NULL; > +#endif > + version= grant_version; /* purecov: inspected */ > +} > + > /* > Check column rights in given security context > > @@ -9533,6 +9626,13 @@ static bool show_global_privileges(THD *thd, ACL_USER_BASE *acl_entry, > want_access= ((ACL_ROLE *)acl_entry)->initial_role_access; > else > want_access= acl_entry->access; > + > + // suppress "GRANT USAGE ON *.* TO `PUBLIC`" why? I agree that it's pointless, but disabling just it is kind of difficult to explain. > + if (!(want_access & ~GRANT_ACL) && > + acl_entry->user.length == public_name.length && > + strcasecmp(acl_entry->user.str, public_name.str) == 0) > + return FALSE; > + > if (test_all_bits(want_access, (GLOBAL_ACLS & ~ GRANT_ACL))) > global.append(STRING_WITH_LEN("ALL PRIVILEGES")); > else if (!(want_access & ~GRANT_ACL)) > @@ -11015,10 +11116,15 @@ bool mysql_drop_user(THD *thd, List <LEX_USER> &list, bool handle_as_role) > { > int rc; > user_name= get_current_user(thd, tmp_user_name, false); > - if (!user_name) > + if (!user_name || (handle_as_role && > + (strcasecmp(user_name->user.str, > + public_name.str) == 0))) better add this condition to the following if(), not to this one. Like: - if (handle_as_role != user_name->is_role()) + if (handle_as_role != user_name->is_role() || + (handle_as_role && check_role_name(...) == ROLE_NAME_INVALID)) > { > thd->clear_error(); > - append_str(&wrong_users, STRING_WITH_LEN("CURRENT_ROLE")); > + if (!user_name) > + append_str(&wrong_users, STRING_WITH_LEN("CURRENT_ROLE")); > + else > + append_str(&wrong_users, public_name.str, public_name.length); > result= TRUE; > continue; > } > diff --git a/sql/sql_show.cc b/sql/sql_show.cc > index 6290a47ee30..143e647f74c 100644 > --- a/sql/sql_show.cc > +++ b/sql/sql_show.cc > @@ -5304,7 +5300,7 @@ int get_all_tables(THD *thd, TABLE_LIST *tables, COND *cond) > &thd->col_access, NULL, 0, 1) || > (!thd->col_access && check_grant_db(thd, db_name->str))) || > sctx->master_access & (DB_ACLS | SHOW_DB_ACL) || > - acl_get(sctx->host, sctx->ip, sctx->priv_user, db_name->str, 0)) > + acl_get_all3(sctx, db_name->str, 0)) Was it a bug? That one couldn't see db's if privileges were granted via roles? > #endif > { > Dynamic_array<LEX_CSTRING*> table_names(PSI_INSTRUMENT_MEM); > diff --git a/mysql-test/main/public_basic.test b/mysql-test/main/public_basic.test > new file mode 100644 > index 00000000000..3f5993dfe97 > --- /dev/null > +++ b/mysql-test/main/public_basic.test > @@ -0,0 +1,93 @@ > +SHOW GRANTS FOR PUBLIC; > + > +--echo # it is not PUBLIC but an user > +--echo # (this should work as it allowed for roles for example) > +create user PUBLIC; > +create user PUBLIC@localhost; > +GRANT SELECT on test.* to PUBLIC@localhost; > +drop user PUBLIC@localhost; > +drop user PUBLIC; > + > +--echo # preinstalled PUBLIC > +GRANT SELECT on test.* to PUBLIC; > +GRANT SELECT on mysql.db to PUBLIC; > +--replace_regex /"version_id"\:[0-9]+/"version_id":VERSION/ > +select * from mysql.global_priv where user="PUBLIC" ; > + > +SHOW GRANTS FOR PUBLIC; > + > +GRANT UPDATE on test.* to PUBLIC; > +GRANT UPDATE on mysql.db to PUBLIC; > + > +SHOW GRANTS FOR PUBLIC; > + > +REVOKE SELECT on test.* from PUBLIC; > +REVOKE SELECT on mysql.db from PUBLIC; > + > +SHOW GRANTS FOR PUBLIC; > + > +REVOKE UPDATE on test.* from PUBLIC; > +REVOKE UPDATE on mysql.db from PUBLIC; > + > +--error ER_NONEXISTING_GRANT > +REVOKE UPDATE on test.* from PUBLIC; > +--error ER_NONEXISTING_TABLE_GRANT > +REVOKE UPDATE on mysql.db from PUBLIC; > + > +SHOW GRANTS FOR PUBLIC; > + > +--echo # automaticly added PUBLIC > +delete from mysql.global_priv where user="PUBLIC"; > +flush privileges; > +select * from mysql.global_priv where user="PUBLIC" ; > +GRANT SELECT on test.* to PUBLIC; > +GRANT SELECT on mysql.db to PUBLIC; > +--replace_regex /"version_id"\:[0-9]+/"version_id":VERSION/ > +select * from mysql.global_priv where user="PUBLIC" ; > + > +SHOW GRANTS FOR PUBLIC; > + > +GRANT UPDATE on test.* to PUBLIC; > +GRANT UPDATE on mysql.db to PUBLIC; > + > +SHOW GRANTS FOR PUBLIC; > + > +REVOKE SELECT on test.* from PUBLIC; > +REVOKE SELECT on mysql.db from PUBLIC; > + > +SHOW GRANTS FOR PUBLIC; > + > +REVOKE UPDATE on test.* from PUBLIC; > +REVOKE UPDATE on mysql.db from PUBLIC; > + > +SHOW GRANTS FOR PUBLIC; > + > +--error ER_INVALID_ROLE > +GRANT XXXXXX TO CURRENT_USER; > +--echo # following should fail with the same error as above > +--error ER_INVALID_ROLE > +GRANT PUBLIC TO CURRENT_USER; > + > +--error ER_INVALID_ROLE > +REVOKE XXXXXX FROM CURRENT_USER; > +--echo # following should fail with the same error as above > +--error ER_INVALID_ROLE > +REVOKE PUBLIC FROM CURRENT_USER; > +--error ER_CANNOT_USER > + > +drop role XXXXXX; > +--echo # following should fail with the same error as above > +--error ER_CANNOT_USER > +drop role PUBLIC; > + > +--error ER_INVALID_ROLE > +SET ROLE XXXXXX; > +--echo # following should fail with the same error as above > +--error ER_INVALID_ROLE > +SET ROLE PUBLIC; > + > +--error ER_INVALID_ROLE > +SET DEFAULT ROLE XXXXXX; > +--echo # following should fail with the same error as above > +--error ER_INVALID_ROLE > +SET DEFAULT ROLE PUBLIC; This is good, but one can update global_priv table directly and put the default_role=PUBLIC there. Please, add a test for that. and also a couple of tests for `public` (in backticks) > diff --git a/mysql-test/main/public_privileges.test b/mysql-test/main/public_privileges.test > new file mode 100644 > index 00000000000..a542376f05c > --- /dev/null > +++ b/mysql-test/main/public_privileges.test > @@ -0,0 +1,293 @@ > +--echo # > +--echo # Test DB/TABLE/COLUMN privileges in queries > +--echo # > + > +SHOW GRANTS FOR PUBLIC; > + > +create user testuser; > +create database testdb1; > +use testdb1; > +create table t1 (a int, b int); > +insert into t1 values (1,2); > +create database testdb2; > +use testdb2; > +create table t2 (a int, b int); > +insert into t2 values (1,2); > +create table t3 (a int, b int); > +insert into t3 values (1,2); > + > +connect (testuser,localhost,testuser,,); > +connection testuser; "connect testuser" also switches to this connection, the following "connection testuser" is redundant > +--error ER_TABLEACCESS_DENIED_ERROR > +select * from testdb1.t1; > +--error ER_TABLEACCESS_DENIED_ERROR > +select * from testdb2.t2; > +--error ER_TABLEACCESS_DENIED_ERROR > +select b from testdb2.t3; > +--error ER_TABLEACCESS_DENIED_ERROR > +select a from testdb2.t3; > + > +connection default; > + > +GRANT SELECT ON testdb1.* to PUBLIC; > +GRANT SELECT ON testdb2.t2 to PUBLIC; > +GRANT SELECT (b) ON testdb2.t3 to PUBLIC; > + > +disconnect testuser; > +connect (testuser,localhost,testuser,,); > +connection testuser; > +select * from testdb1.t1; > +select * from testdb2.t2; > +select b from testdb2.t3; > +--error ER_COLUMNACCESS_DENIED_ERROR > +select a from testdb2.t3; > + > +connection default; > + > +--replace_column 1 # 3 # 6 # 7 # > +SHOW PROCESSLIST; why? > + > +connection default; > + > +use test; > +disconnect testuser; > +REVOKE SELECT ON testdb1.* from PUBLIC; > +REVOKE SELECT ON testdb2.t2 from PUBLIC; > +REVOKE SELECT (b) ON testdb2.t3 from PUBLIC; may be `revoke all privileges, grant option` ? looks like a good place to have it tested > +drop user testuser; > +drop database testdb1; > +drop database testdb2; > + > +--echo # > +--echo # test global process list privilege and EXECUTE db level > +--echo # > + > +create user testuser; > +create database testdb; > +use testdb; > +create procedure p1 () select 1; > + > +connect (testuser,localhost,testuser,,); > +connection testuser; > + > +--replace_column 1 # 3 # 6 # 7 # > +SHOW PROCESSLIST; > +--error ER_PROCACCESS_DENIED_ERROR > +call testdb.p1(); > + > +connection default; > + > +GRANT PROCESS ON *.* to PUBLIC; > +GRANT EXECUTE ON testdb.* to PUBLIC; > + > +disconnect testuser; > +connect (testuser,localhost,testuser,,); > +connection testuser; > + > +--replace_column 1 # 3 # 6 # 7 # > +SHOW PROCESSLIST; > +call testdb.p1(); > + > +connection default; > + > +--replace_column 1 # 3 # 6 # 7 # > +SHOW PROCESSLIST; > + > +connection default; > + > +use test; > +disconnect testuser; > +REVOKE PROCESS ON *.* from PUBLIC; > +REVOKE EXECUTE ON testdb.* from PUBLIC; > +drop user testuser; > +drop database testdb; > + > +--echo # > +--echo # test DB privilege to allow USE statement > +--echo # > + > +create user testuser; > +create database testdb; > + > +connect (testuser,localhost,testuser,,); > +connection testuser; > + > +--error ER_DBACCESS_DENIED_ERROR > +use testdb; > + > +connection default; > + > +GRANT LOCK TABLES ON testdb.* to PUBLIC; > + > +disconnect testuser; > +connect (testuser,localhost,testuser,,); > +connection testuser; > + > +use testdb; > + > +connection default; > + > +use test; > +disconnect testuser; > +REVOKE LOCK TABLES ON testdb.* from PUBLIC; > +drop user testuser; > +drop database testdb; > + > + > +--echo # > +--echo # test DB privilege to allow USE statement (as above) > +--echo # test current db privileges > +--echo # > + > +create user testuser; > +create database testdb; > +use testdb; > +create table t1 (a int); > +insert into t1 values (1); > +GRANT LOCK TABLES ON testdb.* to PUBLIC; > + > +connect (testuser,localhost,testuser,,); > +connection testuser; > + > +use testdb; > +--error ER_TABLEACCESS_DENIED_ERROR > +update t1 set a=a+1; > + > +connection default; > + > +GRANT UPDATE,SELECT ON testdb.* to PUBLIC; > + > +disconnect testuser; > +connect (testuser,localhost,testuser,,); > +connection testuser; > + > +use testdb; > +update t1 set a=a+1; > + > +connection default; > +select * from testdb.t1; > + > +use test; > +disconnect testuser; > +REVOKE LOCK TABLES ON testdb.* from PUBLIC; > +REVOKE UPDATE,SELECT ON testdb.* from PUBLIC; > +drop user testuser; > +drop database testdb; > + > + > +--echo # > +--echo # test DB privilege to allow USE statement (as above) > +--echo # test table/column privileges in current DB > +--echo # > + > +create user testuser; > +create database testdb; > +use testdb; > +create table t1 (a int); > +insert into t1 values (1); > +create table t2 (a int, b int); > +insert into t2 values (1,2); > +GRANT LOCK TABLES ON testdb.* to PUBLIC; > + > +connect (testuser,localhost,testuser,,); > +connection testuser; > + > +use testdb; > +--error ER_TABLEACCESS_DENIED_ERROR > +delete from t1; > +--error ER_TABLEACCESS_DENIED_ERROR > +select b from t2; > +--error ER_TABLEACCESS_DENIED_ERROR > +select a from t2; > + > +connection default; > + > +GRANT DELETE ON testdb.t1 to PUBLIC; > +GRANT SELECT (a) ON testdb.t2 to PUBLIC; > + > +disconnect testuser; > +connect (testuser,localhost,testuser,,); > +connection testuser; > + > +use testdb; > +delete from t1; > +select a from t2; > +--error ER_COLUMNACCESS_DENIED_ERROR > +select b from t2; > + > +connection default; > +select * from testdb.t1; > + > +use test; > +disconnect testuser; > +REVOKE LOCK TABLES ON testdb.* from PUBLIC; > +REVOKE DELETE ON testdb.t1 from PUBLIC; > +REVOKE SELECT (a) ON testdb.t2 from PUBLIC; > +drop user testuser; > +drop database testdb; > + > +--echo # > +--echo # test function privilege > +--echo # > + > +create user testuser; > +create database testdb; > +use testdb; > +create function f1() returns int return 2; > + > +connect (testuser,localhost,testuser,,); > +connection testuser; > + > +--error ER_PROCACCESS_DENIED_ERROR > +alter function testdb.f1 comment "A stupid function"; > +--error ER_PROCACCESS_DENIED_ERROR > +select testdb.f1(); > + > +connection default; > + > +GRANT ALTER ROUTINE ON testdb.* to PUBLIC; > + > +disconnect testuser; > +connect (testuser,localhost,testuser,,); > +connection testuser; > + > +alter function testdb.f1 comment "A stupid function"; > +--error ER_PROCACCESS_DENIED_ERROR > +select testdb.f1(); > + > +connection default; > + > +use test; > +disconnect testuser; > +REVOKE ALTER ROUTINE ON testdb.* from PUBLIC; > +drop function testdb.f1; > +drop user testuser; > +drop database testdb; > + > +--echo # > +--echo # bug with automatically added PUBLIC role > +--echo # > + > +--echo # automaticly added PUBLIC > +delete from mysql.global_priv where user="PUBLIC"; > +flush privileges; > +GRANT SELECT on test.* to PUBLIC; > + > +REVOKE SELECT on test.* from PUBLIC; > + > +create user testuser; > +create database testdb1; > +use testdb1; > +create table t1 (a int, b int); > +insert into t1 values (1,2); > + > +connect (testuser,localhost,testuser,,); > +connection testuser; > +--error ER_TABLEACCESS_DENIED_ERROR > +select * from testdb1.t1; > + > +connection default; > + > +disconnect testuser; > +drop user testuser; > +drop database testdb1; > diff --git a/mysql-test/suite/roles/none_public.test b/mysql-test/suite/roles/none_public.test > index a0ec2315cfc..d0ba62d226b 100644 > --- a/mysql-test/suite/roles/none_public.test > +++ b/mysql-test/suite/roles/none_public.test > @@ -19,8 +19,9 @@ grant select on *.* to none; > grant public to role1; > --error ER_INVALID_ROLE > grant role1 to public; ^^^ this shouldn't fail > ---error ER_INVALID_ROLE > -grant select on *.* to public; > +# PUBLIC is legal role > +#--error ER_INVALID_ROLE > +#grant select on *.* to public; you can just remove it, commenting out isn't better > --error ER_INVALID_ROLE > grant role1 to current_role; > @@ -40,16 +41,18 @@ revoke select on *.* from public; > > --error ER_INVALID_ROLE > show grants for none; > ---error ER_INVALID_ROLE > -show grants for public; > +# PUBLIC is legal role > +#--error ER_INVALID_ROLE > +#show grants for public; > > --error ER_INVALID_ROLE > create definer=none view test.v1 as select 1; > ---error ER_INVALID_ROLE > -create definer=public view test.v1 as select 1; > +# PUBLIC is legal role > +#--error ER_INVALID_ROLE > +#create definer=public view test.v1 as select 1; eh, no, PUBLIC isn't a valid definer, as far as I understand. > > drop role role1; > > -insert mysql.global_priv values ('', 'none', '{"is_role":true}'), ('', 'public', '{"is_role":true}'); > +insert mysql.global_priv values ('', 'none', '{"is_role":true}'); > flush privileges; > -delete from mysql.global_priv where host=''; > +delete from mysql.global_priv where host='' and user='none'; > diff --git a/mysql-test/main/sp_notembedded.result b/mysql-test/main/sp_notembedded.result > index e03361598a6..239f8acb7f3 100644 > --- a/mysql-test/main/sp_notembedded.result > +++ b/mysql-test/main/sp_notembedded.result > @@ -345,7 +345,6 @@ show grants; > Grants for foo1@localhost > GRANT USAGE ON *.* TO `foo1`@`localhost` IDENTIFIED BY PASSWORD '-F3A2A51A9B0F2BE2468926B4132313728C250DBF' > GRANT CREATE ROUTINE ON `test`.* TO `foo1`@`localhost` > -GRANT EXECUTE, ALTER ROUTINE ON PROCEDURE `test`.`spfoo` TO `foo1`@`localhost` Why did this line disappear? > connection default; > disconnect foo; > drop procedure spfoo; Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] bcfc424d859: MDEV-5215 prerequisite: remove test and test_* database hacks in the test suite
by Sergei Golubchik 25 Aug '22

25 Aug '22

Hi, Oleksandr, On Aug 25, Oleksandr Byelkin wrote: > revision-id: bcfc424d859 (mariadb-10.6.1-483-gbcfc424d859) > parent(s): 1c3182a99e5 > author: Oleksandr Byelkin > committer: Oleksandr Byelkin > timestamp: 2022-08-16 11:49:41 +0200 > message: > > MDEV-5215 prerequisite: remove test and test_* database hacks in the test suite let's say, it's ok. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 1c3182a99e5: MDEV-5215 prerequisite of prerequisite: if DB is not mentioned in connect ignore errors of switching to it
by Sergei Golubchik 25 Aug '22

25 Aug '22

Hi, Oleksandr, On Aug 25, Oleksandr Byelkin wrote: > revision-id: 1c3182a99e5 (mariadb-10.6.1-482-g1c3182a99e5) > parent(s): 7253cdf8928 > author: Oleksandr Byelkin > committer: Oleksandr Byelkin > timestamp: 2022-08-16 11:49:41 +0200 > message: > > MDEV-5215 prerequisite of prerequisite: if DB is not mentioned in connect ignore errors of switching to it > > diff --git a/mysql-test/main/init_connect.test b/mysql-test/main/init_connect.test > index e96d02fe0d1..60549c3458e 100644 > --- a/mysql-test/main/init_connect.test > +++ b/mysql-test/main/init_connect.test > @@ -10,31 +10,31 @@ > > --source include/add_anonymous_users.inc > > -connect (con0,localhost,root,,); > +connect (con0,localhost,root,,test); why did you specify a db explicitly (here and in many other places)? I thought the whole point was to modify mysqltest in a way that would avoid the need of changing tests. > connection con0; > select hex(@a); Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] b3844107287: MDEV-16546 System versioning setting to allow history modification
by Sergei Golubchik 24 Aug '22

24 Aug '22

Hi, Aleksey! On Jun 08, Aleksey Midenkov wrote: > revision-id: b3844107287 (mariadb-10.5.2-424-gb3844107287) > parent(s): 27d66d644cf > author: Aleksey Midenkov <midenok(a)gmail.com> > committer: Aleksey Midenkov <midenok(a)gmail.com> > timestamp: 2021-03-01 17:43:34 +0300 > message: > > MDEV-16546 System versioning setting to allow history modification > > 1. force_fields_visible session variable makes system-invisible and > user-invisible fields visible on next table open. FLUSH TABLES > required for already opened tables. * system_versioning_insert_history * allows pseudocolumns ROW_START and ROW_END be specified in INSERT * doesn't affect invisible columns * FLUSH TABLES requirememnt is rather confusing, but let's have it, at least until I could suggest something better > 2. If secure_timestamp allows to modify timestamp variable then > following DML commands: INSERT, INSERT..SELECT and LOAD DATA can > specify row_start and row_end system field values. when system_versioning_insert_history=1 > 3. Cleaned up select_insert::send_data() from setting vers_write as > this parameter is now set on TABLE initialization. I don't see where it's set. This commit removes two table->vers_write= table->versioned(); and adds one from_field->table->vers_write= false; in Item_field::fix_fields. But where is vers_write set to true? Was it done in some other commit? > diff --git a/mysql-test/suite/versioning/r/insert.result b/mysql-test/suite/versioning/r/insert.result > index 2645d0184e8..00d1bb705c4 100644 > --- a/mysql-test/suite/versioning/r/insert.result > +++ b/mysql-test/suite/versioning/r/insert.result > @@ -112,3 +112,95 @@ x y > 9 9001 > drop table t1; > drop table t2; > +# > +# MDEV-16546 System versioning setting to allow history modification > +# > +# restart: --secure-timestamp=NO > +set @@session.time_zone='+00:00'; > +create table t1(x int) with system versioning; > +insert into t1(x) values (1); > +insert into t1(x, row_start, row_end) values (2, '1980-01-01 00:00:00', '1980-01-01 00:00:01'); > +ERROR 42S22: Unknown column 'row_start' in 'field list' > +set session force_fields_visible= 1; > +flush tables; > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `x` int(11) DEFAULT NULL, > + `row_start` timestamp(6) GENERATED ALWAYS AS ROW START, > + `row_end` timestamp(6) GENERATED ALWAYS AS ROW END, shouldn't be visible here, only in insert and load. > + PERIOD FOR SYSTEM_TIME (`row_start`, `row_end`) > +) ENGINE=DEFAULT_ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING > +insert into t1(x, row_start, row_end) values (3, '1980-01-01 00:00:00', '1980-01-01 00:00:01'); please, add tests for insert into t1 values (4) this should work, row_start/row_end must be mentioned explicitly. Also insert into t1 set x=5, row_start=..., row_end=... this should work too. And when only one row_start or row_end is mentioned as well. But update t1 set row_start=... insert into t1 (x, row_start, row_end) values (...) on duplicate key update row_start=... this should fail. While replace load data ... replace should work, but as DELETE+INSERT (I mean that DELETE in versioned tables doesn't actually delete history). If that's too difficult, then an error too. > +update t1 set x= x + 1; > +select x, check_row_ts(row_start, row_end) from t1 for system_time all order by x; ... > diff --git a/mysql-test/suite/versioning/t/insert.opt b/mysql-test/suite/versioning/t/insert.opt > new file mode 100644 > index 00000000000..a74d68957ef > --- /dev/null > +++ b/mysql-test/suite/versioning/t/insert.opt > @@ -0,0 +1 @@ > +--secure-timestamp=yes why is this? > -- source suite/versioning/common_finish.inc > diff --git a/mysql-test/suite/versioning/t/insert2.opt b/mysql-test/suite/versioning/t/insert2.opt > new file mode 100644 > index 00000000000..a74d68957ef > --- /dev/null > +++ b/mysql-test/suite/versioning/t/insert2.opt > @@ -0,0 +1 @@ > +--secure-timestamp=yes and this? > diff --git a/sql/sql_class.cc b/sql/sql_class.cc > index d23b190b2c5..93f8a8434e4 100644 > --- a/sql/sql_class.cc > +++ b/sql/sql_class.cc > @@ -7604,6 +7604,20 @@ void THD::set_last_commit_gtid(rpl_gtid &gtid) > #endif > } > > +bool THD::vers_modify_sys_field() const don't call it vers_modify_sys_field, your previous name vers_modify_history was better. There may be other sys fields that will never be directly writable (like ROWID, for example). > +{ > + if (lex->sql_command != SQLCOM_INSERT && > + lex->sql_command != SQLCOM_INSERT_SELECT && > + lex->sql_command != SQLCOM_LOAD) > + return false; I don't think this is enough to reject insert ... on duplicate key update row_start=xxx > + if (opt_secure_timestamp >= SECTIME_REPL || this is fixed in a later commit, so ok > + (opt_secure_timestamp == SECTIME_SUPER && > + !(security_ctx->master_access & SUPER_ACL))) > + return false; > + return true; > +} > + > + > void > wait_for_commit::reinit() > { > diff --git a/sql/sql_class.h b/sql/sql_class.h > index 2913a4a8350..520ea1d63bf 100644 > --- a/sql/sql_class.h > +++ b/sql/sql_class.h > @@ -820,6 +820,7 @@ typedef struct system_variables > > vers_asof_timestamp_t vers_asof_timestamp; > ulong vers_alter_history; > + my_bool force_fields_visible; system_versioning_insert_history > } SV; > > /** > @@ -5400,6 +5401,7 @@ class THD: public THD_count, /* this must be first */ > Item *sp_prepare_func_item(Item **it_addr, uint cols= 1); > bool sp_eval_expr(Field *result_field, Item **expr_item_ptr); > > + bool vers_modify_sys_field() const; vers_modify_history > }; > > > diff --git a/sql/sql_insert.cc b/sql/sql_insert.cc > index f0b84ab126f..84c2ae4ab36 100644 > --- a/sql/sql_insert.cc > +++ b/sql/sql_insert.cc > @@ -2014,7 +2014,7 @@ int write_record(THD *thd, TABLE *table, COPY_INFO *info, select_result *sink) > !table->file->referenced_by_foreign_key() && > (!table->triggers || !table->triggers->has_delete_triggers())) > { > - if (table->versioned(VERS_TRX_ID)) > + if (table->versioned(VERS_TRX_ID) && table->vers_write) does it even make sense? I mean direct insertion of transaction ids. > { > bitmap_set_bit(table->write_set, table->vers_start_field()->field_index); > table->file->column_bitmaps_signal(); > @@ -2029,7 +2029,7 @@ int write_record(THD *thd, TABLE *table, COPY_INFO *info, select_result *sink) > info->deleted++; > if (!table->file->has_transactions()) > thd->transaction->stmt.modified_non_trans_table= TRUE; > - if (table->versioned(VERS_TIMESTAMP)) > + if (table->versioned(VERS_TIMESTAMP) && table->vers_write) This (and the previous hunk) are in the DUP_REPLACE branch. There should be tests for these changes. > { > store_record(table, record[2]); > error= vers_insert_history_row(table); > diff --git a/sql/sys_vars.cc b/sql/sys_vars.cc > index f817f9086a9..4429308a097 100644 > --- a/sql/sys_vars.cc > +++ b/sql/sys_vars.cc > @@ -6737,3 +6737,7 @@ static Sys_var_ulonglong Sys_max_rowid_filter_size( > SESSION_VAR(max_rowid_filter_size), CMD_LINE(REQUIRED_ARG), > VALID_RANGE(1024, (ulonglong)~(intptr)0), DEFAULT(128*1024), > BLOCK_SIZE(1)); > + > +static Sys_var_mybool Sys_force_fields_visible( > + "force_fields_visible", "Make invisible fields visible on next table open", "system_versioning_insert_history" "Allows direct inserts into ROW_START and ROW_END columns" > + SESSION_VAR(force_fields_visible), CMD_LINE(OPT_ARG), DEFAULT(FALSE)); > diff --git a/sql/table.cc b/sql/table.cc > index f4bdbdeac5a..520f3738c9e 100644 > --- a/sql/table.cc > +++ b/sql/table.cc > @@ -4024,6 +4024,9 @@ enum open_frm_error open_table_from_share(THD *thd, TABLE_SHARE *share, > { > if (!((*field_ptr)= share->field[i]->clone(&outparam->mem_root, outparam))) > goto err; > + if (thd->variables.force_fields_visible && > + (*field_ptr)->invisible <= INVISIBLE_SYSTEM) > + (*field_ptr)->invisible= VISIBLE; Not really, system_versioning_insert_history should not affect visibility of anything besides ROW_START and ROW_END pseudocolumns in the INSERT and LOAD DATA statements. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 5

Re: [Maria-developers] Review request for MDEV-12325 Unexpected data type and truncation when using CTE
by Alexander Barkov 06 Aug '22

06 Aug '22

Hello Monty. The patch looks OK for me. I have just one question below: On 8/5/22 7:15 PM, Michael Widenius wrote: > Hi! > > Just to make things easy, here is the patch: > commit a556c4d972eaa09bb8ad586b9bda6cfb2a7b574a > Author: Monty <monty(a)mariadb.org> > Date: Fri Aug 5 17:57:27 2022 +0300 > > MDEV-12325 Unexpected data type and truncation when using CTE > > When creating a recursive CTE, the column types are taken from the > non recursive part of the CTE (this is according to the SQL standard). > > This patch adds code to abort the CTE if the calculated values in the > recursive part does not fit in the fields in the created temporary table. > > The new code only affects recursive CTE, so it should not cause any notable > problems for old applications. > <cut> > diff --git a/sql/sql_union.cc b/sql/sql_union.cc > index 48d8c16db68..d271a919a4e 100644 > --- a/sql/sql_union.cc > +++ b/sql/sql_union.cc > @@ -111,6 +111,8 @@ int select_unit::send_data(List<Item> &values) > { > int rc; > int not_reported_error= 0; > + enum_check_fields save_count_cuted_fields= thd->count_cuted_fields; > + > if (unit->offset_limit_cnt) > { // using limit offset,count > unit->offset_limit_cnt--; > @@ -118,8 +120,10 @@ int select_unit::send_data(List<Item> &values) > } > if (thd->killed == ABORT_QUERY) > return 0; > + > if (table->no_rows_with_nulls) > table->null_catch_flags= CHECK_ROW_FOR_NULLS_TO_REJECT; > + > if (intersect_mark) > { > fill_record(thd, table, table->field + 1, values, TRUE, FALSE); > @@ -127,6 +131,8 @@ int select_unit::send_data(List<Item> &values) > } > else > fill_record(thd, table, table->field, values, TRUE, FALSE); > + thd->count_cuted_fields= save_count_cuted_fields; > + > if (unlikely(thd->is_error())) > { > rc= 1; > @@ -297,7 +303,13 @@ bool select_unit::send_eof() > Why do you need above changes in select_unit::send_data()? Shouldn't it be enough just to have below changes in select_union_recursive::send_data()? > int select_union_recursive::send_data(List<Item> &values) > { > + Abort_on_warning_instant_set handle_abort(thd, 1); > + enum_check_fields save_count_cuted_fields= thd->count_cuted_fields; > + > + /* For recursive CTE's give warnings for wrong field info */ > + thd->count_cuted_fields= CHECK_FIELD_WARN; > int rc= select_unit::send_data(values); > + thd->count_cuted_fields= save_count_cuted_fields; > > if (rc == 0 && > write_err != HA_ERR_FOUND_DUPP_KEY && <cut>

1 1

Re: [Maria-developers] a507f126b03: MDEV-29159 Patch for MDEV-28918 introduces more inconsistency than it solves, breaks usability
by Sergei Golubchik 05 Aug '22

05 Aug '22

Hi, Alexander, a thought about the error message text and a couple of questions about changes in results, please, see below: On Aug 05, Alexander Barkov wrote: > revision-id: a507f126b03 (mariadb-10.7.4-39-ga507f126b03) > parent(s): 97d16c7544c > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2022-08-05 09:42:46 +0400 > message: > > MDEV-29159 Patch for MDEV-28918 introduces more inconsistency than it solves, breaks usability > > 1. Store assignment failures on incompatible data types now cause errors if: > - STRICT_ALL_TABLES or STRICT_TRANS_TABLES sql_mode is used, and > - IGNORE is not used > > Otherwise, only a warning is raised and the statement continues. > > 2. Changing the error/warning test as follows: > > -ERROR HY000: Illegal parameter data types inet6 and int for operation 'SET' > +ERROR HY000: Unknown CAST(int AS inet6) in assignment of 'target' Better not to write 'CAST(int AS inet6)' if there's no explicit CAST() function, I think it might be confusing. I'd say Cannot cast int as inet6 in assignment of 'target' > so in case of a big table it's easier to see which column has the problem. > The new error text is aslo applied to SP variables. s/aslo/also/ > diff --git a/mysql-test/main/sp-vars.result b/mysql-test/main/sp-vars.result > index 127a0993e2b..feef67ccb1d 100644 > --- a/mysql-test/main/sp-vars.result > +++ b/mysql-test/main/sp-vars.result > @@ -1026,11 +1026,11 @@ BEGIN > SELECT arg; > END| > CALL p1((1, 2)); > -ERROR HY000: Illegal parameter data types tinyint and row for operation 'SET' > +ERROR 21000: Operand should contain 1 column(s) Why not "Unknown CAST" like elsewhere? > diff --git a/mysql-test/main/type_geometry_mix_int.result b/mysql-test/main/type_geometry_mix_int.result > index e5d00c63727..0d0a21a8d29 100644 > --- a/mysql-test/main/type_geometry_mix_int.result > +++ b/mysql-test/main/type_geometry_mix_int.result > @@ -6,6 +6,7 @@ > # > CREATE TABLE t1 (target GEOMETRY DEFAULT POINT(1,1), source INT DEFAULT 0); > # Start of type_store_assignment_incompatible.inc > +SET @sql_mode_save= (SELECT @@sql_mode); strange, why not `SET @sql_mode_save= @@sql_mode` ? > SET @source_type= (SELECT COLUMN_TYPE FROM INFORMATION_SCHEMA.COLUMNS > WHERE COLUMN_NAME='source' > AND TABLE_NAME='t1' > @@ -47,6 +52,7 @@ INSERT INTO t3 VALUES > (2, > (SELECT source FROM t2 ORDER BY id LIMIT 1), > (SELECT source FROM t2 ORDER BY id LIMIT 1)); > +ERROR 22003: Cannot get geometry object from data you send to the GEOMETRY field What's that? It wasn't an error, and now it is? > SELECT * FROM v3; > id target source > 1 POINT(1 1) 0 Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 2

Re: [Maria-developers] f3fc58149dd: MDEV-29159 Patch for MDEV-28918 introduces more inconsistency than it solves, breaks usability
by Sergei Golubchik 05 Aug '22

05 Aug '22

Hi, Alexander, A couple of comments: On Aug 03, Alexander Barkov wrote: > revision-id: f3fc58149dd (mariadb-10.7.4-39-gf3fc58149dd) > parent(s): 97d16c7544c > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2022-08-03 11:28:31 +0400 > message: > > MDEV-29159 Patch for MDEV-28918 introduces more inconsistency than it solves, breaks usability > > Store assignment failures on incompatible data types now cause errors if: > - STRICT_ALL_TABLES or STRICT_TRANS_TABLES sql_mode is used, and > - IGNORE is not used > > Otherwise, only a warning is raised and the statement continues. > > TODO: add MTR tests for "loose" modes. > > diff --git a/sql/field.cc b/sql/field.cc > index 249269a6b1d..e997e2cbd69 100644 > --- a/sql/field.cc > +++ b/sql/field.cc > @@ -964,7 +964,7 @@ Type_handler::aggregate_for_result_traditional(const Type_handler *a, > } > > > -bool Field::check_assignability_from(const Type_handler *from) const > +bool Field::check_assignability_from(const Type_handler *from, bool error) const "error" is very confusing here, I had to read the whole patch to understand what it means. Better names: * only_warning * ignore * strict * issue_error > { > /* > Using type_handler_for_item_field() here to get the data type handler > @@ -982,6 +982,15 @@ bool Field::check_assignability_from(const Type_handler *from) const > type_handler_for_item_field()); > if (th.aggregate_for_result(from->type_handler_for_item_field())) > { > + if (!error) > + { > + THD *thd= get_thd(); > + push_warning_printf(thd, Sql_condition::WARN_LEVEL_WARN, > + ER_ILLEGAL_PARAMETER_DATA_TYPES2_FOR_OPERATION, > + ER_THD(thd, ER_ILLEGAL_PARAMETER_DATA_TYPES2_FOR_OPERATION), > + type_handler()->name().ptr(), from->name().ptr(), "SET"); > + return false; > + } > my_error(ER_ILLEGAL_PARAMETER_DATA_TYPES2_FOR_OPERATION, MYF(0), > type_handler()->name().ptr(), from->name().ptr(), "SET"); > return true; if you wouldn't want to change the error message in a followup commit, I'd suggested something like my_error(ER_ILLEGAL_PARAMETER_DATA_TYPES2_FOR_OPERATION, MYF(error ? 0 : ME_WARNING), type_handler()->name().ptr(), from->name().ptr(), "SET"); return error; Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

[Maria-developers] Review: MDEV-29075 Changing explicit_defaults_for_timestamp within stored procedure works inconsistently
by Alexander Barkov 01 Aug '22

01 Aug '22

Hello Sergei, I have a couple of suggestions: > commit 7b8304045272111a6f4d44196d6b37cbfef06f37 > Author: Sergei Golubchik <serg(a)mariadb.org> > Date: Wed Jul 20 17:31:48 2022 +0200 > > MDEV-29075 Changing explicit_defaults_for_timestamp within stored procedure works inconsistently > > move the OPTION_EXPLICIT_DEF_TIMESTAMP check from the parsing step > to the execution > <cut> > > diff --git a/sql/field.cc b/sql/field.cc > index 2e0c70d3d13..23ebb07b7f7 100644 > --- a/sql/field.cc > +++ b/sql/field.cc > @@ -10824,6 +10824,7 @@ Column_definition::Column_definition(THD *thd, Field *old_field, > comment= old_field->comment; > vcol_info= old_field->vcol_info; > option_list= old_field->option_list; > + explicitly_nullable= !(old_field->flags & NOT_NULL_FLAG); This introduces asymmetry in how NULL and NOT NULL attributes are handled. I suggest considering these ways: 1. Change "bool explicitly_nullable" to "bool explicit_nullability", which will be: - false if nothing was specified - true if NULL or NOT NULL was specified. 2. Or don't store NOT_NULL_FLAG in Column_definition::flags at all. Add a new Column_definition enum member with three values for: - not specified - explicit NULL - explicit NOT NULL As discussed on slack, the latter will need some changes. But I think not that much, as only Column_definition needs changes, while Field does not seem to need at this point. > compression_method_ptr= 0; > versioning= VERSIONING_NOT_SET; > invisible= old_field->invisible; > diff --git a/sql/field.h b/sql/field.h > index 5d5be5204cd..87a30bd9f95 100644 > --- a/sql/field.h > +++ b/sql/field.h > @@ -5281,7 +5281,7 @@ class Column_definition: public Sql_alloc, > uint flags, pack_length; > List<String> interval_list; > engine_option_value *option_list; > - > + bool explicitly_nullable; > > /* > This is additinal data provided for any computed(virtual) field. > @@ -5303,7 +5303,7 @@ class Column_definition: public Sql_alloc, > comment(null_clex_str), > on_update(NULL), invisible(VISIBLE), char_length(0), > flags(0), pack_length(0), > - option_list(NULL), > + option_list(NULL), explicitly_nullable(false), > vcol_info(0), default_value(0), check_constraint(0), > versioning(VERSIONING_NOT_SET), period(NULL) > { > diff --git a/sql/sql_table.cc b/sql/sql_table.cc > index 3eeabe20c71..8c31f988a05 100644 > --- a/sql/sql_table.cc > +++ b/sql/sql_table.cc > @@ -2316,6 +2316,8 @@ void promote_first_timestamp_column(List<Create_field> *column_definitions) > if (column_definition.is_timestamp_type() || // TIMESTAMP > column_definition.unireg_check == Field::TIMESTAMP_OLD_FIELD) // Legacy > { > + if (!column_definition.explicitly_nullable) > + column_definition.flags|= NOT_NULL_FLAG; I think this does not work well with PS. On the very first execution the field becomes NOT NULL forever, if the current OPTION_EXPLICIT_DEF_TIMESTAMP says so. > DBUG_PRINT("info", ("field-ptr:%p", column_definition.field)); > if (first && > (column_definition.flags & NOT_NULL_FLAG) != 0 && // NOT NULL, > diff --git a/sql/sql_type.cc b/sql/sql_type.cc > index 44a51428280..dc0c0b1e0cf 100644 > --- a/sql/sql_type.cc > +++ b/sql/sql_type.cc > @@ -4258,18 +4258,6 @@ void Type_handler_temporal_with_date::Item_update_null_value(Item *item) const > (void) item->get_date(thd, &ltime, Datetime::Options(thd)); > } > > -bool > -Type_handler_timestamp_common:: > -Column_definition_set_attributes(THD *thd, > - Column_definition *def, > - const Lex_field_type_st &attr, > - column_definition_type_t type) const > -{ > - Type_handler::Column_definition_set_attributes(thd, def, attr, type); > - if (!(thd->variables.option_bits & OPTION_EXPLICIT_DEF_TIMESTAMP)) > - def->flags|= NOT_NULL_FLAG; > - return false; > - > > void Type_handler_string_result::Item_update_null_value(Item *item) const > { > diff --git a/sql/sql_type.h b/sql/sql_type.h > index 7ff4bc64679..e5f54535ece 100644 > --- a/sql/sql_type.h > +++ b/sql/sql_type.h > @@ -6655,11 +6655,6 @@ class Type_handler_timestamp_common: public Type_handler_temporal_with_date > bool Item_func_min_max_get_date(THD *thd, Item_func_min_max*, > MYSQL_TIME *, date_mode_t fuzzydate) > const override; > - bool Column_definition_set_attributes(THD *thd, > - Column_definition *def, > - const Lex_field_type_st &attr, > - column_definition_type_t type) > - const override; > }; > > > diff --git a/sql/sql_yacc.yy b/sql/sql_yacc.yy > index 5416cec49b5..f326d8c0cc5 100644 > --- a/sql/sql_yacc.yy > +++ b/sql/sql_yacc.yy > @@ -5831,7 +5831,6 @@ field_def: > | opt_generated_always AS virtual_column_func > { > Lex->last_field->vcol_info= $3; > - Lex->last_field->flags&= ~NOT_NULL_FLAG; // undo automatic NOT NULL for timestamps > } > vcol_opt_specifier vcol_opt_attribute > { > @@ -6313,7 +6312,12 @@ attribute_list: > ; > > attribute: > - NULL_SYM { Lex->last_field->flags&= ~ NOT_NULL_FLAG; $$.init(); } > + NULL_SYM > + { > + Lex->last_field->flags&= ~NOT_NULL_FLAG; > + Lex->last_field->explicitly_nullable= true; > + $$.init(); > + } > | DEFAULT column_default_expr { Lex->last_field->default_value= $2; $$.init(); } > | ON UPDATE_SYM NOW_SYM opt_default_time_precision > {

3 3

[Maria-developers] review for (MDEV-28632) bugfix: DEFAULT NULL was allowed for NOT NULL columns
by Alexander Barkov 29 Jul '22

29 Jul '22

Hello Sergei, > commit 5a362d486b30fdaf3c7a360737331767154b4ee8 > Author: Sergei Golubchik <serg(a)mariadb.org> > Date: Mon Jul 18 22:53:27 2022 +0200 > > bugfix: DEFAULT NULL was allowed for NOT NULL columns > > this was detected during parsing (so NOT NULL DEFAULT NULL was an error), > but if a column was made NOT NULL later (e.g. as a part of primary key, > in mysql_prepare_create_table()), DEFAULT NULL was accepted and ignored. > > to fix this the NOT NULL DEFAULT NULL was moved into > mysql_prepare_create_table(). I'm not sure that in case of auto_increment we should really disallow DEFAULT NULL under terms of this TIMESTAMP task. Look at this script: DROP TABLE IF EXISTS t1; CREATE TABLE t1 (a INT NOT NULL AUTO_INCREMENT PRIMARY KEY); INSERT INTO t1 VALUES (NULL); Query OK, 1 row affected (0.012 sec) NULL can be inserted without errors. Yes, it's inserted with a side effect - it's replaced to the next auto increment value. But why this side effect makes NULL an invalid default? Note, I'm not against this change. Stricter behavior is good :) But I'm worried that: - It can break compatibility on one hand. - It goes without it's own MDEV on the other hand. This change deserves it's own MDEV (with motivation and consequences explained) and should be properly documented and mentioned in release notes. <cut> > index 712b8629c09..d8471894db9 100644 > --- a/mysql-test/main/view.result > +++ b/mysql-test/main/view.result > @@ -4082,12 +4082,12 @@ DROP TABLE t1,t2; > # MDEV-6251: SIGSEGV in query optimizer (in set_check_materialized > # with MERGE view) > # > -CREATE TABLE t1 (a1 INT(11) NOT NULL DEFAULT NULL AUTO_INCREMENT PRIMARY KEY); > -CREATE TABLE t2 (b1 INT(11) NOT NULL DEFAULT NULL AUTO_INCREMENT PRIMARY KEY); > -CREATE TABLE t3 (c1 INT(11) NOT NULL DEFAULT NULL AUTO_INCREMENT PRIMARY KEY); > -CREATE TABLE t4 (d1 INT(11) NOT NULL DEFAULT NULL AUTO_INCREMENT PRIMARY KEY); > -CREATE TABLE t5 (e1 INT(11) NOT NULL DEFAULT NULL AUTO_INCREMENT PRIMARY KEY); > -CREATE TABLE t6 (f1 INT(11) NOT NULL DEFAULT NULL AUTO_INCREMENT PRIMARY KEY); > +CREATE TABLE t1 (a1 INT(11) NOT NULL AUTO_INCREMENT PRIMARY KEY); > +CREATE TABLE t2 (b1 INT(11) NOT NULL AUTO_INCREMENT PRIMARY KEY); > +CREATE TABLE t3 (c1 INT(11) NOT NULL AUTO_INCREMENT PRIMARY KEY); > +CREATE TABLE t4 (d1 INT(11) NOT NULL AUTO_INCREMENT PRIMARY KEY); > +CREATE TABLE t5 (e1 INT(11) NOT NULL AUTO_INCREMENT PRIMARY KEY); > +CREATE TABLE t6 (f1 INT(11) NOT NULL AUTO_INCREMENT PRIMARY KEY); It seems you removed all cases with the old syntax. So now it's not recorded in MTR what happens on this statement: CREATE TABLE t1 (a INT NOT NULL DEFAULT NULL AUTO_INCREMENT PRIMARY KEY); Please cover this scenario in some relevant *.test file. <cut> > diff --git a/sql/sql_table.cc b/sql/sql_table.cc > index cf0bbfb1130..3eeabe20c71 100644 > --- a/sql/sql_table.cc > +++ b/sql/sql_table.cc <cut> > @@ -3647,6 +3643,15 @@ mysql_prepare_create_table(THD *thd, HA_CREATE_INFO *create_info, > { > Field::utype type= (Field::utype) MTYP_TYPENR(sql_field->unireg_check); > > + if (sql_field->default_value && > + sql_field->default_value->expr->type() == Item::NULL_ITEM && > + sql_field->flags & NOT_NULL_FLAG && > + !(sql_field->flags & AUTO_INCREMENT_FLAG)) > + { > + my_error(ER_INVALID_DEFAULT, MYF(0), sql_field->field_name.str); > + DBUG_RETURN(TRUE); > + } ^^^ now obvious errors are caught only during EXECUTE, although it's already clear at the PREPARE stage that the DEFAULT is wrong: MariaDB [test]> PREPARE stmt FROM 'CREATE TABLE T1 (a INT NOT NULL DEFAULT NULL)'; Query OK, 0 rows affected (0.000 sec) Statement prepared MariaDB [test]> EXECUTE stmt; ERROR 1067 (42000): Invalid default value for 'a' I suggest we still catch non-unambiguous cases (when nothing depends on system variables) during PREPARE. Inside some existing or a new Type_handler method. Also, please add PS tests: - for unambiguous scenarios like here - for ambiguous scenarios, e.g. with TIMESTAMP columns changing OPTION_EXPLICIT_DEF_TIMESTAMP between PREPARE and EXECUTE. It seems this code needs to be changed: > /* > Set NO_DEFAULT_VALUE_FLAG if this field doesn't have a default value and > it is NOT NULL, not an AUTO_INCREMENT field, not a TIMESTAMP and not > updated trough a NOW() function. > */ > if (!sql_field->default_value && > !sql_field->has_default_function() && > (sql_field->flags & NOT_NULL_FLAG) && > (!sql_field->is_timestamp_type() || > (thd->variables.option_bits & OPTION_EXPLICIT_DEF_TIMESTAMP))&& > !sql_field->vers_sys_field()) > { > sql_field->flags|= NO_DEFAULT_VALUE_FLAG; > sql_field->pack_flag|= FIELDFLAG_NO_DEFAULT; > } It can only add NO DEFAULT related flags. Shouldn't it also remove NO DEFAULT flags if the current PS EXECUTE time option OPTION_EXPLICIT_DEF_TIMESTAMP is unset? <cut> > diff --git a/storage/spider/mysql-test/spider/bg/t/spider_fixes.test b/storage/spider/mysql-test/spider/bg/t/spider_fixes.test > index b222f494ba1..41f87e61416 100644 > --- a/storage/spider/mysql-test/spider/bg/t/spider_fixes.test > +++ b/storage/spider/mysql-test/spider/bg/t/spider_fixes.test > @@ -294,14 +294,14 @@ if ($USE_CHILD_GROUP2) > --disable_query_log > echo CREATE TABLE ta_l ( > a int(11) NOT NULL DEFAULT '0', > - b char(1) DEFAULT NULL, > - c datetime DEFAULT NULL, > + b char(1), > + c datetime, > PRIMARY KEY (a, b, c) > ) MASTER_1_ENGINE MASTER_1_CHARSET MASTER_1_COMMENT5_2_1; > eval CREATE TABLE ta_l ( > a int(11) NOT NULL DEFAULT '0', > - b char(1) DEFAULT NULL, > - c datetime DEFAULT NULL, > + b char(1), > + c datetime, > PRIMARY KEY (a, b, c) > ) $MASTER_1_ENGINE $MASTER_1_CHARSET $MASTER_1_COMMENT5_2_1; > --enable_query_log Why this change ^^^ ? > diff --git a/storage/spider/mysql-test/spider/t/spider_fixes.test b/storage/spider/mysql-test/spider/t/spider_fixes.test > index 47bc225d614..75f108f1164 100644 > --- a/storage/spider/mysql-test/spider/t/spider_fixes.test > +++ b/storage/spider/mysql-test/spider/t/spider_fixes.test > @@ -294,14 +294,14 @@ if ($USE_CHILD_GROUP2) > --disable_query_log > echo CREATE TABLE ta_l ( > a int(11) NOT NULL DEFAULT '0', > - b char(1) DEFAULT NULL, > - c datetime DEFAULT NULL, > + b char(1), > + c datetime, > PRIMARY KEY (a, b, c) > ) MASTER_1_ENGINE MASTER_1_CHARSET MASTER_1_COMMENT5_2_1; > eval CREATE TABLE ta_l ( > a int(11) NOT NULL DEFAULT '0', > - b char(1) DEFAULT NULL, > - c datetime DEFAULT NULL, > + b char(1), > + c datetime, > PRIMARY KEY (a, b, c) > ) $MASTER_1_ENGINE $MASTER_1_CHARSET $MASTER_1_COMMENT5_2_1; > --enable_query_log Same question.

2 2

Re: [Maria-developers] Move on with Gtid strict mode
by Kristian Nielsen 13 Jul '22

13 Jul '22

Andrei Elkin <andrei.elkin(a)mariadb.com> writes: > Kristian, salve! Hi Andrei, thanks for interesting comments and discussion! >> 3. To solve (2), we recover the old master with >> --tc-heuristic-recover=rollback. This makes it rollback (=discard) any >> transaction that was not fully committed to disk (does it also roll back the >> binary log for such transactions? I don't recall this detail). >> >> 4. Because of (3), there might be a transaction that originated on the old >> master (has it's server_id), but which is now missing from that server (now >> a slave) as it was deliberately discarded. >> >> 5. To solve (4), we will use circular replication > > Well, the circularity is not necessary for (4). The ex-master just needs > to be configured as slave. Right, I see now. > Also let me call in the following this stage of receiving/accepting own > events as 'slave recovery'. But thinking more, there's another point that I don't understand, around the use of --tc-heuristic-recover=rollback. From memory, and also from reading docs on https://mariadb.com/kb/en/heuristic-recovery-with-the-transaction-coordinat…, this will forcibly roll back any prepared-but-not-committed InnoDB transaction, but _not_ roll back any corresponding binlog entry. Doesn't this leave the binlog inconsistent with the storage engine? Of the last few transactions in the binlog, some may be in InnoDB, some may not. And if so, doesn't this make the binlog useless for deciding which of our own transactions to ignore? When the old-master-now-slave gets one of its own events that's also in the binlog, we don't know if it is in the engine. If we apply it, we may end up with it double. If we do not apply it, we may be missing it. Is there some mechanims used in this feature/MDEV that handles this correctly that I'm missing? >> So isn't there a better way to achieve the underlying use case? For example, >> why not just use the new MASTER_DEMOTE_TO_SLAVE option to set the correct >> slave position from which to replicate from in the old master (now >> slave)? > > That's how it should normally do, yet it's an optimization (to save on > traffic) part of the solution. Interesting. Can you elaborate on this optimization - how does it save on traffic? > In the worst case `gtid_slave_pos` can be even empty for the > ex-master's domain, so the ex-master slave might keep receiving a bulk > of own events, only to ignore the most of them. > And that's the patch's base code logics as well. I guess there's something here that I'm not understanding. If MASTER_DEMOTE_TO_SLAVE is used to copy the gtid_binlog_pos into the gtid_slave_pos, how can the gtid_slave_pos be empty for the ex-master's domain? > ... , `gtid_ignore_duplicate = OFF` would have to go in my perception. > And if it goes, we'd remain with the only "choice" of > `gtid_ignore_duplicate = ON` :-) so the option would be gone. Hm. So are you suggesting that the current behaviour when --gtid-ignore-duplicates=ON becomes the default behaviour - in fact the only behaviour? As I remember it, the --gtid-ignore-duplicates=ON option is a very dangerous option unless the servers are managed very strictly and correctly. In fact I was expecting to find a dire warning written by myself about this, but didn't find anything in a quick web search ;-) I think the main problem is if an incorrect transaction done on a slave somewhere introduces an out-of-order GTID sequence, Then the replication will silently ignore potentially a large number of transactions that have their sequence number smaller than the incorrect out-of-order transaction. Out-of-order will normally be an error. But mistakes happen, and it is bad if this causes replication to become silently corrupt, as opposed to flagging the error (assuming --gtid-strict-mode=ON). What do you think? Again, it's cool that things are being worked on in MariaDB GTID, and I hope my comments don't come out as purely negative. While MariaDB/MySQL replication is in some ways overly complex, this is also what gives it the enormous flexibility for advanced users to do crazy^H^H^H^H^Hfantastic stuff! So there's some leeway for allowing complexity. But there are also some very delicate pitfalls that the original design tried hard to avoid, with decisions that may not always be obvious. So anything I can do there to help with history and design principles, I'm happy to assist with... - Kristian.

1 0

Re: [Maria-developers] ce5cc8fb905: MDEV-29021 ALTER TABLE fails when a stored virtual column is dropped+added
by Sergei Golubchik 13 Jul '22

13 Jul '22

Hi, Nikita, This is good, but I think fill_extra_persistent_columns() shouldn't be used at all. It doesn't handle default values. I've changed rpl_alter_extra_persistent test as -alter table t1 add column z1 int as(a+1) virtual, add column z2 int as (a+2) persistent; +alter table t1 add column z1 int as(a+1) virtual, add column z2 int default (a+2); and, of course, new column z2 wasn't updated properly. An easy fix would be to use your code in all cases and remove fill_extra_persistent_columns(). Could you do that too, please? And add the test case too. May be just, append , add column z3 int default (a+2) On Jul 07, Nikita Malyavin wrote: > revision-id: ce5cc8fb905 (mariadb-10.6.1-506-gce5cc8fb905) > parent(s): 49ad875902e > author: Nikita Malyavin > committer: Nikita Malyavin > timestamp: 2022-07-04 15:55:03 +0300 > message: > > MDEV-29021 ALTER TABLE fails when a stored virtual column is dropped+added > > We shouldn't rely on `fill_extra_persistent_columns`, as it only updates > fields which have an index > cols->n_bits (replication bitmap width). > > Normal update_virtual_fields+update_default_fields shoudl be done. > > diff --git a/sql/rpl_record.cc b/sql/rpl_record.cc > index 17731e64600..42f476fc112 100644 > --- a/sql/rpl_record.cc > +++ b/sql/rpl_record.cc > @@ -412,6 +412,26 @@ int unpack_row(rpl_group_info *rgi, TABLE *table, uint const colcnt, > { > copy->do_copy(copy); > } > + if (table->default_field) > + { > + error= table->update_default_fields(table->in_use->lex->ignore); > + if (unlikely(error)) > + DBUG_RETURN(error); > + } > + if (table->vfield) > + { > + error= table->update_virtual_fields(table->file, VCOL_UPDATE_FOR_WRITE); > + if (unlikely(error)) > + DBUG_RETURN(error); > + } > + } > + else > + { > + /* > + Add Extra slave persistent columns > + */ > + if (unlikely(error= fill_extra_persistent_columns(table, cols->n_bits))) > + DBUG_RETURN(error); > } > > /* > @@ -439,12 +459,6 @@ int unpack_row(rpl_group_info *rgi, TABLE *table, uint const colcnt, > } > } > > - /* > - Add Extra slave persistent columns > - */ > - if (unlikely(error= fill_extra_persistent_columns(table, cols->n_bits))) > - DBUG_RETURN(error); > - > /* > We should now have read all the null bytes, otherwise something is > really wrong. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 4

Re: [Maria-developers] 3e2d297830b: MDEV-29013 ER_KEY_NOT_FOUND/lock timeout upon online alter with long unique
by Sergei Golubchik 13 Jul '22

13 Jul '22

Hi, Nikita, On Jul 08, Nikita Malyavin wrote: > revision-id: 3e2d297830b (mariadb-10.6.1-511-g3e2d297830b) > parent(s): c1feb8c3c32 > author: Nikita Malyavin > committer: Nikita Malyavin > timestamp: 2022-07-05 15:20:04 +0300 > message: > > MDEV-29013 ER_KEY_NOT_FOUND/lock timeout upon online alter with long unique > > 1. ER_KEY_NOT_FOUND > Some virtual columns were not updated because were not included in read_set > to the moment of calculation. > > In Row_logs_event::do_apply_event some fields are excluded based on m_cols > value, the number of replicated rows. the number of replicated *columns* > We can't rely on this. Basically, we'd be satisfied, if all columns were > just set, at least for now. > > 2. ER_LOCK_WAIT_TIMEOUT > This is a long unique specific problem. > > Sometimes, lookup_handler is created for to->file. To properly free it, > ha_reset should be called. It is usually done by calling > close_thread_table, but ALTER TABLE makes it differently. Hence, a single > ha_reset call is added to mysql_alter_table. > > Also, the lifetime of lookup_handler is corrected: it's been allocated on > thd->mem_root, which in case of online alter > is event's mem_root, and can vary other ways. > It is changed to table->mem_root, which is more appropriate: > ha_reset is called only when table is closed, or in a few additional cases, > which correspond to a statement end. So memory leaks are unlikely here. > > diff --git a/sql/handler.cc b/sql/handler.cc > index d71c0fc83a3..fbebfd27203 100644 > --- a/sql/handler.cc > +++ b/sql/handler.cc > @@ -3212,7 +3212,7 @@ int handler::create_lookup_handler() > handler *tmp; > if (lookup_handler != this) > return 0; > - if (!(tmp= clone(table->s->normalized_path.str, table->in_use->mem_root))) > + if (!(tmp= clone(table->s->normalized_path.str, &table->mem_root))) that's wrong, the lifetime of a lookup handler is one statement, it's destroyed at the end. You cannot keep allocating new handlers on the table->mem_root for every statement. > return 1; > lookup_handler= tmp; > return lookup_handler->ha_external_lock(table->in_use, F_RDLCK); > diff --git a/sql/log_event_server.cc b/sql/log_event_server.cc > index 89ca2a41937..2c19bf67586 100644 > --- a/sql/log_event_server.cc > +++ b/sql/log_event_server.cc > @@ -6007,6 +6007,11 @@ int Rows_log_event::do_apply_event(rpl_group_info *rgi) > if (m_width == table->s->fields && bitmap_is_set_all(&m_cols)) > set_flags(COMPLETE_ROWS_F); > > + Rpl_table_data rpl_data{}; > + if (rgi) rgi->get_table_data(table, &rpl_data); > + > + if (!rpl_data.copy_fields) > + { > /* > Set tables write and read sets. > > @@ -6027,17 +6027,19 @@ int Rows_log_event::do_apply_event(rpl_group_info *rgi) > bitmap_set_all(table->read_set); > if (get_general_type_code() == DELETE_ROWS_EVENT || > get_general_type_code() == UPDATE_ROWS_EVENT) > bitmap_intersect(table->read_set,&m_cols); > > bitmap_set_all(table->write_set); > > /* WRITE ROWS EVENTS store the bitmap in m_cols instead of m_cols_ai */ > MY_BITMAP *after_image= ((get_general_type_code() == UPDATE_ROWS_EVENT) ? > &m_cols_ai : &m_cols); > bitmap_intersect(table->write_set, after_image); > > this->slave_exec_mode= slave_exec_mode_options; // fix the mode > + } I'm sorry, I don't quite understand. "some fields are excluded based on m_cols value" - what columns were excluded? What was the value of m_cols here? > table->rpl_write_set= table->write_set; > > // Do event specific preparations > error= do_before_row_operations(rli); > @@ -6045,8 +6052,6 @@ int Rows_log_event::do_apply_event(rpl_group_info *rgi) > extra columns on the slave. In that case, do not force > MODE_NO_AUTO_VALUE_ON_ZERO. > */ > - Rpl_table_data rpl_data{}; > - if (rgi) rgi->get_table_data(table, &rpl_data); > sql_mode_t saved_sql_mode= thd->variables.sql_mode; > if (!is_auto_inc_in_extra_columns()) > thd->variables.sql_mode= (rpl_data.copy_fields ? saved_sql_mode : 0) > diff --git a/sql/sql_table.cc b/sql/sql_table.cc > index 29c4d09cd39..a3efcdd9785 100644 > --- a/sql/sql_table.cc > +++ b/sql/sql_table.cc > @@ -11011,6 +11011,9 @@ do_continue:; > !(table->file->ha_table_flags() & HA_REUSES_FILE_NAMES) && > !(new_table->file->ha_table_flags() & > HA_REUSES_FILE_NAMES)); > + > + // Close lookup_handler. > + new_table->file->ha_reset(); please, put it not here, but inside the following thd->drop_temporary_table() call. It'll also fix Sanja's bug MDEV-26433. And please coordinate with him to put it on the same place so that git would detect a conflict and we could merge it to have only one reset call in that method, not two. > /* > Close the intermediate table that will be the new table, but do > not delete it! Even though MERGE tables do not have their children > @@ -11854,8 +11857,11 @@ copy_data_between_tables(THD *thd, TABLE *from, TABLE *to, > > // We'll be filling from->record[0] from row events > bitmap_set_all(from->write_set); > - // We restore bitmaps, because update event is going to mess up with them. > + // Use all columns. UPDATE/DELETE events reset read_set and write_set to > + // def_*_set after each row operation, so using all_set won't work. > to->default_column_bitmaps(); > + bitmap_set_all(&to->def_read_set); > + bitmap_set_all(&to->def_write_set); > > end_read_record(&info); > init_read_record_done= false; > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] 73c4fd41afc: Do not ignore sql_mode when replicating
by Sergei Golubchik 13 Jul '22

13 Jul '22

Hi, Nikita, On Jul 07, Nikita Malyavin wrote: > > diff --git a/sql/log_event_server.cc b/sql/log_event_server.cc > index aba846453e9..89ca2a41937 100644 > --- a/sql/log_event_server.cc > +++ b/sql/log_event_server.cc > @@ -6045,9 +6045,12 @@ int Rows_log_event::do_apply_event(rpl_group_info *rgi) > extra columns on the slave. In that case, do not force > MODE_NO_AUTO_VALUE_ON_ZERO. > */ > + Rpl_table_data rpl_data{}; > + if (rgi) rgi->get_table_data(table, &rpl_data); > sql_mode_t saved_sql_mode= thd->variables.sql_mode; > if (!is_auto_inc_in_extra_columns()) > - thd->variables.sql_mode= MODE_NO_AUTO_VALUE_ON_ZERO; > + thd->variables.sql_mode= (rpl_data.copy_fields ? saved_sql_mode : 0) you start using this condition in many places and it's quite cryptic for those, who don't know the code. I'd suggest to add method, like struct Rpl_table_data { ... bool is_online_alter() { return copy_fields != NULL; } ... }; btw, it fits very well in your Rpl_table_data simplification. > + | MODE_NO_AUTO_VALUE_ON_ZERO; Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] Move on with Gtid strict mode
by Kristian Nielsen 11 Jul '22

11 Jul '22

Hi Andrei, Good to hear from you, and to see that things are still going on with MariaDB GTID replication! I tried looking at the MDEVs that you referred to. Here is how I understand the motivation for this, correct me if I'm wrong: 1. We're considering a slave with "lossless semi-sync slave", where phantom reads are avoided by delaying storage engine commit until at least one slave has acknowledged a transaction. 2. If a master crashes, and comes back up after recovery, we want to prevent it from having any extra transactions that are not replicated to the rest of the replication servers (one of which will have been promoted as a new master). Such a transaction would prevent demoting the old master as a slave, as it would contain a rouge transaction not known to the rest of the replication topology. 3. To solve (2), we recover the old master with --tc-heuristic-recover=rollback. This makes it rollback (=discard) any transaction that was not fully committed to disk (does it also roll back the binary log for such transactions? I don't recall this detail). 4. Because of (3), there might be a transaction that originated on the old master (has it's server_id), but which is now missing from that server (now a slave) as it was deliberately discarded. 5. To solve (4), we will use circular replication so that the missing transaction gets replicated back to the old master (now slave). And we want to have a slave replicate its own transactions (replicate_same_sever_id), which is normally not done. And we want to use the server's binlog to know which transactions to replicate and which to ignore. Is that correctly understood? I must admit, this sounds horribly complex to me. So my question would be, do we really need to do this to solve the fundamental use case? In my belief, the main drawback with MariaDB replication and GTID is the complexity of especially the more advanced setups, which is barely feasible for even a skilled DBA to fully understand. Does this come from trying to port over some specific MySQL replication/GTID feature to MariaDB GTID? One important point is that MariaDB GTID was originally designed to keep the concept of master/binlog and slave/relaylog cleanly separated. A master transaction needs only be written to the binlog, there is no need to update any replication state (=overhead). Similarly, a slave transaction needs only be replicated to the storage engine, there is no need to write it to the binlog (=overhead). IIRC, MySQL (GTID) replication required the binlog (and --log-slave-updates) to be enabled on a slave) This is one of the improvements in the design of MariaDB GTID. So I don't particularly like the idea of now involving the binlog in how a slave replicates transactions. On the contrary, the MASTER_USE_GTID=current_pos feature is one thing I regret from the original GTID design, precisely because it violates the principle. And I was happy to see work to deprecate this, which I think is the prefered direction. So isn't there a better way to achieve the underlying use case? For example, why not just use the new MASTER_DEMOTE_TO_SLAVE option to set the correct slave position from which to replicate from in the old master (now slave)? And then use the existing --gtid-ignore-duplicate option? Since this is GTID-based circular replication, there must already be configured a separate gtid_domain_id for each master? Then you suggest to change the current --gtid-strict-mode and --gtid-ignore-duplicate options to a single option. I feel less strongly about this change; as you say, --gtid-strict-mode=OFF + --gtid-ignore-duplicates=ON may not make much sense. Still, I wonder what is the the need for such a change? This would be a not-backwards-compatible change, with little real benefit, as the semantics would be the same? Also, the --gtid-strict-mode option is an option that is highly recommended for any GTID use, and which should preferably be as simple as possible to understand to be able to be widely used by DBAs. On the contrary --gtid-ignore-duplicates is a very complex option only useful for very complex replication topologies, and with many subtle pitfalls that is very hard to fully understand. So there is real benefit to keeping them separate. As to the --gtid-strict-mode=OFF. The main original motivation for this option (and its default value of "OFF") was to be able to enable GTID by default, for any server installation upgraded (to was it 10.0 that introduced GTID? Long time back!). This is another big improvement (so I think) in MariaDB GTID over MySQL, that GTID is available by default, and compatible with any replication topology that works with old-style MySQL replication. Is it really so that no-one is using old-style, non-GTID replication any more, with all its sloppy setups possible, Sure, then non-strict GTID mode can be deprecated, I suppose - but it seems doubtful that this would be the case for *all* users? I hope I managed to explain where I didn't fully understand the motivation and need for some of these proposals? I'd be very interested to understand better any underlying necessities that I didn't catch from your mail, and maybe refine my comments. And once again, great to see the work still going on with MariaDB replication and GTID! Hope this helps, - Kristian. Andrei Elkin <andrei.elkin(a)mariadb.com> writes: > Two points of concern of this email deal with observations around GTID strict mode. > > > 1. About gtid_ignore_duplicates > > Introduced by > > hash: 2c2478b82260 > author: unknown > date: 2014-03-09 10:27:38 > > MDEV-5804: If same GTID is ... > Before, the arrival of same GTID twice in multi-source replication > would cause double-apply or in gtid strict mode an error. > > with a clear motivation to overcome the gtid strict mode error policy. > That is `gtid_ignore_duplicates = ON' softens `gtid_strict_mode = ON' > to ignore a duplicate gtid without a slave error/interruption. > As `gtid_strict_mode = OFF', `gtid_ignore_duplicates = ON' combination is > hardly ever relevant/practical, the `gtid_ignore_duplicates = ON' then can be fairly > perceived as `gtid_strict_mode=ON' 's "mild" submode to suggest > enum `gtid_stict_mode' { OFF, ON, "ON_BUT_IGNORE_DUPLICATES_RATHER_THAT_ERROR" } > which would obsolete `gtid_ignore_duplicates'. > > Why I'd prefer this change actually also deals with the circular replication > that - MDEV-28609 - forces us to do searching for duplicates > in the slave's binlog. > MDEV-28609 presents a case where a same server-id transaction received > by a semisync slave requires a more sophisticated decision making than > before (the same server-id based) for weather it to be ignored or > accepted. > > The accept policy of even own server-id gtids (transactions) is necessary for > MDEV-21117 semisync slave recovery, and the legacy > `replicate_same_sever_id' filtering can't help being too coarse as we need to consider > the gtid seq_no. > So under those conditions the same server-id transaction is made to be accepted > when it does not exist in the slave's binlog. > Otherwise, when it does exist in binlog, it'd be (like) a duplicate, arriving through > a circular topology from a "source" of the slave local binlog, only > to be ignored, like the same server-id gtids normally are. > The ignore policy in this case corresponds to the proposed > `gtid_stict_mode = "ON_BUT_IGNORE..."'. > > It should be naturally default as it inherits its pre-gtid equivalent of > `replicate_same_sever_id = NO' > and is safe (the slave's gtid state) for a real multi-source configuration as well. > > `gtid_stict_mode = ON' still would make sense as a method to > catch misconfigurations of the likes of Change-Master's do,ignore- domain_id ones. > > Your take? > > 2. But it's not clear what `gtid_strict_mode = OFF' is for. > > The ON mode is pretty flexible allowing gtid seq_no holes that is setting the gtid state > to any greater than the current (logical timestamp) value. > To set the gtid "clock" backward is still possible though requires slave restart. > > So how would you comment on an idea to deprecate OFF? > > Cheers, > > Andrei

1 0

Re: [Maria-developers] [Commits] 78370ea: MDEV-28965 Assertion failure when preparing UPDATE with derived table in WHERE
by Oleksandr Byelkin 11 Jul '22

11 Jul '22

Hi, Igor! Generally all looks good, but see my notes and couple of questions below. On Thu, Jul 7, 2022 at 6:57 AM IgorBabaev <igor(a)mariadb.com> wrote: > revision-id: 78370ea0fd21f66a4084d488ddcebac541249241 > (mariadb-10.6.1-477-g78370ea) > parent(s): 2db18fdb3d68d906fbd188ec570a64502ba55849 > author: Igor Babaev > committer: Igor Babaev > timestamp: 2022-07-06 21:57:23 -0700 > message: > > MDEV-28965 Assertion failure when preparing UPDATE with derived table in > WHERE > > This patch fixes not only the assertion failure in the function > Field_iterator_table_ref::set_field_iterator() but also: > - fixes the problem of forced materialization of derived tables used > in subqueries contained in WHERE clauses of single-table and multi-table > UPDATE and DELETE statements > - fixes the problem of MDEV-17954 that prevented execution of multi-table > DELETE statements if they use in their WHERE clauses references to > the tables that are updated. > > The patch must be considered a complement to the patch for MDEV-28883. > > --- > mysql-test/main/delete_use_source.result | 184 ++++++++++- > mysql-test/main/delete_use_source.test | 120 ++++++++ > mysql-test/main/derived.result | 336 > +++++++++++++++++++++ > mysql-test/main/derived.test | 210 +++++++++++++ > mysql-test/main/derived_cond_pushdown.result | 27 +- > mysql-test/main/derived_cond_pushdown.test | 2 +- > mysql-test/main/multi_update.result | 2 - > mysql-test/main/multi_update.test | 2 - > mysql-test/main/subselect.result | 1 - > mysql-test/main/subselect.test | 1 - > mysql-test/main/subselect_no_exists_to_in.result | 1 - > mysql-test/main/subselect_no_mat.result | 1 - > mysql-test/main/subselect_no_opts.result | 1 - > mysql-test/main/subselect_no_scache.result | 1 - > mysql-test/main/subselect_no_semijoin.result | 1 - > .../engines/iuds/r/update_delete_number.result | 22 +- > .../suite/engines/iuds/t/update_delete_number.test | 20 +- > sql/opt_subselect.cc | 56 +++- > sql/sql_base.cc | 41 ++- > sql/sql_delete.cc | 63 ++-- > sql/sql_delete.h | 14 +- > sql/sql_lex.cc | 28 +- > sql/sql_lex.h | 1 + > sql/sql_update.cc | 21 +- > sql/sql_update.h | 16 +- > sql/sql_yacc.yy | 13 + > sql/table.cc | 10 +- > 27 files changed, 1106 insertions(+), 89 deletions(-) > > [skip] > > --- a/mysql-test/suite/engines/iuds/t/update_delete_number.test > +++ b/mysql-test/suite/engines/iuds/t/update_delete_number.test > @@ -285,8 +285,18 @@ SELECT * FROM t1,t2 WHERE t2.c1=t1.c2; > DELETE FROM a1, a2 USING t1 AS a1 INNER JOIN t2 AS a2 WHERE a2.c1=a1.c2; > --sorted_result > SELECT * FROM t1,t2 WHERE t2.c1=t1.c2; > ---error ER_UPDATE_TABLE_USED > -DELETE FROM t1,t2 using t1,t2 where t1.c1=(select c1 from t1); > +TRUNCATE TABLE t1; > +TRUNCATE TABLE t2; > +INSERT INTO t1 VALUES(254,127,1),(0,-128,2),(1,127,3),(3,NULL,5); > +INSERT INTO t2 VALUES(127,255,1),(127,1,2),(-128,0,3),(-1,NULL,5); > +# After the patch for MDEV-28883 this should not report > +# --error ER_UPDATE_TABLE_USED anymore > Why is the above left as a comment? (the same in other cases) [skip] > diff --git a/sql/opt_subselect.cc b/sql/opt_subselect.cc > index fa338f0..ac47566 100644 > --- a/sql/opt_subselect.cc > +++ b/sql/opt_subselect.cc > @@ -30,6 +30,8 @@ > #include "sql_base.h" > #include "sql_const.h" > #include "sql_select.h" > +#include "sql_update.h" > +#include "sql_delete.h" > it would be nice to have comment why you include it as it only for one function as I can see, something like: #include "sql_update.h" // processing_as_multitable_update_prohibited (the same for delete and for including in #include "sql_update.h" & "sql_base.cc") #include "filesort.h" > #include "opt_subselect.h" > #include "sql_test.h" > @@ -532,6 +534,48 @@ bool is_materialization_applicable(THD *thd, > Item_in_subselect *in_subs, > return FALSE; > } > > [skip] > diff --git a/sql/sql_base.cc b/sql/sql_base.cc > index 2b41b78..447573b 100644 > --- a/sql/sql_base.cc > +++ b/sql/sql_base.cc > @@ -47,6 +47,8 @@ > #include "sql_prepare.h" > #include "sql_statistics.h" > #include "sql_cte.h" > +#include "sql_update.h" > +#include "sql_delete.h" > see above > #include <m_ctype.h> > #include <my_dir.h> > #include <hash.h> > @@ -1164,7 +1166,7 @@ TABLE_LIST* find_dup_table(THD *thd, TABLE_LIST > *table, TABLE_LIST *table_list, > /* > If we found entry of this table or table of SELECT which already > processed in derived table or top select of > multi-update/multi-delete > - (exclude_from_table_unique_test) or prelocking placeholder. > + (exclude_from_table_unique_test) or prelocking placeholder. > Please fix formatting above > */ > DBUG_PRINT("info", > ("found same copy of table or table which we should > skip")); > @@ -1175,16 +1177,43 @@ TABLE_LIST* find_dup_table(THD *thd, TABLE_LIST > *table, TABLE_LIST *table_list, > We come here for queries of type: > INSERT INTO t1 (SELECT tmp.a FROM (select * FROM t1) as tmp); > > - Try to fix by materializing the derived table > + Try to fix by materializing the derived table if one can't do > without it. > */ > TABLE_LIST *derived= res->belong_to_derived; > if (derived->is_merged_derived() && !derived->derived->is_excluded()) > { > - DBUG_PRINT("info", > + bool materialize= true; > + if (thd->lex->sql_command == SQLCOM_UPDATE) > + { > + Sql_cmd_update *cmd= (Sql_cmd_update *) (thd->lex->m_sql_cmd); > + if (cmd->is_multitable()) > + materialize= false; > + else if (!cmd->processing_as_multitable_update_prohibited(thd)) > + { > + cmd->set_as_multitable(); > + materialize= false; > + } > + } > + else if (thd->lex->sql_command == SQLCOM_DELETE) > + { > + Sql_cmd_delete *cmd= (Sql_cmd_delete *) (thd->lex->m_sql_cmd); > + if (cmd->is_multitable()) > + materialize= false; > + if (!cmd->processing_as_multitable_delete_prohibited(thd)) > + { > + cmd->set_as_multitable(); > + materialize= false; > + } > + } > + if (materialize) > + { > + DBUG_PRINT("info", > ("convert merged to materialization to resolve the > conflict")); > - derived->change_refs_to_fields(); > - derived->set_materialized_derived(); > - goto retry; > + derived->change_refs_to_fields(); > + derived->set_materialized_derived(); > + // derived->field_translation= 0; > above comment has no big sens, looks not cleaned up some chenges? > + goto retry; > + } > } > } > DBUG_RETURN(res); > > [skip] > diff --git a/sql/sql_delete.h b/sql/sql_delete.h > index e1d5044..ffb8173 100644 > --- a/sql/sql_delete.h > +++ b/sql/sql_delete.h > @@ -44,11 +44,12 @@ class Sql_cmd_delete final : public Sql_cmd_dml > { > public: > Sql_cmd_delete(bool multitable_arg) > - : multitable(multitable_arg), save_protocol(NULL) {} > + : orig_multitable(multitable_arg), save_protocol(NULL) > + { multitable= orig_multitable; } > why do not use the same initializers? : multitable(multitable_arg), rig_multitable(multitable_arg), save_protocol(NULL) {} > enum_sql_command sql_command_code() const override > { > - return multitable ? SQLCOM_DELETE_MULTI : SQLCOM_DELETE; > + return orig_multitable ? SQLCOM_DELETE_MULTI : SQLCOM_DELETE; > } > > DML_prelocking_strategy *get_dml_prelocking_strategy() > > [skip] > diff --git a/sql/sql_lex.cc b/sql/sql_lex.cc > index 78c067a..4059c0e 100644 > --- a/sql/sql_lex.cc > +++ b/sql/sql_lex.cc > @@ -37,6 +37,8 @@ > #include "sql_partition.h" > #include "sql_partition_admin.h" // > Sql_cmd_alter_table_*_part > #include "event_parse_data.h" > +#include "sql_update.h" > +#include "sql_delete.h" > see above > > void LEX::parse_error(uint err_number) > { > @@ -4037,9 +4039,8 @@ bool LEX::can_use_merged() > SYNOPSIS > LEX::can_not_use_merged() > > - @param no_update_or_delete Set to 1 if we can't use merge with > multiple-table > - updates, like when used from > - TALE_LIST::init_derived() > + @param forced_no_merge_for_update_delete Set to 1 if we can't use merge > with > + multiple-table updates/deletes > > DESCRIPTION > Temporary table algorithm will be used on all SELECT levels for > queries > @@ -4050,7 +4051,7 @@ bool LEX::can_use_merged() > TRUE - VIEWs with MERGE algorithms can be used > */ > > -bool LEX::can_not_use_merged(bool no_update_or_delete) > +bool LEX::can_not_use_merged(bool forced_no_merge_for_update_delete) > { > switch (sql_command) { > case SQLCOM_CREATE_VIEW: > @@ -4064,18 +4065,29 @@ bool LEX::can_not_use_merged(bool > no_update_or_delete) > return TRUE; > > case SQLCOM_UPDATE_MULTI: > - case SQLCOM_DELETE_MULTI: > - if (no_update_or_delete) > + if (forced_no_merge_for_update_delete) > return TRUE; > /* Fall through */ > > case SQLCOM_UPDATE: > - if (no_update_or_delete && m_sql_cmd && > - (m_sql_cmd->sql_command_code() == SQLCOM_UPDATE_MULTI || > + if (forced_no_merge_for_update_delete && > + (((Sql_cmd_update *) m_sql_cmd)->is_multitable() || > query_tables->is_multitable())) > return TRUE; > + return FALSE; > + > + case SQLCOM_DELETE_MULTI: > + if (forced_no_merge_for_update_delete) > + return TRUE; > /* Fall through */ > > + case SQLCOM_DELETE: > + if (forced_no_merge_for_update_delete && > + (((Sql_cmd_delete *) m_sql_cmd)->is_multitable() || > + query_tables->is_multitable())) > + return TRUE; > + return FALSE; > + > default: > return FALSE; > } > diff --git a/sql/sql_lex.h b/sql/sql_lex.h > index e733b4f..402e3bd 100644 > > [skip] > diff --git a/sql/sql_update.cc b/sql/sql_update.cc > index 6b14c4f..7769777 100644 > --- a/sql/sql_update.cc > +++ b/sql/sql_update.cc > @@ -2807,6 +2807,23 @@ bool multi_update::send_eof() > > > /** > + @brief Check whether conversion to multi-table update is prohibited > + > + @param thd global context the processed statement > + @returns true if conversion is prohibited, false otherwise > + > + @todo > + Introduce handler level flag for storage engines that would prohibit > + such conversion for any single-table update. > +*/ > + > +bool Sql_cmd_update::processing_as_multitable_update_prohibited(THD *thd) > +{ > + return false; > +} > + > + > +/** > @brief Perform precheck of table privileges for update statements > > @param thd global context the processed statement > @@ -2889,7 +2906,9 @@ bool Sql_cmd_update::prepare_inner(THD *thd) > "updating and querying the same temporal periods table"); > DBUG_RETURN(TRUE); > } > - multitable= true; > + if (!table_list->is_multitable() && > + !processing_as_multitable_update_prohibited(thd)) > + multitable= true; > } > } > > diff --git a/sql/sql_update.h b/sql/sql_update.h > index d0fc7cb..4aff77a 100644 > --- a/sql/sql_update.h > +++ b/sql/sql_update.h > @@ -46,12 +46,12 @@ class Sql_cmd_update final : public Sql_cmd_dml > { > public: > Sql_cmd_update(bool multitable_arg) > - : multitable(multitable_arg) > - { } > + : orig_multitable(multitable_arg) > + { multitable= orig_multitable; } > why do not use construction for both? Sql_cmd_update(bool multitable_arg) : multitable(multitable_arg) : orig_multitable(multitable_arg) {} > > enum_sql_command sql_command_code() const override > { > - return multitable ? SQLCOM_UPDATE_MULTI : SQLCOM_UPDATE; > + return orig_multitable ? SQLCOM_UPDATE_MULTI : SQLCOM_UPDATE; > } > > DML_prelocking_strategy *get_dml_prelocking_strategy() > @@ -59,6 +59,12 @@ class Sql_cmd_update final : public Sql_cmd_dml > return &multiupdate_prelocking_strategy; > } > > + bool processing_as_multitable_update_prohibited(THD *thd); > + > + bool is_multitable() { return multitable; } > + > + void set_as_multitable() { multitable= true; } > + > protected: > /** > @brief Perform precheck of table privileges for update statements > @@ -89,13 +95,15 @@ class Sql_cmd_update final : public Sql_cmd_dml > */ > bool multitable; > > + /* Original value of the 'multitable' flag set by constructor */ > + const bool orig_multitable; > + > /* The prelocking strategy used when opening the used tables */ > Multiupdate_prelocking_strategy multiupdate_prelocking_strategy; > > public: > /* The list of the updating expressions used in the set clause */ > List<Item> *update_value_list; > - > }; > > #endif /* SQL_UPDATE_INCLUDED */ > diff --git a/sql/sql_yacc.yy b/sql/sql_yacc.yy > index a587a37..13dc602 100644 > --- a/sql/sql_yacc.yy > +++ b/sql/sql_yacc.yy > @@ -13370,8 +13370,21 @@ delete_single_table: > YYPS->m_lock_type, > YYPS->m_mdl_type, > NULL, > + 0))) > + MYSQL_YYABORT; > + Select->table_list.save_and_clear(&Lex->auxiliary_table_list); > + Lex->table_count= 1; > Why it is not enough to have this counter (table_count) set by open_tables? May be it would be better to use mysql_init_select() to set everything needed by SELECT for sure? > + Lex->query_tables= 0; > + Lex->query_tables_last= &Lex->query_tables; > + if (unlikely(!Select-> > + add_table_to_list(thd, $2, NULL, > TL_OPTION_UPDATING, > + YYPS->m_lock_type, > + YYPS->m_mdl_type, > + NULL, > $3))) > MYSQL_YYABORT; > + Lex->auxiliary_table_list.first->correspondent_table= > + Lex->query_tables; > YYPS->m_lock_type= TL_READ_DEFAULT; > YYPS->m_mdl_type= MDL_SHARED_READ; > } > [skip]

1 0

[Maria-developers] Account locked and USERS view?
by Martijn Tonies (Upscene Productions) 06 Jul '22

06 Jul '22

Hi, Any reason why the ‘account locked’ attribute isn’t included in the USERS view? https://mariadb.com/kb/en/account-locking/ It could be added like this: if(json_value(mysql.global_priv.Priv, '$.account_locked') = 1, 'Y', 'N') AS is_locked With regards, Martijn Tonies Upscene Productions https://www.upscene.com Database Workbench - developer tool for Oracle, SQL Server, PostgreSQL, MySQL, InterBase, NexusDB and Firebird.

1 0

Re: [Maria-developers] MDEV-28881 Server crashes in Dep_analysis_context::create_table_value
by Sergey Petrunia 05 Jul '22

05 Jul '22

Hi Oleg, > commit 812f8cbfb3a7a02705f65d8bb7e30ae2faf13b5d > Author: Oleg Smirnov <olernov(a)gmail.com> > Date: Tue Jun 21 16:20:17 2022 +0700 > > MDEV-28881 Server crashes in Dep_analysis_context::create_table_value > > SELECT_LEX::first_select()->join may be NULL for some scenarios. > This commit adds a check for this. Looking into Jira I can infer that the patch fixes two issues, but the commit comment doesn't say that. Please avoid doing this in the future. Please split the commit into two: - first one fixing the SELECT_LEX::first_select()->join == NULL part - the second fixing the memory management. For the second part, I think it's not a good approach when most of the Table Elimination data resides in a MEM_ROOT with exception of std::set which keeps its data on the heap. I was wondering if there's a way to get STL containers to reside on MEM_ROOT as well, and found that yes there is, and MySQL actually already has it: https://dev.mysql.com/worklog/task/?id=6074 https://github.com/mysql/mysql-server/blob/8.0/sql/mem_root_allocator.h Can we copy mem_root_allocator.h from them (in a separate commit) and then fix memory management here by making std::set use it? BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://petrunia.net

1 0

Re: [Maria-developers] 9ec0b36ee8b: MDEV-28838 password_reuse_check plugin mixes username and password
by Sergei Golubchik 04 Jul '22

04 Jul '22

Hi, Oleksandr, On Jul 04, Oleksandr Byelkin wrote: > revision-id: 9ec0b36ee8b (mariadb-10.7.4-27-g9ec0b36ee8b) > parent(s): ac0af4ec4ab > author: Oleksandr Byelkin > committer: Oleksandr Byelkin > timestamp: 2022-06-29 17:25:44 +0200 > message: > > MDEV-28838 password_reuse_check plugin mixes username and password > > To prevent the problem of mixing user name and password and > host name and user name we add length of the hostname and user name > to the hash. > > diff --git a/plugin/password_reuse_check/password_reuse_check.c b/plugin/password_reuse_check/password_reuse_check.c > index ff0364ce007..4d07cda878a 100644 > --- a/plugin/password_reuse_check/password_reuse_check.c > +++ b/plugin/password_reuse_check/password_reuse_check.c > @@ -165,9 +166,11 @@ static int validate(const MYSQL_CONST_LEX_STRING *username, > return 1; > } > > - memcpy(buff, hostname->str, hostname->length); > - memcpy(buff + hostname->length, username->str, username->length); > - memcpy(buff + hostname->length + username->length, password->str, > + int2store(buff, hostname->length); > + int2store(buff + 2, username->length); > + memcpy(buff + 4, hostname->str, hostname->length); > + memcpy(buff + 4 + hostname->length, username->str, username->length); > + memcpy(buff + 4 + hostname->length + username->length, password->str, > password->length); why? let's use the standard length,string format, why risk inventing something unusual here for no benefits at all? I'd do like: static char *store_str(char *to, MYSQL_CONST_LEX_STRING from) { int2store(to, from.length); memcpy(to+2, from.str, from.length); return to+2+from.length; } and buff= store_str(buff, hostname); buff= store_str(buff, username); buff= store_str(buff, password); > buff[key_len]= 0; > memset(hash, 0, sizeof(hash)); > @@ -235,7 +238,7 @@ maria_declare_plugin(password_reuse_check) > 0x0100, ^^^ this should be 0x0101 (or 0x0200 as I suggest below) > NULL, > sysvars, > - "1.0", > + "1.1", make it 2.0 please, this is an incompatible change > MariaDB_PLUGIN_MATURITY_GAMMA > } > maria_declare_plugin_end; > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] e9293e31ec6: MDEV-28930 ALTER TABLE Deadlocks with parallel TL_WRITE
by Sergei Golubchik 30 Jun '22

30 Jun '22

Hi, Nikita, This is very good. Just one comment, see below: On Jun 29, Nikita Malyavin wrote: > revision-id: e9293e31ec6 (mariadb-10.6.1-499-ge9293e31ec6) > parent(s): 6cfcb36d027 > author: Nikita Malyavin > committer: Nikita Malyavin > timestamp: 2022-06-29 20:16:19 +0300 > message: > > MDEV-28930 ALTER TABLE Deadlocks with parallel TL_WRITE > > ALTER ONLINE TABLE acquires table with TL_READ. Myisam normally acquires > TL_WRITE for DML, which makes it hang until table is freed. > > We deadlock once ALTER upgrades its MDL lock. > > Solution: > Unlock table earlier. We don't need to hold TL_READ once we finished > copying. Relay log replication requires no data locks on `from` table. > > diff --git a/sql/log_event_server.cc b/sql/log_event_server.cc > index 425de469a7f..90f6283d9df 100644 > --- a/sql/log_event_server.cc > +++ b/sql/log_event_server.cc > @@ -5734,7 +5734,7 @@ int Rows_log_event::do_apply_event(rpl_group_info *rgi) > used in the transaction and proceed with execution of the actual > event. > */ > - if (!thd->lock) > + if (!thd->lock && rgi->tables_to_lock_count) > { > /* > Lock_tables() reads the contents of thd->lex, so they must be Instead of adding a second condition to the execution path for every row event in the replication, consider this: - if (!thd->lock) + if (!thd->open_tables) this will work identically for replication, I hope, but will be always false in your case, as you still have the table open. I mean, that's the idea. I run the rpl suite with it, but haven't done any further testing, so not completely sure it'll work. When making this change, don't forget to update the comment right above this if(). Ok to push into bb-10.10-MDEV-16329 after that. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 3

[Maria-developers] How to know when a FUNCTION is AGGREGATE?
by Martijn Tonies (Upscene Productions) 17 Jun '22

17 Jun '22

Hi all, Since MariaDB 10.3, you can use an AGGREGATE stored function. https://mariadb.com/kb/en/stored-aggregate-functions/ When examining the meta data, how does one distinguish between aggregate and normal stored functions? Can’t find anything in the information_schema about this. With regards, Martijn Tonies Upscene Productions https://www.upscene.com Database Workbench - developer tool for Oracle, SQL Server, PostgreSQL, MySQL, InterBase, NexusDB and Firebird.

3 3

Re: [Maria-developers] 49ecf935415: MDEV-27009 Add UCA-14.0.0 collations
by Sergei Golubchik 16 Jun '22

16 Jun '22

Hi, Alexander, On Mar 14, Alexander Barkov wrote: > revision-id: 49ecf935415 (mariadb-10.6.1-335-g49ecf935415) > parent(s): c67789f63c8 > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2022-02-28 14:04:58 +0400 > message: > > MDEV-27009 Add UCA-14.0.0 collations please, list all user visible changes there. Mainly that collations are now decoupled from charsets. New syntax in CREATE TABLE, changes in I_S tables, etc. > diff --git a/mysql-test/include/ctype_utf_uca1400_ids.inc b/mysql-test/include/ctype_utf_uca1400_ids.inc > new file mode 100644 > index 00000000000..09cf49fc0e7 > --- /dev/null > +++ b/mysql-test/include/ctype_utf_uca1400_ids.inc > @@ -0,0 +1,17 @@ file names are confusing. better rename ctype_ucs_uca1400_ids.inc to something like ctype_convert_uca1400_ids and ctype_utf_uca1400_ids to ctype_set_names_uca1400_ids or something like that, to show what they do. > + > +--disable_ps_protocol > +--enable_metadata > +DELIMITER $$; > +FOR rec IN (SELECT COLLATION_NAME > + FROM INFORMATION_SCHEMA.COLLATION_CHARACTER_SET_APPLICABILITY > + WHERE CHARACTER_SET_NAME=@charset > + AND COLLATION_NAME RLIKE 'uca1400' > + ORDER BY ID) > +DO > + EXECUTE IMMEDIATE CONCAT('SET NAMES ',@charset,' COLLATE ', rec.COLLATION_NAME); > + SELECT rec.COLLATION_NAME; > +END FOR; > +$$ > +DELIMITER ;$$ > +--disable_metadata > +--enable_ps_protocol > diff --git a/include/m_ctype.h b/include/m_ctype.h > index 4c6628b72b3..706764ead2a 100644 > --- a/include/m_ctype.h > +++ b/include/m_ctype.h > @@ -34,7 +34,9 @@ enum loglevel { > extern "C" { > #endif > > -#define MY_CS_NAME_SIZE 32 > +#define MY_CS_CHARACTER_SET_NAME_SIZE 32 > +#define MY_CS_COLLATION_NAME_SIZE 64 That's FULL_COLLATION_NAME_SIZE, right? > + > #define MY_CS_CTYPE_TABLE_SIZE 257 > #define MY_CS_TO_LOWER_TABLE_SIZE 256 > #define MY_CS_TO_UPPER_TABLE_SIZE 256 > @@ -240,6 +242,46 @@ typedef enum enum_repertoire_t > } my_repertoire_t; > > > +/* ID compatibility */ > +typedef enum enum_collation_id_type > +{ > + MY_COLLATION_ID_TYPE_PRECISE= 0, > + MY_COLLATION_ID_TYPE_COMPAT_100800= 1 > +} my_collation_id_type_t; > + > + > +/* Collation name display modes */ > +typedef enum enum_collation_name_mode > +{ > + MY_COLLATION_NAME_MODE_FULL= 0, > + MY_COLLATION_NAME_MODE_CONTEXT= 1 > +} my_collation_name_mode_t; > + > + > +/* Level flags */ > +#define MY_CS_LEVEL_BIT_PRIMARY 0x00 > +#define MY_CS_LEVEL_BIT_SECONDARY 0x01 > +#define MY_CS_LEVEL_BIT_TERTIARY 0x02 > +#define MY_CS_LEVEL_BIT_QUATERNARY 0x03 > + > +#define MY_CS_COLL_LEVELS_S1 (1<<MY_CS_LEVEL_BIT_PRIMARY) > + > +#define MY_CS_COLL_LEVELS_AI_CS (1<<MY_CS_LEVEL_BIT_PRIMARY)| \ > + (1<<MY_CS_LEVEL_BIT_TERTIARY) > + > +#define MY_CS_COLL_LEVELS_S2 (1<<MY_CS_LEVEL_BIT_PRIMARY)| \ > + (1<<MY_CS_LEVEL_BIT_SECONDARY) > + > +#define MY_CS_COLL_LEVELS_S3 (1<<MY_CS_LEVEL_BIT_PRIMARY)| \ > + (1<<MY_CS_LEVEL_BIT_SECONDARY) | \ > + (1<<MY_CS_LEVEL_BIT_TERTIARY) AI_CS and S3 don't seem to be used yet > + > +#define MY_CS_COLL_LEVELS_S4 (1<<MY_CS_LEVEL_BIT_PRIMARY)| \ > + (1<<MY_CS_LEVEL_BIT_SECONDARY) | \ > + (1<<MY_CS_LEVEL_BIT_TERTIARY) | \ > + (1<<MY_CS_LEVEL_BIT_QUATERNARY) > + > + > /* Flags for strxfrm */ > #define MY_STRXFRM_LEVEL1 0x00000001 /* for primary weights */ > #define MY_STRXFRM_LEVEL2 0x00000002 /* for secondary weights */ > diff --git a/sql/sql_alter.cc b/sql/sql_alter.cc > index 86c6e9a27f8..9ddd482ad57 100644 > --- a/sql/sql_alter.cc > +++ b/sql/sql_alter.cc > @@ -546,6 +546,7 @@ bool Sql_cmd_alter_table::execute(THD *thd) > > result= mysql_alter_table(thd, &select_lex->db, &lex->name, > &create_info, > + lex->create_info.default_charset_collation, I don't see why you need a new argument here. It's create_info.default_charset_collation, so, mysql_alter_table already gets it in create_info. All other mysql_alter_table invocations also take create_info argument and can get default_charset_collation from there > first_table, > &alter_info, > select_lex->order_list.elements, > diff --git a/sql/sql_partition_admin.cc b/sql/sql_partition_admin.cc > index fb1ae0d5fc7..4188dde252b 100644 > --- a/sql/sql_partition_admin.cc > +++ b/sql/sql_partition_admin.cc > @@ -211,6 +211,7 @@ bool compare_table_with_partition(THD *thd, TABLE *table, TABLE *part_table, > part_table->use_all_columns(); > table->use_all_columns(); > if (unlikely(mysql_prepare_alter_table(thd, part_table, &part_create_info, > + Lex_maybe_default_charset_collation(), Same. Can be in part_create_info > &part_alter_info, &part_alter_ctx))) > { > my_error(ER_TABLES_DIFFERENT_METADATA, MYF(0)); > diff --git a/sql/sql_i_s.h b/sql/sql_i_s.h > index bed2e886718..5ff06d32231 100644 > --- a/sql/sql_i_s.h > +++ b/sql/sql_i_s.h > @@ -162,6 +162,11 @@ class Yesno: public Varchar > { > public: > Yesno(): Varchar(3) { } > + static LEX_CSTRING value(bool val) > + { > + return val ? Lex_cstring(STRING_WITH_LEN("Yes")) : > + Lex_cstring(); > + } eh... please, rename the class from Yesno to something like Yesempty or Yes_or_empty, something that says that the second should not be Lex_cstring(STRING_WITH_LEN("No")) > }; > > > diff --git a/sql/table.cc b/sql/table.cc > index a683a78ff49..c28cb2bd928 100644 > --- a/sql/table.cc > +++ b/sql/table.cc > @@ -3491,6 +3493,16 @@ int TABLE_SHARE::init_from_sql_statement_string(THD *thd, bool write, > else > thd->set_n_backup_active_arena(arena, &backup); > > + /* > + THD::reset_db() does not set THD::db_charset, > + so it keeps pointing to the character set and collation > + of the current database, rather than the database of the > + new initialized table. Hmm, is that correct? Could you check other invocation of thd->reset_db()? Perhaps they all need to switch charset? In that case it should be done inside THD::reset_db(). Or may be they have to use mysql_change_db_impl() instead? > + Let's call get_default_db_collation() before reset_db(). > + This forces the db.opt file to be loaded. > + */ > + db_cs= get_default_db_collation(thd, db.str); > + > thd->reset_db(&db); > lex_start(thd); > > @@ -3498,6 +3510,11 @@ int TABLE_SHARE::init_from_sql_statement_string(THD *thd, bool write, > sql_unusable_for_discovery(thd, hton, sql_copy)))) > goto ret; > > + if (!(thd->lex->create_info.default_table_charset= > + thd->lex->create_info.default_charset_collation. > + resolved_to_character_set(db_cs, db_cs))) > + DBUG_RETURN(true); How could this (and similar if()'s in other files) fail? > + > thd->lex->create_info.db_type= hton; > #ifdef WITH_PARTITION_STORAGE_ENGINE > thd->work_part_info= 0; // For partitioning > diff --git a/sql/mysys_charset.h b/sql/mysys_charset.h > new file mode 100644 > index 00000000000..86eaeedd432 > --- /dev/null > +++ b/sql/mysys_charset.h > @@ -0,0 +1,44 @@ > +#ifndef MYSYS_CHARSET > +#define MYSYS_CHARSET > + > +/* Copyright (c) 2021, MariaDB Corporation. > + > + This program is free software; you can redistribute it and/or modify > + it under the terms of the GNU General Public License as published by > + the Free Software Foundation; version 2 of the License. > + > + This program is distributed in the hope that it will be useful, > + but WITHOUT ANY WARRANTY; without even the implied warranty of > + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > + GNU General Public License for more details. > + > + You should have received a copy of the GNU General Public License > + along with this program; if not, write to the Free Software > + Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1335 USA */ > + > + > +#include "my_sys.h" > + > + > +class Charset_loader_mysys: public MY_CHARSET_LOADER > +{ > +public: > + Charset_loader_mysys() > + { > + my_charset_loader_init_mysys(this); > + } > + void raise_unknown_collation_error(const char *name, > + CHARSET_INFO *name_cs) const; > + CHARSET_INFO *get_charset(const char *cs_name, uint cs_flags, myf my_flags); > + CHARSET_INFO *get_exact_collation(const char *name, myf utf8_flag); > + CHARSET_INFO *get_contextually_typed_collation(CHARSET_INFO *cs, > + const char *name); > + CHARSET_INFO *get_contextually_typed_collation(const char *name); > + CHARSET_INFO *get_contextually_typed_collation_or_error(CHARSET_INFO *cs, > + const char *name); > + CHARSET_INFO *find_default_collation(CHARSET_INFO *cs); > + CHARSET_INFO *find_bin_collation_or_error(CHARSET_INFO *cs); > +}; you can have C++ code in mysys too, you know, no need to put it in sql/mysys* > + > +#endif // MYSYS_CHARSET > + > diff --git a/strings/ctype-simple.c b/strings/ctype-simple.c > index b579f0af203..d09dfba86ed 100644 > --- a/strings/ctype-simple.c > +++ b/strings/ctype-simple.c > @@ -1940,13 +1941,26 @@ my_bool my_propagate_complex(CHARSET_INFO *cs __attribute__((unused)), > } > > > +void my_ci_set_strength(struct charset_info_st *cs, uint strength) > +{ > + DBUG_ASSERT(strength > 0 && strength <= MY_STRXFRM_NLEVELS); don't use && in asserts, please create two separate asserts instead: DBUG_ASSERT(strength > 0); DBUG_ASSERT(strength <= MY_STRXFRM_NLEVELS); > + cs->levels_for_order= ((1 << strength) - 1); why do you still use the old concept of "strength"? Why not to use bitmap consistently everywhere? > +} > + > + > +void my_ci_set_level_flags(struct charset_info_st *cs, uint flags) > +{ > + DBUG_ASSERT(flags < (1<<MY_STRXFRM_NLEVELS)); > + cs->levels_for_order= flags; > +} > + > /* > Normalize strxfrm flags > > SYNOPSIS: > my_strxfrm_flag_normalize() > + cs - the CHARSET_INFO pointer > flags - non-normalized flags > - nlevels - number of levels > > NOTES: > If levels are omitted, then 1-maximum is assumed. > diff --git a/sql/handler.h b/sql/handler.h > index 8ad521e189a..1e82f37b1e7 100644 > --- a/sql/handler.h > +++ b/sql/handler.h > @@ -2409,7 +2386,32 @@ struct Table_specification_st: public HA_CREATE_INFO, > { > HA_CREATE_INFO::options= 0; > DDL_options_st::init(); > + default_charset_collation.init(); > + } > + > + bool > + add_alter_list_item_convert_to_charset(const Lex_charset_collation_st &cl) > + { > + /* > + cs cannot be NULL, as sql_yacc.yy translates > + CONVERT TO CHARACTER SET DEFAULT > + to > + CONVERT TO CHARACTER SET <character-set-of-the-current-database> > + TODO: Shouldn't we postpone resolution of DEFAULT until the > + character set of the table owner database is loaded from its db.opt? > + */ > + DBUG_ASSERT(cl.charset_collation()); > + DBUG_ASSERT(!cl.is_contextually_typed_collation()); > + alter_table_convert_to_charset= cl.charset_collation(); > + default_charset_collation.Lex_charset_collation_st::operator=(cl); looks quite ugly. can you do, like, default_charset_collation.set(cl) ? > + used_fields|= (HA_CREATE_USED_CHARSET | HA_CREATE_USED_DEFAULT_CHARSET); > + return false; > } > + bool add_table_option_default_charset(CHARSET_INFO *cs); > + bool add_table_option_default_collation(const Lex_charset_collation_st &cl); > + bool resolve_db_charset_and_collation(THD *thd, > + const LEX_CSTRING &db, > + bool is_alter); > }; > > > diff --git a/strings/ctype-uca1400data.h b/strings/ctype-uca1400data.h > new file mode 100644 > index 00000000000..da95dcfde54 > --- /dev/null > +++ b/strings/ctype-uca1400data.h > @@ -0,0 +1,44151 @@ > +/* > + Generated from allkeys.txt version '14.0.0' > +*/ if it's generated, do you need to check it in? perhaps it should be generated during the build? you've checked in allkeys1400.txt anyway. > +static const uint16 uca1400_p000[]= { /* 0000 (4 weights per char) */ > +0x0000,0x0000,0x0000,0x0000, 0x0000,0x0000,0x0000,0x0000, /* 0000 */ > +0x0000,0x0000,0x0000,0x0000, 0x0000,0x0000,0x0000,0x0000, /* 0002 */ > diff --git a/sql/sql_lex.cc b/sql/sql_lex.cc > index 6ca10267187..d115401a855 100644 > --- a/sql/sql_lex.cc > +++ b/sql/sql_lex.cc > @@ -542,6 +542,30 @@ bool LEX::add_alter_list(LEX_CSTRING name, LEX_CSTRING new_name, bool exists) > } > > > +bool LEX::add_alter_list_item_convert_to_charset( > + THD *thd, > + CHARSET_INFO *cs, > + const Lex_charset_collation_st &cl) > +{ > + if (!cs) > + { > + Lex_charset_collation_st tmp; > + tmp.set_charset_collate_default(thd->variables.collation_database); Hmm, what if one is doing ALTER TABLE db.test CHARSET DEFAULT and current db is not `db` but `test` ? > + if (!(cs= tmp.charset_collation())) > + return true; // Should not actually happen assert? > + } > + > + Lex_explicit_charset_opt_collate tmp(cs, false); > + if (tmp.merge_opt_collate_or_error(cl) || > + create_info.add_alter_list_item_convert_to_charset( > + Lex_charset_collation(tmp))) > + return true; > + > + alter_info.flags|= ALTER_CONVERT_TO; > + return false; > +} > + > + > void LEX::init_last_field(Column_definition *field, > const LEX_CSTRING *field_name) > { > @@ -11871,29 +11869,41 @@ CHARSET_INFO *Lex_collation_st::find_default_collation(CHARSET_INFO *cs) > "def" is the upper level CHARACTER SET clause (e.g. of a table) > */ > CHARSET_INFO * > -Lex_collation_st::resolved_to_character_set(CHARSET_INFO *def) const > +Lex_charset_collation_st::resolved_to_character_set(CHARSET_INFO *def) const > { > DBUG_ASSERT(def); > - if (m_type != TYPE_CONTEXTUALLY_TYPED) > - { > - if (!m_collation) > - return def; // Empty - not typed at all > - return m_collation; // Explicitly typed > + > + switch (m_type) { > + case TYPE_EMPTY: > + return def; > + case TYPE_CHARACTER_SET: > + DBUG_ASSERT(m_ci); > + return m_ci; > + case TYPE_COLLATE_EXACT: > + DBUG_ASSERT(m_ci); > + return m_ci; > + case TYPE_COLLATE_CONTEXTUALLY_TYPED: > + break; > } > > // Contextually typed > - DBUG_ASSERT(m_collation); > + DBUG_ASSERT(m_ci); > > - if (m_collation == &my_charset_bin) // CHAR(10) BINARY > - return find_bin_collation(def); > + Charset_loader_mysys loader; > + if (is_contextually_typed_binary_style()) // CHAR(10) BINARY > + return loader.find_bin_collation_or_error(def); > > - if (m_collation == &my_charset_latin1) // CHAR(10) COLLATE DEFAULT > - return find_default_collation(def); > + if (is_contextually_typed_collate_default()) // CHAR(10) COLLATE DEFAULT > + return loader.find_default_collation(def); > + > + const LEX_CSTRING context_name= collation_name_context_suffix(); I'd rather put this in assert, not in if(). Like - if (!strncasecmp(context_name.str, STRING_WITH_LEN("uca1400_"))) + DBUG_ASSERT(!strncasecmp(context_cl_name.str, STRING_WITH_LEN("uca1400_"))); > + if (!strncasecmp(context_name.str, STRING_WITH_LEN("uca1400_"))) > + return loader.get_contextually_typed_collation_or_error(def, > + context_name.str); > > /* > - Non-binary and non-default contextually typed collation. > + Non-binary, non-default, non-uca1400 contextually typed collation. > We don't have such yet - the parser cannot produce this. > - But will have soon, e.g. "uca1400_as_ci". > */ > DBUG_ASSERT(0); > return NULL; > @@ -11944,58 +11972,106 @@ bool Lex_collation_st:: > CHAR(10) BINARY .. COLLATE latin1_bin > CHAR(10) COLLATE uca1400_as_ci .. COLLATE latin1_bin > */ > - if (collation() == &my_charset_latin1 && > - !(cl.collation()->state & MY_CS_PRIMARY)) > + if (is_contextually_typed_collate_default() && > + !cl.charset_collation()->default_flag()) > { > - my_error(ER_CONFLICTING_DECLARATIONS, MYF(0), > - "COLLATE ", "DEFAULT", "COLLATE ", > - cl.collation()->coll_name.str); > + error_conflicting_collations_or_styles(*this, cl); > return true; > } > - if (collation() == &my_charset_bin && > - !(cl.collation()->state & MY_CS_BINSORT)) > + > + if (is_contextually_typed_binary_style() && > + !cl.charset_collation()->binsort_flag()) > { > - my_error(ER_CONFLICTING_DECLARATIONS, MYF(0), > - "", "BINARY", "COLLATE ", cl.collation()->coll_name.str); > + error_conflicting_collations_or_styles(*this, cl); > return true; > } > *this= cl; > return false; > } > > - if (cl.is_contextually_typed_collation()) > - { > + DBUG_ASSERT(0); > + return false; > +} > + > + > +bool > +Lex_explicit_charset_opt_collate:: > + merge_collate_or_error(const Lex_charset_collation_st &cl) > +{ > + DBUG_ASSERT(cl.type() != Lex_charset_collation_st::TYPE_CHARACTER_SET); > + > + switch (cl.type()) { > + case Lex_charset_collation_st::TYPE_EMPTY: > + return false; > + case Lex_charset_collation_st::TYPE_CHARACTER_SET: > + DBUG_ASSERT(0); > + return false; > + case Lex_charset_collation_st::TYPE_COLLATE_EXACT: > /* > - EXPLICIT + CONTEXT > - CHAR(10) COLLATE latin1_bin .. COLLATE DEFAULT - not supported > - CHAR(10) COLLATE latin1_bin .. COLLATE uca1400_as_ci - not yet > + EXPLICIT + EXPLICIT > + CHAR(10) CHARACTER SET latin1 .. COLLATE latin1_bin > + CHAR(10) CHARACTER SET latin1 COLLATE latin1_bin .. COLLATE latin1_bin > + CHAR(10) COLLATE latin1_bin .. COLLATE latin1_bin > + CHAR(10) COLLATE latin1_bin .. COLLATE latin1_bin > + CHAR(10) CHARACTER SET latin1 BINARY .. COLLATE latin1_bin > */ > - DBUG_ASSERT(0); // Not possible yet > + if (m_with_collate && m_ci != cl.charset_collation()) > + { > + my_error(ER_CONFLICTING_DECLARATIONS, MYF(0), > + "COLLATE ", m_ci->coll_name.str, > + "COLLATE ", cl.charset_collation()->coll_name.str); > + return true; > + } > + if (!my_charset_same(m_ci, cl.charset_collation())) > + { > + my_error(ER_COLLATION_CHARSET_MISMATCH, MYF(0), > + cl.charset_collation()->coll_name.str, m_ci->cs_name.str); > + return true; > + } > + m_ci= cl.charset_collation(); > + m_with_collate= true; > return false; > - } > > - /* > - EXPLICIT + EXPLICIT > - CHAR(10) CHARACTER SET latin1 .. COLLATE latin1_bin > - CHAR(10) CHARACTER SET latin1 COLLATE latin1_bin .. COLLATE latin1_bin > - CHAR(10) COLLATE latin1_bin .. COLLATE latin1_bin > - CHAR(10) COLLATE latin1_bin .. COLLATE latin1_bin > - CHAR(10) CHARACTER SET latin1 BINARY .. COLLATE latin1_bin > - */ > - if (type() == TYPE_EXPLICIT && collation() != cl.collation()) > - { > - my_error(ER_CONFLICTING_DECLARATIONS, MYF(0), > - "COLLATE ", collation()->coll_name.str, > - "COLLATE ", cl.collation()->coll_name.str); > - return true; > - } > - if (!my_charset_same(collation(), cl.collation())) > - { > - my_error(ER_COLLATION_CHARSET_MISMATCH, MYF(0), > - cl.collation()->coll_name.str, collation()->cs_name.str); > - return true; > + case Lex_charset_collation_st::TYPE_COLLATE_CONTEXTUALLY_TYPED: > + if (cl.is_contextually_typed_collate_default()) > + { > + /* > + SET NAMES latin1 COLLATE DEFAULT; > + ALTER TABLE t1 CONVERT TO CHARACTER SET latin1 COLLATE DEFAULT; > + */ > + CHARSET_INFO *tmp= Charset_loader_mysys().find_default_collation(m_ci); > + if (!tmp) > + return true; > + m_ci= tmp; > + m_with_collate= true; > + return false; > + } > + else > + { > + /* > + EXPLICIT + CONTEXT > + CHAR(10) COLLATE latin1_bin .. COLLATE DEFAULT not possible yet > + CHAR(10) COLLATE latin1_bin .. COLLATE uca1400_as_ci > + */ > + > + const LEX_CSTRING context_cl_name= cl.collation_name_context_suffix(); > + if (!strncasecmp(context_cl_name.str, STRING_WITH_LEN("uca1400_"))) Like above, better DBUG_ASSERT(!strncasecmp(context_cl_name.str, STRING_WITH_LEN("uca1400_"))) > + { > + CHARSET_INFO *tmp; > + Charset_loader_mysys loader; > + if (!(tmp= loader.get_contextually_typed_collation_or_error(m_ci, > + context_cl_name.str))) > + return true; > + m_with_collate= true; > + m_ci= tmp; > + return false; > + } > + > + DBUG_ASSERT(0); // Not possible yet > + return false; > + } > } > - *this= cl; > + DBUG_ASSERT(0); > return false; > } > > diff --git a/strings/ctype-uca.c b/strings/ctype-uca.c > index b89916f3b20..3e6b4e4ce43 100644 > --- a/strings/ctype-uca.c > +++ b/strings/ctype-uca.c > @@ -30542,7 +30613,7 @@ static const char vietnamese[]= > Myanmar, according to CLDR Revision 8900. > http://unicode.org/cldr/trac/browser/trunk/common/collation/my.xml > */ > -static const char myanmar[]= "[shift-after-method expand][version 5.2.0]" > +static const char myanmar[]= "[shift-after-method expand]" What's going on with myanmar? You removed a version here and added &my_uca_v520 below in its charset_info_st. What does this change mean? > /* Tones */ > "&\\u108C" > "<\\u1037" > @@ -37627,7 +37825,7 @@ struct charset_info_st my_charset_utf32_myanmar_uca_ci= > NULL, /* to_lower */ > NULL, /* to_upper */ > NULL, /* sort_order */ > - NULL, /* uca */ > + &my_uca_v520, /* uca */ What does this change? > NULL, /* tab_to_uni */ > NULL, /* tab_from_uni */ > &my_unicase_unicode520,/* caseinfo */ Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 3

[Maria-developers] Review #3 for: MDEV-26278 Elimination: consider GROUP BY as unique key
by Sergey Petrunia 13 Jun '22

13 Jun '22

Hi Oleg, Please fix compile errors on this platform: https://buildbot.mariadb.org/#/builders/146/builds/13126/steps/5/logs/stdio also please apply the attached diff that fixes the coding style and wording of comments (input and/or adjustments are welcome). After this, the patch is ok to push. The next steps are to create a branch with name like preview-10.10-mdev26278-table-elimination and to officially submit it for preview tree build and testing. > commit 7275bf4939248959b5017314d6b0393296831c0c (HEAD, origin/bb-10.9-MDEV-26278-look) > Author: Oleg Smirnov <olernov(a)gmail.com> > Date: Thu Feb 17 22:53:37 2022 +0700 > > MDEV-26278 Add functionality to eliminate derived tables from the query > > Elimination of unnecessary tables from SQL queries is already present > in MariaDB. But it only works for regular tables and not for derived ones. > > Imagine we have a view: > CREATE VIEW v1 AS SELECT a, b, max(c) AS maxc FROM t1 GROUP BY a, b > > Due to "GROUP BY a, b" the values of combinations {a, b} are unique, > and this fact can be treated as like derived table "v1" has a unique key > on fields {a, b}. > > Suppose we have a SQL query: > SELECT t2.* FROM t2 LEFT JOIN v1 ON t2.a=v1.a and t2.b=v1.b > > 1. Since {v1.a, v1.b} is unique and both these fields are bound to t2, > "v1" is functionally dependent on t2. > This means every record of "t2" will be either joined with > a single record of "v1" or NULL-complemented. > 2. No fields of "v1" are present on the SELECT list > > These two facts allow the server to completely exclude (eliminate) > the derived table "v1" from the query. BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://petrunia.net

2 1

Re: [Maria-developers] 8c2c0e6079a: MDEV-27265 Improve contraction performance in UCA collations
by Sergei Golubchik 11 Jun '22

11 Jun '22

Hi, Alexander, Why haven't you used HASH? On Jun 11, Alexander Barkov wrote: > revision-id: 8c2c0e6079a (mariadb-10.9.1-13-g8c2c0e6079a) > parent(s): 27fa5c10eff > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2022-05-26 13:17:34 +0400 > message: > > MDEV-27265 Improve contraction performance in UCA collations > > Adding a hash table for contractions. > > The old code iterated through all items in MY_CONTRACTIONS, > and was much slower, especially for those contractions > in the end of the list. > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] MDEV-26278 Elimination: consider GROUP BY as unique key
by Sergey Petrunia 09 Jun '22

09 Jun '22

Hi Oleg, Please find the review below. > commit 20609d374b3a82c46291ee5e87cfadd26de47817 > Author: Oleg Smirnov <olernov(a)gmail.com> > Date: Thu Feb 17 22:53:37 2022 +0700 > > MDEV-26278 Elimination: consider GROUP BY as unique key Please provide a more verbose description. We should aim for it to provide sufficient info for one to understand what the optimization does and how that is achieved. Some general comments: The patch uses this formatting for multi-line comments: > + /* GROUP BY expression is considered as a synthetic "unique key" > + for the derived table. Add this key as a dependency */ While the coding style and actual MariaDB code would use this: /* GROUP BY expression is considered as a synthetic "unique key" for the derived table. Add this key as a dependency */ Please use this style for all comments. Also, the patch uses the term "synthetic unique key". I think the term is not very good: It can be confused with "synthetic primary key", also I read "synthetic" as something that actually exists, while in our case the key doesn't exist. Would using "Pseudo-key" be better? > diff --git a/sql/table.h b/sql/table.h > index 497502b2d06..fd598556cc3 100644 > --- a/sql/table.h > +++ b/sql/table.h > @@ -2872,6 +2872,9 @@ struct TABLE_LIST > } > } > > + void mark_derived_as_eliminated() { m_is_derived_eliminated= true; } > + bool is_derived_eliminated() const { return m_is_derived_eliminated; } > + So, TABLE_LIST objects are created non-eliminated and then they are eliminated, and that is forever. TABLE_LIST object has the same lifetime as the statement, that is, it will survive multiple executions of a prepared statement. On the other hand, Table Elimination optimization is done in eliminate_tables() call which is invoked for every statement execution. This looks like a dangerous mismatch. I'll continue below > private: > bool prep_check_option(THD *thd, uint8 check_opt_type); > bool prep_where(THD *thd, Item **conds, bool no_where_clause); > diff --git a/sql/sql_lex.cc b/sql/sql_lex.cc > index 3bf6d9abf57..02ef5cd7ad7 100644 > --- a/sql/sql_lex.cc > +++ b/sql/sql_lex.cc > @@ -11786,7 +11786,7 @@ bool SELECT_LEX_UNIT::explainable() const > EXPLAIN/ANALYZE unit, when: > (1) if it's a subquery - it's not part of eliminated WHERE/ON clause. > (2) if it's a CTE - it's not hanging (needed for execution) > - (3) if it's a derived - it's not merged > + (3) if it's a derived - it's not merged or eliminated > if it's not 1/2/3 - it's some weird internal thing, ignore it > */ > return item ? > @@ -11795,6 +11795,7 @@ bool SELECT_LEX_UNIT::explainable() const > derived && derived->derived_result && > !with_element->is_hanging_recursive(): // (2) > derived ? > - derived->is_materialized_derived() : // (3) > + derived->is_materialized_derived() && // (3) > + !derived->is_derived_eliminated() : > false; > } I think, there's no need for m_is_derived_eliminated. You can have this check in is_derived_eliminated(): this->derived->table->map & this->outer_select()->join->eliminated_tables Please try doing this. > diff --git a/mysql-test/main/table_elim.result b/mysql-test/main/table_elim.result > index deff0623370..842330f3713 100644 > --- a/mysql-test/main/table_elim.result > +++ b/mysql-test/main/table_elim.result > @@ -704,3 +704,252 @@ LIMIT 1; > PostID Voted > 1 NULL > DROP TABLE t1,t2; > +# > +# MDEV-26278: Table elimination does not work across derived tables > +# > +create table t1 (a int, b int); > +insert into t1 select seq, seq+10 from seq_1_to_10; > +create table t11 ( > +a int not null, > +b int, > +key(a) > +); > +insert into t11 select A.seq, A.seq+B.seq > +from > +seq_1_to_100 A, > +seq_1_to_1000 B; Do you really need 100K rows table? I think 10K rows will be just as good. > diff --git a/sql/opt_table_elimination.cc b/sql/opt_table_elimination.cc > index 8c4720bdec4..fe2155191ca 100644 > --- a/sql/opt_table_elimination.cc > +++ b/sql/opt_table_elimination.cc ... > @@ -278,6 +286,8 @@ class Dep_value_field : public Dep_value > Dep_module_key *key_dep; > /* Otherwise, this and advance */ > uint equality_no; > + > + Dep_module_synthetic_key *synth_key_dep; Please add a comment for the above that we should try that one, too. @@ -447,6 +463,55 @@ const size_t Dep_module::iterator_size= > MY_MAX(Dep_module_expr::iterator_size, Dep_module_key::iterator_size); > > > +/* A synthetic unique key module for a derived table. > + For example, a derived table > + SELECT t11.a, count(*) from t1 LEFT JOIN t2 ON t1.id = t2.fk GROUP BY t11.a > + has unique values in its first field (t11.a) due to GROUP BY expression > + so this can be considered as a unique key for this derived table > +*/ > + > +class Dep_module_synthetic_key : public Dep_module > +{ > +public: > + Dep_module_synthetic_key(Dep_value_table *table_arg, > + std::vector<field_index_t>&& field_indexes) > + : table(table_arg), derived_table_field_indexes(field_indexes) > + { > + unbound_args= static_cast<uint>(field_indexes.size()); > + } > + > + Dep_value_table * table; Coding style: "*table". ... > @@ -1603,7 +1674,73 @@ Dep_value_table *Dep_analysis_context::create_table_value(TABLE *table) > key_list= &(key_dep->next_table_key); > } > } > - return table_deps[table->tablenr]= tbl_dep; > + > + auto select_unit= table_list->get_unit(); > + SELECT_LEX *first_select= nullptr; > + if (select_unit) Please move the code that creates the pseudo-key into a separate function. ... > + > +int Dep_analysis_context::find_field_in_list(List<Item> &fields_list, > + Item *field) Please add a one-line comment describing what function does. > @@ -1786,6 +1941,18 @@ Dep_module* Dep_value_field::get_next_unbound_module(Dep_analysis_context *dac, > } > else > di->key_dep= NULL; > + > + Dep_module_synthetic_key *synth_key_dep= di->synth_key_dep; > + if (synth_key_dep && !synth_key_dep->is_applicable() && > + std::find(synth_key_dep->derived_table_field_indexes.begin(), > + synth_key_dep->derived_table_field_indexes.end(), > + field->field_index) != std::end(synth_key_dep->derived_table_field_indexes)) Please introduce a method like bool Dep_module_synthetic_key::covers_field(int field_index); Will this also allow to make Dep_module_synthetic_key::derived_table_field_indexes a private member? BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://petrunia.net

1 1

[Maria-developers] Introduction of Weijun
by Weijun Huang 07 Jun '22

07 Jun '22

Hi everyone! My name is Weijun Huang and I was selected for GSoC to work with MariaDB. I'm from China and I am studying at Technical University of Denmark, majoring in Human-centered Artificial Intelligence. My nickname on Zulip is Weijun Huang, and on Github it's Weijun-H. My project for GSoC in 2022 is called "Add Regression Functions for MariaDB." I've already started to work with MariaDB Server. Now I'm doing the implementation of REGR_SXX, REGR_SYY, and REGR_SXY I'm looking forward to working on this project. It would be an excellent experience for me! Best regard Weijun Huang

1 0

Re: [Maria-developers] 2fdcfaaba09: MDEV-27832 disable binary logging for SQL SERVICE.
by Sergei Golubchik 06 May '22

06 May '22

Hi, Alexey, Looks good. Minor comments below. Ok to push after addressing them. On May 06, Alexey Botchkov wrote: > revision-id: 2fdcfaaba09 (mariadb-10.7.2-8-g2fdcfaaba09) > parent(s): 33fd136c61b > author: Alexey Botchkov > committer: Alexey Botchkov > timestamp: 2022-05-05 15:03:38 +0400 > message: > > MDEV-27832 disable binary logging for SQL SERVICE. > > Binary logging is now disabled for the queries run by SQL SERVICE. > The binlogging can be turned on with the 'SET SQL_LOG_BIN=On' query. > > diff --git a/mysql-test/suite/plugins/t/test_sql_service.test b/mysql-test/suite/plugins/t/test_sql_service.test > index b80d78fe6e5..0fd86fccd1a 100644 > --- a/mysql-test/suite/plugins/t/test_sql_service.test > +++ b/mysql-test/suite/plugins/t/test_sql_service.test > @@ -58,3 +73,7 @@ drop table t1; > > uninstall plugin test_sql_service; > > +--replace_column 2 # 5 # > +--replace_regex /xid=[0-9]+/xid=XX/ /GTID [0-9]+-[0-9]+-[0-9]+/GTID #-#-#/ > +SHOW BINLOG EVENTS LIMIT 3,100; use --source include/show_binlog_events.inc It does all the replaces, provides a unified output everywhere, it'll be updated if show binlog event output changes. No need to duplicate that. > + > diff --git a/sql/sql_prepare.cc b/sql/sql_prepare.cc > index e025147c71e..d3b77fbece3 100644 > --- a/sql/sql_prepare.cc > +++ b/sql/sql_prepare.cc > @@ -5617,12 +5617,26 @@ class Protocol_local : public Protocol_text > THD *new_thd; > Security_context empty_ctx; > > + my_bool do_log_bin; > + > Protocol_local(THD *thd_arg, THD *new_thd_arg, ulong prealloc) : > Protocol_text(thd_arg, prealloc), > cur_data(0), first_data(0), data_tail(&first_data), alloc(0), > - new_thd(new_thd_arg) > + new_thd(new_thd_arg), do_log_bin(FALSE) > {} > > + void set_binlog_vars(my_bool *sav_log_bin) > + { > + *sav_log_bin= thd->variables.sql_log_bin; > + thd->variables.sql_log_bin= do_log_bin; > + thd->set_binlog_bit(); > + } > + void restore_binlog_vars(my_bool &sav_log_bin) this doesn't need to be a reference, just a normal pass-by-value will do. > + { > + do_log_bin= thd->variables.sql_log_bin; > + thd->variables.sql_log_bin= sav_log_bin; > + thd->set_binlog_bit(); > + } > protected: > bool net_store_data(const uchar *from, size_t length); > bool net_store_data_cs(const uchar *from, size_t length, Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 9d6be81c0da: MDEV-27832 disable binary logging for SQL SERVICE.
by Sergei Golubchik 05 May '22

05 May '22

Hi, Alexey, On Mar 02, Alexey Botchkov wrote: > revision-id: 9d6be81c0da (mariadb-10.7.2-8-g9d6be81c0da) > parent(s): 33fd136c61b > author: Alexey Botchkov > committer: Alexey Botchkov > timestamp: 2022-02-15 13:37:59 +0400 > message: > > MDEV-27832 disable binary logging for SQL SERVICE. > > Binary logging is now disabled for the queries run by SQL SERVICE. > The binlogging can be turned on with the 'SET SQL_LOG_BIN=On' query. > > diff --git a/mysql-test/suite/plugins/r/test_sql_service.result b/mysql-test/suite/plugins/r/test_sql_service.result > index 00f0411b665..943bb239e5f 100644 > --- a/mysql-test/suite/plugins/r/test_sql_service.result > +++ b/mysql-test/suite/plugins/r/test_sql_service.result > @@ -2,6 +2,19 @@ install plugin test_sql_service soname 'test_sql_service'; > show status like 'test_sql_service_passed'; > Variable_name Value > Test_sql_service_passed 1 > +set global test_sql_service_execute_sql_global= 'create table test.t1 select 1 as a, @@SQL_LOG_BIN'; > +set global test_sql_service_execute_sql_local= 'insert into test.t1 select 2 as a, @@SQL_LOG_BIN'; > +set global test_sql_service_execute_sql_global= 'SET SQL_LOG_BIN=1'; > +set global test_sql_service_execute_sql_global= 'insert into test.t1 select 3 as a, @@SQL_LOG_BIN'; > +set global test_sql_service_execute_sql_global= 'SET SQL_LOG_BIN=0'; > +set global test_sql_service_execute_sql_global= 'insert into test.t1 select 4 as a, @@SQL_LOG_BIN'; > +select * from t1 order by a; this is not a test. Please, enable binlog and see whether queries are actually logged. Also, add a test for, say, sql_auto_is_null: set global test_sql_service_execute_sql_global='set sql_auto_is_null=1'; set global test_sql_service_execute_sql_global='insert test.t1 select 5, @@sql_auto_is_null'; select * from t1 where a=5; > +a @@SQL_LOG_BIN > +1 0 > +2 0 > +3 1 > +4 0 > +drop table t1; > set global test_sql_service_run_test= 1; > show status like 'test_sql_service_passed'; > Variable_name Value > diff --git a/sql/sql_prepare.cc b/sql/sql_prepare.cc > index e025147c71e..48681ac0b6f 100644 > --- a/sql/sql_prepare.cc > +++ b/sql/sql_prepare.cc > @@ -5617,12 +5617,30 @@ class Protocol_local : public Protocol_text > THD *new_thd; > Security_context empty_ctx; > > + my_bool do_log_bin; > + > Protocol_local(THD *thd_arg, THD *new_thd_arg, ulong prealloc) : > Protocol_text(thd_arg, prealloc), > cur_data(0), first_data(0), data_tail(&first_data), alloc(0), > - new_thd(new_thd_arg) > + new_thd(new_thd_arg), do_log_bin(FALSE) > {} > > + void set_binlog_vars(my_bool *sav_log_bin, ulonglong *sav_bits) > + { > + *sav_log_bin= thd->variables.sql_log_bin; > + *sav_bits= thd->variables.option_bits; > + > + if ((thd->variables.sql_log_bin= do_log_bin)) > + thd->variables.option_bits|= OPTION_BIN_LOG; > + else > + thd->variables.option_bits&= ~OPTION_BIN_LOG; > + } > + void restore_binlog_vars(my_bool &sav_log_bin, ulonglong &sav_bits) > + { > + do_log_bin= thd->variables.sql_log_bin; > + thd->variables.sql_log_bin= sav_log_bin; > + thd->variables.option_bits= sav_bits; that's what sql_auto_is_null test is for.You destroy all other option_bits here. Better don't save/restore option_bits, but use the same if() as in set_binlog_vars(). Or, even better, reuse fix_sql_log_bin_after_update(). I'd suggest to make it a method in THD > + } > protected: > bool net_store_data(const uchar *from, size_t length); > bool net_store_data_cs(const uchar *from, size_t length, > @@ -6230,12 +6248,17 @@ loc_advanced_command(MYSQL *mysql, enum enum_server_command command, > Ed_connection con(p->thd); > Security_context *ctx_orig= p->thd->security_ctx; > MYSQL_LEX_STRING sql_text; > + my_bool log_bin_orig; > + ulonglong option_bits_orig; > + p->set_binlog_vars(&log_bin_orig, &option_bits_orig); > + > DBUG_ASSERT(current_thd == p->thd); > sql_text.str= (char *) arg; > sql_text.length= arg_length; > p->thd->security_ctx= &p->empty_ctx; > result= con.execute_direct(p, sql_text); > p->thd->security_ctx= ctx_orig; > + p->restore_binlog_vars(log_bin_orig, option_bits_orig); > } > if (skip_check) > result= 0; > @@ -6391,6 +6414,9 @@ extern "C" MYSQL *mysql_real_connect_local(MYSQL *mysql) > new_thd->security_ctx->skip_grants(); > new_thd->query_cache_is_applicable= 0; > new_thd->variables.wsrep_on= 0; > + new_thd->variables.sql_log_bin= 0; > + new_thd->variables.option_bits&= ~OPTION_BIN_LOG; > + > /* > TOSO: decide if we should turn the auditing off > for such threads. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] 4ba5b444b4b: MDEV-28437: Assertion `!eliminated' failed in Item_subselect::exec
by Sergei Golubchik 04 May '22

04 May '22

Hi, Sergei, On May 04, Sergei Petrunia wrote: > revision-id: 4ba5b444b4b (mariadb-10.2.43-97-g4ba5b444b4b) > parent(s): 70555454b4c > author: Sergei Petrunia > committer: Sergei Petrunia > timestamp: 2022-05-04 12:24:48 +0300 > message: > > MDEV-28437: Assertion `!eliminated' failed in Item_subselect::exec > > (This is the assert that was added in fix for MDEV-26047) > > Table elimination may remove an ON expression from an outer join. > However SELECT_LEX::update_used_tables() will still call > > item->walk(&Item::eval_not_null_tables) > > for eliminated expressions. If the subquery is constant and cheap > Item_cond_and will attempt to evaluate it, which will trigger an > assert. > The fix is not to call update_used_tables() or eval_not_null_tables() > for ON expressions that were eliminated. > > diff --git a/sql/sql_lex.cc b/sql/sql_lex.cc > index 22ee8801e3a..8e718f2a942 100644 > --- a/sql/sql_lex.cc > +++ b/sql/sql_lex.cc > @@ -4229,7 +4229,7 @@ void SELECT_LEX::update_used_tables() > } > } > while ((embedding= embedding->embedding)); > - if (tl->on_expr) > + if (tl->on_expr && !is_eliminated_table(join->eliminated_tables, tl)) > { > tl->on_expr->update_used_tables(); > tl->on_expr->walk(&Item::eval_not_null_tables, 0, NULL); > @@ -4253,8 +4253,11 @@ void SELECT_LEX::update_used_tables() > if (embedding->on_expr && > embedding->nested_join->join_list.head() == tl) > { can embedding be eliminated? if yes - does it make sense to do anything in tl's below that eliminated embedding? Perhaps it should be (in the do..while above), like do { bool maybe_null; if ((maybe_null= MY_TEST(embedding->outer_join))) { tl->table->maybe_null= maybe_null; break; } + if (is_eliminated_table(join->eliminated_tables, embedding)) + tl= embedding->embedding; } while ((embedding= embedding->embedding)); > + if (!is_eliminated_table(join->eliminated_tables, embedding)) > + { > embedding->on_expr->update_used_tables(); > embedding->on_expr->walk(&Item::eval_not_null_tables, 0, NULL); > + } > } > tl= embedding; > embedding= tl->embedding; Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 85cc56875e9: MDEV-19398: Assertion `item1->type() == Item::FIELD_ITEM ...
by Sergei Golubchik 03 May '22

03 May '22

Hi, Sergei, The solutions looks fine Two comments below, about the implementation. ok to push after addressing them On May 03, Sergei Petrunia wrote: > revision-id: 85cc56875e9 (mariadb-10.2.43-75-g85cc56875e9) > parent(s): 3b6c04f44c4 > author: Sergei Petrunia > committer: Sergei Petrunia > timestamp: 2022-05-03 14:06:27 +0300 > message: > > MDEV-19398: Assertion `item1->type() == Item::FIELD_ITEM ... > > Window Functions code tries to minimize the number of times it > needs to sort the select's resultset by finding "compatible" > OVER (PARTITION BY ... ORDER BY ...) clauses. > > This employs compare_order_elements(). That function assumed that > the order expressions are Item_field-derived objects (that refer > to a temp.table). But this is not always the case: one can > construct queries order expressions are arbitrary item expressions. > > Add handling for such expressions: sort them according to the window > specification they appeared in. > This means we cannot detect that two compatible PARTITION BY clauses > that use expressions can share the sorting step. > But at least we won't crash. > can share the same sort > > diff --git a/sql/sql_parse.cc b/sql/sql_parse.cc > index 989ca0c8803..457849a7569 100644 > --- a/sql/sql_parse.cc > +++ b/sql/sql_parse.cc > @@ -8624,6 +8624,7 @@ bool st_select_lex::add_window_def(THD *thd, > fields_in_window_functions+= win_part_list_ptr->elements + > win_order_list_ptr->elements; > } > + win_def->win_spec_number= window_specs.elements; > return (win_def == NULL || window_specs.push_back(win_def)); > } > > @@ -8651,6 +8652,7 @@ bool st_select_lex::add_window_spec(THD *thd, > win_order_list_ptr->elements; > } > thd->lex->win_spec= win_spec; > + win_spec->win_spec_number= window_specs.elements; > return (win_spec == NULL || window_specs.push_back(win_spec)); > } > > diff --git a/sql/sql_window.cc b/sql/sql_window.cc > index 3ef751bc5b9..fbf7aca0197 100644 > --- a/sql/sql_window.cc > +++ b/sql/sql_window.cc > @@ -312,15 +312,44 @@ setup_windows(THD *thd, Ref_ptr_array ref_pointer_array, TABLE_LIST *tables, > #define CMP_GT 2 // Greater then > > static > -int compare_order_elements(ORDER *ord1, ORDER *ord2) > +int compare_order_elements(ORDER *ord1, int weight1, > + ORDER *ord2, int weight2) > { > if (*ord1->item == *ord2->item && ord1->direction == ord2->direction) > return CMP_EQ; > Item *item1= (*ord1->item)->real_item(); > Item *item2= (*ord2->item)->real_item(); > - DBUG_ASSERT(item1->type() == Item::FIELD_ITEM && > - item2->type() == Item::FIELD_ITEM); > - ptrdiff_t cmp= ((Item_field *) item1)->field - ((Item_field *) item2)->field; > + > + bool item1_field= (item1->type() == Item::FIELD_ITEM); > + bool item2_field= (item2->type() == Item::FIELD_ITEM); > + > + ptrdiff_t cmp; > + if (item1_field && item2_field) > + cmp= ((Item_field *) item1)->field - ((Item_field *) item2)->field; 1. why is this order stable? I'd rather sort by item->field->field_index 2. please, add an assert, that item1->field->table == item2->field->table > + else if (item1_field && !item2_field) > + return CMP_LT; > + else if (!item1_field && item2_field) > + return CMP_LT; > + else > + { > + /* > + Ok, item1_field==NULL and item2_field==NULL. > + We're not able to compare Item expressions. Order them according to > + their passed "weight" (which comes from Window_spec::win_spec_number): > + */ > + if (weight1 != weight2) > + cmp= weight1 - weight2; > + else > + { > + /* > + The weight is the same. That is, the elements come from the same > + window specification... This shouldn't happen. > + */ > + DBUG_ASSERT(0); > + cmp= item1 - item2; > + } > + } > + > if (cmp == 0) > { Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 1c0731e0544: MDEV-26695: Number of an invalid row is not calculated for table value constructor
by Sergei Golubchik 26 Apr '22

26 Apr '22

Hi, Rucha, Looks good. please also check that row_number values match what's in a warning, and then ok to push! On Apr 26, Rucha Deodhar wrote: > On Tue, Apr 26, 2022 at 12:41 AM Sergei Golubchik <serg(a)mariadb.org> wrote: > > what if you combine it with union? > > create table t (a char(1)) values ('a'),('b') union values ('foo'); > > > > please, add a test for that. > > > > > It gives 3 as row the number (in warning). Added a test for it. > Please find it here: 1c0731e0544db4d2f5c107473776bbd6d4d3e2f0 > > > On Apr 25, Rucha Deodhar wrote: > > > revision-id: 1c0731e0544 (mariadb-10.7.1-33-g1c0731e0544) > > > parent(s): 5e6fd4e8043 > > > author: Rucha Deodhar > > > committer: Rucha Deodhar > > > timestamp: 2022-01-23 13:48:18 +0530 > > > message: > > > > > > MDEV-26695: Number of an invalid row is not calculated for table value > > > constructor > > > > > > Analysis: counter does not increment while sending rows for table value > > > constructor and so row_number assumes the default value (0 in this case). > > > Fix: Increment the counter to avoid counter using default value. > > > > > > diff --git a/sql/sql_tvc.cc b/sql/sql_tvc.cc > > > index bdaf6829fbd..7db0ab217ca 100644 > > > --- a/sql/sql_tvc.cc > > > +++ b/sql/sql_tvc.cc > > > @@ -422,7 +422,9 @@ bool table_value_constr::exec(SELECT_LEX *sl) > > > DBUG_ENTER("table_value_constr::exec"); > > > List_iterator_fast<List_item> li(lists_of_values); > > > List_item *elem; > > > + THD *cur_thd= sl->parent_lex->thd; > > > ha_rows send_records= 0; > > > + int rc=0; > > > > > > if (select_options & SELECT_DESCRIBE) > > > DBUG_RETURN(false); > > > @@ -438,10 +440,10 @@ bool table_value_constr::exec(SELECT_LEX *sl) > > > > > > while ((elem= li++)) > > > { > > > + cur_thd->get_stmt_da()->inc_current_row_for_warning(); > > > if (send_records >= sl->master_unit()->lim.get_select_limit()) > > > break; > > > - int rc= > > > - result->send_data_with_check(*elem, sl->master_unit(), > > send_records); > > > + rc= result->send_data_with_check(*elem, sl->master_unit(), > > send_records); > > > > you don't need to do that anymore (it was needed in your first patch, > > but not here) > > > > Removed. > > > > if (!rc) > > > send_records++; > > > else if (rc > 0) > > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 1c0731e0544: MDEV-26695: Number of an invalid row is not calculated for table value constructor
by Sergei Golubchik 25 Apr '22

25 Apr '22

Hi, Rucha, what if you combine it with union? create table t (a char(1)) values ('a'),('b') union values ('foo'); please, add a test for that. On Apr 25, Rucha Deodhar wrote: > revision-id: 1c0731e0544 (mariadb-10.7.1-33-g1c0731e0544) > parent(s): 5e6fd4e8043 > author: Rucha Deodhar > committer: Rucha Deodhar > timestamp: 2022-01-23 13:48:18 +0530 > message: > > MDEV-26695: Number of an invalid row is not calculated for table value > constructor > > Analysis: counter does not increment while sending rows for table value > constructor and so row_number assumes the default value (0 in this case). > Fix: Increment the counter to avoid counter using default value. > > diff --git a/sql/sql_tvc.cc b/sql/sql_tvc.cc > index bdaf6829fbd..7db0ab217ca 100644 > --- a/sql/sql_tvc.cc > +++ b/sql/sql_tvc.cc > @@ -422,7 +422,9 @@ bool table_value_constr::exec(SELECT_LEX *sl) > DBUG_ENTER("table_value_constr::exec"); > List_iterator_fast<List_item> li(lists_of_values); > List_item *elem; > + THD *cur_thd= sl->parent_lex->thd; > ha_rows send_records= 0; > + int rc=0; > > if (select_options & SELECT_DESCRIBE) > DBUG_RETURN(false); > @@ -438,10 +440,10 @@ bool table_value_constr::exec(SELECT_LEX *sl) > > while ((elem= li++)) > { > + cur_thd->get_stmt_da()->inc_current_row_for_warning(); > if (send_records >= sl->master_unit()->lim.get_select_limit()) > break; > - int rc= > - result->send_data_with_check(*elem, sl->master_unit(), send_records); > + rc= result->send_data_with_check(*elem, sl->master_unit(), send_records); you don't need to do that anymore (it was needed in your first patch, but not here) > if (!rc) > send_records++; > else if (rc > 0) Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 585cd1f52e7: MDEV-26875: Wrong user in SET DEFAULT ROLE error
by Sergei Golubchik 25 Apr '22

25 Apr '22

Hi, Anel, On Apr 25, Anel Husakovic wrote: > revision-id: 585cd1f52e7 (mariadb-10.5.14-10-g585cd1f52e7) > parent(s): 52b32c60c26 > author: Anel Husakovic > committer: Anel Husakovic > timestamp: 2022-02-14 13:59:24 +0100 > message: > > MDEV-26875: Wrong user in SET DEFAULT ROLE error > > - Caused by 7c02e8717de5, where 957cb7b7ba35 introduced the bug. > > Reviewed by: > > diff --git a/sql/sql_acl.cc b/sql/sql_acl.cc > index c4e66cf5d73..e83bc5635dc 100644 > --- a/sql/sql_acl.cc > +++ b/sql/sql_acl.cc > @@ -3277,10 +3277,14 @@ static int check_user_can_set_role(THD *thd, const char *user, > check_role_is_granted_callback, > NULL) == -1)) > { > - /* Role is not granted but current user can see the role */ > - my_printf_error(ER_INVALID_ROLE, "User %`s@%`s has not been granted role %`s", > - MYF(0), thd->security_ctx->priv_user, > - thd->security_ctx->priv_host, rolename); > + /* If the SET ROLE is applied on the anonymous user, host is null */ > + if (!host) > + my_printf_error(ER_INVALID_ROLE, "User %`s@%`s has not been granted role %`s", > + MYF(0), thd->security_ctx->priv_user, thd->security_ctx->priv_host, rolename); I don't understand it. 1) what does it mean? why for anonymous user you print priv_user@priv_host ? 2) do you have any tests for that? There are none in the commit > + else > + /* Role is not granted but current user can see the role */ > + my_printf_error(ER_INVALID_ROLE, "User %`s@%`s has not been granted role %`s", > + MYF(0), user, host, rolename); > } > else > { Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 5f8ffc7271a: MDEV-20207: Assertion `! is_set()' failed in Diagnostics_area::set_eof_status upon HANDLER READ
by Sergei Golubchik 25 Apr '22

25 Apr '22

Hi, Rucha, The fix is ok to push. But as a general comment, better try to define variables in the smallest scope where they're used. In particular here it could've been { MY_BITMAP *old_map= dbug_tmp_use_all_columns(table, &table->write_set); - (void) item->save_in_field(key_part->field, 1); + int res= item->save_in_field(key_part->field, 1); because you don't need `res` outside of that scope. Makes the code easier to read. Of course, there can be exceptions. For example, if the variable is an object and its initialization is very expensive. But whether you move `int res` declaration inside or not - it's ok to push, no need to ask for another review. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org On Apr 25, Rucha Deodhar wrote: > revision-id: 5f8ffc7271a (mariadb-10.2.40-251-g5f8ffc7271a) > parent(s): 2cbf92522b5 > author: Rucha Deodhar > committer: Rucha Deodhar > timestamp: 2022-01-27 01:18:19 +0530 > message: > > MDEV-20207: Assertion `! is_set()' failed in Diagnostics_area::set_eof_status > upon HANDLER READ > > Analysis: The error state is not stored while checking condition and key name. > Fix: Return true while checking condition and key name if error is reported > because geometry object can't be created from the data in the index value for > HANDLER READ. > > diff --git a/sql/sql_handler.cc b/sql/sql_handler.cc > index ec3756eceba..7c002a68d9e 100644 > --- a/sql/sql_handler.cc > +++ b/sql/sql_handler.cc > @@ -554,6 +554,7 @@ mysql_ha_fix_cond_and_key(SQL_HANDLER *handler, > { > THD *thd= handler->thd; > TABLE *table= handler->table; > + int res= 0; > if (cond) > { > /* This can only be true for temp tables */ > @@ -613,8 +614,10 @@ mysql_ha_fix_cond_and_key(SQL_HANDLER *handler, > if (!in_prepare) > { > MY_BITMAP *old_map= dbug_tmp_use_all_columns(table, &table->write_set); > - (void) item->save_in_field(key_part->field, 1); > + res= item->save_in_field(key_part->field, 1); > dbug_tmp_restore_column_map(&table->write_set, old_map); > + if (res) > + return 1; > } > key_len+= key_part->store_length; > keypart_map= (keypart_map << 1) | 1;

1 0

Re: [Maria-developers] 14cc679c95e: MDEV-27831 Let the SQL SERVICE user set the current user name.
by Sergei Golubchik 22 Apr '22

22 Apr '22

Hi, Alexey, On Mar 04, Alexey Botchkov wrote: > revision-id: 14cc679c95e (mariadb-10.7.2-8-g14cc679c95e) > parent(s): 33fd136c61b > author: Alexey Botchkov > committer: Alexey Botchkov > timestamp: 2022-02-15 11:35:18 +0400 > message: > > MDEV-27831 Let the SQL SERVICE user set the current user name. > > The 'user' argument added to the mysql_real_connect_local. I think this is wrong on many levels. A plugin name is already known inside the plugin, you shouldn't force the plugin to pass its own name as an argument in multiple places, the server should determine it automatically. You should not set current_user to an arbitrary string for audit plugin to see it. current_user is the name of the user account and it's used in many places as such. Try, for example, to create a view or a stored procedure. Who will be a definer? Setting only user() might be ok. Setting @@proxy_user or @@external_user is even better, if your audit plugin can show them. @@external_user would be the best, I think it's purely informational. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 3

Re: [Maria-developers] 96b40a5e2f5: MDEV-27744 InnoDB: Failing assertion: !cursor->index->is_committed() in row0ins.cc (from row_ins_sec_index_entry_by_modify) | Assertion `0' failed in row_upd_sec_index_entry (debug) | Corruption
by Sergei Golubchik 22 Apr '22

22 Apr '22

Hi, Alexander, Some comments/questions here. I'll need to look at the patch again after I understand why you did it this way better. On Apr 18, Alexander Barkov wrote: > revision-id: 96b40a5e2f5 (mariadb-10.3.33-61-g96b40a5e2f5) > parent(s): 7355f7b1f5c > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2022-04-07 16:22:17 +0400 > message: > > MDEV-27744 InnoDB: Failing assertion: !cursor->index->is_committed() in row0ins.cc (from row_ins_sec_index_entry_by_modify) | Assertion `0' failed in row_upd_sec_index_entry (debug) | Corruption A detailed comment here is a must. What was the problem, how it was fixed. > diff --git a/sql/item_func.h b/sql/item_func.h > index 754b1cd1eb2..d844e45280a 100644 > --- a/sql/item_func.h > +++ b/sql/item_func.h > @@ -35,6 +35,36 @@ extern "C" /* Bug in BSDI include file */ > #include <cmath> > > > +template<class FUNC> > +FUNC* item_func_create_2_3_args(THD *thd, const LEX_CSTRING &name, > + List<Item> *item_list) > +{ > + int arg_count= item_list ? item_list->elements : 0; > + > + switch (arg_count) { > + case 2: > + { > + Item *param_1= item_list->pop(); > + Item *param_2= item_list->pop(); > + return new (get_thd_memroot(thd)) FUNC(thd, param_1, param_2); > + } > + case 3: > + { > + Item *param_1= item_list->pop(); > + Item *param_2= item_list->pop(); > + Item *param_3= item_list->pop(); > + return new (get_thd_memroot(thd)) FUNC(thd, param_1, param_2, param_3); A constructor that takes a List<Item> wouldn't need a switch. And wouldn't need a template either, your create_check_args_ge() isn't a template. > + break; > + } > + default: > + my_error(ER_WRONG_PARAMCOUNT_TO_NATIVE_FCT, MYF(0), name.str); > + break; > + } > + > + return NULL; > +} > + > + > class Item_func :public Item_func_or_sum > { > void sync_with_sum_func_and_with_field(List<Item> &list); > @@ -56,8 +86,41 @@ class Item_func :public Item_func_or_sum > bool check_argument_types_can_return_text(uint start, uint end) const; > bool check_argument_types_can_return_date(uint start, uint end) const; > bool check_argument_types_can_return_time(uint start, uint end) const; > + > + void print_schema_name_if_needed(String *to) const > + { > + const LEX_CSTRING schema_name= schema_name_cstring(); > + if (schema_name.length) > + { > + to->append(schema_name); > + to->append('.'); > + } this should check that schema_name isn't the same as the schema in the path (implicit path, assumed by the current sql_mode). > + } > + void print_maybe_qualified_name(String *to) const > + { > + print_schema_name_if_needed(to); > + to->append(func_name()); > + } > + > public: > > + // Print an error message for a builtin-schema qualified function call > + static void wrong_param_count_error(const LEX_CSTRING &schema_name, > + const LEX_CSTRING &func_name); > + > + // Check that the number of arguments is greater or equal to "expected" > + static bool create_check_args_ge(const LEX_CSTRING &name, > + const List<Item> *item_list, > + uint expected) > + { > + if (!item_list || item_list->elements < expected) > + { > + my_error(ER_WRONG_PARAMCOUNT_TO_NATIVE_FCT, MYF(0), name.str); what's the point of your wrong_param_count_error() if you aren't using it? > + return true; > + } > + return false; > + } > + > table_map not_null_tables_cache; > > enum Functype { UNKNOWN_FUNC,EQ_FUNC,EQUAL_FUNC,NE_FUNC,LT_FUNC,LE_FUNC, > diff --git a/sql/item_strfunc.cc b/sql/item_strfunc.cc > index 75abd9906cd..e45fe6ef6d7 100644 > --- a/sql/item_strfunc.cc > +++ b/sql/item_strfunc.cc > @@ -3204,6 +3239,39 @@ static String *default_pad_str(String *pad_str, CHARSET_INFO *collation) > return pad_str; > } > > + > +Item_func_lpad* > +Item_func_lpad::create(THD *thd, const LEX_CSTRING &name, > + List<Item> *item_list) > +{ > + return item_func_create_2_3_args<Item_func_lpad>(thd, name, item_list); may be item_func_create_2_3_args should be in Item_func? Like create_check_args_ge() is? and may be create_check_args_ge() should be a template too? > +} > + > + > +Item_func_lpad_oracle* > +Item_func_lpad_oracle::create(THD *thd, const LEX_CSTRING &name, > + List<Item> *item_list) > +{ > + return item_func_create_2_3_args<Item_func_lpad_oracle>(thd, name, item_list); > +} > + > + > +Item_func_rpad* > +Item_func_rpad::create(THD *thd, const LEX_CSTRING &name, > + List<Item> *item_list) > +{ > + return item_func_create_2_3_args<Item_func_rpad>(thd, name, item_list); > +} > + > + > +Item_func_rpad_oracle* > +Item_func_rpad_oracle::create(THD *thd, const LEX_CSTRING &name, > + List<Item> *item_list) > +{ > + return item_func_create_2_3_args<Item_func_rpad_oracle>(thd, name, item_list); > +} > + > + > bool Item_func_pad::fix_length_and_dec() > { > if (arg_count == 3) > diff --git a/sql/sql_lex.cc b/sql/sql_lex.cc > index 7f1b362d91d..6b0a351f76e 100644 > --- a/sql/sql_lex.cc > +++ b/sql/sql_lex.cc > @@ -1542,7 +1541,18 @@ int Lex_input_stream::lex_one_token(YYSTYPE *yylval, THD *thd) > if (lex->parsing_options.lookup_keywords_after_qualifier) > next_state= MY_LEX_IDENT_OR_KEYWORD; > else > - next_state= MY_LEX_IDENT_START; // Next is ident (not keyword) > + { > + /* > + Next is: > + - A qualified func with a special syntax: > + mariadb_schema.REPLACE('a','b','c') > + mariadb_schema.SUSTRING('a',1,2) > + mariadb_schema.TRIM('a') > + - Or an identifier otherwise. No keyword lookup is done, > + all keywords are treated as identifiers. > + */ > + next_state= MY_LEX_IDENT_OR_QUALIFIED_SPECIAL_FUNC; I don't understand why you did it in the lexer and not in the parser, like - | REPLACE '(' expr ',' expr ',' expr ')' + | opt_schema REPLACE '(' expr ',' expr ',' expr ')' or (if the above won't work in LARL(1) parser) - | ident_cli '.' ident_cli '(' opt_expr_list ')' + | ident_cli '.' func_2nd_part with + func_2nd_part: ident_cli '(' opt_expr_list ')' + | REPLACE '(' expr ',' expr ',' expr ')' etc (not reviewing lexer changes anymore, until I understand the above) > + } > if (!ident_map[(uchar) yyPeek()]) // Probably ` or " > next_state= MY_LEX_START; > return((int) c); Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 4

Re: [Maria-developers] 17548c8a8b6: MDEV-25477 Auto-create breaks replication when triggering event was not replicated
by Sergei Golubchik 20 Apr '22

20 Apr '22

Hi, Aleksey, Basically, it's fine, but I still didn't like that there's a new flag in THD to mark a statement that has to be logged. So I tried to change to log_current_statement. I know you tried that and you told me what tests fails, rpl.create_or_replace_*. I debugged that a bit, and I think the problem is that select_create inherits from select_insert, so CREATE ... SELECT runs your code that is only supposed to be run for INSERT ODKU or REPLACE. This change makes all tests pass (after removing vers_created_partitions): @@ -4225,7 +4225,8 @@ bool select_insert::prepare_eof() thd->clear_error(); else errcode= query_error_code(thd, killed_status == NOT_KILLED); - StatementBinlog stmt_binlog(thd, thd->binlog_need_stmt_format(trans_table)); + StatementBinlog stmt_binlog(thd, !can_rollback_data() && + thd->binlog_need_stmt_format(trans_table)); res= thd->binlog_query(THD::ROW_QUERY_TYPE, thd->query(), thd->query_length(), trans_table, FALSE, FALSE, errcode); On Apr 10, Aleksey Midenkov wrote: > revision-id: 17548c8a8b6 (mariadb-10.6.1-250-g17548c8a8b6) > parent(s): 6282e025a0e > author: Aleksey Midenkov > committer: Aleksey Midenkov > timestamp: 2022-03-07 23:49:35 +0300 > message: > > MDEV-25477 Auto-create breaks replication when triggering event was not replicated > > diff --git a/sql/sql_class.h b/sql/sql_class.h > index 686e6e70766..4fed1bf7355 100644 > --- a/sql/sql_class.h > +++ b/sql/sql_class.h > @@ -2914,6 +2914,13 @@ class THD: public THD_count, /* this must be first */ > int binlog_flush_pending_rows_event(bool stmt_end, bool is_transactional); > int binlog_remove_pending_rows_event(bool clear_maps, bool is_transactional); > > + bool binlog_need_stmt_format(bool is_transactional) const > + { > + return vers_created_partitions && !binlog_get_pending_rows_event(is_transactional); > + } > + > + bool binlog_for_noop_dml(bool transactional_table); > + > /** > Determine the binlog format of the current statement. > > @@ -3557,6 +3564,7 @@ class THD: public THD_count, /* this must be first */ > bool tablespace_op; /* This is TRUE in DISCARD/IMPORT TABLESPACE */ > /* True if we have to log the current statement */ > bool log_current_statement; > + uint vers_created_partitions; > /** > True if a slave error. Causes the slave to stop. Not the same > as the statement execution error (is_error()), since > diff --git a/sql/sql_insert.cc b/sql/sql_insert.cc > index 707b8a0d3bf..25551dfa197 100644 > --- a/sql/sql_insert.cc > +++ b/sql/sql_insert.cc > @@ -4223,6 +4225,7 @@ bool select_insert::prepare_eof() > thd->clear_error(); > else > errcode= query_error_code(thd, killed_status == NOT_KILLED); > + StatementBinlog stmt_binlog(thd, thd->binlog_need_stmt_format(trans_table)); > res= thd->binlog_query(THD::ROW_QUERY_TYPE, > thd->query(), thd->query_length(), > trans_table, FALSE, FALSE, errcode); Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 8

Re: [Maria-developers] 93493a0e9b5: MDEV-24176 Server crashes after insert in the table with virtual column generated using date_format() and if()
by Sergei Golubchik 16 Apr '22

16 Apr '22

Hi, Aleksey! Could you please explain in the commit comment what the bug was and what you are fixing? I wasn't able to understad that, so I had to debug those crashes myself. So, the reason is, objects allocated in the wrong arena, right? And it was done inconsistently, on different code paths the "re-fix" context was different. I agree that encapsulating all the context in one Vcol_expr_context is a good idea, it'll allow to have the same, correct, context everywhere. But always using expr_arena doesn't necessarily seem to be a correct choice. E.g. Arg_comparator creates Item_cache_temporal on every fix_field. If vcol expr is re-fixed all the time, expr_arena will grow continuously. Same for charset conversion, CASE, IN, etc. I suspect that if the vcol expr is always re-fixed, it has to use stmt arena. Why do you re-fix fields? I can hardly imagine that an Item_field can find itself in a different table. And if it's always in the same table, why to force it to find this table over and over? We've already discussed VCOL_TABLE_ALIAS, no need to go there again. On Jan 06, Aleksey Midenkov wrote: > revision-id: 93493a0e9b5 (mariadb-10.2.40-194-g93493a0e9b5) > parent(s): a565e63c54c > author: Aleksey Midenkov > committer: Aleksey Midenkov > timestamp: 2021-12-29 00:46:31 +0300 > message: > > MDEV-24176 Server crashes after insert in the table with virtual > column generated using date_format() and if() > > When table is reopened from cache vcol_info contains stale > expression. We refresh expression via TABLE::vcol_fix_exprs() but > first we must prepare a proper context (Vcol_expr_context) and do some > preparations which meet some requirements: > > 1. fields must not be fixed, we unfix them with cleanup() via > cleanup_processor. > > 2. Item_ident must have cached_table cleared. Again, this is stale > data which was already freed. cached_table interferes with > find_field_in_tables(). > > We cannot clear cached_table directly in Item_ident::cleanup() > method. Lots of spaghetti logic depends on cached_table existence even > after cleanup() done. > > 3. If Item_ident has table_name non-NULL we trigger expr update. That > is done via Item_ident::check_vcol_func_processor() and > VCOL_TABLE_ALIAS flag. > > (4-7) The below conditions are prepared by Vcol_expr_context and used in > both fix_session_expr_for_read() and TABLE::vcol_fix_exprs(): > > 4. Any fix_expr() must be done on expr_arena as there may be new items > created. It was a bug in fix_session_expr_for_read(). It was just not > reproduced because there were no second refix. Now refix is done for > more cases so it does reproduce. Tests affected: vcol.binlog > > 5. Also name resolution context must be narrowed to the single table > with all crutches in place. Tests affected: vcol.update > > 6. sql_mode must be clean and not fail expr update. > > sql_mode such as MODE_NO_BACKSLASH_ESCAPES, MODE_NO_ZERO_IN_DATE, etc > must not affect vcol expression update. If the table was created > successfully any further evaluation must not fail. Tests affected: > main.func_like > > 7. Warnings are disabled during expr update. It is assumed that these > warnings were printed before during initialization phase or later > during execution phase. Tests affected: vcol.wrong_arena > > Dictionary: refresh, update and refix are the synonyms for > vcol_fix_exprs() or fix_session_expr_for_read() calls. > > 8. cleanup_excluding_fields_processor() removed. It was just a quick > hack to exclude wrongly working Item_field from processing. Now it > works due to correct execution environment (Vcol_expr_context). > Related to MDEV-10355. > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 12

Re: [Maria-developers] 62d3496969d: MDEV-24176 Server crashes after insert in the table with virtual column generated using date_format() and if()
by Sergei Golubchik 12 Apr '22

12 Apr '22

Hi, Aleksey, This is a review of `git diff 5fbbdbc85ba3 62d3496969db` Basically, no new comments, see below: On Apr 07, Aleksey Midenkov wrote: > diff --git a/libmariadb b/libmariadb > index f6c3d9fd2af..735a7299dba 160000 > --- a/libmariadb > +++ b/libmariadb > @@ -1 +1 @@ > -Subproject commit f6c3d9fd2af5d17db64cc996574aa312efd70fcf > +Subproject commit 735a7299dbae19cc2b82b9697becaf90e9b43047 you've mistakenly checked in a submodule > diff --git a/sql/field.cc b/sql/field.cc > index f18fb25ebe3..04c492e6c69 100644 > --- a/sql/field.cc > +++ b/sql/field.cc > @@ -2416,6 +2416,13 @@ int Field::set_default() > if (default_value) > { > Query_arena backup_arena; > + /* > + TODO: this imposes memory leak until table flush when save_in_field() > + does expr_arena allocation. F.ex. case from main.default: > + > + CREATE TABLE t1 (a INT DEFAULT CONCAT('1 ')); > + INSERT INTO t1 VALUES (DEFAULT); I wasn't able to repeat that. In your branch (didn't try in vanilla 10.3) and for the above test case I see no additional allocations in table->mem_root after the table is opened. > + */ > table->in_use->set_n_backup_active_arena(table->expr_arena, &backup_arena); > int rc= default_value->expr->save_in_field(this, 0); > table->in_use->restore_active_arena(table->expr_arena, &backup_arena); > diff --git a/sql/table.cc b/sql/table.cc > index 08d91678b25..d22da17c31a 100644 > --- a/sql/table.cc > +++ b/sql/table.cc > @@ -2970,36 +2988,133 @@ static bool fix_vcol_expr(THD *thd, Virtual_column_info *vcol) ... > +bool Vcol_expr_context::init() > +{ > + /* > + As this is vcol expression we must narrow down name resolution to > + single table. > + */ > + if (init_lex_with_single_table(thd, table, &lex)) I've already asked about it in a separate email, so here I'm just marking a line that we haven't fully agreed on yet > + { > + my_error(ER_OUT_OF_RESOURCES, MYF(0)); > + table->map= old_map; > + return true; > + } > + > + lex.sql_command= old_lex->sql_command; > + thd->variables.sql_mode= 0; > + > + TABLE_LIST const *tl= table->pos_in_table_list; > + DBUG_ASSERT(table->pos_in_table_list); > + > + if (table->pos_in_table_list->security_ctx) > thd->security_ctx= tl->security_ctx; > - bool res= fix_session_vcol_expr(thd, vcol); > + > + inited= true; > + return false; > +} > + Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] e3e59c9e38d: MDEV-28266 Crash in Field_string::type_handler when calling procedures
by Sergei Golubchik 10 Apr '22

10 Apr '22

Hi, Oleksandr, On Apr 10, Oleksandr Byelkin wrote: > revision-id: e3e59c9e38d (mariadb-10.6.6-83-ge3e59c9e38d) > parent(s): 4e1ca388381 > author: Oleksandr Byelkin > committer: Oleksandr Byelkin > timestamp: 2022-04-08 10:37:17 +0200 > message: > > MDEV-28266 Crash in Field_string::type_handler when calling procedures > > In case of only metadata request the cursor lack the call to redirect > Field::orig_table to the temporary table. > > diff --git a/sql/sql_cursor.cc b/sql/sql_cursor.cc > index 0add5845558..155346341f1 100644 > --- a/sql/sql_cursor.cc > +++ b/sql/sql_cursor.cc > @@ -183,6 +183,16 @@ int mysql_open_cursor(THD *thd, select_result *result, > { > Materialized_cursor *materialized_cursor= > result_materialize->materialized_cursor; > + if (result_materialize->view_structure_only()) > + { > + /* > + If it was view_structure_only then only send metadata call was made > + and there was no other call. So we have to "close" the cursor > + explicitly (which redirect orig_table in the fields, and untie > + it with closed tables) > + */ > + materialized_cursor->on_table_fill_finished(); > + } > > if ((rc= materialized_cursor->open(0))) > { > delete materialized_cursor; > goto end; > } normally on_table_fill_finished() is called from Select_materialize::send_eof() and Select_materialize::abort_result_set(). Note that directly after your added code there's materialized_cursor->open(), and there we have int Materialized_cursor::open(JOIN *join __attribute__((unused))) { ... if (!rc) { thd->server_status|= SERVER_STATUS_CURSOR_EXISTS; result->send_eof(); } else { result->abort_result_set(); } That is, one could expect that even without your added code, on_table_fill_finished() would be called. The problem is that result here is sp_cursor::Select_fetch_into_spvars, and sp_cursor::Select_fetch_into_spvars::send_eof() is simply { return FALSE; } So, as a fix I'd suggest to remove on_table_fill_finished() from send_eof/abort_result_set, and put it unconditionally after the if(), to make it work no matter what select_result was chosen. Like this: ====================================== --- a/sql/sql_cursor.cc +++ b/sql/sql_cursor.cc @@ -80,19 +80,7 @@ class Select_materialize: public select_unit Select_materialize(THD *thd_arg, select_result *result_arg): select_unit(thd_arg), result(result_arg), materialized_cursor(0) {} virtual bool send_result_set_metadata(List<Item> &list, uint flags); - bool send_eof() - { - if (materialized_cursor) - materialized_cursor->on_table_fill_finished(); - return false; - } - - void abort_result_set() - { - if (materialized_cursor) - materialized_cursor->on_table_fill_finished(); - } - + bool send_eof() { return false; } bool view_structure_only() const { return result->view_structure_only(); @@ -333,6 +321,8 @@ int Materialized_cursor::open(JOIN *join result->abort_result_set(); } + on_table_fill_finished(); + return rc; } ====================================== with the commit comment saying that "on_table_fill_finished() should always be done at the end of open(), even if result is not Select_materialize, but, for example, Select_fetch_into_spvars" Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] a10facb656b: MDEV-27065 Partitioning tables with custom data directories moves data back to default directory
by Sergei Golubchik 29 Mar '22

29 Mar '22

Hi, Nayuta, On Mar 29, Nayuta Yanagisawa wrote: > revision-id: a10facb656b (mariadb-10.2.43-48-ga10facb656b) > parent(s): 33ff18627ea > author: Nayuta Yanagisawa > committer: Nayuta Yanagisawa > timestamp: 2022-03-29 23:43:16 +0900 > message: > > MDEV-27065 Partitioning tables with custom data directories moves data back to default directory > > --- > .../suite/parts/r/partition_exch_qa_14.result | 22 ++++++++++++++++++++++ > sql/ha_partition.cc | 16 ++++++++++++++++ > 2 files changed, 38 insertions(+) > > diff --git a/mysql-test/suite/parts/r/partition_exch_qa_14.result b/mysql-test/suite/parts/r/partition_exch_qa_14.result > index 1420982436a..52839644f55 100644 > --- a/mysql-test/suite/parts/r/partition_exch_qa_14.result > +++ b/mysql-test/suite/parts/r/partition_exch_qa_14.result > @@ -9,6 +9,9 @@ PARTITION BY RANGE (a) > (PARTITION p0 VALUES LESS THAN (10) DATA DIRECTORY = 'MYSQLTEST_VARDIR/mysql-test-data-dir' INDEX DIRECTORY = 'MYSQLTEST_VARDIR/mysql-test-idx-dir', > PARTITION p1 VALUES LESS THAN (100) DATA DIRECTORY = 'MYSQLTEST_VARDIR/mysql-test-data-dir' INDEX DIRECTORY = 'MYSQLTEST_VARDIR/mysql-test-idx-dir', > PARTITION p2 VALUES LESS THAN (1000) DATA DIRECTORY = 'MYSQLTEST_VARDIR/mysql-test-data-dir' INDEX DIRECTORY = 'MYSQLTEST_VARDIR/mysql-test-idx-dir'); > +Warnings: > +Warning 1618 <DATA DIRECTORY> option ignored > +Warning 1618 <INDEX DIRECTORY> option ignored This is very confusing. There are four DATA DIRECTORY and four INDEX DIRECTORY in the statement. Which one is ignored? (yes, *I* know, the table level one) > @@ -148,6 +154,9 @@ PARTITION BY RANGE (a) > (PARTITION p0 VALUES LESS THAN (10) , > PARTITION p1 VALUES LESS THAN (100) , > PARTITION p2 VALUES LESS THAN (1000) ); > +Warnings: > +Warning 1618 <DATA DIRECTORY> option ignored > +Warning 1618 <INDEX DIRECTORY> option ignored and here DATA/INDEX DIRECTORY behaves not like other options. For a normal option table-level value would apply to all partitions. That is CREATE TABLE t1 (...) OPT="VAL (PARTITION p0 VALUES LESS THAN (10) , PARTITION p1 VALUES LESS THAN (100) , PARTITION p2 VALUES LESS THAN (1000) ); would be the same as CREATE TABLE t1 (...) (PARTITION p0 VALUES LESS THAN (10) OPT="VAL, PARTITION p1 VALUES LESS THAN (100) OPT="VAL, PARTITION p2 VALUES LESS THAN (1000) OPT="VAL); Is that correct? > CREATE TABLE tsp (a INT, b VARCHAR(55), PRIMARY KEY (a)) DATA DIRECTORY = 'MYSQLTEST_VARDIR/mysql-test-data-dir' INDEX DIRECTORY = 'MYSQLTEST_VARDIR/mysql-test-idx-dir' ENGINE = MYISAM > PARTITION BY RANGE (a) > SUBPARTITION BY HASH(a) Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 88f8aa20bba: MDEV-23486 RBR can bypass secure_timestamp=YES
by Sergei Golubchik 26 Mar '22

26 Mar '22

Hi, Aleksey! On Jan 09, Aleksey Midenkov wrote: > revision-id: 88f8aa20bba (mariadb-10.3.31-79-g88f8aa20bba) > parent(s): d5451867af1 > author: Aleksey Midenkov > committer: Aleksey Midenkov > timestamp: 2021-12-23 04:15:39 +0300 > message: > > MDEV-23486 RBR can bypass secure_timestamp=YES > > Pass to a slave whether master table is system-versioned via > TM_SYSTEM_VERSIONED flag. That deprecates m_vers_from_plain > detection. If secure_timestamp == YES or master did not supply row_end > data treat the row as current data and generate timestamps for it. I don't understand why does it matter whether the master is system versioned or not. secure_timestamp=YES means the table has to store actual timestamps when rows were changed. It doesn't trust the master to set timestamps. So it should not matter whether master table is versioned or not, if you don't trust the master, you cannot trust its TM_SYSTEM_VERSIONED flag either. Even if the table was system versioned on the master, the master could've had secure_timestamp=NO and a completely fake history. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] MDEV-24560 SIGSEGV in st_join_table::cleanup
by Sergey Petrunia 25 Mar '22

25 Mar '22

Hi Oleg, > commit 4f41909c3ee1d7e42c56947c9a9156fb78c22d8a > Author: Oleg Smirnov <olernov(a)gmail.com> > Date: Fri Mar 11 21:18:34 2022 +0700 > > MDEV-24560 SIGSEGV in st_join_table::cleanup > > If JOIN::create_postjoin_aggr_table encounters errors during execution > then free_tmp_table() is then called twice for JOIN_TAB::aggr. > The solution is to initialize JOIN_TAB::aggr and JOIN_TAB::table only > on successful completion of JOIN::create_postjoin_aggr_table > > diff --git a/sql/sql_select.cc b/sql/sql_select.cc > index c044656e916..8c07014d981 100644 > --- a/sql/sql_select.cc > +++ b/sql/sql_select.cc > @@ -4037,10 +4037,6 @@ JOIN::create_postjoin_aggr_table(JOIN_TAB *tab, List<Item> *table_fields, > !tables_list); > if (tab > join_tab) > (tab - 1)->next_select= sub_select_postjoin_aggr; > - if (!(tab->aggr= new (thd->mem_root) AGGR_OP(tab))) > - goto err; > - tab->table= table; > - table->reginfo.join_tab= tab; > > /* if group or order on first table, sort first */ > if ((group_list && simple_group) || > @@ -4089,7 +4085,10 @@ JOIN::create_postjoin_aggr_table(JOIN_TAB *tab, List<Item> *table_fields, > order= NULL; > } > } > - > + if (!(tab->aggr= new (thd->mem_root) AGGR_OP(tab))) > + goto err; > + tab->table= table; > + table->reginfo.join_tab= tab; > DBUG_RETURN(false); I see a potential issue in the patch: The moved code does this assignment: > tab->table= table; Let's look inside this if-branch: >> if ((group_list && simple_group) || >> (implicit_grouping && select_lex->have_window_funcs())) The code inside if() { ... } part seems to be fine: it does not touch the contents of 'tab', in particular tab->table. Let's look at the coed inside the else {... } part: It has this condition: >> if (!group_list && !table->distinct && order && simple_order && >> tab == join_tab + const_tables) >> { Note that tab==join_tab+const_tables. Then, a few lines below: >> add_sorting_to_table(join_tab + const_tables, order)) This is essentially add_sorting_to_table(tab, order). Inside JOIN::add_sorting_to_table one can find this: TABLE *table= tab->table; if ((tab == join_tab + const_tables) && table->pos_in_table_list && table->pos_in_table_list->is_sjm_scan_table()) that is, we access tab->table and tab->table->pos_in_table_list. Which are not assigned yet (because your patch moves the assignment to happen later). I've tried to construct an example to demonstrate this issue, but was not successful. I've made this change: diff --git a/sql/sql_select.cc b/sql/sql_select.cc index c044656e916..5a9a7d7e73f 100644 --- a/sql/sql_select.cc +++ b/sql/sql_select.cc @@ -4080,6 +4080,8 @@ JOIN::create_postjoin_aggr_table(JOIN_TAB *tab, List<Item> *table_fields, tab == join_tab + const_tables) { DBUG_PRINT("info",("Sorting for order")); + fprintf(stderr,"AAABBBCCC:%s\n", thd->query()); + DBUG_ASSERT(0); THD_STAGE_INFO(thd, stage_sorting_for_order); if (ordered_index_usage != ordered_index_order_by && and ran the entire testsuite with it. There were no failures. I'm not sure if this is deadcode or we just don't have enough test coverage. I suggest to assume the latter and change the patch to avoid this issue. BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://petrunia.net

1 0

Re: [Maria-developers] 3be1e491fb5: MDEV-27743 Remove Lex::charset
by Sergei Golubchik 22 Mar '22

22 Mar '22

Hi, Alexander, All comments from the previous review were addressed and there were no other changes. So, ok to push. On Mar 22, Alexander Barkov wrote: > revision-id: 3be1e491fb5 (mariadb-10.6.1-383-g3be1e491fb5) > parent(s): c69defd3b61 > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2022-03-21 21:33:30 +0400 > message: > > MDEV-27743 Remove Lex::charset > > This patch also fixes: > > MDEV-27690 Crash on `CHARACTER SET csname COLLATE DEFAULT` in column definition > MDEV-27853 Wrong data type on column `COLLATE DEFAULT` and table `COLLATE some_non_default_collation` > MDEV-28067 Multiple conflicting column COLLATE clauses are not rejected > MDEV-28118 Wrong collation of `CAST(.. AS CHAR COLLATE DEFAULT)` > MDEV-28119 Wrong column collation on MODIFY + CONVERT > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 026e52bad84: MDEV-27743 Remove Lex::charset
by Sergei Golubchik 21 Mar '22

21 Mar '22

Hi, Alexander, Please, see below On Mar 18, Alexander Barkov wrote: > revision-id: 026e52bad84 (mariadb-10.6.1-369-g026e52bad84) > parent(s): 369498d6b8c > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2022-03-18 12:26:03 +0400 > message: > > MDEV-27743 Remove Lex::charset > > This patch also fixes: > > MDEV-27690 Crash on `CHARACTER SET csname COLLATE DEFAULT` in column definition > MDEV-27853 Wrong data type on column `COLLATE DEFAULT` and table `COLLATE some_non_default_collation` > MDEV-28067 Multiple conflicting column COLLATE clauses are not rejected > MDEV-28118 Wrong collation of `CAST(.. AS CHAR COLLATE DEFAULT)` > MDEV-28119 Wrong column collation on MODIFY + CONVERT > > diff --git a/mysql-test/main/ctype_collate_column.test b/mysql-test/main/ctype_collate_column.test > new file mode 100644 > index 00000000000..6a7cf2cc84c > --- /dev/null > +++ b/mysql-test/main/ctype_collate_column.test > @@ -0,0 +1,637 @@ ... > + > +DELIMITER $$; > +CREATE PROCEDURE p1(dt TEXT, cs TEXT, cl0 TEXT, cl1 TEXT, cl2 TEXT, tcs TEXT) > +BEGIN > + DECLARE errstate TEXT DEFAULT NULL; > + DECLARE errno INT DEFAULT NULL; > + DECLARE errmsg TEXT DEFAULT NULL; > + DECLARE query TEXT; > + DECLARE result TEXT; > + DECLARE CONTINUE HANDLER FOR SQLEXCEPTION > + BEGIN > + GET DIAGNOSTICS CONDITION 1 errstate = RETURNED_SQLSTATE, > + errno = MYSQL_ERRNO, errmsg = MESSAGE_TEXT; > + END; > + SET query= CONCAT('CREATE TABLE t1 (a ', dt, ' ', cs, ' ', cl0, > + ' NOT NULL ',cl1, > + ' DEFAULT '''' ', cl2, > + ') ', tcs, ' ENGINE=Memory'); > + EXECUTE IMMEDIATE query; > + IF errmsg IS NOT NULL > + THEN > + SET result=CONCAT('ERROR: ', > + COALESCE(errstate,'<NULL>'), ' ', > + COALESCE(errno,'<NULL>'), ' ', > + COALESCE(errmsg,'<NULL>')); > + INSERT INTO results (dt,cs,cl0,cl1,cl2,tcs,query,result) > + VALUES (dt,cs,cl0,cl1,cl2,tcs,query,result); > + ELSE > + FOR row IN (SELECT CONCAT(COLUMN_TYPE, > + ' CHARACTER SET ', CHARACTER_SET_NAME, > + ' COLLATE ', COLLATION_NAME) AS result > + FROM INFORMATION_SCHEMA.COLUMNS > + WHERE TABLE_SCHEMA='test' AND TABLE_NAME='t1') > + DO > + INSERT INTO results (dt,cs,cl0,cl1,cl2,tcs,query,result) > + VALUES (dt,cs,cl0,cl1,cl2,tcs,query,row.result); > + END FOR; > + DROP TABLE t1; > + END IF; > +END; > +$$ > +DELIMITER ;$$ > + > + > +DELIMITER $$; > +CREATE PROCEDURE p3(dt TEXT) > +BEGIN > + FOR row IN ( > + SELECT cs, cl0, cl1.cl1 AS cl1, cl2.cl1 AS cl2, tcs > + FROM cs, cl0, cl1, cl1 AS cl2, tcs > + ORDER BY cs, cl0, cl1, cl2, tcs > + ) > + DO > + CALL p1('char(10)', row.cs, row.cl0, row.cl1, row.cl2, row.tcs); minor detail: here ^^^ you want to use dt, not 'char(10)' > + END FOR; > +END; > +$$ > +DELIMITER ;$$ > + > + > +--disable_column_names > +CALL p3('char(10)'); > +--enable_column_names > + > +SELECT * FROM results_statistics; isn't it redundant? you select all rows from result anyway. If something will change, the result of that select will already cause the the to fail. Why do you need second mismatch in the result/reject. > + > +--vertical_results > +SELECT query, result, '' AS `` FROM results > +ORDER BY dt, cs, cl0, cl1, cl2, tcs; > +--horizontal_results > + > +DROP PROCEDURE p1; > +DROP PROCEDURE p3; > + > +DROP TABLE cs, cl0, cl1, tcs; ... > +# CREATE TABLE t1 > +# ( > +# a CHAR(10) BINARY NOT NULL DEFAULT '' > +# ) CHARACTER SET cs COLLATE cs_bin; > + > +DELETE FROM results > +WHERE cs='' > + AND cl0='BINARY' > + AND cl1='' > + AND cl2='' > + AND tcs LIKE 'CHARACTER SET%' > + AND result NOT LIKE 'ERROR%' > + AND result RLIKE CONCAT('COLLATE ', tcs_character_set_name, '_bin'); > +SELECT ROW_COUNT(); ... > +DROP FUNCTION is_collate_clause_with_explicit_default_collation; > +DROP FUNCTION is_collate_clause_with_explicit_collation; > +DROP FUNCTION is_conflicting_charset_explicit_collate_explicit; > +DROP FUNCTION is_conflicting_collate_explicit2; > +DROP FUNCTION is_conflicting_collate_default_collate_explicit; > +DROP FUNCTION collate_cs_default_collation; I think the main result is `select * from results` (with order by, etc). It's great, that you added that. Everything else is redundant, I suggest to remove other selects and deletes. > + > + > +--echo # > +--echo # End of 10.9 tests > +--echo # > diff --git a/sql/lex_charset.h b/sql/lex_charset.h > new file mode 100644 > index 00000000000..d95f0bc1920 > --- /dev/null > +++ b/sql/lex_charset.h > @@ -0,0 +1,202 @@ ... > +#define LEX_CHARSET_COLLATION_TYPE_BITS 2 > + static_assert(((1<<LEX_CHARSET_COLLATION_TYPE_BITS)-1) >= > + TYPE_COLLATE_CONTEXTUALLY_TYPED, > + "Lex_charset_collation_st::Type bits check"); > + > +protected: > + CHARSET_INFO *m_ci; > + Type m_type; > +public: > + static CHARSET_INFO *find_bin_collation(CHARSET_INFO *cs); > + static CHARSET_INFO *find_default_collation(CHARSET_INFO *cs); > +public: ... > + void set_contextually_typed_binary_style() > + { > + m_ci= &my_collation_contextually_typed_binary; > + m_type= TYPE_COLLATE_CONTEXTUALLY_TYPED; > + } > + bool is_contextually_typed_collate_default() const > + { > + return m_ci == &my_collation_contextually_typed_default && > + m_type == TYPE_COLLATE_CONTEXTUALLY_TYPED; what does it mean when only one of these two expressions is true? Say, the first true, the second false? Or the first false, the second true? > + } > + bool is_contextually_typed_binary_style() const > + { > + return m_ci == &my_collation_contextually_typed_binary && > + m_type == TYPE_COLLATE_CONTEXTUALLY_TYPED; > + } ... > diff --git a/strings/ctype-bin.c b/strings/ctype-bin.c > index bb746ad90b0..a5c9129ebea 100644 > --- a/strings/ctype-bin.c > +++ b/strings/ctype-bin.c > @@ -630,3 +630,69 @@ struct charset_info_st my_charset_bin = > &my_charset_handler, > &my_collation_binary_handler > }; > + > + > +struct charset_info_st my_collation_contextually_typed_binary= > +{ > + 0,0,0, /* number */ I thought you will never need to resove the &my_collation_contextually_typed_binary pointer, that is, I thought the values in the my_collation_contextually_typed_binary struct should not matter as they'll never be used. Was that wrong? Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 2

Re: [Maria-developers] d8e915a88fc: MDEV-26009 Server crash when calling twice procedure using FOR-loop
by Sergei Golubchik 20 Mar '22

20 Mar '22

Hi, Oleksandr, ok to push also, see below On Mar 20, Oleksandr Byelkin wrote: > revision-id: d8e915a88fc (mariadb-10.3.33-32-gd8e915a88fc) > parent(s): 6a2d88c1322 > author: Oleksandr Byelkin > committer: Oleksandr Byelkin > timestamp: 2022-03-18 11:13:09 +0100 > message: > > MDEV-26009 Server crash when calling twice procedure using FOR-loop > > The problem was that instructions sp_instr_cursor_copy_struct and > sp_instr_copen uses the same lex, adding and removing "tail" of > prelocked tables and forgetting that tail of all tables is kept in > LEX::query_tables_last. If the LEX used only by one instruction > or the query do not have prelocked tables it is not important. > But to work correctly in all cases LEX::query_tables_last should > be reset to make new tables added in the correct list (after last > table in the LEX instead after last table of the prelocking "tail" > which was cut). > > diff --git a/sql/sp_head.cc b/sql/sp_head.cc > index 57ab31d9edf..96dc78dcf49 100644 > --- a/sql/sp_head.cc > +++ b/sql/sp_head.cc > @@ -3486,6 +3486,11 @@ sp_lex_keeper::reset_lex_and_exec_core(THD *thd, uint *nextp, > lex_query_tables_own_last= m_lex->query_tables_own_last; > prelocking_tables= *lex_query_tables_own_last; > *lex_query_tables_own_last= NULL; > + /* > + Here we cut "tail" of prelocked tables, so have to return last table > + to the original position (wuthout the tail > + */ > + m_lex->query_tables_last= m_lex->query_tables_own_last; 1. the comment looks truncated 2. I don't even think this line needs a comment, this code block removes the list of prelocked tables from the query_tables list. Looking at how query_tables_last and query_tables_own_last are documented: /* Pointer to next_global member of last element in the previous list. */ TABLE_LIST **query_tables_last; /* If non-0 then indicates that query requires prelocking and points to next_global member of last own element in query table list (i.e. last table which was not added to it as part of preparation to prelocking). 0 - indicates that this query does not need prelocking. */ TABLE_LIST **query_tables_own_last; it is pretty clear that after the "tail" is removed, one has to set query_tables_last to point to the end of the list. The fact that it wasn't done was clearly an omission. That is, existing code and existing comments document it pretty well, your comment looks redundant and even confusing to me. So, please, remove the comment and ok to push > m_lex->mark_as_requiring_prelocking(NULL); > } > thd->rollback_item_tree_changes(); Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] Google Summer of Code 2022
by Vicențiu Ciorbaru 20 Mar '22

20 Mar '22

Hi Sergey! I'm happy to see that you have experience in MariaDB code base already. The MDEV-18827 is definitely up for grabs for GSoC 2022. The project is mostly a part-time project (requiring a commitment of around 175 hours), however depending on your availability we can increase the scope of the project and bump it up to a full time project. My suggestion for this task is to start looking at code parsing FRM files. Search for the function TABLE_SHARE::init_from_binary_frm_image to get an idea how an .frm is structured. After that, think how you would like to implement the feature described in MDEV-18827. Feel free to submit a draft proposal and we can iterate on it. Not 100% sure how the GSoC interface works this year, but last time you were able to share with potential mentors a Google Doc with the possibility to comment. Have a look at the proposal guidelines https://mariadb.org/gsoc2022-instructions/ and start writing it. When the program application opens you can submit it (4th of April). In the meantime, feel free to ask any questions you might have here. I encourage you to keep all discussions on the mailing list as much as possible so others can chime in. Vicențiu On Sat, 19 Mar 2022 at 09:54, Сергей Ваниславский <serzhmn(a)gmail.com> wrote: > Hello Vicentiu! > My name is Vanislavsky Sergey. Your colleague Nikita Malyavin > recommended contacting you about GSoC.I would like to do your task > MDEV-18827. I am a 3rd year student at FEFU. Here is my last pull request: > https://github.com/MariaDB/server/pull/2016 > Thank you for your consideration. >

1 0

[Maria-developers] Relevance of MDEV-19556
by Vladisav Kakurin 17 Mar '22

17 Mar '22

Hi, Vicențiu! With my mentor Nikita Malyavin we chose the topic of my bachelor's graduation work, and I was really interested in the MDEV-19556 <https://jira.mariadb.org/browse/MDEV-19556> ticket. Could you say some words about this, maybe the ticket ain’t actual, or something about work that was done by one of the developers to this direction?

1 0

Re: [Maria-developers] 2467eb2d314: MDEV-27743 Remove Lex::charset
by Sergei Golubchik 16 Mar '22

16 Mar '22

Hi, Alexander, I had only a couple of questions/comments, see below: On Mar 11, Alexander Barkov wrote: > revision-id: 2467eb2d314 (mariadb-10.6.1-330-g2467eb2d314) > parent(s): 61b72ed3dde > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2022-02-10 10:27:55 +0400 > message: > > MDEV-27743 Remove Lex::charset > > This patch also fixes: > MDEV-27690 Crash on `CHARACTER SET csname COLLATE DEFAULT` in column definition how do you plan to fix it in 10.2-10.8 ? > > diff --git a/mysql-test/main/ctype_collate_column.result b/mysql-test/main/ctype_collate_column.result > new file mode 100644 > index 00000000000..b20be5d3a48 > --- /dev/null > +++ b/mysql-test/main/ctype_collate_column.result > @@ -0,0 +1,450 @@ > +# > +# Start of 10.9 tests > +# > +# > +# MDEV-27743 Remove Lex::charset 1. Jira doesn't explain what user-visible changes one can expect. If there are none, it's not clear why you needed this big test case 2. the test is totally unreadable. Could you do a normal test instead? Like, put `select` instead of `execute immediate` and convert the output to a .test file? > +# > +CREATE TABLE cs (cs VARCHAR(64) NOT NULL); > +INSERT INTO cs (cs) VALUES > +(''), > +('CHARACTER SET latin1'), > +('CHARACTER SET utf8mb4'); > +CREATE TABLE cl0 (cl0 VARCHAR(64) NOT NULL); > +INSERT INTO cl0 (cl0) VALUES > +(''), > +('BINARY'), > +('COLLATE DEFAULT'), > +('COLLATE latin1_bin'), > +('COLLATE latin1_swedish_ci'), > +('COLLATE utf8mb4_bin'), > +('COLLATE utf8mb4_general_ci'); > +CREATE TABLE cl1 (cl1 VARCHAR(64) NOT NULL); > +INSERT INTO cl1 (cl1) VALUES > +(''), > +('COLLATE DEFAULT'), > +('COLLATE latin1_bin'), > +('COLLATE latin1_swedish_ci'), > +('COLLATE utf8mb4_bin'), > +('COLLATE utf8mb4_general_ci'); ... > diff --git a/mysql-test/main/ctype_latin1.result b/mysql-test/main/ctype_latin1.result > index 0b16d1cf6f2..70577f2c22c 100644 > --- a/mysql-test/main/ctype_latin1.result > +++ b/mysql-test/main/ctype_latin1.result > @@ -8889,6 +8889,38 @@ a b > 111 111 > DROP TABLE t1; > # > +# MDEV-27690 Crash on `CHARACTER SET csname COLLATE DEFAULT` in column definition > +# > +CREATE TABLE t1 (a CHAR(10) CHARACTER SET latin1 COLLATE DEFAULT); > +SHOW CREATE TABLE t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` char(10) DEFAULT NULL > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +DROP TABLE t1; > +SELECT CAST('a' AS CHAR(10) CHARACTER SET latin1 COLLATE DEFAULT); > +CAST('a' AS CHAR(10) CHARACTER SET latin1 COLLATE DEFAULT) > +a > +CREATE TABLE t1 AS > +SELECT CAST('a' AS CHAR(10) CHARACTER SET latin1 COLLATE DEFAULT) AS c1; > +SHOW CREATE TABLE t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `c1` varchar(10) DEFAULT NULL > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +DROP TABLE t1; > +SELECT COLUMN_GET(COLUMN_CREATE(0, 'string'),0 AS CHAR CHARACTER SET latin1 COLLATE DEFAULT) AS c1; > +c1 > +string > +CREATE TABLE t1 AS > +SELECT COLUMN_GET(COLUMN_CREATE(0, 'string'),0 AS CHAR CHARACTER SET latin1 COLLATE DEFAULT) AS c1; > +SHOW CREATE TABLE t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `c1` longtext DEFAULT NULL > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +DROP TABLE t1; 1. why is it in "10.2 tests" ? 2. shouldn't here be at least one test that COLLATE DEFAULT is not a no-op? Like CREATE TABLE (... COLLATE DEFAULT ...) COLLATE _non_default ? Or is it in your generated ctype_collate_column.test? > +# > # End of 10.2 tests > # > # > diff --git a/mysql-test/suite/roles/create_and_drop_role_invalid_user_table.test b/mysql-test/suite/roles/create_and_drop_role_invalid_user_table.test > index 9cf0d7b4aff..dac6eab21e9 100644 > --- a/mysql-test/suite/roles/create_and_drop_role_invalid_user_table.test > +++ b/mysql-test/suite/roles/create_and_drop_role_invalid_user_table.test > @@ -36,7 +36,7 @@ set default role test_role for root@localhost; > drop role test_role; > drop user test_user@localhost; > > -alter table user add column default_role char(80) binary default '' not null > +alter table user add column default_role char(80) default '' not null > COLLATE utf8_general_ci Heh, interesting that it worked before good catch > after is_role; > alter table user add max_statement_time decimal(12,6) default 0 not null > diff --git a/sql/structs.h b/sql/structs.h > index 9a5006e8b47..db065c49931 100644 > --- a/sql/structs.h > +++ b/sql/structs.h > @@ -599,11 +599,140 @@ class DDL_options: public DDL_options_st ... > +class Lex_collation: public Lex_collation_st > +{ > +public: > + Lex_collation(CHARSET_INFO *collation, Type type) > + { > + m_collation= collation; > + m_type= type; > + } > + static Lex_collation national(bool bin_mod) > + { > + if (bin_mod) > + return Lex_collation(&my_charset_utf8mb3_bin, TYPE_EXPLICIT); > + return Lex_collation(&my_charset_utf8mb3_general_ci, TYPE_IMPLICIT); utf8mb3? not utf8? > + } > +}; > + > + > struct Lex_length_and_dec_st > { > -private: > +protected: > uint32 m_length; > uint8 m_dec; > + uint8 m_collation_type:2; > bool m_has_explicit_length:1; > bool m_has_explicit_dec:1; > public: > diff --git a/sql/sql_lex.cc b/sql/sql_lex.cc > index ae6ae9a0cc0..6ca10267187 100644 > --- a/sql/sql_lex.cc > +++ b/sql/sql_lex.cc > @@ -11816,3 +11784,257 @@ bool SELECT_LEX_UNIT::explainable() const ... > +CHARSET_INFO * > +Lex_collation_st::resolved_to_character_set(CHARSET_INFO *def) const > +{ > + DBUG_ASSERT(def); > + if (m_type != TYPE_CONTEXTUALLY_TYPED) > + { > + if (!m_collation) > + return def; // Empty - not typed at all > + return m_collation; // Explicitly typed > + } > + > + // Contextually typed > + DBUG_ASSERT(m_collation); > + > + if (m_collation == &my_charset_bin) // CHAR(10) BINARY > + return find_bin_collation(def); > + > + if (m_collation == &my_charset_latin1) // CHAR(10) COLLATE DEFAULT > + return find_default_collation(def); Uhm. Could you create two dummy CHARSET_INFO variables for that instead? Like CHARSET_INFO binary_collaton, default_collation; no need to initialize them, but to have distinct values for binary/default and not reuse my_charset_bin and my_charset_latin1. > + > + /* > + Non-binary and non-default contextually typed collation. > + We don't have such yet - the parser cannot produce this. > + But will have soon, e.g. "uca1400_as_ci". > + */ > + DBUG_ASSERT(0); > + return NULL; > +} > + > + ... > diff --git a/sql/field.h b/sql/field.h > index 2ed02b37cfd..fe5ba435202 100644 > --- a/sql/field.h > +++ b/sql/field.h > @@ -5493,6 +5492,22 @@ class Column_definition: public Sql_alloc, > { return compression_method_ptr; } > > bool check_vcol_for_key(THD *thd) const; > + > + void set_lex_collation(const Lex_collation_st &lc) > + { > + charset= lc.collation(); > + if (lc.is_contextually_typed_collation()) > + flags|= BINCMP_FLAG; > + else > + flags&= ~BINCMP_FLAG; Isn't COLLATE DEFAULT also contextually typed? > + } > + Lex_collation lex_collation() const > + { > + return Lex_collation(charset, > + flags & BINCMP_FLAG ? > + Lex_collation_st::TYPE_CONTEXTUALLY_TYPED : > + Lex_collation_st::TYPE_IMPLICIT); > + } > }; > > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 2

Re: [Maria-developers] 75aa95ada6b: Tricky static asserts did not compile on some platforms. Commenting out.
by Sergei Golubchik 16 Mar '22

16 Mar '22

Hi, Alexander, On Mar 16, Alexander Barkov wrote: > revision-id: 75aa95ada6b (mariadb-10.6.1-338-g75aa95ada6b) > parent(s): e90062c20f1 > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2022-03-09 13:07:35 +0400 > message: > > Tricky static asserts did not compile on some platforms. Commenting out. Details? What doesn't compile and where? I'd suggest to try to keep the assert in some form > diff --git a/sql/structs.h b/sql/structs.h > index abfafb635b1..23958490c7b 100644 > --- a/sql/structs.h > +++ b/sql/structs.h > @@ -602,13 +602,16 @@ class DDL_options: public DDL_options_st > > struct Lex_length_and_dec_st > { > + /* > + Does not compile on some platforms: > static constexpr uint collation_type_bits= 2; > static_assert(((1<<collation_type_bits)-1) >= > Lex_charset_collation_st::TYPE_LAST); > + */ > protected: > uint32 m_length; > uint8 m_dec; > - uint8 m_collation_type:collation_type_bits; > + uint8 m_collation_type:2; /*collation_type_bits;*/ > bool m_has_explicit_length:1; > bool m_has_explicit_dec:1; > public: Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] e90062c20f1: MDEV-27266 Improve UCA collation performance for utf8mb3 and utf8mb4
by Sergei Golubchik 16 Mar '22

16 Mar '22

Hi, Alexander, also ok On Mar 14, Alexander Barkov wrote: > revision-id: e90062c20f1 (mariadb-10.6.1-337-ge90062c20f1) > parent(s): 6a3201d3769 > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2022-02-28 14:43:43 +0400 > message: > > MDEV-27266 Improve UCA collation performance for utf8mb3 and utf8mb4 > > Adding two levels of optimization: > > include/m_ctype.h | 53 ++++ > strings/ctype-uca-scanner_next.inl | 39 +++ > strings/ctype-uca.c | 548 ++++++++++++++++++++++++++++++++++++- > strings/ctype-uca.ic | 30 ++ I thought all *.ic files were renamed to *.inl ? > unittest/strings/strings-t.c | 2 +- > 5 files changed, 660 insertions(+), 12 deletions(-) Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 6a3201d3769: MDEV-27265 Improve contraction performance in UCA collations
by Sergei Golubchik 16 Mar '22

16 Mar '22

Hi, Alexander, that ok, up to you. I'm not looking into the strings/ctype*c code. On Mar 14, Alexander Barkov wrote: > revision-id: 6a3201d3769 (mariadb-10.6.1-336-g6a3201d3769) > parent(s): 49ecf935415 > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2022-02-28 14:05:08 +0400 > message: > > MDEV-27265 Improve contraction performance in UCA collations > > Adding a hash table for contractions. > > The old code iterated through all items in MY_CONTRACTIONS, > and was much slower, especially for those contractions > in the end of the list. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] c67789f63c8: MDEV-27009 Add UCA-14.0.0 collations - adding version aware implicit weight handling
by Sergei Golubchik 14 Mar '22

14 Mar '22

Hi, Alexander, this is ok too, thanks On Mar 14, Alexander Barkov wrote: > revision-id: c67789f63c8 (mariadb-10.6.1-334-gc67789f63c8) > parent(s): c1010fdfac2 > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2022-02-28 10:31:31 +0400 > message: > > MDEV-27009 Add UCA-14.0.0 collations - adding version aware implicit weight handling > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] c1010fdfac2: MDEV-27009 Add UCA-14.0.0 collations - dump logical positions and contractions
by Sergei Golubchik 14 Mar '22

14 Mar '22

Hi, Alexander, this is ok too On Mar 14, Alexander Barkov wrote: > revision-id: c1010fdfac2 (mariadb-10.6.1-333-gc1010fdfac2) > parent(s): ab62059bbd4 > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2022-02-28 10:31:31 +0400 > message: > > MDEV-27009 Add UCA-14.0.0 collations - dump logical positions and contractions > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] ab62059bbd4: MDEV-27009 Add UCA-14.0.0 collations - Adding implicit weight handling for Unicode-14.0.0
by Sergei Golubchik 14 Mar '22

14 Mar '22

Hi, Alexander, I believe it's ok too, but see a couple of questions below On Mar 14, Alexander Barkov wrote: > revision-id: ab62059bbd4 (mariadb-10.6.1-332-gab62059bbd4) > parent(s): 5fe33342399 > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2022-02-28 10:31:06 +0400 > message: > > MDEV-27009 Add UCA-14.0.0 collations - Adding implicit weight handling for Unicode-14.0.0 > > diff --git a/strings/ctype-uca.c b/strings/ctype-uca.c > index ad0b46ea296..eb6bc2eb9e6 100644 > --- a/strings/ctype-uca.c > +++ b/strings/ctype-uca.c > @@ -31694,7 +31694,7 @@ static inline void > my_uca_implicit_weight_put(uint16 *to, my_wc_t code, uint level) > { > MY_UCA_IMPLICIT_WEIGHT weight; > - weight= my_uca_520_implicit_weight_on_level(code, level); > + weight= my_uca_implicit_weight_on_level(520, code, level); will the version always be a constant in every invocation of my_uca_implicit_weight_put() ? > to[0]= weight.weight[0]; > to[1]= weight.weight[1]; > to[2]= 0; > diff --git a/strings/ctype-uca.h b/strings/ctype-uca.h > index 01f655b06cb..d439b8c578c 100644 > --- a/strings/ctype-uca.h > +++ b/strings/ctype-uca.h > @@ -136,11 +149,11 @@ my_uca_implicit_weight_quaternary() > > > static inline MY_UCA_IMPLICIT_WEIGHT > -my_uca_520_implicit_weight_on_level(my_wc_t code, uint level) > +my_uca_implicit_weight_on_level(uint version, my_wc_t code, uint level) > { > switch (level) { > case 0: > - return my_uca_520_implicit_weight_primary(code); > + return my_uca_implicit_weight_primary(version, code); > case 1: > return my_uca_implicit_weight_secondary(); > case 2: do you mean that secondary/tertiary/etc weights do not depend on the version? Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 5fe33342399: MDEV-27009 Add UCA-14.0.0 collations - adding uca-dump into build targets
by Sergei Golubchik 14 Mar '22

14 Mar '22

Hi, Alexander, this is ok (not ok to push as is just yet, please wait until we agree on commits in the branch) On Mar 14, Alexander Barkov wrote: > revision-id: 5fe33342399 (mariadb-10.6.1-331-g5fe33342399) > parent(s): 2467eb2d314 > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2022-02-10 11:53:47 +0400 > message: > > MDEV-27009 Add UCA-14.0.0 collations - adding uca-dump into build targets > > - Adding uca-dump into build targets > - Adding ctype-uca.h and moving implicit weight related routines there > - Reusing implicit weight routines in ctype-uca.c and uca-dump.c > - Adding handling of command line arguments to uca-dump > - Fixing some compile-time warnings in uca-dump.c > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0