- developers - lists.mariadb.org

Re: [Maria-developers] a00410a2ee4: MDEV-26238: Remove inconsistent behaviour of --default-* options in my_print_defaults
by Sergei Golubchik 11 Sep '21

11 Sep '21

Hi, Rucha! On Sep 11, Rucha Deodhar wrote: > revision-id: a00410a2ee4 (mariadb-10.6.1-77-ga00410a2ee4) > parent(s): 76149650764 > author: Rucha Deodhar > committer: Rucha Deodhar > timestamp: 2021-09-11 16:30:38 +0530 > message: > > MDEV-26238: Remove inconsistent behaviour of --default-* options > in my_print_defaults > > Analysis: --defaults* option is recognized anywhere in the commandline > instead of only at the beginning because handle_options() recognizes > options in any order. > Fix: use get_defaults_options() which recognizes --defaults* options only at > the beginning. After this is done, we only want to recognize other options > given in any order which can be done using handle_options(). So (only) skip > --defaults* options, combine it with program name to pass it to handle_options() > (because we skip program name in handle_options() too). Also, removed -e, -g and > -c because only my_print_defaults supports them. > > diff --git a/extra/my_print_defaults.c b/extra/my_print_defaults.c > index b7f52382721..0253614b410 100644 > --- a/extra/my_print_defaults.c > ++ b/extra/my_print_defaults.c > @@ -141,50 +124,49 @@ static int get_options(int *argc,char ***argv) > return 0; > } > > int main(int argc, char **argv) > { > int count, error, args_used; > char **load_default_groups= 0, *tmp_arguments[6]; > char **argument, **arguments, **org_argv, **new_argv, **pos; > int nargs, i= 0, index=1; > MY_INIT(argv[0]); > > org_argv= argv; > args_used= get_defaults_options(argv); > > /* Copy defaults-xxx arguments & program name */ > count= args_used; > arguments= tmp_arguments; > memcpy((char*) arguments, (char*) org_argv, count*sizeof(*org_argv)); > arguments[count]= 0; you're still overdoing it. you can do just argv+= args_used-1; argc+= args_used-1; get_options(&argc,&argv); no need to allocate or copy anything. Isn't it simpler? > /* > We already process --defaults* options at the beginning in > get_defaults_options(). So skip only --defaults* options and > pass program name and remaining options to handle_options(). > */ > > new_argv= (char**)malloc(argc*sizeof(*org_argv)); > memcpy((char*)new_argv, (char*)org_argv, 1*(sizeof(*new_argv))); > org_argv++; > > /* We want to skip --defaults* option only */ > if (!is_prefix(*org_argv, "--no-defaults")) > org_argv+= args_used-1; > > for (pos= org_argv; *pos; pos++) > new_argv[index++]= *(pos); > > new_argv[index]= 0; > argc= index; > > memcpy((char*)argv, (char*)new_argv, index*(sizeof(*new_argv))); > > /* Check out the args. */ > if (get_options(&argc,&argv)) > cleanup_and_exit(1); > > nargs= argc + 1; > if (opt_mysqld) > nargs+= array_elements(mysqld_groups); Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] eb121bbe93e: Vanilla cleanups and refactorings
by Sergei Golubchik 11 Sep '21

11 Sep '21

Hi, Aleksey! On Sep 10, Aleksey Midenkov wrote: > revision-id: eb121bbe93e (mariadb-10.6.1-78-geb121bbe93e) > parent(s): d90f54e2206 > author: Aleksey Midenkov > committer: Aleksey Midenkov > timestamp: 2021-09-10 17:42:10 +0300 > message: > > diff --git a/dbug/tests.c b/dbug/tests.c > index 3e77bf82236..100388891eb 100644 > --- a/dbug/tests.c > +++ b/dbug/tests.c > @@ -13,6 +13,17 @@ char *push1=0; > #include <my_pthread.h> > #include <string.h> > > + > +#ifndef DBUG_OFF > +#define DBUG_EVALUATE(keyword,a1,a2) \ > + (_db_keyword_(0,(keyword), 0) ? (a1) : (a2)) > +#define DBUG_EVALUATE_IF(keyword,a1,a2) \ > + (_db_keyword_(0,(keyword), 1) ? (a1) : (a2)) > +#else > +#define DBUG_EVALUATE(keyword,a1,a2) (a2) > +#define DBUG_EVALUATE_IF(keyword,a1,a2) (a2) > +#endif > + could you fix the test properly, please? > const char *func3() > { > DBUG_ENTER("func3"); Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 3

Re: [Maria-developers] 0e09bc41cba: MDEV-9245 password "reuse prevention" validation plugin
by Sergei Golubchik 11 Sep '21

11 Sep '21

Hi, Oleksandr! Looks very good. See comments/questions below. The main thing - please add some tests for SQL errors in the plugin. On Sep 11, Oleksandr Byelkin wrote: > revision-id: 0e09bc41cba (mariadb-10.6.1-78-g0e09bc41cba) > parent(s): 67cd5da8c34 > author: Oleksandr Byelkin > committer: Oleksandr Byelkin > timestamp: 2021-09-10 13:47:07 +0200 > message: > > MDEV-9245 password "reuse prevention" validation plugin > > diff --git a/mysql-test/suite/plugins/t/password_reuse_check.test b/mysql-test/suite/plugins/t/password_reuse_check.test > --- /dev/null > +++ b/mysql-test/suite/plugins/t/password_reuse_check.test > @@ -0,0 +1,58 @@ > +--source include/not_embedded.inc > + > +if (!$PASSWORD_REUSE_CHECK_SO) { > + skip No PASSWORD_REUSE_CHECK plugin; > +} > + > +install soname "password_reuse_check"; > + > +set @save_password_reuse_check_interval= @@password_reuse_check_interval; this is kind of pointless, if you uninstall the plugin anyway. it's not harmless, just useless. > + > +set global password_reuse_check_interval= 0; ... > diff --git a/plugin/password_reuse_check/CMakeLists.txt b/plugin/password_reuse_check/CMakeLists.txt > --- /dev/null > +++ b/plugin/password_reuse_check/CMakeLists.txt > @@ -0,0 +1,2 @@ > + > +MYSQL_ADD_PLUGIN(password_reuse_check password_reuse_check.c) did you try to compile it statically? > diff --git a/plugin/password_reuse_check/password_reuse_check.c b/plugin/password_reuse_check/password_reuse_check.c > --- /dev/null > +++ b/plugin/password_reuse_check/password_reuse_check.c ... > +static void bin_to_hex512(char *to, const unsigned char *str) > +{ > + const unsigned char *str_end= str + (512/8); > + for (; str != str_end; ++str) > + { > + *to++= digits[((unsigned char) *str) >> 4]; > + *to++= digits[((unsigned char) *str) & 0x0F]; > + } > +} > + > + > +/** > + Send SQL error as ER_UNKNOWN_ERROR for information > + > + @param mysql Connection handler > +*/ > + > +static void report_sql_error(MYSQL *mysql) > +{ > + my_printf_error(ER_UNKNOWN_ERROR, "password_reuse_check:[%d] %s", ME_WARNING, > + mysql_errno(mysql), mysql_error(mysql)); > +} No tests for that? You can, for example, drop the table, and create a non-insertable view with the same name. CREATE VIEW mysql.password_reuse_check_history AS SELECT 1; That should make the plugin to fail, right? > + > + > +/** > + Create the history of passwords table for this plugin. > + > + @param mysql Connection handler > + > + @retval 1 - Error > + @retval 0 - OK > +*/ Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 818407474d6: MDEV-22166 CONVERT PARTITION: move out partition into a table
by Sergei Golubchik 10 Sep '21

10 Sep '21

Hi, Aleksey! I presume that changes to the logging code is what Monty asked. So I'll only comment on a couple of minor issues: On Sep 10, Aleksey Midenkov wrote: > revision-id: 818407474d6 (mariadb-10.6.1-80-g818407474d6) > parent(s): bb4b95975f7 > author: Aleksey Midenkov > committer: Aleksey Midenkov > timestamp: 2021-09-10 18:47:31 +0300 > message: > > MDEV-22166 CONVERT PARTITION: move out partition into a table > > diff --git a/sql/sql_partition.cc b/sql/sql_partition.cc > index db371d23ba7..cbdb40472d8 100644 > --- a/sql/sql_partition.cc > +++ b/sql/sql_partition.cc > @@ -5371,8 +5367,12 @@ that are reorganised. > tab_part_info->is_auto_partitioned= FALSE; > } > } > - else if (alter_info->partition_flags & ALTER_PARTITION_DROP) > + else if ((alter_info->partition_flags & ALTER_PARTITION_DROP) | > + (alter_info->partition_flags & ALTER_PARTITION_CONVERT_OUT)) > { > + const char * const cmd= > + (alter_info->partition_flags & ALTER_PARTITION_CONVERT_OUT) ? > + "EXTRACT" : "DROP"; "CONVERT" > /* > Drop a partition from a range partition and list partitioning is > always safe and can be made more or less immediate. It is necessary > diff --git a/sql/sql_table.h b/sql/sql_table.h > index aacb6c99f15..a580c7d348d 100644 > --- a/sql/sql_table.h > +++ b/sql/sql_table.h > @@ -20,6 +20,17 @@ > #include <my_sys.h> // pthread_mutex_t > #include "m_string.h" // LEX_CUSTRING > > +inline bool suicide() > +{ > + DBUG_SUICIDE(); > + return false; > +} as I wrote earlier, this is wrong. DBUG_SUICIDE() must always be an expression. > + > +#define ERROR_INJECT_CRASH(code) \ > + (DBUG_IF(code) && suicide()) > +#define ERROR_INJECT_ERROR(code) \ > + (DBUG_IF(code) && (my_error(ER_UNKNOWN_ERROR, MYF(0)), 1)) > + > class Alter_info; > class Alter_table_ctx; > class Column_definition; Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 2

Re: [Maria-developers] ff0ece70855: MDEV-26531: mtr shows some hunks failed
by Sergei Golubchik 10 Sep '21

10 Sep '21

Hi, Anel! On Sep 10, Anel Husakovic wrote: > revision-id: ff0ece70855 (mariadb-10.6.1-73-gff0ece70855) > parent(s): 1bc82aaf0a7 > author: Anel Husakovic > committer: Anel Husakovic > timestamp: 2021-09-03 21:15:58 +0200 > message: > > MDEV-26531: mtr shows some hunks failed > > Reviewed by: wlad(a)mariadb.com > > --- > mysql-test/mariadb-test-run.pl | 10 +++++++--- > 1 file changed, 7 insertions(+), 3 deletions(-) > > diff --git a/mysql-test/mariadb-test-run.pl b/mysql-test/mariadb-test-run.pl > index 6dcd25a9c86..7e0282b89e4 100755 > --- a/mysql-test/mariadb-test-run.pl > +++ b/mysql-test/mariadb-test-run.pl > @@ -3278,11 +3278,15 @@ sub do_before_run_mysqltest($) > # don't rebuild a file if it's up to date > unless (-e $dest and -M $dest < -M $resfile > and -M $dest < -M $base_result) { > - run_system(@cmd); > + if(run_system(@cmd) != 0){ > + die("system(@cmd) : $?"); I'd rather see it done more perl-ish, like run_system(@cmd) or die "system(@cmd): $?" > + } > } > } else { > $cmd[-3] = $dest = $opt_tmpdir . '/' . basename($dest); > - run_system(@cmd); > + if(run_system(@cmd) != 0){ > + die("system(@cmd) : $?"); > + } > } > $tinfo->{result_file} = $dest; > } > @@ -4887,7 +4891,7 @@ sub report_failure_and_restart ($) { > > sub run_system(@) { > mtr_verbose("Running '$_[0]'"); > - my $ret= system(@_) >> 8; > + my $ret= system(@_); this seems wrong. `perldoc -f system` says The return value is the exit status of the program as returned by the "wait" call. To get the actual exit value, shift right by eight (see below). > return $ret; > } Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 4645a1e1950: MDEV-26238: Remove inconsistent behaviour of --default-* options in my_print_defaults
by Sergei Golubchik 10 Sep '21

10 Sep '21

Hi, Rucha! On Sep 08, Rucha Deodhar wrote: > Hi, Sergei ! > Thanks for the review. > > > > diff --git a/extra/my_print_defaults.c b/extra/my_print_defaults.c > > > index b7f52382721..e2ec0b00923 100644 > > > --- a/extra/my_print_defaults.c > > > +++ b/extra/my_print_defaults.c > > > @@ -48,20 +48,6 @@ static struct my_option my_long_options[] = > > > > you forgot the "config-file", which is here, just about "debug" > > I am sorry, I don't understand. Oops. Sorry. I looked at my_print_defaults.c in the checked out branch and I saw: {"config-file", 'c', "Deprecated, please use --defaults-file instead. " "Name of config file to read; if no extension is given, default " "extension (e.g., .ini or .cnf) will be added", (char**) &config_file, (char**) &config_file, 0, GET_STR, REQUIRED_ARG, 0, 0, 0, 0, 0, 0}, #ifdef DBUG_OFF {"debug", '#', "This is a non-debug version. Catch this and exit", 0,0, 0, GET_DISABLED, OPT_ARG, 0, 0, 0, 0, 0, 0}, #else so I wrote that "config-file" should also be removed. But now I see that it's present in 10.2, but not in 10.4, it was already removed. So, ignore that my comment, you didn't forget config-file, it was removed in 10.4, I looked at the wrong branch. > > > {"debug", '#', "Output debug log", (char**) &default_dbug_option, > > > (char**) &default_dbug_option, 0, GET_STR, OPT_ARG, 0, 0, 0, 0, 0, 0}, > > > #endif > > > - {"defaults-file", 'c', > > > - "Read this file only, do not read global or per-user config " > > > - "files; should be the first option", > > > - (char**) &config_file, (char*) &config_file, 0, GET_STR, REQUIRED_ARG, > > > @@ -141,49 +124,44 @@ static int get_options(int *argc,char ***argv) > > > return 0; > > > } > > > > > > -static char *make_args(const char *s1, const char *s2) > > > -{ > > > - char *s= malloc(strlen(s1) + strlen(s2) + 1); > > > - strmov(strmov(s, s1), s2); > > > - return s; > > > -} > > > - > > > int main(int argc, char **argv) > > > { > > > - int count= 0, error, no_defaults= 0; > > > + int count, error, args_used; > > > char **load_default_groups= 0, *tmp_arguments[6]; > > > - char **argument, **arguments, **org_argv; > > > - int nargs, i= 0; > > > + char **argument, **arguments, **org_argv, **new_argv, **pos; > > > + int nargs, i= 0, index=0; > > > MY_INIT(argv[0]); > > > > > > org_argv= argv; > > > - if (*argv && !strcmp(*argv, "--no-defaults")) > > > - { > > > - argv++; > > > - ++count; > > > - no_defaults= 1; > > > - } > > > - /* Copy program name and --no-defaults if present*/ > > > + args_used= get_defaults_options(argv); > > > + > > > + /* Copy defaults-xxx arguments & program name */ > > > + count= args_used; > > > arguments= tmp_arguments; > > > - memcpy((char*) arguments, (char*) org_argv, (++count)*sizeof(*org_argv)); > > > + memcpy((char*) arguments, (char*) org_argv, count*sizeof(*org_argv)); > > > arguments[count]= 0; > > > > > > - /* Check out the args */ > > > - if (get_options(&argc,&argv)) > > > - cleanup_and_exit(1); > > > - > > > - if (!no_defaults) > > > + /* > > > + We already process --defaults* options at the beginning in > > > + get_defaults_options(). So, now we we make a new commandline which > > > + has all options except --defaults* options and pass it to handle_options() > > > + instead of original commandline. > > > + */ > > > + new_argv= (char**)malloc(argc*sizeof(*org_argv)); > > > + for (pos= org_argv; *pos; pos++) > > > { > > > + if (!is_prefix(*pos, "--defaults-extra-file=") && > > > + !is_prefix(*pos, "--defaults-file=") && > > > + !is_prefix(*pos, "--defaults-group-suffix=")) > > > + new_argv[index++]= *(pos); > > > > Why are you doing it? > > After get_defaults_options, you know how many arguments were default, > > just skip args_used, and invoke handle_options on the rest. > > I did it to filter all the --defaults* options, anywhere on the command > line. > > get_defaults_options() counts the program name in addition to --defaults* > options, so args_used will at least be 1. So skipping argv using agrs_used > will at least skip the program name even before it is passed to > handle_options(). But argv is also skipped in handle_options() to skip the > program name. > So one extra argument will get skipped too. correct, you need to skip args_used-1, good point. > Say --defaults* options are somewhere at the end, they will not get counted > in get_defaults_options(). So --defaults* will be passed to > handle_options() without getting skipped. And handle_options() will give an > error even if the options are valid because > it will not find --defaults* options in my_long_options[]. This is what we want, right? That's why you're doing it. Try `mariadb` client or `mariadb-dump` or `mariadbd` server, anything, and put --defaults* option not at the beginning. It'll be "unknown option". The idea is to make my_print_defaults *not special*, to behave like any other executable. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 6768e3a2830: MDEV-22166 MIGRATE PARTITION: move out partition into a table
by Sergei Golubchik 10 Sep '21

10 Sep '21

Hi, Aleksey! It's great that you made it atomic. Frankly, I didn't think of it, MDEV defines the feature as being exactly equivalent to the 3-command sequence, so non-atomic. But, of course, making it atomic makes perfect sense. See other comments below. On Aug 23, Aleksey Midenkov wrote: > commit 6768e3a2830 > Author: Aleksey Midenkov <midenok(a)gmail.com> > Date: Thu Jul 29 22:54:12 2021 +0300 > > diff --git a/sql/lex.h b/sql/lex.h > index cbf9d9d51b2..55eefc39a7e 100644 > --- a/sql/lex.h > +++ b/sql/lex.h > @@ -240,6 +240,7 @@ SYMBOL symbols[] = { > { "EXPLAIN", SYM(DESCRIBE)}, > { "EXTENDED", SYM(EXTENDED_SYM)}, > { "EXTENT_SIZE", SYM(EXTENT_SIZE_SYM)}, > + { "EXTRACT", SYM(EXTRACT_SYM)}, So, you did EXTRACT after all, not MIGRATE? ADD/EXTRACT aren't opposites in SQL. It would be rather consistent to have ALTER ... ADD PARTITION (...) FROM table_name and ALTER ... DROP PARTITION ... TO table_name except that DROP...TO looks kind of strange, and ADD...FROM does not convey the meaning that the table disappears. The second issue also applies to IMPORT/EXPORT. Another option, also kind of strange, but also consistent and makes it clear that the table/partition disappears: ALTER ... MOVE TABLE table_name TO PARTITION (...) ALTER ... MOVE PARTITION ... TO TABLE table_name or the same with CHANGE instead of MOVE to reuse an existing keyword. I'm not saying anything of the above is perfect, and better variants are very much welcome. But pretty much anything is better than ADD/EXTRACT. > { "FALSE", SYM(FALSE_SYM)}, > { "FAST", SYM(FAST_SYM)}, > { "FAULTS", SYM(FAULTS_SYM)}, > diff --git a/sql/sql_yacc.yy b/sql/sql_yacc.yy > index dc2eb4425f0..a41ea7d3e4a 100644 > --- a/sql/sql_yacc.yy > +++ b/sql/sql_yacc.yy > @@ -7605,6 +7605,25 @@ alter_commands: > if (Lex->stmt_alter_table_exchange_partition($6)) > MYSQL_YYABORT; > } > + | EXTRACT_SYM PARTITION_SYM alt_part_name_item > + AS TABLE_SYM table_ident have_partitioning > + { > + LEX *lex= Lex; > + lex->first_select_lex()->db= $6->db; > + if (lex->first_select_lex()->db.str == NULL && > + lex->copy_db_to(&lex->first_select_lex()->db)) > + MYSQL_YYABORT; > + if (unlikely(check_table_name($6->table.str,$6->table.length, > + FALSE)) || > + ($6->db.str && unlikely(check_db_name((LEX_STRING*) &$6->db)))) > + my_yyabort_error((ER_WRONG_TABLE_NAME, MYF(0), $6->table.str)); good point, LEX::stmt_alter_table_exchange_partition() doesn't seem to do it > + lex->name= $6->table; > + > + lex->m_sql_cmd= new (thd->mem_root) Sql_cmd_alter_table(); > + if (unlikely(lex->m_sql_cmd == NULL)) > + MYSQL_YYABORT; > + lex->alter_info.partition_flags|= ALTER_PARTITION_EXTRACT; why wouldn't you combine the common code from here and EXCHANGE PARTITION? less code duplication and as a bonus EXCHANGE will get proper table/db name checks. > + } > ; > > remove_partitioning: > diff --git a/sql/sql_alter.h b/sql/sql_alter.h > index a0f89c28d2a..6aa65d232ca 100644 > --- a/sql/sql_alter.h > +++ b/sql/sql_alter.h > @@ -345,6 +348,8 @@ class Alter_table_ctx > /** Indicates that we are altering temporary table */ > bool tmp_table; > > + DDL_LOG_STATE *ddl_log_state; What do you need this for? > + > private: > char new_filename[FN_REFLEN + 1]; > char new_alias_buff[NAME_LEN + 1]; > diff --git a/sql/sql_table.cc b/sql/sql_table.cc > index 779d125af70..033d77a827d 100644 > --- a/sql/sql_table.cc > +++ b/sql/sql_table.cc > @@ -10016,7 +10146,7 @@ bool mysql_alter_table(THD *thd, const LEX_CSTRING *new_db, > } > > // In-place execution of ALTER TABLE for partitioning. > - DBUG_RETURN(fast_alter_partition_table(thd, table, alter_info, > + DBUG_RETURN(fast_alter_partition_table(thd, table, alter_info, &alter_ctx, > create_info, table_list, > &alter_ctx.db, > &alter_ctx.table_name)); you no longer need to pass alter_ctx.db and alter_ctx.table_name, if you pass the whole alter_ctx down. > diff --git a/include/my_dbug.h b/include/my_dbug.h > index e25bfcf28a7..cf969ce5750 100644 > --- a/include/my_dbug.h > +++ b/include/my_dbug.h > @@ -104,6 +104,8 @@ extern int (*dbug_sanity)(void); > (_db_keyword_(0,(keyword), 0) ? (a1) : (a2)) > #define DBUG_EVALUATE_IF(keyword,a1,a2) \ > (_db_keyword_(0,(keyword), 1) ? (a1) : (a2)) > +#define DBUG_TRUE_IF(keyword) DBUG_EVALUATE_IF(keyword, 1, 0) > +#define DBUG_FALSE_IF(keyword) DBUG_EVALUATE_IF(keyword, 0, 1) hmm. DBUG_FALSE_IF is not used anywhere, DBUG_TRUE_IF is used once as if (DBUG_TRUE_IF("error_extract_partition_00") || unlikely(error= file->ha_rename_table(from_name, to_name))) and this is typically written as if (DBUG_EVALUATE_IF("error_extract_partition_00", 1, error= file->ha_rename_table(from_name, to_name))) so I see no need for either DBUG_TRUE_IF or DBUG_FALSE_IF, please, remove both > #define DBUG_PRINT(keyword,arglist) \ > do if (_db_pargs_(__LINE__,keyword)) _db_doprnt_ arglist; while(0) > > diff --git a/sql/sql_partition.cc b/sql/sql_partition.cc > index 85d880cbbb4..4172a61812e 100644 > --- a/sql/sql_partition.cc > +++ b/sql/sql_partition.cc > @@ -5432,7 +5432,7 @@ uint prep_alter_part_table(THD *thd, TABLE *table, Alter_info *alter_info, > } while (++part_count < tab_part_info->num_parts); > if (num_parts_found != num_parts_dropped) > { > - my_error(ER_DROP_PARTITION_NON_EXISTENT, MYF(0), "DROP"); > + my_error(ER_DROP_PARTITION_NON_EXISTENT, MYF(0), cmd); Not part of your patch, I know, but the error message here is "Error in list of partitions to %-.64s" shouldn't it say "Partition does not exist" or something? > goto err; > } > if (table->file->is_fk_defined_on_table_or_index(MAX_KEY)) > @@ -7268,6 +7420,48 @@ uint fast_alter_partition_table(THD *thd, TABLE *table, > if (alter_partition_lock_handling(lpt)) > goto err; > } > + else if (alter_info->partition_flags & ALTER_PARTITION_EXTRACT) > + { > + if (mysql_write_frm(lpt, WFRM_WRITE_EXTRACTED) || > + write_log_drop_shadow_frm(lpt) || > + ERROR_INJECT_CRASH("crash_extract_partition_1") || > + ERROR_INJECT_ERROR("fail_extract_partition_1") || > + mysql_write_frm(lpt, WFRM_WRITE_SHADOW) || > + ERROR_INJECT_CRASH("crash_extract_partition_2") || > + ERROR_INJECT_ERROR("fail_extract_partition_2") || > + wait_while_table_is_used(thd, table, HA_EXTRA_NOT_USED) || > + ERROR_INJECT_CRASH("crash_extract_partition_3") || > + ERROR_INJECT_ERROR("fail_extract_partition_3") || > + write_log_extract_partition(lpt) || > + ERROR_INJECT_CRASH("crash_extract_partition_4") || > + ERROR_INJECT_ERROR("fail_extract_partition_4") || > + alter_close_table(lpt) || > + ERROR_INJECT_CRASH("crash_extract_partition_5") || > + ERROR_INJECT_ERROR("fail_extract_partition_5") || > + alter_partition_extract(lpt) || > + ERROR_INJECT_CRASH("crash_extract_partition_7") || > + ERROR_INJECT_ERROR("fail_extract_partition_7") || > + (frm_install= TRUE, FALSE) || > + mysql_write_frm(lpt, WFRM_INSTALL_SHADOW|WFRM_BACKUP_ORIGINAL) || > + log_partition_alter_to_ddl_log(lpt) || > + (frm_install= FALSE, FALSE) || > + ERROR_INJECT_CRASH("crash_extract_partition_8") || > + ERROR_INJECT_ERROR("fail_extract_partition_8") || > + (ddl_log_complete(lpt->part_info), false) || as far as I understand this ^^^ should be done *after* binlogging. I suspect that if you crash here, blnlog won't get this ALTER TABLE. > + ((!thd->lex->no_write_to_binlog) && > + (write_bin_log(thd, FALSE, > + thd->query(), thd->query_length()), FALSE)) || if you crash here, what will roll forward on recovery and drop the backup? The log is already closed here, isn't it? I don't understand why you need a backup. Why not to install shadow after the binlog was written? > + mysql_write_frm(lpt, WFRM_DROP_BACKUP) || > + ERROR_INJECT_CRASH("crash_extract_partition_9") || > + ERROR_INJECT_ERROR("fail_extract_partition_9")) > + { > + (void) ddl_log_revert(thd, lpt->part_info); > + handle_alter_part_error(lpt, true, true, frm_install); > + goto err; > + } > + if (alter_partition_lock_handling(lpt)) > + goto err; > + } > else if ((alter_info->partition_flags & ALTER_PARTITION_ADD) && > (part_info->part_type == RANGE_PARTITION || > part_info->part_type == LIST_PARTITION)) > diff --git a/mysql-test/include/lcase_names.inc b/mysql-test/include/lcase_names.inc > new file mode 100644 > index 00000000000..e69de29bb2d better put here some comment, like in other dummy *.inc files see e.g. platform.inc or protocol.inc > diff --git a/mysql-test/suite/atomic/alter_partition,aria.rdiff b/mysql-test/suite/atomic/alter_partition,aria.rdiff > new file mode 100644 > index 00000000000..20c0510c5ed > --- /dev/null > +++ b/mysql-test/suite/atomic/alter_partition,aria.rdiff > @@ -0,0 +1,219 @@ > +--- alter_partition.result > ++++ alter_partition,aria.reject > +@@ -1,4 +1,4 @@ > +-set @@default_storage_engine=MyISAM; > ++set @@default_storage_engine=Aria; > + # Crash recovery > + create or replace procedure prepare_table() > + begin > +@@ -13,12 +13,12 @@ > + end $ > + # QUERY: ALTER TABLE t1 EXTRACT PARTITION p1 AS TABLE tp1 > + # CRASH POINT: ddl_log_create_before_create_frm > +-t1#P#p0.MYD > +-t1#P#p0.MYI > +-t1#P#p1.MYD may be better `replace_result MAI MYI MAD MYD` ? if that's the only difference. To avoid big rdiff files. > +-t1#P#p1.MYI > +-t1#P#pn.MYD > +-t1#P#pn.MYI > ++t1#P#p0.MAD > ++t1#P#p0.MAI Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 20

Re: [Maria-developers] ca4ef7185da: MDEV-9245 password "reuse prevention" validation plugin
by Sergei Golubchik 09 Sep '21

09 Sep '21

Hi, Oleksandr! you forgot to check in the result file On Sep 07, Oleksandr Byelkin wrote: > revision-id: ca4ef7185da (mariadb-10.6.1-79-gca4ef7185da) > parent(s): 271afbc88e4 > author: Oleksandr Byelkin > committer: Oleksandr Byelkin > timestamp: 2021-09-07 14:35:05 +0200 > message: > > MDEV-9245 password "reuse prevention" validation plugin > > diff --git a/plugin/reuse_password_check/CMakeLists.txt b/plugin/reuse_password_check/CMakeLists.txt > --- /dev/null > +++ b/plugin/reuse_password_check/CMakeLists.txt > @@ -0,0 +1,4 @@ > + > +ADD_DEFINITIONS(-DMYSQL_SERVER) as I wrote to HF, this should not be needed. > + > +MYSQL_ADD_PLUGIN(reuse_password_check reuse_password_check.c MODULE_ONLY) Unless you have a reason why this must be "MODULE_ONLY", please remove MODULE_ONLY and try to compile and test with -DPLUGIN_REUSE_PASSWORD_CHECK=STATIC also, I'd suggest "password_reuse_check" not "reuse_password_check". > diff --git a/plugin/reuse_password_check/reuse_password_check.c b/plugin/reuse_password_check/reuse_password_check.c > --- /dev/null > +++ b/plugin/reuse_password_check/reuse_password_check.c > @@ -0,0 +1,236 @@ > +/* Copyright (c) 2021, Oleksandr Byelkin and MariaDB > + > + This program is free software; you can redistribute it and/or modify > + it under the terms of the GNU General Public License as published by > + the Free Software Foundation; version 2 of the License. > + > + This program is distributed in the hope that it will be useful, > + but WITHOUT ANY WARRANTY; without even the implied warranty of > + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > + GNU General Public License for more details. > + > + You should have received a copy of the GNU General Public License > + along with this program; if not, write to the Free Software > + Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1335 USA */ > + > +#include <my_config.h> > +#include <my_global.h> > +#include <my_base.h> > +#include <mysql/plugin_password_validation.h> > +#include <mysql.h> only plugin_password_validation.h should be needed here. > + > + > +#include <mysqld_error.h> this might be needed all right > +#include <mysql/service_sha2.h> but this isn't needed either > + > +#define HISTORY_DB_NAME "reuse_password_check_history" > + > +#define SQL_BUFF_LEN 2048 > + > +// 0 - unlimited, otherwise number of days to check > +static unsigned interval= 0; > + > +// helping string for bin_to_hex512 > +static char digits[]= "0123456789ABCDEF"; > + > + > +/** > + Convert string of 512 bits (64 bytes) to hex representation > + > + @param to pointer to the result puffer > + (should be at least 64*2 bytes) > + @param str pointer to 512 bits (64 bytes string) > +*/ > + > +static void bin_to_hex512(char *to, const unsigned char *str) > +{ > + const unsigned char *str_end= str + (512/8); > + for (; str != str_end; ++str) > + { > + *to++= digits[((uchar) *str) >> 4]; > + *to++= digits[((uchar) *str) & 0x0F]; > + } > +} > + > + > +/** > + Send SQL error as ER_UNKNOWN_ERROR for information > + > + @param mysql Connection handler > +*/ > + > +static void report_sql_error(MYSQL *mysql) > +{ > + my_printf_error(ER_UNKNOWN_ERROR, "[%d] %s", ME_WARNING, > + mysql_errno(mysql), mysql_error(mysql)); may be include plugin name here? > +} > + > + > +/** > + Create the history of passwords table for this plugin. > + > + @param mysql Connection handler > + > + @retval 1 - Error > + @retval 0 - OK > +*/ > + > +static int create_table(MYSQL *mysql) > +{ > + if (mysql_real_query(mysql, > + // 512/8 = 64 > + STRING_WITH_LEN("CREATE TABLE mysql." HISTORY_DB_NAME > + " ( hash binary(64)," > + " time timestamp," make it "default current_timestamp", just to be explicit. even though it's implicitly default current_timestamp already. > + " primary key (hash), index tm (time) )" > + " ENGINE=Aria"))) > + { > + report_sql_error(mysql); > + return 1; > + } > + return 0; > +} > + > + > +/** > + Run this query and create table if needed > + > + @param mysql Connection handler > + @param query The query to run > + @param len length of the query text > + > + @retval 1 - Error > + @retval 0 - OK > +*/ > + > +static int run_query_with_table_creation(MYSQL *mysql, const char *query, > + size_t len) > +{ > + if (unlikely(mysql_real_query(mysql, query, len))) > + { > + unsigned int rc= mysql_errno(mysql); > + if (rc != ER_NO_SUCH_TABLE) > + { > + // suppress this error in case of try to add the same password twice > + if (rc != ER_DUP_ENTRY) > + report_sql_error(mysql); > + return 1; > + } > + if (create_table(mysql)) > + return 1; > + if (unlikely(mysql_real_query(mysql, query, len))) > + { > + report_sql_error(mysql); > + return 1; > + } > + } > + return 0; > +} > + > + > +/** > + Password validator > + > + @param username User name (part of whole login name) > + @param password Password to validate > + @param hostname Host name (part of whole login name) > + > + @retval 1 - Password is not OK or an error happened > + @retval 0 - Password is OK > +*/ > + > +static int validate(const MYSQL_CONST_LEX_STRING *username, > + const MYSQL_CONST_LEX_STRING *password, > + const MYSQL_CONST_LEX_STRING *hostname) > +{ > + MYSQL *mysql= NULL; > + size_t key_len= username->length + password->length + hostname->length; > + size_t buff_len= (key_len > SQL_BUFF_LEN ? key_len : SQL_BUFF_LEN); > + size_t len; > + char *buff= malloc(buff_len); > + unsigned char hash[512/8]; > + char escaped_hash[512/8*2 + 1]; > + if (!buff) > + return 1; > + > + mysql= mysql_init(NULL); > + if (!mysql) > + { > + free(buff); > + return 1; > + } > + > + memcpy(buff, hostname->str, hostname->length); > + memcpy(buff + hostname->length, username->str, username->length); > + memcpy(buff + hostname->length + username->length, password->str, > + password->length); > + buff[key_len]= 0; > + bzero(hash, sizeof(hash)); > + my_sha512(hash, buff, key_len); > + if (mysql_real_connect_local(mysql, NULL, NULL, NULL, 0) == NULL) > + goto sql_error; > + > + if (interval) > + { > + // trim the table > + len= snprintf(buff, buff_len, > + "DELETE FROM mysql." HISTORY_DB_NAME > + " WHERE time < DATE_SUB(NOW(), interval %d day)", > + interval); > + if (unlikely(run_query_with_table_creation(mysql, buff, len))) > + goto sql_error; > + } > + > + bin_to_hex512(escaped_hash, hash); > + escaped_hash[512/8*2]= '\0'; > + len= snprintf(buff, buff_len, > + "INSERT INTO mysql." HISTORY_DB_NAME "(hash) " > + "values (x'%s')", > + escaped_hash); > + if (unlikely(run_query_with_table_creation(mysql, buff, len))) > + goto sql_error; > + > + free(buff); > + mysql_close(mysql); > + return 0; // OK > + > +sql_error: > + free(buff); > + if (mysql) > + mysql_close(mysql); > + return 1; // Error > +} > + > +static MYSQL_SYSVAR_UINT(interval, interval, PLUGIN_VAR_RQCMDARG, > + "How old (in days) passwords to check (0 means unlimited)", NULL, NULL, better "Password history retention period in days (0 means unlimited)" > > + 0, 0, 365*100, 1); > + > + > +static struct st_mysql_sys_var* sysvars[]= { > + MYSQL_SYSVAR(interval), > + NULL > +}; > + > +static struct st_mariadb_password_validation info= > +{ > + MariaDB_PASSWORD_VALIDATION_INTERFACE_VERSION, > + validate > +}; > + > +maria_declare_plugin(simple_password_check) ^^^ must be the same name as in CMakeLists.txt, otherwise static linking won't work. > +{ > + MariaDB_PASSWORD_VALIDATION_PLUGIN, > + &info, > + "reuse_password_check", > + "Oleksandr Byelkin", > + "Reusage of password check", Better "Password reuse check" or, as it's a description, "Prevent password reuse" > + PLUGIN_LICENSE_GPL, > + NULL, > + NULL, > + 0x0100, > + NULL, > + sysvars, > + "1.0", > + MariaDB_PLUGIN_MATURITY_ALPHA > +} > +maria_declare_plugin_end; > > diff --git a/mysql-test/suite/plugins/t/reuse_password_check.test b/mysql-test/suite/plugins/t/reuse_password_check.test > --- /dev/null > +++ b/mysql-test/suite/plugins/t/reuse_password_check.test > @@ -0,0 +1,58 @@ > +--source include/not_embedded.inc > + > +if (!$REUSE_PASSWORD_CHECK_SO) { > + skip No REUSE_PASSWORD_CHECK plugin; > +} > + > +install soname "reuse_password_check"; > + > +set @save_reuse_password_check_interval= @@reuse_password_check_interval; > + > +set global reuse_password_check_interval= 0; > + > +--echo # Default value (sould be unlimited i.e. 0) > +SHOW GLOBAL VARIABLES like "reuse_password_check%"; > + > +--echo # insert user > +grant select on *.* to user_name@localhost identified by 'test_pwd'; what about sql mode that doesn't allow to create users with grant? I don't see where you've disabled that. > + > +#--error ER_NOT_VALID_PASSWORD > +#grant select on *.* to user_name@localhost identified by 'test_pwd'; > +#show warnings; why is that commented out? > + > +--error ER_CANNOT_USER > +alter user user_name@localhost identified by 'test_pwd'; > +show warnings; > + > +# Plugin does not work for it > +#--error ER_NOT_VALID_PASSWORD > +#SET PASSWORD FOR user_name@localhost = PASSWORD('test_pwd'); > + > +--echo # check exparation > + > +set global reuse_password_check_interval= 10; > + > +--error ER_CANNOT_USER > +alter user user_name@localhost identified by 'test_pwd'; > +show warnings; > +select hex(hash) from mysql.reuse_password_check_history; > + > +--echo # emulate old password > +update mysql.reuse_password_check_history set time= date_sub(now(), interval > +11 day); > + > +alter user user_name@localhost identified by 'test_pwd'; > +show warnings; > + > +drop user user_name@localhost; > + > +show create table mysql.reuse_password_check_history; > +select count(*) from mysql.reuse_password_check_history; > + > +drop table mysql.reuse_password_check_history; > + > +set @save_reuse_password_check_interval= @@reuse_password_check_interval; > + > +set global reuse_password_check_interval= @save_reuse_password_check_interval; > + > +uninstall plugin reuse_password_check; Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] 9e1c15355a9: resolve the possible deadlock issue through intern_plugin_lock
by Sergei Golubchik 08 Sep '21

08 Sep '21

Hi, Nikita! Looks good so far but I noticed too late that it's just one patch on top of others, not a complete thing. See few comments below. Shall I review everything? Is it feature complete except that it skips inactive threads? On Sep 07, Nikita Malyavin wrote: > revision-id: 9e1c15355a9 (mariadb-10.6.3-44-g9e1c15355a9) > parent(s): d7c36600144 > author: Nikita Malyavin > committer: Nikita Malyavin > timestamp: 2021-09-07 18:37:51 +0300 > message: > > resolve the possible deadlock issue through intern_plugin_lock > > diff --git a/sql/sql_plugin.cc b/sql/sql_plugin.cc > index be043bf229a..ecd885ed296 100644 > --- a/sql/sql_plugin.cc > +++ b/sql/sql_plugin.cc > @@ -1053,6 +1053,15 @@ plugin_ref plugin_lock(THD *thd, plugin_ref ptr) > } > this needs a function comment. the function is quite specialized and very different from normal plugin lock functions. it requires a THD (normally it's optional), it does not return a locked plugin like the others (because, exactly, it's stored in the THD). because it's not what people usually expect from a plugin_lock* function, the comment should describe all these differences. > +void plugin_lock_by_var_if_different(THD *thd, sys_var *var, sys_var *var_prev) > +{ > + sys_var_pluginvar *pv= var->cast_pluginvar(); > + sys_var_pluginvar *pv_prev= var_prev ? var_prev->cast_pluginvar() : NULL; > + if (pv && (!pv_prev || pv->plugin != pv_prev->plugin)) > + intern_plugin_lock(thd->lex, plugin_int_to_ref(pv->plugin)); > +} > + > + > /* > Notes on lifetime: > > diff --git a/storage/perfschema/pfs_variable.cc b/storage/perfschema/pfs_variable.cc > index 984b68e699b..85656532c72 100644 > --- a/storage/perfschema/pfs_variable.cc > +++ b/storage/perfschema/pfs_variable.cc > @@ -100,6 +102,15 @@ bool PFS_system_variable_cache::init_show_var_array(enum_var_type scope, bool st > > mysql_prlock_unlock(&LOCK_system_variables_hash); > > + for (uint i= 0; i < m_show_var_array.elements(); i++) why separately, not in the loop above? > + { > + sys_var *var= (sys_var *)m_show_var_array.at(i).value; > + if (!var) continue; can var be NULL? > + sys_var *prev= i == 0 ? NULL : (sys_var *)m_show_var_array.at(i-1).value; hmm, normally this is done with sys_var *prev= NULL; before the loop and prev= var; in the loop, after plugin_lock_by_var_if_different(). > + > + plugin_lock_by_var_if_different(m_current_thd, var, prev); > + } > + > /* Increase cache size if necessary. */ > m_cache.reserve(m_show_var_array.elements()); > > @@ -316,7 +327,27 @@ class PFS_system_variable_cache_apc: public Apc_target::Apc_call > > void call_in_target_thread() override > { > - (m_pfs->*m_func)(m_param); > + call(m_pfs, m_func, m_param); > + } > +public: > + static void call(PFS_system_variable_cache *pfs, Request func, uint param) > + { > + THD *safe_thd= pfs->safe_thd(); > + > + if (pfs->query_scope() == OPT_SESSION) what's the point doing apc call if the query_scope() is not OPT_SESSION? It seems like here you need a DBUG_ASSERT, and the if() should be much earlier up the stack. > + { > + mysql_mutex_lock(&LOCK_global_system_variables); > + if (!safe_thd->variables.dynamic_variables_ptr || > + global_system_variables.dynamic_variables_head > > + safe_thd->variables.dynamic_variables_head) > + { > + mysql_prlock_rdlock(&LOCK_system_variables_hash); > + sync_dynamic_session_variables(safe_thd, false); > + mysql_prlock_unlock(&LOCK_system_variables_hash); > + } > + mysql_mutex_unlock(&LOCK_global_system_variables); > + } > + (pfs->*func)(param); > } > }; Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 2

Re: [Maria-developers] 4645a1e1950: MDEV-26238: Remove inconsistent behaviour of --default-* options in my_print_defaults
by Sergei Golubchik 07 Sep '21

07 Sep '21

Hi, Rucha! On Sep 07, Rucha Deodhar wrote: > revision-id: 4645a1e1950 (mariadb-10.6.1-72-g4645a1e1950) > parent(s): 21ce69123c1 > author: Rucha Deodhar > committer: Rucha Deodhar > timestamp: 2021-09-02 22:10:37 +0530 > message: > > MDEV-26238: Remove inconsistent behaviour of --default-* options > in my_print_defaults > > Analysis: --defaults* option is recognized anywhere in the commandline > instead of only at the beginning because handle_options() recognizes > options in any order. > Fix: use get_defaults_options() which recognizes --defaults* options only at > the beginning. After this is done, we only want to recognize other options > given in any order which can be done using handle_options(). But > handle_options() returns error and exits if an option is not found. > We don't want to exit and give error because --defaults* options are still > valid. So make a new commandline which doesn't have --defaults* > option and pass it to handle_options(). Also, removed -e, -g and -c because > only my_print_defaults supports them. > > diff --git a/extra/my_print_defaults.c b/extra/my_print_defaults.c > index b7f52382721..e2ec0b00923 100644 > --- a/extra/my_print_defaults.c > +++ b/extra/my_print_defaults.c > @@ -48,20 +48,6 @@ static struct my_option my_long_options[] = you forgot the "config-file", which is here, just about "debug" > {"debug", '#', "Output debug log", (char**) &default_dbug_option, > (char**) &default_dbug_option, 0, GET_STR, OPT_ARG, 0, 0, 0, 0, 0, 0}, > #endif > - {"defaults-file", 'c', > - "Read this file only, do not read global or per-user config " > - "files; should be the first option", > - (char**) &config_file, (char*) &config_file, 0, GET_STR, REQUIRED_ARG, > - 0, 0, 0, 0, 0, 0}, > - {"defaults-extra-file", 'e', > - "Read this file after the global config file and before the config " > - "file in the users home directory; should be the first option", > - (void *)&my_defaults_extra_file, (void *)&my_defaults_extra_file, 0, > - GET_STR, REQUIRED_ARG, 0, 0, 0, 0, 0, 0}, > - {"defaults-group-suffix", 'g', > - "In addition to the given groups, read also groups with this suffix", > - (char**) &my_defaults_group_suffix, (char**) &my_defaults_group_suffix, > - 0, GET_STR, REQUIRED_ARG, 0, 0, 0, 0, 0, 0}, > {"mysqld", 0, "Read the same set of groups that the mysqld binary does.", > &opt_mysqld, &opt_mysqld, 0, GET_BOOL, NO_ARG, 0, 0, 0, 0, 0, 0}, > {"no-defaults", 'n', "Return an empty string (useful for scripts).", > @@ -109,9 +95,6 @@ get_one_option(const struct my_option *opt __attribute__((unused)), > const char *filename __attribute__((unused))) > { > switch (opt->id) { > - case 'c': > - opt_defaults_file_used= 1; > - break; > case 'n': > cleanup_and_exit(0); > case 'I': > @@ -141,49 +124,44 @@ static int get_options(int *argc,char ***argv) > return 0; > } > > -static char *make_args(const char *s1, const char *s2) > -{ > - char *s= malloc(strlen(s1) + strlen(s2) + 1); > - strmov(strmov(s, s1), s2); > - return s; > -} > - > int main(int argc, char **argv) > { > - int count= 0, error, no_defaults= 0; > + int count, error, args_used; > char **load_default_groups= 0, *tmp_arguments[6]; > - char **argument, **arguments, **org_argv; > - int nargs, i= 0; > + char **argument, **arguments, **org_argv, **new_argv, **pos; > + int nargs, i= 0, index=0; > MY_INIT(argv[0]); > > org_argv= argv; > - if (*argv && !strcmp(*argv, "--no-defaults")) > - { > - argv++; > - ++count; > - no_defaults= 1; > - } > - /* Copy program name and --no-defaults if present*/ > + args_used= get_defaults_options(argv); > + > + /* Copy defaults-xxx arguments & program name */ > + count= args_used; > arguments= tmp_arguments; > - memcpy((char*) arguments, (char*) org_argv, (++count)*sizeof(*org_argv)); > + memcpy((char*) arguments, (char*) org_argv, count*sizeof(*org_argv)); > arguments[count]= 0; > > - /* Check out the args */ > - if (get_options(&argc,&argv)) > - cleanup_and_exit(1); > - > - if (!no_defaults) > + /* > + We already process --defaults* options at the beginning in > + get_defaults_options(). So, now we we make a new commandline which > + has all options except --defaults* options and pass it to handle_options() > + instead of original commandline. > + */ > + new_argv= (char**)malloc(argc*sizeof(*org_argv)); > + for (pos= org_argv; *pos; pos++) > { > + if (!is_prefix(*pos, "--defaults-extra-file=") && > + !is_prefix(*pos, "--defaults-file=") && > + !is_prefix(*pos, "--defaults-group-suffix=")) > + new_argv[index++]= *(pos); Why are you doing it? After get_defaults_options, you know how many arguments were default, just skip args_used, and invoke handle_options on the rest. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 271afbc88e4: Pre requiste MDEV-9245: add host to validate password interface
by Sergei Golubchik 07 Sep '21

07 Sep '21

Hi, Oleksandr! this is ok! On Sep 07, Oleksandr Byelkin wrote: > revision-id: 271afbc88e4 (mariadb-10.6.1-78-g271afbc88e4) > parent(s): 47207ea291a > author: Oleksandr Byelkin > committer: Oleksandr Byelkin > timestamp: 2021-09-07 11:52:28 +0200 > message: > > Pre requiste MDEV-9245: add host to validate password interface > > Hostname added to the interface. > > Cracklib plugin fixed according to the new interface. > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] b48dfdedd59: MDEV-26307 multi-source-replication support mysql syntax(for channel)
by Sergei Golubchik 02 Sep '21

02 Sep '21

Hi, Michael! On Sep 02, Michael Widenius wrote: > revision-id: b48dfdedd59 (mariadb-10.6.1-64-gb48dfdedd59) > parent(s): 5f0b6b017a8 > author: Michael Widenius > committer: Michael Widenius > timestamp: 2021-08-24 11:22:29 +0300 > message: > > MDEV-26307 multi-source-replication support mysql syntax(for channel) > > diff --git a/sql/sql_yacc.yy b/sql/sql_yacc.yy > index 0c1c68cd644..704812e5dd8 100644 > --- a/sql/sql_yacc.yy > +++ b/sql/sql_yacc.yy > @@ -2307,6 +2309,34 @@ connection_name: > } > ; > > +optional_for_channel: > + /* empty */ > + { > + /*do nothing */ > + } > + | for_channel > + > + ; > + > +for_channel: > + FOR_SYM CHANNEL_SYM TEXT_STRING_sys > + { > + if (Lex->mi.connection_name.str != NULL) > + { > + my_yyabort_error((ER_WRONG_ARGUMENTS, MYF(0), "CONNECTION_NAME AND FOR CHANNEL CAN NOT BE SPECIFIED AT THE SAME TIME)")); No, please, no LONG ENGLISH MESSAGES IN UPPERCASE WITH STRAY PARENTHESYS) embedded as a part of the error message. In your tests it looks like ERROR HY000: Incorrect arguments to CONNECTION_NAME AND FOR CHANNEL CAN NOT BE SPECIFIED AT THE SAME TIME) or, for example, ERROR HY000: Felaktiga argument till CONNECTION_NAME AND FOR CHANNEL CAN NOT BE SPECIFIED AT THE SAME TIME) The correct approach is to have the parser ensure the syntax is correct. Meaning, it should be START_SYM SLAVE optional_connection_name slave_thread_opts | START_SYM SLAVE slave_thread_opts optional_for_channel the incorrect but still kind of acceptable approach is to fake this and check in the code, like you did, but use my_parse_error, not ER_WRONG_ARGUMENTS > + } > + else > + { > + Lex->mi.connection_name= $3; > +#ifdef HAVE_REPLICATION > + if (unlikely(check_master_connection_name(&$3))) > + my_yyabort_error((ER_WRONG_ARGUMENTS, MYF(0), "MASTER_CONNECTION_NAME")); > +#endif > + } > + > + } > + ; > + > /* create a table */ > > create: > @@ -8030,7 +8060,7 @@ opt_to: > ; > > slave: > - START_SYM SLAVE optional_connection_name slave_thread_opts > + START_SYM SLAVE optional_connection_name slave_thread_opts optional_for_channel > { > LEX *lex=Lex; > lex->sql_command = SQLCOM_SLAVE_START; Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] dc2ace70f1b: Syntax with MIGRATE keyword
by Sergei Golubchik 01 Sep '21

01 Sep '21

Hi, Aleksey! On Sep 01, Aleksey Midenkov wrote: > revision-id: dc2ace70f1b (mariadb-10.6.1-68-gdc2ace70f1b) > parent(s): 19fbfab084f > author: Aleksey Midenkov > committer: Aleksey Midenkov > timestamp: 2021-08-30 22:37:08 +0300 > message: > > Syntax with MIGRATE keyword > > ALTER TABLE tbl_name > [alter_option [, alter_option] ...] | > [partition_options] > > partition_option: { > ... > | MIGRATE [OUT] PARTITION partition_name TO [TABLE] tbl_name > } > > Examples: > > ALTER TABLE t1 MIGRATE PARTITION p1 TO tp1; > ALTER TABLE t1 MIGRATE PARTITION p2 TO TABLE tp2; > ALTER TABLE t1 MIGRATE OUT PARTITION p3 TO TABLE tp3; > > diff --git a/mysql-test/suite/parts/r/alter_table.result b/mysql-test/suite/parts/r/alter_table.result > index bc1bf661d4c..7a2d45c51c1 100644 > --- a/mysql-test/suite/parts/r/alter_table.result > +++ b/mysql-test/suite/parts/r/alter_table.result > @@ -258,9 +258,46 @@ show grants for current_user; > Grants for alan@% > GRANT USAGE ON *.* TO `alan`@`%` > GRANT INSERT, CREATE, DROP, ALTER ON `test`.* TO `alan`@`%` > -alter table t1 extract partition p1 as table tp1; > +alter table t1 migrate partition p1 to table tp1; > disconnect alan; > connection default; > drop database EXISTENT; > drop user alan; > drop tables t1, tp1, tp2, tp4; > +# Cunning syntax what's so cunning here? > +create or replace table t1 (x int) > +partition by range(x) ( > diff --git a/sql/handler.h b/sql/handler.h > index c455625aafe..7ac9e929ecc 100644 > --- a/sql/handler.h > +++ b/sql/handler.h > @@ -824,7 +824,7 @@ typedef bool Log_func(THD*, TABLE*, bool, const uchar*, const uchar*); > #define ALTER_PARTITION_TRUNCATE (1ULL << 11) > // Set for REORGANIZE PARTITION > #define ALTER_PARTITION_TABLE_REORG (1ULL << 12) > -#define ALTER_PARTITION_EXTRACT (1ULL << 13) > +#define ALTER_PARTITION_MIGRATE_OUT (1ULL << 13) > > /* > This is master database for most of system tables. However there > diff --git a/sql/sql_yacc.yy b/sql/sql_yacc.yy > index 49dd244f2ff..e1be6ea9557 100644 > --- a/sql/sql_yacc.yy > +++ b/sql/sql_yacc.yy > @@ -7492,6 +7492,16 @@ ident_or_empty: > | ident > ; > > +move_out_partition: > + MIGRATE_SYM PARTITION_SYM > + | MIGRATE_SYM OUT_SYM PARTITION_SYM Where does this OUT_SYM came out from? It doesn't add any values, it doesn't make the syntax more natural. Why did you add it? > + ; > + > +to_table: > + TO_SYM TABLE_SYM > + | TO_SYM Let's keep the TABLE_SYM. Saying MIGRATE PARTITION p1 TO p2 is very vague, the syntax should explicitly say that it migrates a PARTITION to a TABLE. The statement mentions many different types of objects, if should be clear about what object type every identifier refers to. > + ; > + > alter_commands: > /* empty */ > | DISCARD TABLESPACE > @@ -7619,15 +7629,15 @@ alter_commands: > MYSQL_YYABORT; > Lex->alter_info.partition_flags|= ALTER_PARTITION_EXCHANGE; > } > - | EXTRACT_SYM PARTITION_SYM alt_part_name_item > - AS TABLE_SYM table_ident have_partitioning > + | move_out_partition alt_part_name_item > + to_table table_ident have_partitioning please, let it be just MIGRATE_SYM PARTITION_SYM alt_part_name_item TO_SYM TABLE_SYM table_ident have_partitioning > { > - if (Lex->stmt_alter_table($6)) > + if (Lex->stmt_alter_table($4)) > MYSQL_YYABORT; > Lex->m_sql_cmd= new (thd->mem_root) Sql_cmd_alter_table(); > if (unlikely(Lex->m_sql_cmd == NULL)) > MYSQL_YYABORT; > - Lex->alter_info.partition_flags|= ALTER_PARTITION_EXTRACT; > + Lex->alter_info.partition_flags|= ALTER_PARTITION_MIGRATE_OUT; > } > ; Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 5

Re: [Maria-developers] 19fbfab084f: Compilation fix
by Sergei Golubchik 01 Sep '21

01 Sep '21

Hi, Aleksey! On Sep 01, Aleksey Midenkov wrote: > revision-id: 19fbfab084f (mariadb-10.6.1-67-g19fbfab084f) > parent(s): 8009680fbc3 > author: Aleksey Midenkov > committer: Aleksey Midenkov > timestamp: 2021-08-30 22:02:40 +0300 > message: > > Compilation fix > > diff --git a/sql/sql_table.h b/sql/sql_table.h > index 83e88c82f0e..8cb9ebe0445 100644 > --- a/sql/sql_table.h > +++ b/sql/sql_table.h > @@ -20,8 +20,14 @@ > #include <my_sys.h> // pthread_mutex_t > #include "m_string.h" // LEX_CUSTRING > > +inline bool suicide() > +{ > + DBUG_SUICIDE(); > + return false; > +} > + > #define ERROR_INJECT_CRASH(code) \ > - (DBUG_IF(code) && (DBUG_SUICIDE(), 0)) > + (DBUG_IF(code) && suicide()) What did that change? A compiler didn't like (void,0) ? Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 2

Re: [Maria-developers] 87953156c76: Vanilla cleanup: DBUG_EVALUATE_IF() macro replaced by shorter form DBUG_IF()
by Sergei Golubchik 01 Sep '21

01 Sep '21

Hi, Aleksey! On Sep 01, Aleksey Midenkov wrote: > revision-id: 87953156c76 (mariadb-10.6.1-61-g87953156c76) > parent(s): b67fe0c009d > author: Aleksey Midenkov > committer: Aleksey Midenkov > timestamp: 2021-08-30 21:24:35 +0300 > message: > > Vanilla cleanup: DBUG_EVALUATE_IF() macro replaced by shorter form DBUG_IF() Thanks, I agree, this looks much better > diff --git a/dbug/user.r b/dbug/user.r > index 8d8a9ce6910..3740a60851c 100644 > --- a/dbug/user.r > +++ b/dbug/user.r > @@ -785,18 +785,20 @@ EX:\fC > +.B 0 > +Like DBUG_EXECUTE_IF this could be used to conditionally execute > "dangerous" actions. > .SP 1 > EX:\fC > .br > if (prepare_transaction () || > .br > - DBUG_EVALUATE ("crashme", (DBUG_ABORT(), 0), 0) || > + (DBUG_IF("crashme") && (DBUG_ABORT(), 0)) || oh, a bug in the manual! good catch. > .br > commit_transaction () )\fR > .SP 1 > diff --git a/include/my_dbug.h b/include/my_dbug.h > index e25bfcf28a7..2afcb010e7c 100644 > --- a/include/my_dbug.h > +++ b/include/my_dbug.h > @@ -102,8 +102,7 @@ extern int (*dbug_sanity)(void); > do {if (_db_keyword_(0, (keyword), 1)) { a1 }} while(0) > #define DBUG_EVALUATE(keyword,a1,a2) \ > (_db_keyword_(0,(keyword), 0) ? (a1) : (a2)) you've kept DBUG_EVALUATE, although it was only added for consistency together with DBUG_EVALUATE_IF. And DBUG_EVALUATE is not used anywhere, I've just grepped. Remove it too? > -#define DBUG_EVALUATE_IF(keyword,a1,a2) \ > - (_db_keyword_(0,(keyword), 1) ? (a1) : (a2)) > +#define DBUG_IF(keyword) _db_keyword_(0, (keyword), 1) > #define DBUG_PRINT(keyword,arglist) \ > do if (_db_pargs_(__LINE__,keyword)) _db_doprnt_ arglist; while(0) > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] f14cfd22115: Vanilla refactoring: binlog query in DDL_LOG_RENAME_ACTION
by Aleksey Midenkov 24 Aug '21

24 Aug '21

Hi Sergei! Really, this is not pure refactoring. I've been planning to remove it if it will not be useful for MDEV-22165 (that task should be based on this branch). Meanwhile, can you please review other commits? On Mon, Aug 23, 2021 at 7:35 PM Sergei Golubchik <serg(a)mariadb.com> wrote: > > Hi, Aleksey! > > On Aug 23, Aleksey Midenkov wrote: > > revision-id: f14cfd22115 (mariadb-10.6.1-63-gf14cfd22115) > > parent(s): 03225474d60 > > author: Aleksey Midenkov > > committer: Aleksey Midenkov > > timestamp: 2021-08-17 20:01:48 +0300 > > message: > > > > Vanilla refactoring: binlog query in DDL_LOG_RENAME_ACTION > > > > This turned out to be not needed for MDEV-22166. But it still might be > > useful for MDEV-22165 or later. > > There are three changes here. > Adding 'locked' to ddl_log_store_query, creating > st_ddl_recovery::binlog_query(), adding 'case RENAME_PHASE_BINLOG' > > st_ddl_recovery::binlog_query() is ok, if you want to keep it. > But the other two aren't a refactoring, it's creating new functionality > which isn't used in this patch, dead code basically. And your commit > comment seems to imply that this new functionality isn't needed > in the following feature patch either. > > While your other refactoring patches are indeed pure refactoring, this > one isn't, so, please, remove it from the patchset if it isn't used > there. You can keep st_ddl_recovery::binlog_query() if you'd like - it > is indeed pure refactoring. > > Regards, > Sergei > VP of MariaDB Server Engineering > and security(a)mariadb.org -- All the best, Aleksey Midenkov @midenok

2 1

Re: [Maria-developers] bc55e3c438f: MDEV-4742 - provide function to sort string in "natural" order
by Sergei Golubchik 20 Aug '21

20 Aug '21

Hi, Vladislav! On Aug 20, Vladislav Vaintroub wrote: > commit bc55e3c438f > Author: Vladislav Vaintroub <wlad(a)mariadb.com> > Date: Thu Jul 29 13:28:11 2021 +0200 > > MDEV-4742 - provide function to sort string in "natural" order > > The numbers should be compared as numbers, while the rest should be compared > as string. > > Introduce natural_sort_key() function that transforms original string > so that the lexicographic order of such keys is suitable for > natural sort. This comment doesn't mention the second optional argument. Neither does Jira. > diff --git a/mysql-test/main/natural_sort_key.test b/mysql-test/main/natural_sort_key.test > new file mode 100644 > index 00000000000..bee0d8e76bd > --- /dev/null > +++ b/mysql-test/main/natural_sort_key.test > @@ -0,0 +1,64 @@ > +--source include/have_innodb.inc > +--source include/have_sequence.inc > + > +SET NAMES utf8mb4; > +SELECT NATURAL_SORT_KEY(NULL); > + > +# Sort without tables > +SELECT '' c WHERE 0 UNION VALUES('a10'),('a9'),('a1000'), ('a0'),('b'),('b0') ORDER BY NATURAL_SORT_KEY(c); > + > +#Test that max packet overflow produces NULL plus warning > +SELECT NATURAL_SORT_KEY(repeat('a1',@@max_allowed_packet/2-1)); > + > +#Test with virtual(index only) key > +CREATE TABLE t1( > + c VARCHAR(30) CHARACTER SET latin1 COLLATE latin1_bin, > + k VARCHAR(45) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci AS (NATURAL_SORT_KEY(CONVERT(c USING utf8mb4))) INVISIBLE, > + KEY(k,c)) ENGINE=InnoDB; > +INSERT INTO t1 values > + ('A1'),('a1'),('A100'),('a100'),('A2'),('ä2'),('a2'),('A99'), > + ('äb'),('B1'),('B100'),('B9'),('C'),('100'); > +EXPLAIN SELECT c FROM t1 ORDER BY k,c; > +-- echo #Natural sort order. > +# We sort by 2 colums, for stable sort,as we do not currenly have a case and accent insensitive Unicode collation. > +SELECT c FROM t1 ORDER BY k,c; > +-- echo #Unnatural but unicode aware) sort order > +SELECT c FROM t1 ORDER BY CONVERT(c USING utf8mb4) COLLATE utf8mb4_unicode_ci,c; > +# CREATE TABLE AS SELECT, to see that length of the column is correct. > +CREATE TABLE t2 AS SELECT c, NATURAL_SORT_KEY(c) FROM t1 WHERE 0; you could've simply used --enable_metadata > +SHOW CREATE TABLE t2; > +DROP TABLE t1,t2; > + > +#Show encoding of numbers, including fractions, and leading whitespace. > +SELECT RPAD(val,21,' ') value , RPAD(NATURAL_SORT_KEY(val),26,' ') sortkey , LENGTH(NATURAL_SORT_KEY(val)) - LENGTH(val) encoding_overhead > +FROM > +( > +SELECT 0 val > +UNION VALUES ('0.0'),('0.1'), ('0,15'),('1.001'),('1.002'),('1.010'),('1.02'),('1.1'),('1.3'),('1'),('01'),('0001') > +UNION SELECT CONCAT('1',repeat('0',seq)) FROM seq_1_to_20 > +) AS numbers ORDER BY sortkey; > + > +--echo # Disable fractions handling by passing NULL as second parameter to NATURAL_SORT_KEY > +SELECT val value > +FROM > +( > +SELECT 0 val > +UNION VALUES ('0.1'),('1.001'),('1.002'),('1.010'),('1.02'),('1.1'),('1.3'),('1'), ('0,1'),('1,001'),('1,002'),('1,010'),('1,02'),('1,1'),('1,3'),('1')) > +AS numbers ORDER BY NATURAL_SORT_KEY(val, NULL); > + > +--echo # Use ',' as decimal separator for NATURAL_SORT_KEY > +SELECT val value, NATURAL_SORT_KEY(val,',') sortkey > +FROM > +( > +SELECT 0 val > +UNION VALUES ('0,1'),('1,001'),('1,002'),('1,010'),('1,02'),('1,1'),('1,3'),('1')) > +AS numbers ORDER BY sortkey; > + > +--echo # Use '.' as decimal separator for NATURAL_SORT_KEY > +SELECT val value,NATURAL_SORT_KEY(val,'.') sortkey > +FROM > +( > +SELECT 0 val > +UNION VALUES ('0.1'),('1.001'),('1.002'),('1.010'),('1.02'),('1.1'),('1.3'),('1')) > +AS numbers ORDER BY sortkey; > +SET NAMES DEFAULT; > diff --git a/sql/item_strfunc.cc b/sql/item_strfunc.cc > index 7e8ff667e75..a2efd414750 100644 > --- a/sql/item_strfunc.cc > +++ b/sql/item_strfunc.cc > @@ -5293,6 +5293,232 @@ String *Item_temptable_rowid::val_str(String *str) > return &str_value; > } > > +/** > + Here is how we generate a key suitable for lexicographic comparison > + from a numeric string (representing positive number). > + > + We prepend variable length prefix, that only encodes string length > + The longer the string, the lexicographically larger is the prefix. > + > + Rules for generating prefix is following. > + > + 1. Small length (1 to 9) > + Prefix is single char '0' + length -1 > + This gives the range '0' - '8' > + 2. Medium length(10 to 18) > + Prefix is 2 chars - concat('9', '0'+length-10)' > + This gives range '90'-'98' > + 3. Large length(19) > + Prefix is "990" > + 4. Huge length (20 to 2^32-1) > + Prefix stars with '99', then log10 of the length is added as char > + then the number itself is added > + The range is '99120' - '9994294967295' please also explain here how you handle leading zeros > +*/ > + > +#define NATSORT_BUFSIZE 16 > +static size_t natsort_num2str(size_t n, char *s) > +{ > + if (n < 10) > + { > + s[0]= '0' + (char)n -1; > + return 1; > + } > + > + if (n < 19) > + { > + s[0]= '9'; > + s[1]= '0'+ (char)n- 10; > + return 2; > + } > + > + if (n == 19) > + { > + s[0]= '9'; > + s[1]= '9'; > + s[2]= '0'; > + return 3; > + } > + > + /* Here, n > 19*/ > + s[0]= '9'; > + s[1]= '9'; > + size_t log10n= 0; > + for (size_t tmp= n/10; tmp; tmp /= 10) > + log10n++; > + DBUG_ASSERT(log10n && log10n < 10); first, I don't see where the caller ensures that log10n < 10 (in other words, that n < 9999999999) second, please, never write assert(a && b), always do two separate asserts, so that when it fires, it'd be clear what condition was false > + s[2]='0' + (char)log10n; > + return ll2str(n,s+3,10,0) - s; > +} > + > +/** > + Convert a string to natural sort key. > + @param[in] in - input string > + @param[out] s - output string > + > + We assume that memory is preallocated for the output > + string, so that appends do not fail. > + > + The lead zeros count, if positive, is stored after > + the numeric string. > + > + Fractional parts are stored and compared as strings. > + Even if they have numbers in them, they should just > + be compared as strings. > +*/ > +static void to_natsort_key(const String *in, String *s, my_wc_t decimal_sep) > +{ > + CHARSET_INFO *cs= in->charset(); > + uchar *pos= (uchar *) in->ptr(); > + uchar *end= pos + in->length(); > + size_t num_len= 0; > + size_t leading_zeros= 0; > + uchar *num_start= nullptr; > + bool in_fraction= false; > + size_t fraction_len=0; > + > + for (;;) > + { > + my_wc_t c= 0; > + int charlen= cs->mb_wc(&c, pos, end); > + bool is_digit= charlen >= 1 && (c >= '0' && c <= '9'); what do you mean `>= 1` ? how can it be `> 1` and still satisfy the secon part of the condition? > + > + if (!is_digit && (num_start || leading_zeros)) > + { > + /* > + Handle end of digits run. > + > + Write prefix, natsort_num2str(length) is without leading zeros > + Write numeric value, without leading zeros > + If there were leading zeros, write variable length suffix > + */ > + char buf[NATSORT_BUFSIZE]; > + if (!num_len) > + { > + // number was zeros only > + leading_zeros--; > + num_len= 1; > + } > + > + s->append(buf, natsort_num2str(num_len, buf)); > + if (!num_start) > + s->append('0'); > + else > + s->append((const char *) num_start, pos - num_start, cs); > + > + if (leading_zeros) > + s->append(buf, natsort_num2str(leading_zeros, buf)); hmm. How would it sort "1;", "001;", "1a" ? > + > + /* Reset state.*/ > + num_len= 0; > + num_start= nullptr; > + leading_zeros= 0; > + in_fraction= c == decimal_sep; I don't really like this idea. First, see https://natsort.readthedocs.io/en/master/index.html there are many ways one can potentially want to customize natsort. (of course, none of them have to be in this MDEV) One of those customizations, indeed, is treating floating point numbers as floating point numbers. But comma vs dot is locale thing, and a one string argument it's not how you should tell the function what decimal separator the current locale uses. So. Simple approach - remove the second argument, no floating point support. Complex approach - have a second argument that enables floating point support, but that can be used in the future to specify more customizations, not that is hard-wired to specify decimal separator duplicating and bypassing the locale. > + fraction_len= 0; > + } > + > + if (charlen < 1) > + break; > + > + if (is_digit && !in_fraction) > + { > + /* Do not count leading zeros in num_len, and num_pos */ > + if (c != '0' || num_len) > + { > + num_len++; > + if (!num_start) > + num_start= pos; > + } > + else > + leading_zeros++; > + } > + else > + { > + /* Append a non-digit, or fractional part of the number (handling the same as string) */ > + s->append((const char *) pos, charlen); > + if (in_fraction) > + { > + if (!is_digit && fraction_len) > + in_fraction= false; > + fraction_len++; > + } > + } > + pos+= charlen; > + } > +} > + > + > +String *Item_func_natural_sort_key::val_str(String *s) > +{ > + if (args[0]->is_null()) > + { > + null_value= true; > + return nullptr; > + } > + > + String *in= args[0]->val_str(); > + CHARSET_INFO *cs= in->charset(); > + ulong max_allowed_packet= current_thd->variables.max_allowed_packet; > + size_t reserve_length= (size_t) in->length() + in->length() / 2 + 2; > + if (s->alloc((uint32)reserve_length)) > + goto error_exit; > + s->length(0); > + s->set_charset(cs); > + > + to_natsort_key(in, s, decimal_separator()); > + > + DBUG_ASSERT(s->length() < reserve_length); > + > + if (s->length() > max_allowed_packet) better check this inside to_natsort_key(). max_allowed_packet is supposed to prevent the string from growing too large, instead of just letting it grow and erroring out afterwards. > + { > + push_warning_printf(current_thd, Sql_condition::WARN_LEVEL_WARN, > + ER_WARN_ALLOWED_PACKET_OVERFLOWED, > + ER(ER_WARN_ALLOWED_PACKET_OVERFLOWED), func_name(), > + max_allowed_packet); > + goto error_exit; > + } > + return s; > + > +error_exit: > + null_value=true; > + return nullptr; > +} > + > + > +my_wc_t Item_func_natural_sort_key::decimal_separator() > +{ > + if (m_decimal_separator != DECIMAL_SEP_UNDEFINED) > + return m_decimal_separator; > + > + DBUG_ASSERT(arg_count > 1); > + String *s= args[1]->val_str(); > + if (s && s->length()) > + { > + my_wc_t c; > + if (s->charset()->mb_wc(&c, (const uchar *) s->ptr(), > + (const uchar *) s->end())) > + return c; > + } > + return DECIMAL_SEP_NONE; > +} > + > +bool Item_func_natural_sort_key::fix_length_and_dec(void) > +{ > + if (agg_arg_charsets_for_string_result(collation, args, 1)) > + return true; > + DBUG_ASSERT(collation.collation != NULL); > + uint32 max_clen= args[0]->max_char_length(); > + fix_char_length(max_clen + max_clen/2 + ((max_clen <= 3)?1:0)); > + set_maybe_null(); better, only set maybe_null if the first argument maybe_null or the first argument maybe_too_long > + > + if (arg_count> 1 && args[1]->can_eval_in_optimize()) > + { > + /* Initialize decimal separator */ > + m_decimal_separator= decimal_separator(); > + } > + return false; > +} > + > #ifdef WITH_WSREP > > #include "wsrep_mysqld.h" Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] a19495728de: MDEV-16355 Add option for mysqldump to read data as of specific timestamp from system-versioned tables
by Sergei Golubchik 15 Aug '21

15 Aug '21

Hi, Aleksey! I'm reviewing it separately from other commits in the branch, because it looks to be functionally independent from, say, MDEV-16029 mysqldump: dump and restore historical data and doesn't have to wait for the latter to be pushed first. I think I was able to guess what the patch would look like, if you'd apply it to vanilla 10.7 without MDEV-16029. But just to be sure, please, let me look it over again after the cherry-pick. Otherwise no comments code-wise. A couple of comments related to naming, see below. On Aug 15, Aleksey Midenkov wrote: > revision-id: a19495728de (versioning-1.0.5-45-ga19495728de) > parent(s): be4605ae086 > author: Aleksey Midenkov > committer: Aleksey Midenkov > timestamp: 2018-07-19 14:47:35 +0300 > message: > > MDEV-16355 Add option for mysqldump to read data as of specific timestamp from system-versioned tables > > mysqldump changes: > > * --asof-timestamp option specifies historical point; "asof" isn't a word. I'd suggest --as-of-timestamp or simply --as-of > * with --dump-history it dumps all prior history, without it dumps only single point in time; > * query forging protection for --asof-timestamp parameter. > > diff --git a/client/mysqldump.c b/client/mysqldump.c > index eee12555271..f426e8c58ef 100644 > --- a/client/mysqldump.c > +++ b/client/mysqldump.c > @@ -1099,6 +1102,13 @@ static int get_options(int *argc, char ***argv) > my_progname_short); > return(EX_USAGE); > } > + if (opt_asof_timestamp && strchr(opt_asof_timestamp, '\'')) > + { > + fprintf(stderr, > + "%s: --asof-timestamp wrong argument!\n", > + my_progname_short); May be, use the wording that is closer to the error one would get from any other invalid character, like, if one would do --asof-timestamp='!@#$%' That is, instead of "wrong argument", something like fprintf(stderr, "%s: Incorrect DATETIME value: '%s'\n", opt_asof_timestamp, my_progname_short); Because MariaDB [test]> select timestamp'2010!@#$%'; ERROR 1525 (HY000): Incorrect DATETIME value: '2010!@#$%' > + return(EX_USAGE); > + } > if (strcmp(default_charset, charset_info->csname) && > !(charset_info= get_charset_by_csname(default_charset, > MY_CS_PRIMARY, MYF(MY_WME)))) Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] cdb6db8ce38: MDEV-25444 mysql --binary-mode is not able to replay some mysqlbinlog outputs
by Sergei Golubchik 14 Aug '21

14 Aug '21

Hi, Sachin! On Jul 12, Sachin Kumar wrote: > revision-id: cdb6db8ce38 (mariadb-10.3.26-155-gcdb6db8ce38) > parent(s): 72753d2b65f > author: Sachin Kumar <sachin.setiya(a)mariadb.com> > committer: Sachin Kumar <sachin.setiya(a)mariadb.com> > timestamp: 2021-05-19 15:46:57 +0100 > message: > > MDEV-25444 mysql --binary-mode is not able to replay some mysqlbinlog outputs > > Problem:- Some binary data is inserted into the table using Jconnector. When > binlog dump of the data is applied using mysql cleint it gives syntax error. > > Reason:- > After investigating it turns out to be a issue of mysql client not able to properly > handle \\\0 <0 in binary>. In all binary files where mysql client fails to insert > these 2 bytes are commom (0x5c00) > > Solution:- > I have changed mysql.cc to include for the possibility that binary string can > have \\\0 in it. > > --- > client/mysql.cc | 8 ++++++-- > mysql-test/main/binary_zero_insert.result | 6 ++++++ > mysql-test/main/binary_zero_insert.test | 15 +++++++++++++++ > mysql-test/std_data/binary_zero_insert.bin | Bin 0 -> 87 bytes > 4 files changed, 27 insertions(+), 2 deletions(-) > > diff --git a/client/mysql.cc b/client/mysql.cc > index 2a7c4eaf3e5..8530c105820 100644 > --- a/client/mysql.cc > +++ b/client/mysql.cc > @@ -2319,8 +2319,12 @@ static bool add_line(String &buffer, char *line, size_t line_length, > { > // Found possbile one character command like \c > > + inchar = (uchar) *++pos; > + // In Binary mode , when in_string is not null \0 should not be treated as > + // end statement. This can happen when we are in middle of binary data which > + // can contain \0 and its quoted with ' '. > + if (!real_binary_mode && !*in_string && !inchar) Two comments here. 1. if you check the history where this "real_binary_mode" came from, you'd arrive at commit 2db4392bf4cb and the test file mysql_binary_mode.test So, I'd say it make sense to add your test there too. And in the same style, wthout a separate binary file. in my experimening I did it as simple as --- a/mysql-test/main/mysql_binary_mode.test +++ b/mysql-test/main/mysql_binary_mode.test @@ -16,14 +16,14 @@ let $table_name_right= `SELECT 0x410D0A42`; let $table_name_wrong= `SELECT 0x410A42`; # 0x410042 => 'A\0B' -let $char0= `SELECT 0x410042`; +let $char0= `SELECT 0x4100425C0043`; (there already was a test for a \0 byte, but not for an escaped \\\0) but if you'd like you can add a separate test with a header and everything. > - if (!(inchar = (uchar) *++pos)) > - break; // readline adds one '\' 2. What does the old code mean? Why returning if inchar==0 ? What does the comment mean? It's the code from the last millenium, does it even apply now? For example, there can be no \0 bytes in the string, because of if (!real_binary_mode && strlen(line) != line_length) ... in other words, shouldn't we just remove the if? I tried, at least the main suite didn't break. > + break; // readline adds one '\' > if (*in_string || inchar == 'N') // \N is short for NULL > { // Don't allow commands in string > *out++='\\'; Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 2

[Maria-developers] Pushing down aggregation functions
by Berrocal, Eduardo 04 Aug '21

04 Aug '21

Hello!, Continuing in my series of emails about engine condition pushdown... I wonder if it is possible to push down aggregation functions such as SUM(), MIN(), MAX() or AVG() to the storage engine. Currently, and as far as I can see (I may be wrong here), it is not possible. However, this is something that can be beneficial if the storage engine can perform some of these basic computations itself, like suming over a whole column and returning just the result. The storage engine can even be smart enough to use some hardware advantages like accelerators or computational storage. If you guys think this is indeed possible without drastic code changes, it would be amazing to be pointed at where this is. At first, I would only push down functions if they operate over just one column of the same table. Things can be made more complicated in the future, of course. Thanks a lot!, Eduardo.

2 1

Re: [Maria-developers] 203aa38f9b6: MDEV-26102: gcov 9.1 doesn't support intermediate format any more, but json instead - refactoring needed
by Sergei Golubchik 29 Jul '21

29 Jul '21

Hi, Anel! On Jul 28, Anel Husakovic wrote: > revision-id: 203aa38f9b6 (mariadb-10.2.39-96-g203aa38f9b6) > parent(s): afe00bb7cce > author: Anel Husakovic > committer: Anel Husakovic > timestamp: 2021-07-27 16:09:41 +0200 > message: > > MDEV-26102: gcov 9.1 doesn't support intermediate format any more, but json instead - refactoring needed I've renamed the MDEV to have a shorter title, use the new one in your next commit > > Reviewed by: serg(a)mariadb.com that's not right, I haven't reviewed it yet :) > diff --git a/mysql-test/dgcov.pl b/mysql-test/dgcov.pl > index 2c00c64d1ff..e2aadec33e0 100755 > --- a/mysql-test/dgcov.pl > +++ b/mysql-test/dgcov.pl > @@ -25,6 +25,8 @@ use warnings; > use Getopt::Long; > use File::Find; > use Cwd qw/realpath/; > +use IO::Uncompress::Gunzip qw(gunzip $GunzipError); > +use JSON::PP; move this down, dcgov should not try to load these modules for for gcc < 9 > my $opt_verbose=0; > my $opt_generate; > @@ -32,6 +34,7 @@ my $opt_help; > my $opt_purge; > my $opt_only_gcov; > my $opt_skip_gcov; > +my $fname; and this > > my %cov; > my $file_no=0; > @@ -62,12 +65,15 @@ my $res; > my $cmd; > if ($opt_purge) > { > - $cmd= "find . -name '*.da' -o -name '*.gcda' -o -name '*.gcov' -o ". > + $cmd= "find . -name '*.da' -o -name '*.gcda*' -o -name '*.gcov' -o ". > "-name '*.dgcov' | grep -v 'README\.gcov' | xargs rm -f ''"; > logv "Running: $cmd"; > system($cmd)==0 or die "system($cmd): $? $!"; > exit 0; > } > + > +my $gcc_version= `gcc -dumpversion`; > +$gcc_version=~ s/(\d).*$/$1/; you don't need this second line, I suspect > > find(\&gcov_one_file, $root); > find(\&write_coverage, $root) if $opt_generate; > @@ -162,6 +168,7 @@ sub gcov_one_file { > } > > # now, read the generated file > + if ($gcc_version <9){ > for my $gcov_file (<$_*.gcov>) { > open FH, '<', "$gcov_file" or die "open(<$gcov_file): $!"; > my $fname; > @@ -183,6 +189,18 @@ sub gcov_one_file { > $cov{$fname}->{$1}+=$2; > } > close(FH); > + } > + } > + else { put both `use` lines here > + my $gcov_file_json; > + gunzip "$_.gcov.json.gz" => \$gcov_file_json or die "gunzip($_.gcov.json.gz): $GunzipError"; > + my $obj= decode_json $gcov_file_json; > + for my $file (@{$obj->{files}}) { > + $fname= $file->{file}; and write `my $fname` here > + for my $line (@{$file->{lines}}){ > + $cov{$fname}->{$line->{line_number}}= $line->{count}; shouldn't it be += $line->{count} ? The point is to accumulate all counters, see how it's done for non-json intermediate format. > + } > + } > } > } > > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 3

Re: [Maria-developers] ce289840131: MDEV-24511 null field is created with CREATE..SELECT
by Nikita Malyavin 28 Jul '21

28 Jul '21

Hello, Sergei! Looks fine for me. So if you both prefer it more, let's use this variant. Regards, Nikita On Wed, Jul 28, 2021 at 12:08 PM Sergei Golubchik <serg(a)mariadb.org> wrote: > Hi, Nikita! > > On Jul 27, Nikita Malyavin wrote: > > revision-id: ce289840131 (mariadb-10.3.30-31-gce289840131) > > parent(s): b50ea900636 > > author: Nikita Malyavin > > committer: Nikita Malyavin > > timestamp: 2021-07-22 22:38:31 +0300 > > message: > > > > MDEV-24511 null field is created with CREATE..SELECT > > > > The regression has been initially appeared by MDEV-12588 patch > (441349aa). > > > > The reason CREATE... SELECT NULL does not create null field is that > > it calls handle_select(), where the field is created directly from Item, > > by calling type_handler->type_handler_for_tmp_table(), instead of plain > > type_handler. > > > > In UNION case, we'll already have Item_field's containing UNION'ed tmp > > table fields to the moment of select_create::prepare() call. > > > > There, item's type_handler is used directly in > > st_select_lex_unit::join_union_item_types(), instead of > > type_handler_for_tmp_table() or type_handler_for_union(). > > > > Using those two involve too many changes in CREATE...SELECT...UNION > > behavior, which may be reasonably undesired to be pushed in 10.3 > > > > However, patching Field_null::type_handler() directly seems to be > accurate > > enough. > > > > diff --git a/mysql-test/main/union.result b/mysql-test/main/union.result > > index a892f6c9e40..b19427948a6 100644 > > --- a/mysql-test/main/union.result > > +++ b/mysql-test/main/union.result > > @@ -1609,7 +1609,7 @@ NULL binary(0) YES NULL > > CREATE TABLE t5 SELECT NULL UNION SELECT NULL; > > DESC t5; > > Field Type Null Key Default Extra > > -NULL null YES NULL > > +NULL binary(0) YES NULL > > CREATE TABLE t6 > > SELECT * FROM (SELECT * FROM (SELECT NULL)a) b UNION SELECT a FROM t1; > > DESC t6; > > diff --git a/sql/field.h b/sql/field.h > > index a3385736c0a..2af5346bea9 100644 > > --- a/sql/field.h > > +++ b/sql/field.h > > @@ -2526,7 +2526,7 @@ class Field_null :public Field_str { > > :Field_str(ptr_arg, len_arg, null, 1, > > unireg_check_arg, field_name_arg, collation) > > {} > > - const Type_handler *type_handler() const { return &type_handler_null; > } > > + const Type_handler *type_handler() const { return > &type_handler_string; } > > Information_schema_character_attributes > > information_schema_character_attributes() const > > { > > This looks suspiciously risky to me. The problem was specifically in > UNION, but you change the very basic idea of what type the Field_null > is. > > I cannot clearly see what else might this possibly affect, field's type > is used everywhere. > > It would be safer and more logical to fix specifically field creation > for UNION. > > I've discussed it with Bar, who implemented this whole Type_handler > hierarchy, he suggested to use Type_handler::union_element_finalize() > for that. I've attached the patch that does that. > > union_element_finalize() is for 10.4, so I also backported it to 10.3, > it's very straightforward though. > > What do you think? > > Regards, > Sergei > VP of MariaDB Server Engineering > and security(a)mariadb.org > -- Nikita Malyavin, Software Engineer MariaDB Corporation

1 0

Re: [Maria-developers] ce289840131: MDEV-24511 null field is created with CREATE..SELECT
by Sergei Golubchik 28 Jul '21

28 Jul '21

Hi, Nikita! On Jul 27, Nikita Malyavin wrote: > revision-id: ce289840131 (mariadb-10.3.30-31-gce289840131) > parent(s): b50ea900636 > author: Nikita Malyavin > committer: Nikita Malyavin > timestamp: 2021-07-22 22:38:31 +0300 > message: > > MDEV-24511 null field is created with CREATE..SELECT > > The regression has been initially appeared by MDEV-12588 patch (441349aa). > > The reason CREATE... SELECT NULL does not create null field is that > it calls handle_select(), where the field is created directly from Item, > by calling type_handler->type_handler_for_tmp_table(), instead of plain > type_handler. > > In UNION case, we'll already have Item_field's containing UNION'ed tmp > table fields to the moment of select_create::prepare() call. > > There, item's type_handler is used directly in > st_select_lex_unit::join_union_item_types(), instead of > type_handler_for_tmp_table() or type_handler_for_union(). > > Using those two involve too many changes in CREATE...SELECT...UNION > behavior, which may be reasonably undesired to be pushed in 10.3 > > However, patching Field_null::type_handler() directly seems to be accurate > enough. > > diff --git a/mysql-test/main/union.result b/mysql-test/main/union.result > index a892f6c9e40..b19427948a6 100644 > --- a/mysql-test/main/union.result > +++ b/mysql-test/main/union.result > @@ -1609,7 +1609,7 @@ NULL binary(0) YES NULL > CREATE TABLE t5 SELECT NULL UNION SELECT NULL; > DESC t5; > Field Type Null Key Default Extra > -NULL null YES NULL > +NULL binary(0) YES NULL > CREATE TABLE t6 > SELECT * FROM (SELECT * FROM (SELECT NULL)a) b UNION SELECT a FROM t1; > DESC t6; > diff --git a/sql/field.h b/sql/field.h > index a3385736c0a..2af5346bea9 100644 > --- a/sql/field.h > +++ b/sql/field.h > @@ -2526,7 +2526,7 @@ class Field_null :public Field_str { > :Field_str(ptr_arg, len_arg, null, 1, > unireg_check_arg, field_name_arg, collation) > {} > - const Type_handler *type_handler() const { return &type_handler_null; } > + const Type_handler *type_handler() const { return &type_handler_string; } > Information_schema_character_attributes > information_schema_character_attributes() const > { This looks suspiciously risky to me. The problem was specifically in UNION, but you change the very basic idea of what type the Field_null is. I cannot clearly see what else might this possibly affect, field's type is used everywhere. It would be safer and more logical to fix specifically field creation for UNION. I've discussed it with Bar, who implemented this whole Type_handler hierarchy, he suggested to use Type_handler::union_element_finalize() for that. I've attached the patch that does that. union_element_finalize() is for 10.4, so I also backported it to 10.3, it's very straightforward though. What do you think? Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] c34533186ab: MDEV-18734 ASAN heap-use-after-free upon sorting by blob column from partitioned table
by Sergei Golubchik 26 Jul '21

26 Jul '21

Hi, Aleksey! As before, despite the email subject, it's git diff dfa2d0bc13..e533b5fb307 Don't be confused :) On Jul 21, Aleksey Midenkov wrote: > diff --git a/sql/ha_partition.cc b/sql/ha_partition.cc > index c53732a2b51..70ed14448ea 100644 > --- a/sql/ha_partition.cc > +++ b/sql/ha_partition.cc > @@ -5130,7 +5135,18 @@ bool ha_partition::init_record_priority_queue() > i= bitmap_get_next_set(&m_part_info->read_partitions, i)) > { > DBUG_PRINT("info", ("init rec-buf for part %u", i)); > - int2store(ptr, i); > + if (table->s->blob_fields) > + { > + for (uint j= 0; j < table->s->blob_fields; ++j, ++objs) > + { > + blob_storage[j]= new (objs) Ordered_blob_storage; > + if (!blob_storage[j]) > + DBUG_RETURN(true); is this possible? > + } > + *((Ordered_blob_storage ***) ptr)= blob_storage; > + blob_storage+= table->s->blob_fields; > + } > + int2store(ptr + sizeof(String **), i); > ptr+= m_priority_queue_rec_len; > } > m_start_key.key= (const uchar*)ptr; > @@ -6291,6 +6333,43 @@ int ha_partition::handle_ordered_index_scan_key_not_found() > } > > > +void ha_partition::swap_blobs(uchar * rec_buf, Ordered_blob_storage ** storage, bool restore) > +{ > + uint *ptr, *end; > + uint blob_n= 0; > + table->move_fields(table->field, rec_buf, table->record[0]); > + for (ptr= table->s->blob_field, end= ptr + table->s->blob_fields; > + ptr != end; ++ptr, ++blob_n) > + { > + DBUG_ASSERT(*ptr < table->s->fields); > + Field_blob *blob= (Field_blob*) table->field[*ptr]; > + DBUG_ASSERT(blob->flags & BLOB_FLAG); > + DBUG_ASSERT(blob->field_index == *ptr); > + if (!bitmap_is_set(table->read_set, *ptr) || blob->is_null()) > + continue; > + > + Ordered_blob_storage &s= *storage[blob_n]; > + > + if (restore) > + { > + if (!s.blob.is_empty()) > + blob->swap(s.blob, s.set_read_value); don't you need here something like else blob->reset(); to make blob empty? > + } > + else > + { > + bool set_read_value; > + String *cached= blob->cached(set_read_value); > + if (cached) > + { > + cached->swap(s.blob); > + s.set_read_value= set_read_value; > + } > + } > + } > + table->move_fields(table->field, table->record[0], rec_buf); On swap_blobs() you always move all fields from rec_buf to table->record[0]? Why? This looks very suspicious. > +} > + > + > /* > Common routine to handle index_next with ordered results > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 3

Re: [Maria-developers] c4835af7e8c: MDEV-16026: Forbid global system_versioning_asof in non-default time zone
by Sergei Golubchik 26 Jul '21

26 Jul '21

Hi, Nikita! I suggest to squash commits from c4835af7e8c to 80e3007a5a1 into one. It'll be simpler, no intermediate juggling with MYSQL_TIME anymore. This squashed commit is ok to push (make sure to mention both MDEVs in the commit comment, but not necessarily in the first line) On Jul 26, Nikita Malyavin wrote: > revision-id: c4835af7e8c (mariadb-10.3.30-30-gc4835af7e8c) > parent(s): ec3289b6de4 > author: Nikita Malyavin > committer: Nikita Malyavin > timestamp: 2021-07-21 00:21:31 +0300 > message: > > MDEV-16026: Forbid global system_versioning_asof in non-default time zone > > * store `system_versioning_asof` in unix time; > * both session and global vars are processed in session timezone; > * setting `default` does not copy global varibale abymore. Instead, it sets > system_time to SYSTEM_TIME_UNSPECIFIED, which means that no 'AS OF' time > is applied and `now()` can be assumed > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] a4a5c2d5963: MDEV-25908: -e does not work for my_print_defaults
by Sergei Golubchik 26 Jul '21

26 Jul '21

Hi, Rucha! On Jul 26, Rucha Deodhar wrote: > revision-id: a4a5c2d5963 (mariadb-10.5.11-52-ga4a5c2d5963) > parent(s): 675b72d0da1 > author: Rucha Deodhar > committer: Rucha Deodhar > timestamp: 2021-07-23 19:51:57 +0530 > message: > > MDEV-25908: -e does not work for my_print_defaults > > Analysis: get_defaults_options() rewrites the value for my_defaults_file, > my_defaults_extra_file and my_defaults_group_suffix to 0. So the config file > can't be read. > Fix: Let handle_options() handle --defaults* option and create a new command > line to pass it to load_defaults(). > > diff --git a/libmariadb b/libmariadb > index 802ce584a26..180c543704d 160000 > --- a/libmariadb > +++ b/libmariadb > @@ -1 +1 @@ > -Subproject commit 802ce584a26fdc0ba67fcf35e277bf3c7440956a > +Subproject commit 180c543704d627a50a52aaf60e24ca14e0ec4686 ^^^^^ This is wrong, please, remove it from your commit Otherwise ok to push Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

[Maria-developers] Accessing read records during join for condition pushdown
by Berrocal, Eduardo 23 Jul '21

23 Jul '21

Hello list (first email here, hopefully sending it to the right address), I have been trying to solve this problem for a while now, and I am stuck. I wonder if what I am trying to do is even possible. I am implementing a new storage engine which handles engine condition pushdown. Now, when MariaDB is working on a JOIN using more than one table (for simplicity, let's assume 2), how can I access the read records for table 1 when working on a condition pushdown for table 2? When joining like this: table1.field1 = table2.field1, I can see 2 Item objects of type FIELD in the condition. However, I can't find where the read records for table1 are so I can effectively filter records in table2 by those values. Thanks a lot for your help, [A close up of a sign Description automatically generated] Eduardo Berrocal García de Carellán Senior Software Engineer Intel Corporation | intel.com<http://intel.com/>

3 9

Re: [Maria-developers] 412e696fd2b: MDEV-25395 server recovery hits replication event checksum error
by Sergei Golubchik 23 Jul '21

23 Jul '21

Hi, Andrei! fdle changes - ok, I've seen them in 21117. cur_log/etc - clear. What are the changes around prev_event_pos for? On Jul 23, Andrei Elkin wrote: > revision-id: 412e696fd2b (mariadb-10.3.26-136-g412e696fd2b) > parent(s): 2fa526e26e8 > author: Andrei Elkin > committer: Andrei Elkin > timestamp: 2021-04-15 17:11:07 +0300 > message: > > MDEV-25395 server recovery hits replication event checksum error > > When @@global.binlog_checksum is changing from none to > crc32 and only the first of two Binlog_checkpoint_log_event gets written to > the crc32 rotated binlog file before the server crashes, the recovery faces a checksum > verification error. > > The reason was that recovery algorithm did not realize > potentially varying binlog checksum at scanning of > a sequence of binlog files. > > That is corrected now. > The master-checksum-verify server recovery aborts now and stops > the server due to the checksum error. > A simulation test confirms the fixes. > > Induced changes: > > rpl_cant_read_event_incident got affected because the test > adopts an external not properly closed binlog file that contains > events which can't pass checksum verification. > A work-around is done to accept that binlog and evade recovery. > > binlog_recover_checksum_error needed refinement as > the master-checksum-verify configured server is made to stop > due to the checksum error. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 1c7d4f9c06f: MDEV-25282 Auto-shutdown on idle when socket-activated
by Sergei Golubchik 23 Jul '21

23 Jul '21

Hi, Daniel! The approach is fine, see some comments below On Jul 23, Daniel Black wrote: > revision-id: 1c7d4f9c06f (mariadb-10.6.1-78-g1c7d4f9c06f) > parent(s): 71964c76fe0 > author: Daniel Black > committer: Daniel Black > timestamp: 2021-06-16 17:15:59 +1000 > message: > > MDEV-25282 Auto-shutdown on idle when socket-activated > > Adds max_idle_execution system variable with a UINT_MAX > default value that corresponds to the time in seconds > under which the mariadbd executable will run in an idle state > with no connections. > > Under systemd socket activiation this variable will get a 10 minute > default value. This will enable a service to be activated on > connection and fall back to a shutdown state after 10 minutes of > no queries. The systemd socket activation can restart the service > on the next connection transparently. > > A global variable of server_last_activity is updated on > accepted connections (where max_idle_execution < UINT_MAX) > and when the connection count (standard or extra) is down > to <= 1 to keep the number of updates on a single variable > low (an not create another performance MDEV-21212 problem). > > When the main accept loop times out on the max_idle_execution > seconds, and then the server_last_activity is checked along > with if current connection count (standard + extra) is 0 > (in case a recently started connection hasn't finished > a query). > > To make this neater, in main accept loop moved code to > handle_new_socket_connection that encompases accepting a connection > and the timeout mechanism has been separated too. > > Changed when looping though possible connections, loop until > the end of the connection list and hereby assume two connection can > occur on the same poll/select call and both will be accepted. > > diff --git a/mysql-test/main/max_idle_execution.test b/mysql-test/main/max_idle_execution.test > new file mode 100644 > --- /dev/null > +++ b/mysql-test/main/max_idle_execution.test > @@ -0,0 +1,38 @@ > +--source include/not_embedded.inc > + > +--let $_server_id= `SELECT @@server_id` > +--let $_expect_file_name= $MYSQLTEST_VARDIR/tmp/mysqld.$_server_id.expect > +--exec echo "wait" > $_expect_file_name there's no reason to exec echo, as you can write directly --write_file $_expect_file_name wait EOF > +disable_reconnect; > +disconnect default; > + > +--echo 'allow server to time out in 10 seconds' > + > +--sleep 10 better wait_until_disconnected.inc here and everywhere below > + > +--echo 'should have timed out' > +# mtr weirdness > +--replace_regex /[\/0-9]+.*// > +--error 0,ER_SERVER_SHUTDOWN,ER_CONNECTION_KILLED,2002,2006,201 > +--connect fail_me,localhost,root you won't need all that ^^^ > + > +--exec echo "restart" > $_expect_file_name > +--sleep 5 > +--connect con0,localhost,root,,test > + > +--source include/wait_until_connected_again.inc > +SELECT VARIABLE_VALUE AS THREAD_CONNECTED FROM INFORMATION_SCHEMA.GLOBAL_STATUS WHERE VARIABLE_NAME='THREADS_CONNECTED'; > +SELECT 'we are back'; > +SELECT VARIABLE_VALUE < 10 AS UPTIME_LESS_THAN_10_SECONDS FROM INFORMATION_SCHEMA.GLOBAL_STATUS WHERE VARIABLE_NAME='UPTIME'; > + > +--sleep 13 > +SELECT 'still here because the connection is open, but disconnecting now'; > +disconnect con0; > +--sleep 5 > + > +--connect con2,127.0.0.1,root,,test,$MASTER_EXTRA_PORT, > +SELECT "extra connection just created: still here, only 5 seconds since last query"; > + > +--sleep 13 > +connect default,localhost,root,,test,,; > +SELECT 'active extra connection kept the server up'; This will be a fairly slow test. 46 seconds of waits. May be you can cut away some? E.g. second test, whether a connection keeps the server up, can reasonably be considered a subset of the third test, whether an extra connection keeps the server up. So you can remove the second test. > diff --git a/mysql-test/main/mysqld--help.result b/mysql-test/main/mysqld--help.result > --- a/mysql-test/main/mysqld--help.result > +++ b/mysql-test/main/mysqld--help.result > @@ -576,6 +576,10 @@ The following specify which files/extra groups are read (specified before remain > --max-error-count=# Max number of errors/warnings to store for a statement > --max-heap-table-size=# > Don't allow creation of heap tables bigger than this > + --max-idle-execution=# > + If no new connections or running queries within this time > + (in seconds) shutdown the server (Automatically > + configured unless set explicitly) > --max-join-size=# Joins that are probably going to read more than > max_join_size records return an error > --max-length-for-sort-data=# > diff --git a/sql/handle_connections_win.cc b/sql/handle_connections_win.cc > --- a/sql/handle_connections_win.cc > +++ b/sql/handle_connections_win.cc > @@ -593,7 +593,7 @@ void network_init_win() > } > } > > -void handle_connections_win() > +void handle_connections_win(Atomic_counter<uint> *connection_count) why do you pass it as an argument? first, you always invoke handle_connections_win with the same value of the argument, second other global variables like server_last_activity are used directly. > { > int n_waits; > > @@ -631,12 +631,26 @@ void handle_connections_win() > { > DBUG_ASSERT(wait_events.size() <= MAXIMUM_WAIT_OBJECTS); > DWORD idx = WaitForMultipleObjects((DWORD)wait_events.size(), > - wait_events.data(), FALSE, INFINITE); > + wait_events.data(), FALSE, > + max_idle_execution < UINT_MAX ? > + max_idle_execution * 1000 : INFINITE); > - DBUG_ASSERT((int)idx >= 0 && (int)idx < (int)wait_events.size()); > + DBUG_ASSERT(idx == WAIT_TIMEOUT || > + ((int)idx >= 0 && (int)idx < (int)wait_events.size())); did you test it on Windows? > > if (idx == SHUTDOWN_IDX) > break; > > + if (idx == WAIT_TIMEOUT) > + { > + if (*connection_count == 0 && > + microsecond_interval_timer() > (server_last_activity + max_idle_execution * 1000000)) > + { > + sql_print_information("max_idle_execution time reached starting shutdown"); > + break; > + } > + continue; > + } > + > all_listeners[idx - LISTENER_START_IDX]->completion_callback(); > } > > diff --git a/sql/mysqld.cc b/sql/mysqld.cc > --- a/sql/mysqld.cc > +++ b/sql/mysqld.cc > @@ -474,7 +474,8 @@ ulong malloc_calls; > ulong specialflag=0; > ulong binlog_cache_use= 0, binlog_cache_disk_use= 0; > ulong binlog_stmt_cache_use= 0, binlog_stmt_cache_disk_use= 0; > -ulong max_connections, max_connect_errors; > +ulong max_connections, max_connect_errors, max_idle_execution; > +ulonglong server_last_activity; > uint max_password_errors; > ulong extra_max_connections; > uint max_digest_length= 0; > @@ -4093,6 +4094,13 @@ static int init_common_variables() > SYSVAR_AUTOSIZE(back_log, MY_MIN(900, (50 + max_connections / 5))); > } > > + /* > + max_idle_execution, defaults to 10mins under systemd socket activation, > + otherwise 136 years or so. > + */ > + if (IS_SYSVAR_AUTOSIZE(&max_idle_execution)) > + SYSVAR_AUTOSIZE(max_idle_execution, sd_listen_fds(0) ? 6000 : UINT_MAX); is that really 10 minutes? > + > unireg_init(opt_specialflag); /* Set up extern variabels */ > if (!(my_default_lc_messages= > my_locale_by_name(lc_messages))) > @@ -6064,17 +6072,60 @@ static void set_non_blocking_if_supported(MYSQL_SOCKET sock) > } > > > +static void handle_socket_timeout() > +{ please add a comment, explaining why a race condition is impossible here, that is, why no other thread can increment connection_count or extra_connection_count just after you saw them being zeros. > + if (connection_count == 0 && extra_connection_count == 0 && > + microsecond_interval_timer() > (server_last_activity + max_idle_execution * 1000000)) * 1000000ULL ? > + { > + sql_print_information("max_idle_execution time reached starting shutdown"); > + abort_loop= 1; > + } > +} > + > + > +static void handle_new_socket_connection(MYSQL_SOCKET sock) > +{ > + struct sockaddr_storage cAddr; > + uint error_count= 0; > + > + for (uint retry= 0; retry < MAX_ACCEPT_RETRY; retry++) > + { > + size_socket length= sizeof(struct sockaddr_storage); > + MYSQL_SOCKET new_sock; > + > + new_sock= mysql_socket_accept(key_socket_client_connection, sock, > + (struct sockaddr *)(&cAddr), > + &length); > + if (mysql_socket_getfd(new_sock) != INVALID_SOCKET) > + handle_accepted_socket(new_sock, sock); > + else if (socket_errno != SOCKET_EINTR && socket_errno != SOCKET_EAGAIN) > + { > + /* > + accept(2) failed on the listening port. > + There is not much details to report about the client, > + increment the server global status variable. > + */ > + statistic_increment(connection_errors_accept, &LOCK_status); > + if ((error_count++ & 255) == 0) // This can happen often > + sql_perror("Error in accept"); > + if (socket_errno == SOCKET_ENFILE || socket_errno == SOCKET_EMFILE) > + sleep(1); // Give other threads some time > + break; > + } > + } > +} > + > + > void handle_connections_sockets() > { > - MYSQL_SOCKET sock= mysql_socket_invalid(); > + MYSQL_SOCKET sock; > - uint error_count=0; > - struct sockaddr_storage cAddr; > int retval; > #ifdef HAVE_POLL > // for ip_sock, unix_sock and extra_ip_sock > Dynamic_array<struct pollfd> fds(PSI_INSTRUMENT_MEM); > #else > fd_set readFDs,clientFDs; > + struct timespec timeout; > #endif > > DBUG_ENTER("handle_connections_sockets"); > @@ -6099,6 +6150,7 @@ void handle_connections_sockets() > } > #endif > > + server_last_activity= microsecond_interval_timer(); > sd_notify(0, "READY=1\n" > "STATUS=Taking your SQL requests now...\n"); > > @@ -6106,10 +6158,12 @@ void handle_connections_sockets() > while (!abort_loop) > { > #ifdef HAVE_POLL > - retval= poll(fds.get_pos(0), fds.size(), -1); > + /* poll timeout in milliseconds */ > + retval= poll(fds.get_pos(0), fds.size(), max_idle_execution * 1000); max value of max_idle_execution is UINT_MAX. The third argument of poll() is int. Although it seems to "work", that is, you get a negative number which poll() interprets as infinite, I seriously suspect it's a UB. > #else > + timeout= { max_idle_execution, 0}; > readFDs=clientFDs; > - retval= select(FD_SETSIZE, &readFDs, NULL, NULL, NULL); > + retval= select(FD_SETSIZE, &readFDs, NULL, NULL, &timeout); > #endif > > if (retval < 0) > @@ -6132,50 +6186,27 @@ void handle_connections_sockets() > break; > > /* Is this a new connection request ? */ > -#ifdef HAVE_POLL > - for (size_t i= 0; i < fds.size(); ++i) > + sock= mysql_socket_invalid(); > + for (size_t i= 0; i < listen_sockets.size(); i++) > { > +#ifdef HAVE_POLL > if (fds.at(i).revents & POLLIN) > - { > - sock= listen_sockets.at(i); > - break; > - } > - } > #else // HAVE_POLL > - for (size_t i=0; i < listen_sockets.size(); i++) > - { > if (FD_ISSET(mysql_socket_getfd(listen_sockets.at(i)), &readFDs)) > +#endif // HAVE_POLL > { > sock= listen_sockets.at(i); > - break; > + handle_new_socket_connection(sock); > } > } > -#endif // HAVE_POLL > - > - for (uint retry=0; retry < MAX_ACCEPT_RETRY; retry++) > + /* timeout */ > + if (mysql_socket_getfd(sock) == INVALID_SOCKET) > { > - size_socket length= sizeof(struct sockaddr_storage); > - MYSQL_SOCKET new_sock; > - > - new_sock= mysql_socket_accept(key_socket_client_connection, sock, > - (struct sockaddr *)(&cAddr), > - &length); > - if (mysql_socket_getfd(new_sock) != INVALID_SOCKET) > - handle_accepted_socket(new_sock, sock); > - else if (socket_errno != SOCKET_EINTR && socket_errno != SOCKET_EAGAIN) > - { > - /* > - accept(2) failed on the listening port. > - There is not much details to report about the client, > - increment the server global status variable. > - */ > - statistic_increment(connection_errors_accept, &LOCK_status); > - if ((error_count++ & 255) == 0) // This can happen often > - sql_perror("Error in accept"); > - if (socket_errno == SOCKET_ENFILE || socket_errno == SOCKET_EMFILE) > - sleep(1); // Give other threads some time > - break; > - } > + handle_socket_timeout(); > + } > + else if (max_idle_execution < UINT_MAX) > + { > + server_last_activity= microsecond_interval_timer(); > } > } > sd_notify(0, "STOPPING=1\n" > diff --git a/sql/sql_class.h b/sql/sql_class.h > --- a/sql/sql_class.h > +++ b/sql/sql_class.h > @@ -3949,6 +3950,9 @@ class THD: public THD_count, /* this must be first */ > set_time_for_next_stage(); > if (utime_after_query >= utime_after_lock + variables.long_query_time) > server_status|= SERVER_QUERY_WAS_SLOW; > + /* If we're tracking idle execution, and we're down to the last connection */ > + if (max_idle_execution < UINT_MAX && *scheduler->connection_count <= 1) > + server_last_activity= utime_after_query; 1. you update it concurrenty. Must be atomic, right? 2. do you really want to shutdown the server if there's an idle connection? I'd think it'd be less surprising to shutdown only when there are no connections at all. If there is a runaway idle connection, it'll die after interactive_timeout or wait_timeout. And then you shutdown the server after max_idle_execution. So instead of shutting down the server when there are connections, I'd suggest to autoset interactive_timeout and wait_timeout to be in line with max_idle_execution. Like, 10 mins, or, may be if (IS_SYSVAR_AUTOSIZE(net_wait_timeout) && net_wait_timeout > max_idle_execution) SYSVAR_AUTOSIZE(net_wait_timeout, max_idle_execution); > } > inline ulonglong found_rows(void) > { > diff --git a/sql/sys_vars.cc b/sql/sys_vars.cc > --- a/sql/sys_vars.cc > +++ b/sql/sys_vars.cc > @@ -1724,6 +1724,14 @@ Sys_max_connect_errors( > VALID_RANGE(1, UINT_MAX), DEFAULT(MAX_CONNECT_ERRORS), > BLOCK_SIZE(1)); > > +static Sys_var_ulong Sys_max_idle_execution( ulong is non portable. why not make the variable uint, if max value is UINT_MAX anyway? > + "max_idle_execution", > + "If no new connections or running queries within this time (in seconds) " > + "shutdown the server", may be something like "Default to 10 minutes in systemd socket activation setup" ? I know, it means asking for troubles :( > + AUTO_SET GLOBAL_VAR(max_idle_execution), CMD_LINE(REQUIRED_ARG), > + VALID_RANGE(10, UINT_MAX), DEFAULT(UINT_MAX), > + BLOCK_SIZE(1)); > + > static Sys_var_on_access_global<Sys_var_uint, > PRIV_SET_SYSTEM_GLOBAL_VAR_MAX_PASSWORD_ERRORS> > Sys_max_password_errors( > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] ae1f4e4d727: MDEV-23836: Assertion `! is_set() || m_can_overwrite_status' in Diagnostics_area::set_error_status (interrupted ALTER TABLE under LOCK)
by Sergei Golubchik 23 Jul '21

23 Jul '21

Hi, Rucha! On Jul 23, Rucha Deodhar wrote: > On Fri, Jul 23, 2021 at 1:55 PM Sergei Golubchik <serg(a)mariadb.org> wrote: > > > > Consider this - thd->killed can be changed literally any time. We cannot > > test for it after every second line, can we? Clearly some places are > > better for this test than others. > > > There was no specific reason except it seemed a better choice for this > particular test case. > But yes, it can also arrive before warning or later. Now I think it would > be better check it before exiting handle_if_exists_option() > and return appropriate value. Yes, you're right, it would be better. but KILL can come after handle_if_exists_options() too :) Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

[Maria-developers] MariaDB Connector/J 3.x Status and Plans
by Ondrej Dubaj 23 Jul '21

23 Jul '21

Hello, I would like to ask about the current status of MariaDB Connector/J 3.x. I maintain the package in Fedora and RHEL. Currently, we are facing a decision about which major version to pack to RHEL 9, which is currently being developed. We would like to ship RHEL 9 with the latest stable version of the Java connector, and avoid rebasing to any next major release during the RHEL 9 life, if possible. However the 3.x version is only available in ALPHA quality and I can't see any plan for it to reach further stages in the Jira roadmap for the next half a year. https://jira.mariadb.org/secure/Dashboard.jspa I believe the best solution for us would be you releasing the GA, or at least BETA version ASAP, or at least in the near future. We still have time to update the package from the ALPHA version to GA version before the RHEL 9 development freeze, but the decision whether to take this road has to be made ASAP. After RHEL 9 GA, it will be very complicated for us to deal with rebasing to a new, incompatible version of mariadb-connector-j, where this rebase can break existing user applications and brings a lot of problems with it. Currently mariadb-java-client-2.7.2 is present in the repositories, but as already said, we want to aim for the 3.x version. You, of course, might have very different plans for the connector and different idea about how ready it is to become BETA or GA. We would like to hear your plans with the Java connector 3.x version and to discuss our possible paths to take. Thank you for your time in advance. Regards, Ondrej Dubaj

2 2

Re: [Maria-developers] ae1f4e4d727: MDEV-23836: Assertion `! is_set() || m_can_overwrite_status' in Diagnostics_area::set_error_status (interrupted ALTER TABLE under LOCK)
by Sergei Golubchik 23 Jul '21

23 Jul '21

Hi, Rucha! On Jul 23, Rucha Deodhar wrote: > > > diff --git a/sql/sql_table.cc b/sql/sql_table.cc > > > index f58ce8f997d..0b722984fca 100644 > > > --- a/sql/sql_table.cc > > > +++ b/sql/sql_table.cc > > > @@ -6489,6 +6489,12 @@ handle_if_exists_options(THD *thd, TABLE *table, Alter_info *alter_info, > > > ER_BAD_FIELD_ERROR, > > > ER_THD(thd, ER_BAD_FIELD_ERROR), > > > sql_field->change.str, table->s->table_name.str); > > > + /* > > > + if thread is killed (for example because local memory used execeeds > > > + maximum session memory used), return TRUE and do rollback (done later). > > > + */ > > > + if (thd->killed) > > > + DBUG_RETURN(true); > > > it.remove(); > > > > Looks like a completely arbitrary place to put a check in. > > What if the kill signal arrives later? > > We want to stop execution of the query as soon as we get kill. In this case > it happens when we are giving out a warning. > So I put the check here to return error right after giving warning. You can get KILL anytime. Was there any reason to put this check here and not few lines earlier or later? What if KILL would arrive before the warning? or after it.remove() ? Consider this - thd->killed can be changed literally any time. We cannot test for it after every second line, can we? Clearly some places are better for this test than others. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] ae1f4e4d727: MDEV-23836: Assertion `! is_set() || m_can_overwrite_status' in Diagnostics_area::set_error_status (interrupted ALTER TABLE under LOCK)
by Sergei Golubchik 22 Jul '21

22 Jul '21

Hi, Rucha! On Jul 22, Rucha Deodhar wrote: > revision-id: ae1f4e4d727 (mariadb-10.5.11-1-gae1f4e4d727) > parent(s): dc82effa5df > author: Rucha Deodhar > committer: Rucha Deodhar > timestamp: 2021-06-21 20:24:43 +0530 > message: > > MDEV-23836: Assertion `! is_set() || m_can_overwrite_status' in > Diagnostics_area::set_error_status (interrupted ALTER TABLE under LOCK) > > Analysis: KILL_QUERY is not ignored when local memory used exceeds maximum > session memory. Hence the query proceeds, OK is sent and we end up > reopening tables that are marked for reopen. During this, kill status is > eventually checked and assertion failure happens during trying to send error > message because OK has already been sent. > Fix: If local memory used exceeds maximum session memory and query is killed > before it has completed execution, return TRUE. > diff --git a/sql/sql_table.cc b/sql/sql_table.cc > index f58ce8f997d..0b722984fca 100644 > --- a/sql/sql_table.cc > +++ b/sql/sql_table.cc > @@ -6489,6 +6489,12 @@ handle_if_exists_options(THD *thd, TABLE *table, Alter_info *alter_info, > ER_BAD_FIELD_ERROR, > ER_THD(thd, ER_BAD_FIELD_ERROR), > sql_field->change.str, table->s->table_name.str); > + /* > + if thread is killed (for example because local memory used execeeds > + maximum session memory used), return TRUE and do rollback (done later). > + */ > + if (thd->killed) > + DBUG_RETURN(true); > it.remove(); Looks like a completely arbitrary place to put a check in. What if the kill signal arrives later? Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] f5196787294: MDEV-23178: Qualified asterisk not supported in INSERT .. RETURNING
by Sergei Golubchik 22 Jul '21

22 Jul '21

Hi, Rucha! ok to push! thanks On Jul 22, Rucha Deodhar wrote: > revision-id: f5196787294 (mariadb-10.5.11-28-gf5196787294) > parent(s): b1e57179242 > author: Rucha Deodhar > committer: Rucha Deodhar > timestamp: 2021-07-22 18:54:36 +0530 > message: > > MDEV-23178: Qualified asterisk not supported in INSERT .. RETURNING > > Analysis: When we have INSERT/REPLACE returning with qualified asterisk in the > RETURNING clause, '*' is not resolved properly because of wrong context. > context->table_list is NULL or has incorrect table because context->table_list > has tables from the FROM clause. For INSERT/REPLACE...SELECT...RETURNING, > context->table_list has table we are inserting from. While in other > INSERT/REPLACE syntax, context->table_list is NULL because there is no FROM > clause. > Fix: If filling fields instead of '*' for qualified asterisk in RETURNING, > use first_name_resolution_table for correct resolution of item. > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] d91c4eea668: MDEV-23178: Qualified asterisk not supported in INSERT .. RETURNING
by Sergei Golubchik 22 Jul '21

22 Jul '21

Hi, Rucha! On Jul 22, Rucha Deodhar wrote: > revision-id: d91c4eea668 (mariadb-10.5.11-28-gd91c4eea668) > parent(s): b1e57179242 > author: Rucha Deodhar > committer: Rucha Deodhar > timestamp: 2021-07-17 18:47:15 +0530 > message: > > MDEV-23178: Qualified asterisk not supported in INSERT .. RETURNING > > Analysis: When we have INSERT/REPLACE returning with qualified asterisk in the > RETURNING clause, '*' is not resolved properly because of wrong context. > context->table_list is NULL or has incorrect table because context->table_list > has tables from the FROM clause. For INSERT/REPLACE...SELECT...RETURNING, > context->table_list has table we are inserting from. While in other > INSERT/REPLACE syntax, context->table_list is NULL because there is no FROM > clause. > Fix: If filling fields instead of '*' for qualified asterisk in RETURNING, > use first_name_resolution_table for correct resolution of item. > @@ -271,6 +289,8 @@ INSERT INTO t2(id2, val2) SELECT * FROM t1 WHERE id1=2 RETURNING(SELECT > --error ER_SUBQUERY_NO_1_ROW > INSERT INTO t2(id2,val2) SELECT * FROM t1 WHERE id1=2 RETURNING (SELECT > id2 FROM t2); > +#--error ER_BAD_TABLE_ERROR > +#INSERT INTO t2(id2,val2) SELECT t1.* FROM t1 WHERE id1=2 RETURNING t1.*; Why is this test commented out? (same for REPLACE) > @@ -8031,10 +8034,20 @@ insert_fields(THD *thd, Name_resolution_context *context, const char *db_name, > If table names are qualified, then loop over all tables used in the query, > else treat natural joins as leaves and do not iterate over their underlying > tables. > + If filling fields instead of '*' for qualified asterisk in RETURNING clause, > + use first_name_resolution_table for correct resolution of item because > + context->table_list is either NULL or has incorrect table because > + context->table_list has tables from the FROM clause. > + In case of INSERT/REPLACE...SELECT..RETURNING context->table_list will have > + table we are inserting and NULL for other cases because there is no > + FROM clause. > */ > - TABLE_LIST *first= context->first_name_resolution_table; > - TABLE_LIST *TABLE_LIST::* next= &TABLE_LIST::next_name_resolution_table; > - if (table_name) > + if (returning_field || !table_name) > + { > + first= context->first_name_resolution_table; > + next= &TABLE_LIST::next_name_resolution_table; These are the values already set before, so assignments are redundant. Either remove initializers, like TABLE_LIST *first; TABLE_LIST *TABLE_LIST::* next; if (returning_field || !table_name) or reverse your condition: TABLE_LIST *first= context->first_name_resolution_table; TABLE_LIST *TABLE_LIST::* next= &TABLE_LIST::next_name_resolution_table; if (table_name && !returning_field) > + } > + else > { > first= context->table_list; > next= &TABLE_LIST::next_local; Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 5e16a49f5d9: MDEV-26013 distinct not work properly in some cases for spider tables
by Sergei Golubchik 22 Jul '21

22 Jul '21

Hi, Nayuta! On Jul 21, Nayuta Yanagisawa wrote: > revision-id: 5e16a49f5d9 (mariadb-10.4.20-32-g5e16a49f5d9) > parent(s): 78735dcaf75 > author: Nayuta Yanagisawa > committer: Nayuta Yanagisawa > timestamp: 2021-07-21 09:58:34 +0000 > message: > > MDEV-26013 distinct not work properly in some cases for spider tables > > diff --git a/sql/sql_select.cc b/sql/sql_select.cc > index d8116b20cbe..cfd04d93662 100644 > --- a/sql/sql_select.cc > +++ b/sql/sql_select.cc > @@ -3274,9 +3274,17 @@ bool JOIN::make_aggr_tables_info() > > if (ht && ht->create_group_by) > { > - /* Check if the storage engine can intercept the query */ > - Query query= {&all_fields, select_distinct, tables_list, conds, > - group_list, order ? order : group_list, having}; > + /* > + Check if the storage engine can intercept the query. > + > + JOIN::optimize_stage2() might convert DISTINCT into GROUP BY and then > + optimize away GROUP BY. In such a case, we need to notify a storage engine > + supporting a group by handler of the existence of the original DISTINCT. > + Thus, we set select_distinct || (no_order && group_optimized_away) to > + Query::distinct. > + */ > + Query query= {&all_fields, select_distinct || (no_order && group_optimized_away), > + tables_list, conds, group_list, order ? order : group_list, having}; If DISTINCT was coverted to a GROUP BY, why would the engine need to know whether there was DISTINCT or not originally? There is no DISTINCT on the execution plan now, that should be sufficient, shouldn't it? Why does the query fail with GROUP BY? > group_by_handler *gbh= ht->create_group_by(thd, &query); > > if (gbh) Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 4

Re: [Maria-developers] 2f9bbf392072: MDEV-18734 optimization by taking ownership via String::swap()
by Sergei Golubchik 21 Jul '21

21 Jul '21

Hi, Aleksey! Don't be confused, the commit header and the comment come from your last commit, but the diff below includes all three commits that mention MDEV-18734. On Nov 02, Aleksey Midenkov wrote: > revision-id: 2f9bbf392072 (mariadb-10.2.31-543-g2f9bbf392072) > parent(s): 1f4960e3f2ac > author: Aleksey Midenkov <midenok(a)gmail.com> > committer: Aleksey Midenkov <midenok(a)gmail.com> > timestamp: 2020-11-02 14:26:04 +0300 > message: > > MDEV-18734 optimization by taking ownership via String::swap() > > ha_partition stores records in array of m_ordered_rec_buffer and uses > it for prio queue in ordered index scan. When the records are restored > from the array the blob buffers may be already freed or > rewritten. This can happen when blob buffers are cached in > Field_blob::value or Field_blob::read_value. > > Previous solution worked via copying blob buffer into > mem_root-allocated buffer. That was not optimal as it requires more > memory and processing time. > > This patch avoids memory copy by taking temporary ownership of cached > blob buffers via String::swap(). It also fixes problem when > value.str_length is wrong after Field_blob::store(). > diff --git a/mysql-test/suite/federated/federated_partition.test b/mysql-test/suite/federated/federated_partition.test > index ef1e27ec5054..33ee025442f4 100644 > --- a/mysql-test/suite/federated/federated_partition.test > +++ b/mysql-test/suite/federated/federated_partition.test > @@ -50,4 +50,30 @@ drop table federated.t1_2; > > --echo End of 5.1 tests > > +--echo # > +--echo # MDEV-18734 ASAN heap-use-after-free upon sorting by blob column from partitioned table > +--echo # > +connection slave; > +use federated; > +create table t1_1 (x int, b text, key(x)); > +create table t1_2 (x int, b text, key(x)); > +connection master; > +--replace_result $SLAVE_MYPORT SLAVE_PORT > +eval create table t1 (x int, b text, key(x)) engine=federated > + partition by range columns (x) ( > + partition p1 values less than (40) connection='mysql://root@127.0.0.1:$SLAVE_MYPORT/federated/t1_1', > + partition pn values less than (maxvalue) connection='mysql://root@127.0.0.1:$SLAVE_MYPORT/federated/t1_2' > +); > +insert t1 values (1, 1), (2, 2), (3, 3), (4, 4), (5, 5), (6, 6), (7, 7), (8, 8); > +insert t1 select x + 8, x + 8 from t1; > +insert t1 select x + 16, x + 16 from t1; > +insert t1 select x + 49, repeat(x + 49, 100) from t1; > +--echo # This produces wrong result before MDEV-17573 > +select * from t1; What do you mean, simple `select * from t1` produces incorrect result? > +flush tables; > +select x, b from t1 where x > 30 and x < 60 order by b; Is this ASAN-only test? > +drop table t1; > +connection slave; > +drop table t1_1, t1_2; > + > source include/federated_cleanup.inc; > diff --git a/mysql-test/suite/vcol/t/partition.test b/mysql-test/suite/vcol/t/partition.test > index 889724fb1c5c..f10ee0491ccc 100644 > --- a/mysql-test/suite/vcol/t/partition.test > +++ b/mysql-test/suite/vcol/t/partition.test > @@ -30,3 +30,28 @@ subpartition by hash(v) subpartitions 3 ( > insert t1 set i= 0; > set statement sql_mode= '' for update t1 set i= 1, v= 2; > drop table t1; > + > +--echo # > +--echo # MDEV-18734 ASAN heap-use-after-free in my_strnxfrm_simple_internal upon update on versioned partitioned table > +--echo # > +--echo # Cover queue_fix() in ha_partition::handle_ordered_index_scan() > +create table t1 ( > + x int auto_increment primary key, > + b text, v mediumtext as (b) virtual, > + index (v(10)) > +) partition by range columns (x) ( > + partition p1 values less than (3), > + partition p2 values less than (6), > + partition pn values less than (maxvalue)); > +insert into t1 (b) values ('q'), ('a'), ('b'); > +update t1 set b= 'bar' where v > 'a'; > +drop table t1; This test didn't fail for me in an ASAN build on the vanilla 10.5 without your fix. That is, I'm not sure it actually tests the fix. > + > +--echo # Cover return_top_record() in ha_partition::handle_ordered_index_scan() > +create table t1 (x int primary key, b tinytext, v text as (b) virtual) > +partition by range columns (x) ( > + partition p1 values less than (4), > + partition pn values less than (maxvalue)); > +insert into t1 (x, b) values (1, ''), (2, ''), (3, 'a'), (4, 'b'); > +update t1 set b= 'bar' where x > 0 order by v limit 2; This fails without a fix all right. Still, I wonder whether it's possible to create a test that'll fail in a normal optimized build w/o ASAN > +drop table t1; > diff --git a/sql/field.cc b/sql/field.cc > index bdaaecc20269..0fd40c979d2c 100644 > --- a/sql/field.cc > +++ b/sql/field.cc > @@ -8310,6 +8310,7 @@ int Field_blob::store(const char *from,uint length,CHARSET_INFO *cs) > copy_length= copier.well_formed_copy(field_charset, > (char*) value.ptr(), new_length, > cs, from, length); > + value.length(copy_length); good! Could this be a distinct bug with its own test case? > Field_blob::store_length(copy_length); > bmove(ptr+packlength,(uchar*) &tmp,sizeof(char*)); > > diff --git a/sql/ha_partition.cc b/sql/ha_partition.cc > index 9d82df0235c7..7c658082d397 100644 > --- a/sql/ha_partition.cc > +++ b/sql/ha_partition.cc > @@ -5125,14 +5128,14 @@ bool ha_partition::init_record_priority_queue() > uint alloc_len; > uint used_parts= bitmap_bits_set(&m_part_info->read_partitions); > /* Allocate record buffer for each used partition. */ > - m_priority_queue_rec_len= m_rec_length + PARTITION_BYTES_IN_POS; > + m_priority_queue_rec_len= m_rec_length + ORDERED_REC_OFFSET; > if (!m_using_extended_keys) > m_priority_queue_rec_len += m_file[0]->ref_length; > alloc_len= used_parts * m_priority_queue_rec_len; > /* Allocate a key for temporary use when setting up the scan. */ > alloc_len+= table_share->max_key_length; > > - if (!(m_ordered_rec_buffer= (uchar*)my_malloc(alloc_len, MYF(MY_WME)))) > + if (!(m_ordered_rec_buffer= (uchar*) alloc_root(&m_ordered_root, alloc_len))) I don't see why you need a memroot here. memroot is needed when you allocate later at difeerent points in time some initially unpredictable amount of memory that has the same lifetime and needs to be freed all at once. But here it seems that you only allocate once. You could just make m_priority_queue_rec_len = m_rec_length + PARTITION_BYTES_IN_POS + table->s->blob_fields * sizeof(Ordered_blob_storage) (plus necessary alignment). > DBUG_RETURN(true); > > /* > @@ -6178,7 +6203,11 @@ int ha_partition::handle_ordered_index_scan(uchar *buf, bool reverse_order) > */ > error= file->read_range_first(m_start_key.key? &m_start_key: NULL, > end_range, eq_range, TRUE); > - memcpy(rec_buf_ptr, table->record[0], m_rec_length); > + if (!error) > + { > + memcpy(rec_buf_ptr, table->record[0], m_rec_length); > + } did you have a bug because of that? something didn't work? > + > reverse_order= FALSE; > break; > } > @@ -6310,6 +6349,43 @@ int ha_partition::handle_ordered_index_scan_key_not_found() > } > > > +void ha_partition::swap_blobs(uchar * rec_buf, Ordered_blob_storage ** storage, bool restore) > +{ > + uint *ptr, *end; > + uint blob_n= 0; > + table->move_fields(table->field, rec_buf, table->record[0]); > + for (ptr= table->s->blob_field, end= ptr + table->s->blob_fields; > + ptr != end; ++ptr, ++blob_n) > + { > + DBUG_ASSERT(*ptr < table->s->fields); > + Field_blob *blob= (Field_blob*) table->field[*ptr]; > + DBUG_ASSERT(blob->flags & BLOB_FLAG); > + DBUG_ASSERT(blob->field_index == *ptr); > + if (!bitmap_is_set(table->read_set, *ptr) || blob->is_null()) > + continue; > + > + Ordered_blob_storage &s= *storage[blob_n]; > + > + if (restore) > + { > + if (!s.blob.is_empty()) > + blob->swap(s.blob, s.set_read_value); looks somewhat risky, why blob.length == 0 means that you don't need to restore the value? > + } > + else > + { > + bool set_read_value; > + String *cached= blob->cached(set_read_value); > + if (cached) > + { > + cached->swap(s.blob); > + s.set_read_value= set_read_value; is it indeed possible here for a value to be either in `value` or in `read_value` ? When happens what? > + } > + } > + } > + table->move_fields(table->field, table->record[0], rec_buf); > +} > + > + > /* > Common routine to handle index_next with ordered results > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 11

Re: [Maria-developers] 5d19829df5f: MDEV-25908: -e does not work for my_print_defaults
by Sergei Golubchik 21 Jul '21

21 Jul '21

Hi, Rucha! Here the test is ok, buildbot is green. But you've only removed -e. What about -c, -n, -g ? On Jul 21, Rucha Deodhar wrote: > revision-id: 5d19829df5f (mariadb-10.6.1-39-g5d19829df5f) > parent(s): 2c4d1fb5440 > author: Rucha Deodhar > committer: Rucha Deodhar > timestamp: 2021-07-15 13:29:25 +0530 > message: > > MDEV-25908: -e does not work for my_print_defaults > > Analysis: -e is very inconsistent with how --default* options work in any other > executable. > Fix: Removed the support for -e starting from 10.7 to avoid inconsistency in the > long term. > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 61dcf8d940c: MDEV-25908: -e does not work for my_print_defaults
by Sergei Golubchik 21 Jul '21

21 Jul '21

Hi, Rucha! On Jul 21, Rucha Deodhar wrote: > revision-id: 61dcf8d940c (mariadb-10.5.11-35-g61dcf8d940c) > parent(s): f88d130e710 > author: Rucha Deodhar > committer: Rucha Deodhar > timestamp: 2021-07-15 16:07:18 +0530 > message: > > MDEV-25908: -e does not work for my_print_defaults > > Analysis: get_defaults_options() rewrites the value for my_defaults_file, > my_defaults_extra_file and my_defaults_group_suffix to 0. So the config file > can't be read. > Fix: Let handle_options() handle --defaults* option and create a new command > line to pass it to load_defaults(). The code change looks ok. But I thought we've seen from your previous push that this test doesn't work. Haven't we? See how it failed all over in buildbot: https://buildbot.askmonty.org/buildbot/grid?category=main&branch=bb-10.5-MD… Please, generally, first check that your commit passes buildbot, then ask for a review. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] cf6d83e7d68: MDEV-24760 SELECT..CASE statement syntax error at Spider Engine table
by Sergei Golubchik 21 Jul '21

21 Jul '21

Hi, Nayuta! Great, ok to push! On Jul 21, Nayuta Yanagisawa wrote: > revision-id: cf6d83e7d68 (mariadb-10.3.30-28-gcf6d83e7d68) > parent(s): efa311ab8ea > author: Nayuta Yanagisawa > committer: Nayuta Yanagisawa > timestamp: 2021-07-19 15:16:25 +0000 > message: > > MDEV-24760 SELECT..CASE statement syntax error at Spider Engine table > > The root cause of the bug is in `spider_db_mbase_util::open_item_func()`. > The function handles an instance of the `Item_func` class based on its > `Item_func::Functype`. > The `Functype` of `CASE WHEN ... THEN` is `CASE_SEARCHED_FUNC`. > However, the Spider SE doesn't recognize this `Functype` because > `CASE_SEARCHED_FUNC` is newly added by 4de0d92. This results in the wrong > handling of `CASE WHEN ... THEN`. > > The above also applies to `CASE_SIMPLE_FUNC`. > > --- /dev/null > +++ b/storage/spider/mysql-test/spider/bugfix/r/mdev_24760.result ... > +) ENGINE=Spider DEFAULT CHARSET=utf8 COMMENT='table "tbl_a"' > +PARTITION BY HASH(id) ( > +PARTITION pt1 COMMENT='srv "s_2_1"' > +); > +SELECT id, CASE WHEN name='Alice' THEN "A" WHEN name='Bob' THEN "B" END FROM tbl_a; > +id CASE WHEN name='Alice' THEN "A" WHEN name='Bob' THEN "B" END > +1 A > +2 B > +SELECT id, CASE name WHEN 'Alice' THEN "A" WHEN 'Bob' THEN "B" END FROM tbl_a; > +id CASE name WHEN 'Alice' THEN "A" WHEN 'Bob' THEN "B" END > +1 A > +2 B a _very_ slight preference - could you use different names for local (spider) and remote tables? That is, by all means, please, follow the conventions that are used in existing spider tests. But if it's possible to use different tables names for remote and local without violating spider test conventions, I think it'd make tests easier to read. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 38a888da0f1: MDEV-17554 Auto-create new partition for system versioned tables with history partitioned by INTERVAL/LIMIT
by Sergei Golubchik 20 Jul '21

20 Jul '21

Hi, Aleksey! Like last time, it's a review of the diff bc04ded2353 22158b3c05a, so not only commit 38a888da0f1. The main thing below - I didn't understand a couple of changes and asked for clarifications. On May 30, Aleksey Midenkov wrote: > revision-id: 38a888da0f1 (mariadb-10.5.2-653-g38a888da0f1) > parent(s): bc04ded2353 > author: Aleksey Midenkov <midenok(a)gmail.com> > committer: Aleksey Midenkov <midenok(a)gmail.com> > timestamp: 2021-04-22 23:35:06 +0300 > message: > > MDEV-17554 Auto-create new partition for system versioned tables with history partitioned by INTERVAL/LIMIT > diff --git a/mysql-test/suite/versioning/r/delete_history.result b/mysql-test/suite/versioning/r/delete_history.result > --- a/mysql-test/suite/versioning/r/delete_history.result > +++ b/mysql-test/suite/versioning/r/delete_history.result > @@ -154,3 +152,21 @@ select * from t1; > a > 1 > drop table t1; > +# > +# MDEV-17554 Auto-create new partition for system versioned tables with history partitioned by INTERVAL/LIMIT > +# > +# Don't auto-create new partition on DELETE HISTORY: > +set timestamp= unix_timestamp('2000-01-01 00:00:00'); > +create or replace table t (a int) with system versioning > +partition by system_time interval 1 hour auto; > +set timestamp= unix_timestamp('2000-01-01 10:00:00'); perhaps, make it only 02:00:00 to avoid any chance that a partition won't be created because of max-auto-create limit. it's possible we put the limit at 10, but 2 is very unlikely. or does max-auto-create limit always result in a warning? > +delete history from t; > +set timestamp= default; > +show create table t; > +Table Create Table > +t CREATE TABLE `t` ( > + `a` int(11) DEFAULT NULL > +) ENGINE=DEFAULT_ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING > + PARTITION BY SYSTEM_TIME INTERVAL 1 HOUR STARTS TIMESTAMP'2000-01-01 00:00:00' AUTO > +PARTITIONS 2 > +drop table t; > diff --git a/mysql-test/suite/versioning/r/partition.result b/mysql-test/suite/versioning/r/partition.result > --- a/mysql-test/suite/versioning/r/partition.result > +++ b/mysql-test/suite/versioning/r/partition.result > @@ -1236,27 +1270,826 @@ t1 CREATE TABLE `t1` ( > PARTITION `p5` HISTORY ENGINE = DEFAULT_ENGINE, > PARTITION `pn` CURRENT ENGINE = DEFAULT_ENGINE) > alter table t1 add partition partitions 8; > +drop table t1; > +# > +# End of 10.5 tests > +# > +# > +# MDEV-17554 Auto-create new partition for system versioned tables with history partitioned by INTERVAL/LIMIT > +# > +create or replace table t1 (x int) with system versioning > +partition by system_time limit 1 auto; > show create table t1; > Table Create Table > t1 CREATE TABLE `t1` ( > `x` int(11) DEFAULT NULL > ) ENGINE=DEFAULT_ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING > - PARTITION BY SYSTEM_TIME LIMIT 1 > -(PARTITION `p2` HISTORY ENGINE = DEFAULT_ENGINE, > - PARTITION `p8` HISTORY ENGINE = DEFAULT_ENGINE, > + PARTITION BY SYSTEM_TIME LIMIT 1 AUTO > +PARTITIONS 2 > +insert into t1 values (1); > +create or replace table t2 (y int); > +insert into t2 values (2); > +insert into t1 select * from t2; > +insert into t2 select * from t1; > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `x` int(11) DEFAULT NULL > +) ENGINE=DEFAULT_ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING > + PARTITION BY SYSTEM_TIME LIMIT 1 AUTO > +PARTITIONS 2 > +# Too many partitions error > +set timestamp= unix_timestamp('2000-01-01 00:00:00'); > +create or replace table t1 (x int) with system versioning > +partition by system_time interval 1 hour auto; > +set timestamp= unix_timestamp('2001-01-01 00:01:00'); > +update t1 set x= x + 1; > +ERROR HY000: Too many partitions (including subpartitions) were defined > +# Partition overflow error and manual fix seems to be a wrong comment? no manual fix here. > +set timestamp= unix_timestamp('2000-01-01 00:00:00'); > +create or replace table t1 (x int) with system versioning > +partition by system_time interval 1 hour; > +insert into t1 values (333); > +set timestamp= unix_timestamp('2000-01-01 01:00:00'); > +update t1 set x= x + 1; > +Warnings: > +Warning 4114 Versioned table `test`.`t1`: last HISTORY partition (`p0`) is out of INTERVAL, need more HISTORY partitions > +drop table t1; > +# Partition overflow error and manual fix > +set timestamp= unix_timestamp('2000-01-01 00:00:00'); > +create or replace table t1 (x int) with system versioning > +partition by system_time interval 1 hour; > +insert into t1 values (440); > +set timestamp= unix_timestamp('2000-01-01 00:10:00'); > +update t1 set x= x + 1; > +# Check how pruning boundaries work > +explain partitions select * from t1 for system_time as of '2000-01-01 00:59:58'; > +id select_type table partitions type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 p0,pn # NULL NULL NULL NULL # # > +explain partitions select * from t1 for system_time as of '2000-01-01 00:59:59'; > +id select_type table partitions type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 pn # NULL NULL NULL NULL # # Why the first select looks in both partitions and the second one - only in pn? > +explain partitions select * from t1 for system_time as of '2000-01-01 01:00:00'; > +id select_type table partitions type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 pn # NULL NULL NULL NULL # # > +select * from t1 for system_time as of '2000-01-01 00:09:59'; > +x > +440 > +set timestamp= unix_timestamp('2000-01-01 02:00:00'); > +update t1 set x= x + 1; > +Warnings: > +Warning 4114 Versioned table `test`.`t1`: last HISTORY partition (`p0`) is out of INTERVAL, need more HISTORY partitions > +select * from t1 for system_time as of '2000-01-01 01:00:00'; > +x > +select * from t1 partition (p0) order by x; > +x > +440 > +441 > +alter table t1 add partition partitions 3; > +select * from t1 for system_time as of '2000-01-01 01:00:00'; > +x > +441 > +select * from t1 partition (p0) order by x; > +x > +440 would be good to have EXPLAINs after the overflow and after the manual fix. to show how ALTER changes what partition will be pruned > +drop table t1; > +create or replace table t1 (x int) with system versioning > +partition by system_time interval 3600 second > +starts '2000-01-01 00:00:00' auto partitions 3; > +affected rows: 0 > +insert into t1 values (1); > +affected rows: 1 > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `x` int(11) DEFAULT NULL > +) ENGINE=DEFAULT_ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING > + PARTITION BY SYSTEM_TIME INTERVAL 3600 SECOND STARTS TIMESTAMP'2000-01-01 00:00:00' AUTO > +PARTITIONS 3 > ... > diff --git a/mysql-test/suite/versioning/r/rpl_mix.result b/mysql-test/suite/versioning/r/rpl_mix.result > --- a/mysql-test/suite/versioning/r/rpl_mix.result > +++ b/mysql-test/suite/versioning/r/rpl_mix.result > @@ -8,4 +8,59 @@ DELETE HISTORY FROM t1; > connection slave; > connection master; > drop table t1; > +# > +# MDEV-25347 DML events for auto-partitioned tables are written into binary log twice > +# > +flush binary logs; > +create table t1 (a int) with system versioning > +partition by system_time limit 1 auto; > +insert into t1 values (1); > +update t1 set a= a + 1; > +update t1 set a= a + 1; better do a+2 the second time. otherwise it's a bit confusing, the bug says "written twice" and you indeed have update a=a+1 written twice :) with clearly different statements the test will prove that every event is written once, and not one is skipped and the other is duplicated > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` int(11) DEFAULT NULL > +) ENGINE=ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING > + PARTITION BY SYSTEM_TIME LIMIT 1 AUTO > +PARTITIONS 3 > +select * from t1; > +a > +3 > +include/show_binlog_events.inc > +Log_name Pos Event_type Server_id End_log_pos Info > +master-bin.000002 # Binlog_checkpoint # # master-bin.000002 > +master-bin.000002 # Gtid # # GTID #-#-# > +master-bin.000002 # Query # # use `test`; create table t1 (a int) with system versioning > +partition by system_time limit 1 auto > +master-bin.000002 # Gtid # # BEGIN GTID #-#-# > +master-bin.000002 # Annotate_rows # # insert into t1 values (1) > +master-bin.000002 # Table_map # # table_id: # (test.t1) > +master-bin.000002 # Write_rows_v1 # # table_id: # flags: STMT_END_F > +master-bin.000002 # Query # # COMMIT > +master-bin.000002 # Gtid # # BEGIN GTID #-#-# > +master-bin.000002 # Annotate_rows # # update t1 set a= a + 1 > +master-bin.000002 # Table_map # # table_id: # (test.t1) > +master-bin.000002 # Update_rows_v1 # # table_id: # > +master-bin.000002 # Write_rows_v1 # # table_id: # flags: STMT_END_F > +master-bin.000002 # Query # # COMMIT > +master-bin.000002 # Gtid # # BEGIN GTID #-#-# > +master-bin.000002 # Annotate_rows # # update t1 set a= a + 1 > +master-bin.000002 # Table_map # # table_id: # (test.t1) > +master-bin.000002 # Update_rows_v1 # # table_id: # > +master-bin.000002 # Write_rows_v1 # # table_id: # flags: STMT_END_F > +master-bin.000002 # Query # # COMMIT > +connection slave; > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` int(11) DEFAULT NULL > +) ENGINE=ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING > + PARTITION BY SYSTEM_TIME LIMIT 1 AUTO > +PARTITIONS 3 > +select * from t1; > +a > +3 > +connection master; > +drop table t1; > include/rpl_end.inc > diff --git a/mysql-test/suite/versioning/r/rpl_row.result b/mysql-test/suite/versioning/r/rpl_row.result > --- a/mysql-test/suite/versioning/r/rpl_row.result > +++ b/mysql-test/suite/versioning/r/rpl_row.result > @@ -11,4 +11,59 @@ connection slave; > connection master; > drop table t1; > set binlog_row_image= @old_row_image; > +# > +# MDEV-25347 DML events for auto-partitioned tables are written into binary log twice > +# > +flush binary logs; > +create table t1 (a int) with system versioning > +partition by system_time limit 1 auto; > +insert into t1 values (1); > +update t1 set a= a + 1; > +update t1 set a= a + 1; same. as a general rule it's helpful to use slightly different statement, to be able to map them to binlog events uniquely. a+2, or 1+a, or whatever. > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` int(11) DEFAULT NULL > +) ENGINE=ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING > + PARTITION BY SYSTEM_TIME LIMIT 1 AUTO > +PARTITIONS 3 > +select * from t1; > +a > +3 > +include/show_binlog_events.inc > +Log_name Pos Event_type Server_id End_log_pos Info > +master-bin.000002 # Binlog_checkpoint # # master-bin.000002 > +master-bin.000002 # Gtid # # GTID #-#-# > +master-bin.000002 # Query # # use `test`; create table t1 (a int) with system versioning > +partition by system_time limit 1 auto > +master-bin.000002 # Gtid # # BEGIN GTID #-#-# > +master-bin.000002 # Annotate_rows # # insert into t1 values (1) > +master-bin.000002 # Table_map # # table_id: # (test.t1) > +master-bin.000002 # Write_rows_v1 # # table_id: # flags: STMT_END_F > +master-bin.000002 # Query # # COMMIT > +master-bin.000002 # Gtid # # BEGIN GTID #-#-# > +master-bin.000002 # Annotate_rows # # update t1 set a= a + 1 > +master-bin.000002 # Table_map # # table_id: # (test.t1) > +master-bin.000002 # Update_rows_v1 # # table_id: # > +master-bin.000002 # Write_rows_v1 # # table_id: # flags: STMT_END_F > +master-bin.000002 # Query # # COMMIT > +master-bin.000002 # Gtid # # BEGIN GTID #-#-# > +master-bin.000002 # Annotate_rows # # update t1 set a= a + 1 > +master-bin.000002 # Table_map # # table_id: # (test.t1) > +master-bin.000002 # Update_rows_v1 # # table_id: # > +master-bin.000002 # Write_rows_v1 # # table_id: # flags: STMT_END_F > +master-bin.000002 # Query # # COMMIT > +connection slave; > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` int(11) DEFAULT NULL > +) ENGINE=ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING > + PARTITION BY SYSTEM_TIME LIMIT 1 AUTO > +PARTITIONS 3 > +select * from t1; > +a > +3 > +connection master; > +drop table t1; > include/rpl_end.inc > diff --git a/mysql-test/suite/versioning/t/partition.test b/mysql-test/suite/versioning/t/partition.test > --- a/mysql-test/suite/versioning/t/partition.test > +++ b/mysql-test/suite/versioning/t/partition.test > @@ -1068,13 +1078,456 @@ alter table t1 add partition partitions 2; > --replace_result $default_engine DEFAULT_ENGINE > show create table t1; > alter table t1 add partition partitions 8; > +drop table t1; > + > +--echo # > +--echo # End of 10.5 tests > +--echo # > + > +--echo # > +--echo # MDEV-17554 Auto-create new partition for system versioned tables with history partitioned by INTERVAL/LIMIT > +--echo # > +create or replace table t1 (x int) with system versioning > +partition by system_time limit 1 auto; > +--replace_result $default_engine DEFAULT_ENGINE > +show create table t1; > + in the previous version of this patch you had clarifying comments here, like --echo # INSERT, INSERT .. SELECT don't auto-create partitions or --echo # Number of partitions goes from 3 to 5 I kind of miss them now, they were helpful > +insert into t1 values (1); > + > +create or replace table t2 (y int); > +insert into t2 values (2); > + > +insert into t1 select * from t2; > +insert into t2 select * from t1; > +--replace_result $default_engine DEFAULT_ENGINE > +show create table t1; > + > ... > diff --git a/sql/ha_partition.h b/sql/ha_partition.h > --- a/sql/ha_partition.h > +++ b/sql/ha_partition.h > @@ -1607,7 +1607,7 @@ class ha_partition :public handler > for (; part_id < part_id_end; ++part_id) > { > handler *file= m_file[part_id]; > - DBUG_ASSERT(bitmap_is_set(&(m_part_info->read_partitions), part_id)); > + bitmap_set_bit(&(m_part_info->read_partitions), part_id); would be good to have a comment before the method that it's only used in vers_set_hist_part(), if (vers_info->limit). because if it's a generic method to return the number of rows in a partition (incl. subpartitions) then it's totally not clear why it modifies read_partitions bitmap. or may be you'd want to rename the method to make it look more specific for LIMIT partitions in vers_set_hist_part() ? > file->info(HA_STATUS_VARIABLE | HA_STATUS_NO_LOCK | HA_STATUS_OPEN); > part_recs+= file->stats.records; > } > diff --git a/sql/handler.cc b/sql/handler.cc > --- a/sql/handler.cc > +++ b/sql/handler.cc > @@ -1574,6 +1574,8 @@ int ha_commit_trans(THD *thd, bool all) > DBUG_ASSERT(thd->transaction->stmt.ha_list == NULL || > trans == &thd->transaction->stmt); > > + DBUG_ASSERT(!thd->in_sub_stmt); > if (thd->in_sub_stmt) > { > DBUG_ASSERT(0); This is a bit redundant, don't you think? One DBUG_ASSERT would be enough. (personally, I'd prefer the one you added, not DBUG_ASSERT(0)) > diff --git a/sql/lock.cc b/sql/lock.cc > --- a/sql/lock.cc > +++ b/sql/lock.cc > @@ -662,16 +662,28 @@ MYSQL_LOCK *mysql_lock_merge(MYSQL_LOCK *a,MYSQL_LOCK *b) > DBUG_PRINT("enter", ("a->lock_count: %u b->lock_count: %u", > a->lock_count, b->lock_count)); > > - if (!(sql_lock= (MYSQL_LOCK*) > - my_malloc(key_memory_MYSQL_LOCK, sizeof(*sql_lock) + > - sizeof(THR_LOCK_DATA*)*((a->lock_count+b->lock_count)*2) + > - sizeof(TABLE*)*(a->table_count+b->table_count),MYF(MY_WME)))) > - DBUG_RETURN(0); // Fatal error > + const size_t lock_size= sizeof(*sql_lock) + > + sizeof(THR_LOCK_DATA *) * ((a->lock_count + b->lock_count) * 2) + > + sizeof(TABLE *) * (a->table_count + b->table_count); > + if (thd) > + { > + sql_lock= (MYSQL_LOCK *) thd->alloc(lock_size); > + if (!sql_lock) > + DBUG_RETURN(0); > + sql_lock->flags= GET_LOCK_ON_THD; why? the commit comment for MDEV-23639 doesn't mention it. generally, I think it's a good idea and, likely, should be used more than just in recover_from_failed_open(). But why right now and in this commit? Does this commit (22158b3c) rely on locks being on THD memroot? How so? > + } > + else > + { > + sql_lock= (MYSQL_LOCK *) > + my_malloc(key_memory_MYSQL_LOCK, lock_size, MYF(MY_WME)); > + if (!sql_lock) > + DBUG_RETURN(0); > + sql_lock->flags= 0; > + } > sql_lock->lock_count=a->lock_count+b->lock_count; > sql_lock->table_count=a->table_count+b->table_count; > sql_lock->locks=(THR_LOCK_DATA**) (sql_lock+1); > sql_lock->table=(TABLE**) (sql_lock->locks+sql_lock->lock_count*2); > - sql_lock->flags= 0; > memcpy(sql_lock->locks,a->locks,a->lock_count*sizeof(*a->locks)); > memcpy(sql_lock->locks+a->lock_count,b->locks, > b->lock_count*sizeof(*b->locks)); > @@ -705,8 +717,10 @@ MYSQL_LOCK *mysql_lock_merge(MYSQL_LOCK *a,MYSQL_LOCK *b) > a->lock_count, b->lock_count); > > /* Delete old, not needed locks */ > - my_free(a); > - my_free(b); > + if (!(a->flags & GET_LOCK_ON_THD)) > + my_free(a); > + if (!(b->flags & GET_LOCK_ON_THD)) > + my_free(b); this even looks like a bug, no check that old locks weren't on THD memroot. (probably impossible in this particular case, but still rather fragile) thanks! > DBUG_RETURN(sql_lock); > } > > diff --git a/sql/share/errmsg-utf8.txt b/sql/share/errmsg-utf8.txt > --- a/sql/share/errmsg-utf8.txt > +++ b/sql/share/errmsg-utf8.txt > @@ -7985,3 +7985,5 @@ ER_JSON_TABLE_MULTIPLE_MATCHES > eng "Can't store multiple matches of the path in the column '%s' of JSON_TABLE '%s'." > ER_WITH_TIES_NEEDS_ORDER > eng "FETCH ... WITH TIES requires ORDER BY clause to be present" > +ER_VERS_HIST_PART_FAILED > + eng "Versioned table %`s.%`s: adding HISTORY partition failed" Can you come up with a test for this error? > diff --git a/sql/sql_base.cc b/sql/sql_base.cc > --- a/sql/sql_base.cc > +++ b/sql/sql_base.cc > @@ -931,7 +934,7 @@ void close_thread_table(THD *thd, TABLE **table_ptr) > DBUG_PRINT("tcache", ("table: '%s'.'%s' %p", table->s->db.str, > table->s->table_name.str, table)); > DBUG_ASSERT(!table->file->keyread_enabled()); > - DBUG_ASSERT(!table->file || table->file->inited == handler::NONE); > + DBUG_ASSERT(table->file->inited == handler::NONE); indeed > > /* > The metadata lock must be released after giving back > @@ -1625,6 +1625,125 @@ bool is_locked_view(THD *thd, TABLE_LIST *t) > } > > > +#ifdef WITH_PARTITION_STORAGE_ENGINE > +/** > + Switch part_info->hist_part and request partition creation if needed. > + > + @retval true Error or partition creation was requested. > + @retval false No error > +*/ > +bool TABLE::vers_switch_partition(THD *thd, TABLE_LIST *table_list, > + Open_table_context *ot_ctx) > +{ > + if (!part_info || part_info->part_type != VERSIONING_PARTITION || > + table_list->vers_conditions.delete_history || > + thd->stmt_arena->is_stmt_prepare() || > + table_list->lock_type < TL_WRITE_ALLOW_WRITE || > + table_list->mdl_request.type < MDL_SHARED_WRITE || > + table_list->mdl_request.type == MDL_EXCLUSIVE) > + { > + return false; > + } > + > + switch (thd->lex->sql_command) > + { > + case SQLCOM_INSERT: > + if (thd->lex->duplicates != DUP_UPDATE) > + return false; > + break; > + case SQLCOM_LOAD: > + if (thd->lex->duplicates != DUP_REPLACE) > + return false; > + break; > + case SQLCOM_LOCK_TABLES: > + case SQLCOM_DELETE: > + case SQLCOM_UPDATE: > + case SQLCOM_REPLACE: > + case SQLCOM_REPLACE_SELECT: > + case SQLCOM_DELETE_MULTI: > + case SQLCOM_UPDATE_MULTI: > + break; > + default: > + /* > + TODO: make row events set thd->lex->sql_command appropriately. > + > + Sergei Golubchik: f.ex. currently row events increment > + thd->status_var.com_stat[] each event for its own SQLCOM_xxx, it won't be > + needed if they'll just set thd->lex->sql_command. > + */ > + if (thd->rgi_slave && thd->rgi_slave->current_event && > + thd->lex->sql_command == SQLCOM_END) > + { > + switch (thd->rgi_slave->current_event->get_type_code()) > + { > + case UPDATE_ROWS_EVENT: > + case UPDATE_ROWS_EVENT_V1: > + case DELETE_ROWS_EVENT: > + case DELETE_ROWS_EVENT_V1: > + break; > + default:; > + return false; > + } > + } > + break; > + } > + > + TABLE *table= this; > + > + /* > + NOTE: The semantics of vers_set_hist_part() is double: even when we twofold > + don't need auto-create, we need to update part_info->hist_part. > + */ > + uint *create_count= table_list->vers_skip_create ? > + NULL : &ot_ctx->vers_create_count; > + table_list->vers_skip_create= true; > + if (table->part_info->vers_set_hist_part(thd, create_count)) > + { > + MYSQL_UNBIND_TABLE(table->file); > + tc_release_table(table); > + return true; > + } > + if (ot_ctx->vers_create_count) > + { > + Open_table_context::enum_open_table_action action; > + TABLE_LIST *table_arg; > + mysql_mutex_lock(&table->s->LOCK_share); > + if (!table->s->vers_skip_auto_create) > + { > + table->s->vers_skip_auto_create= true; > + action= Open_table_context::OT_ADD_HISTORY_PARTITION; > + table_arg= table_list; > + } > + else > + { > + /* > + NOTE: this may repeat multiple times until creating thread acquires > + MDL_EXCLUSIVE. Since auto-creation is rare operation this is acceptable. > + We could suspend this thread on cond-var but we must first exit > + MDL_SHARED_WRITE first and we cannot store cond-var into TABLE_SHARE > + because it is already released and there is no guarantee that it will > + be same instance if we acquire it again. > + */ > + table_list->vers_skip_create= false; > + ot_ctx->vers_create_count= 0; > + action= Open_table_context::OT_REOPEN_TABLES; > + table_arg= NULL; > + } I'm afraid I don't understand. All this business with vers_skip_create and vers_skip_auto_create, it wasn't in the previous version of the patch. So, I believe it was a fix for one of the MDEV bugs reported and fixed meanwhile. What MDEV was it? Could you explain what was the issue and what was the fix, please? > + mysql_mutex_unlock(&table->s->LOCK_share); > + if (!thd->locked_tables_mode) > + { > + MYSQL_UNBIND_TABLE(table->file); > + tc_release_table(table); > + } > + ot_ctx->request_backoff_action(action, table_arg); > + return true; > + } > + > + return false; > +} > +#endif /* WITH_PARTITION_STORAGE_ENGINE */ > + > + > /** > Open a base table. > > @@ -2572,11 +2699,13 @@ void Locked_tables_list::mark_table_for_reopen(THD *thd, TABLE *table) > table_list; table_list= table_list->next_global) > { > if (table_list->table->s == share) > + { > table_list->table->internal_set_needs_reopen(true); > + some_table_marked_for_reopen= 1; > + } > } > /* This is needed in the case where lock tables where not used */ > table->internal_set_needs_reopen(true); why ^^^ this doesn't count as "some_table_marked_for_reopen" ? > - some_table_marked_for_reopen= 1; > } > > > @@ -3172,13 +3304,49 @@ Open_table_context::recover_from_failed_open() > break; > case OT_DISCOVER: > case OT_REPAIR: > - if ((result= lock_table_names(m_thd, m_thd->lex->create_info, > - m_failed_table, NULL, > - get_timeout(), 0))) > + case OT_ADD_HISTORY_PARTITION: > + if (!m_thd->locked_tables_mode) > + result= lock_table_names(m_thd, m_thd->lex->create_info, m_failed_table, > + NULL, get_timeout(), 0); > + else > + { > + DBUG_ASSERT(!result); > + DBUG_ASSERT(m_action == OT_ADD_HISTORY_PARTITION); It seems you've solved recover_from_failed_open under LOCK TABLES, so it could be possible to make OT_DISCOVER and OT_REPAIR to work there too. Is that right? It's just a question, I don't imply it should be part of this MDEV. > + } > + /* > + We are now under MDL_EXCLUSIVE mode. Other threads have no table share > + acquired: they are blocked either at open_table_get_mdl_lock() in > + open_table() or at lock_table_names() here. > + */ > + if (result) > + { > + if (m_action == OT_ADD_HISTORY_PARTITION) > + { > + TABLE_SHARE *share= tdc_acquire_share(m_thd, m_failed_table, > + GTS_TABLE, NULL); > + if (share) > + { > + share->vers_skip_auto_create= false; > + tdc_release_share(share); > + } > + if (m_thd->get_stmt_da()->sql_errno() == ER_LOCK_WAIT_TIMEOUT) > + { > + // MDEV-23642 Locking timeout caused by auto-creation affects original DML > + m_thd->clear_error(); > + vers_create_count= 0; > + result= false; > + } > + } > break; > + } > > - tdc_remove_table(m_thd, m_failed_table->db.str, > - m_failed_table->table_name.str); > + /* > + We don't need to remove share under OT_ADD_HISTORY_PARTITION. > + Moreover fast_alter_partition_table() works with TABLE instance. > + */ > + if (m_action != OT_ADD_HISTORY_PARTITION) > + tdc_remove_table(m_thd, m_failed_table->db.str, > + m_failed_table->table_name.str); > > switch (m_action) > { > @@ -3206,6 +3374,70 @@ Open_table_context::recover_from_failed_open() > case OT_REPAIR: > result= auto_repair_table(m_thd, m_failed_table); > break; > + case OT_ADD_HISTORY_PARTITION: > + { > + result= false; > + TABLE *table= open_ltable(m_thd, m_failed_table, TL_WRITE, > + MYSQL_OPEN_HAS_MDL_LOCK | MYSQL_OPEN_IGNORE_LOGGING_FORMAT); > + if (table == NULL) > + { > + m_thd->clear_error(); > + break; > + } > + > + DBUG_ASSERT(vers_create_count); > + result= vers_create_partitions(m_thd, m_failed_table, vers_create_count); > + vers_create_count= 0; > + if (!m_thd->transaction->stmt.is_empty()) > + trans_commit_stmt(m_thd); > + DBUG_ASSERT(!result || > + !m_thd->locked_tables_mode || > + m_thd->lock->lock_count); > + if (result) > + break; > + if (!m_thd->locked_tables_mode) > + { > + /* > + alter_partition_lock_handling() does mysql_lock_remove() but > + does not clear thd->lock completely. > + */ > + DBUG_ASSERT(m_thd->lock->lock_count == 0); > + if (!(m_thd->lock->flags & GET_LOCK_ON_THD)) > + my_free(m_thd->lock); > + m_thd->lock= NULL; > + } > + else if (m_thd->locked_tables_mode == LTM_PRELOCKED) > + { > + MYSQL_LOCK *lock; > + MYSQL_LOCK *merged_lock; > + > + /* > + In LTM_LOCK_TABLES table was reopened via locked_tables_list, > + but not in prelocked environment where we have to reopen > + the table manually. > + */ > + Open_table_context ot_ctx(m_thd, MYSQL_OPEN_REOPEN); > + if (open_table(m_thd, m_failed_table, &ot_ctx)) > + { > + result= true; > + break; > + } > + TABLE *table= m_failed_table->table; > + table->reginfo.lock_type= m_thd->update_lock_default; > + m_thd->in_lock_tables= 1; > + lock= mysql_lock_tables(m_thd, &table, 1, > + MYSQL_OPEN_REOPEN | MYSQL_LOCK_USE_MALLOC); > + m_thd->in_lock_tables= 0; > + if (lock == NULL || > + !(merged_lock= mysql_lock_merge(m_thd->lock, lock, m_thd))) > + { > + result= true; > + break; > + } > + m_thd->lock= merged_lock; What MDEV does it fix? I wonder why other cases in recover_from_failed_open doesn't do that > + } > + break; > + } > case OT_BACKOFF_AND_RETRY: > case OT_REOPEN_TABLES: > case OT_NO_ACTION: > @@ -4370,6 +4611,8 @@ bool open_tables(THD *thd, const DDL_options_st &options, > { > if (ot_ctx.can_recover_from_failed_open()) > { > + // FIXME: is this really used? > + DBUG_ASSERT(0); I take it, you weren't able to get this assert to fire? > close_tables_for_reopen(thd, start, > ot_ctx.start_of_statement_svp()); > if (ot_ctx.recover_from_failed_open()) Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 13

[Maria-developers] Progress of improve IO_CACHE (MDEV-24676)
by Vladisav Kakurin 20 Jul '21

20 Jul '21

To this moment I have implemented medium-grained granularity, and also move IO_CACHE to C++ entitled RingBuffer. In future, it will be fine-grained with slots-pattern. https://github.com/MagHErmit/server/commit/7c066a97403115f6d7702049b741d466…

1 0

Re: [Maria-developers] [Commits] 6f1628b: MDEV-25565 Crash on 2-nd execution of SP/PS for query calculating window functions
by Vicențiu Ciorbaru 20 Jul '21

20 Jul '21

Hi Igor! The patch looks good, however I suggest the following changes to make it smaller and easier to read & maintain. 1. You do not need to have special if statements for save_partition_list and save_order_list when you do the cleanup. 2. You do not need to have special code that saves the partition list and the order list *only* when it is overwritten. You can instead always set save_partition_list to be equal to partition_list in the Window_spec's constructor. Same for save_order_list and order_list. Then, during cleanup, you can always execute order_list= save_order_list; partition_list= save_partition_list; The advantage of this change is that it removes all the references to save_xxxx_list in sql_window.cc. You then only have a simple cleanup function sql_union.cc. 3. I would prefer to name the variables "initial_order_list" and "initial_partition_list". The patch then becomes: diff --git a/sql/sql_union.cc b/sql/sql_union.cc index e5648e6989b..54a33bb7863 100644 --- a/sql/sql_union.cc +++ b/sql/sql_union.cc @@ -30,6 +30,7 @@ #include "filesort.h" // filesort_free_buffers #include "sql_view.h" #include "sql_cte.h" +#include "item_windowfunc.h" bool mysql_union(THD *thd, LEX *lex, select_result *result, SELECT_LEX_UNIT *unit, ulong setup_tables_done_option) @@ -1551,6 +1552,21 @@ static void cleanup_order(ORDER *order) } +static void cleanup_window_funcs(List<Item_window_func> &win_funcs) +{ + List_iterator_fast<Item_window_func> it(win_funcs); + Item_window_func *win_func; + while ((win_func= it++)) + { + Window_spec *win_spec= win_func->window_spec; + if (!win_spec) + continue; + win_spec->partition_list= win_spec->initial_partition_list; + win_spec->order_list= win_spec->initial_order_list; + } +} + + bool st_select_lex::cleanup() { bool error= FALSE; @@ -1559,6 +1575,8 @@ bool st_select_lex::cleanup() cleanup_order(order_list.first); cleanup_order(group_list.first); + cleanup_window_funcs(window_funcs); + if (join) { List_iterator<TABLE_LIST> ti(leaf_tables); diff --git a/sql/sql_window.h b/sql/sql_window.h index e0c1563e5bb..cf84e26c822 100644 --- a/sql/sql_window.h +++ b/sql/sql_window.h @@ -99,8 +99,10 @@ class Window_spec : public Sql_alloc LEX_STRING *window_ref; SQL_I_List<ORDER> *partition_list; + SQL_I_List<ORDER> *initial_partition_list; SQL_I_List<ORDER> *order_list; + SQL_I_List<ORDER> *initial_order_list; Window_frame *window_frame; @@ -111,7 +113,8 @@ class Window_spec : public Sql_alloc SQL_I_List<ORDER> *ord_list, Window_frame *win_frame) : window_names_are_checked(false), window_ref(win_ref), - partition_list(part_list), order_list(ord_list), + partition_list(part_list), initial_partition_list(part_list), + order_list(ord_list), initial_order_list(ord_list), window_frame(win_frame), referenced_win_spec(NULL) {} virtual char *name() { return NULL; } On Sat, 10 Jul 2021 at 04:57, IgorBabaev <igor(a)mariadb.com> wrote: > revision-id: 6f1628b917d365ecfc9c4c9951011613f4212592 > (mariadb-10.2.31-1048-g6f1628b) > parent(s): fb0b28932ce82903f2fcfb690a71bff52355507f > author: Igor Babaev > committer: Igor Babaev > timestamp: 2021-07-09 18:56:34 -0700 > message: > > MDEV-25565 Crash on 2-nd execution of SP/PS for query calculating window > functions > from view > > A crash of the server happened when executing a stored procedure whose the > only query calculated window functions over a mergeable view specified > as a select from non-mergeable view. The crash could be reproduced if > the window specifications of the window functions were identical and both > contained PARTITION lists and ORDER BY lists. A crash also happened on > the second execution of the prepared statement created for such query. > If to use derived tables or CTE instead of views the problem still > manifests itself crashing the server. > > When optimizing the window specifications of a window function the > server can substitute the partition lists and the order lists for > the corresponding lists from another window specification in the case > when the lists are identical. This substitution is not permanent and should > be rolled back before the second execution. It was not done and this > ultimately led to a crash when resolving the column names at the second > execution of SP/PS. > > --- > mysql-test/r/win.result | 287 > ++++++++++++++++++++++++++++++++++++++++++++++++ > mysql-test/t/win.test | 147 +++++++++++++++++++++++++ > sql/sql_union.cc | 26 +++++ > sql/sql_window.cc | 12 ++ > sql/sql_window.h | 5 +- > 5 files changed, 476 insertions(+), 1 deletion(-) > > diff --git a/mysql-test/r/win.result b/mysql-test/r/win.result > index 8a31dcc..bc017ea 100644 > --- a/mysql-test/r/win.result > +++ b/mysql-test/r/win.result > @@ -3911,5 +3911,292 @@ sum(i) over () IN ( SELECT 1 FROM t1 a) > 0 > DROP TABLE t1; > # > +# MDEV-25565: 2-nd call of SP with SELECT from view / derived table / CTE > +# returning the result of calculation of 2 window > +# functions that use the same window specification > +# > +create table t1 (a int); > +insert into t1 values (3), (7), (1), (7), (1), (1), (3), (1), (5); > +create view v2 as select a from t1 group by a; > +create view v1 as select * from v2; > +create procedure sp1() select v1.a, > +sum(v1.a) over (partition by v1.a order by v1.a) as k, > +avg(v1.a) over (partition by v1.a order by v1.a) as m > +from v1; > +call sp1(); > +a k m > +1 1 1.0000 > +3 3 3.0000 > +5 5 5.0000 > +7 7 7.0000 > +call sp1(); > +a k m > +1 1 1.0000 > +3 3 3.0000 > +5 5 5.0000 > +7 7 7.0000 > +prepare stmt from "select v1.a, > +sum(v1.a) over (partition by v1.a order by v1.a) as k, > +avg(v1.a) over (partition by v1.a order by v1.a) as m > +from v1"; > +execute stmt; > +a k m > +1 1 1.0000 > +3 3 3.0000 > +5 5 5.0000 > +7 7 7.0000 > +execute stmt; > +a k m > +1 1 1.0000 > +3 3 3.0000 > +5 5 5.0000 > +7 7 7.0000 > +deallocate prepare stmt; > +create procedure sp2() select * from > +( select dt1.a, > +sum(dt1.a) over (partition by dt1.a order by dt1.a) as k, > +avg(dt1.a) over (partition by dt1.a order by dt1.a) as m > +from (select * from v2) as dt1 > +) as dt; > +call sp2(); > +a k m > +1 1 1.0000 > +3 3 3.0000 > +5 5 5.0000 > +7 7 7.0000 > +call sp2(); > +a k m > +1 1 1.0000 > +3 3 3.0000 > +5 5 5.0000 > +7 7 7.0000 > +prepare stmt from "select * from > +( select dt1.a, > +sum(dt1.a) over (partition by dt1.a order by dt1.a) as k, > +avg(dt1.a) over (partition by dt1.a order by dt1.a) as m > +from (select * from v2) as dt1 > +) as dt"; > +execute stmt; > +a k m > +1 1 1.0000 > +3 3 3.0000 > +5 5 5.0000 > +7 7 7.0000 > +execute stmt; > +a k m > +1 1 1.0000 > +3 3 3.0000 > +5 5 5.0000 > +7 7 7.0000 > +deallocate prepare stmt; > +create procedure sp3() select * from > +( select dt1.a, > +sum(dt1.a) over (partition by dt1.a order by dt1.a) as k, > +avg(dt1.a) over (partition by dt1.a order by dt1.a) as m > +from ( select * from (select * from t1 group by a) as dt2 ) as dt1 > +) as dt; > +call sp3(); > +a k m > +1 1 1.0000 > +3 3 3.0000 > +5 5 5.0000 > +7 7 7.0000 > +call sp3(); > +a k m > +1 1 1.0000 > +3 3 3.0000 > +5 5 5.0000 > +7 7 7.0000 > +prepare stmt from "select * from > +( select dt1.a, > +sum(dt1.a) over (partition by dt1.a order by dt1.a) as k, > +avg(dt1.a) over (partition by dt1.a order by dt1.a) as m > +from ( select * from (select * from t1 group by a) as dt2 ) as dt1 > +) as dt"; > +execute stmt; > +a k m > +1 1 1.0000 > +3 3 3.0000 > +5 5 5.0000 > +7 7 7.0000 > +execute stmt; > +a k m > +1 1 1.0000 > +3 3 3.0000 > +5 5 5.0000 > +7 7 7.0000 > +deallocate prepare stmt; > +create procedure sp4() with cte1 as (select * from (select * from t1 > group by a) as dt2), > +cte as > +( select cte1.a, > +sum(cte1.a) over (partition by cte1.a order by cte1.a) as k, > +avg(cte1.a) over (partition by cte1.a order by cte1.a) as m > +from cte1 ) > +select * from cte; > +call sp4(); > +a k m > +1 1 1.0000 > +3 3 3.0000 > +5 5 5.0000 > +7 7 7.0000 > +call sp4(); > +a k m > +1 1 1.0000 > +3 3 3.0000 > +5 5 5.0000 > +7 7 7.0000 > +prepare stmt from "with cte1 as (select * from (select * from t1 group by > a) as dt2), > +cte as > +( select cte1.a, > +sum(cte1.a) over (partition by cte1.a order by cte1.a) as k, > +avg(cte1.a) over (partition by cte1.a order by cte1.a) as m > +from cte1 ) > +select * from cte"; > +execute stmt; > +a k m > +1 1 1.0000 > +3 3 3.0000 > +5 5 5.0000 > +7 7 7.0000 > +execute stmt; > +a k m > +1 1 1.0000 > +3 3 3.0000 > +5 5 5.0000 > +7 7 7.0000 > +deallocate prepare stmt; > +create procedure sp5() with cte1 as (select * from v2), > +cte as > +( select cte1.a, > +sum(cte1.a) over (partition by cte1.a order by cte1.a) as k, > +avg(cte1.a) over (partition by cte1.a order by cte1.a) as m > +from cte1 ) > +select * from cte; > +call sp5(); > +a k m > +1 1 1.0000 > +3 3 3.0000 > +5 5 5.0000 > +7 7 7.0000 > +call sp5(); > +a k m > +1 1 1.0000 > +3 3 3.0000 > +5 5 5.0000 > +7 7 7.0000 > +prepare stmt from "with cte1 as (select * from v2), > +cte as > +( select cte1.a, > +sum(cte1.a) over (partition by cte1.a order by cte1.a) as k, > +avg(cte1.a) over (partition by cte1.a order by cte1.a) as m > +from cte1 ) > +select * from cte"; > +execute stmt; > +a k m > +1 1 1.0000 > +3 3 3.0000 > +5 5 5.0000 > +7 7 7.0000 > +execute stmt; > +a k m > +1 1 1.0000 > +3 3 3.0000 > +5 5 5.0000 > +7 7 7.0000 > +deallocate prepare stmt; > +create procedure sp6() with > +cte1 as (with cte2 as (select * from t1 group by a) select * from cte2), > +cte as > +( select cte1.a, > +sum(cte1.a) over (partition by cte1.a order by cte1.a) as k, > +avg(cte1.a) over (partition by cte1.a order by cte1.a) as m > +from cte1 ) > +select * from cte; > +call sp6(); > +a k m > +1 1 1.0000 > +3 3 3.0000 > +5 5 5.0000 > +7 7 7.0000 > +call sp6(); > +a k m > +1 1 1.0000 > +3 3 3.0000 > +5 5 5.0000 > +7 7 7.0000 > +prepare stmt from "with > +cte1 as (with cte2 as (select * from t1 group by a) select * from cte2), > +cte as > +( select cte1.a, > +sum(cte1.a) over (partition by cte1.a order by cte1.a) as k, > +avg(cte1.a) over (partition by cte1.a order by cte1.a) as m > +from cte1 ) > +select * from cte"; > +execute stmt; > +a k m > +1 1 1.0000 > +3 3 3.0000 > +5 5 5.0000 > +7 7 7.0000 > +execute stmt; > +a k m > +1 1 1.0000 > +3 3 3.0000 > +5 5 5.0000 > +7 7 7.0000 > +deallocate prepare stmt; > +create procedure sp7() with > +cte2 as (select * from v1), > +cte1 as (select * from cte2), > +cte as > +( select cte1.a, > +sum(cte1.a) over (partition by cte1.a order by cte1.a) as k, > +avg(cte1.a) over (partition by cte1.a order by cte1.a) as m > +from cte1 ) > +select * from cte; > +call sp7(); > +a k m > +1 1 1.0000 > +3 3 3.0000 > +5 5 5.0000 > +7 7 7.0000 > +call sp7(); > +a k m > +1 1 1.0000 > +3 3 3.0000 > +5 5 5.0000 > +7 7 7.0000 > +prepare stmt from "with > +cte2 as (select * from v1), > +cte1 as (select * from cte2), > +cte as > +( select cte1.a, > +sum(cte1.a) over (partition by cte1.a order by cte1.a) as k, > +avg(cte1.a) over (partition by cte1.a order by cte1.a) as m > +from cte1 ) > +select * from cte"; > +execute stmt; > +a k m > +1 1 1.0000 > +3 3 3.0000 > +5 5 5.0000 > +7 7 7.0000 > +execute stmt; > +a k m > +1 1 1.0000 > +3 3 3.0000 > +5 5 5.0000 > +7 7 7.0000 > +deallocate prepare stmt; > +drop procedure sp1; > +drop procedure sp2; > +drop procedure sp3; > +drop procedure sp4; > +drop procedure sp5; > +drop procedure sp6; > +drop procedure sp7; > +drop view v1,v2; > +drop table t1; > +# > # End of 10.2 tests > # > diff --git a/mysql-test/t/win.test b/mysql-test/t/win.test > index c07a81f..72e789d 100644 > --- a/mysql-test/t/win.test > +++ b/mysql-test/t/win.test > @@ -2557,5 +2557,152 @@ SELECT sum(i) over () IN ( SELECT 1 FROM t1 a) > FROM t1; > DROP TABLE t1; > > --echo # > +--echo # MDEV-25565: 2-nd call of SP with SELECT from view / derived > table / CTE > +--echo # returning the result of calculation of 2 window > +--echo # functions that use the same window specification > +--echo # > + > +create table t1 (a int); > +insert into t1 values (3), (7), (1), (7), (1), (1), (3), (1), (5); > + > +create view v2 as select a from t1 group by a; > +create view v1 as select * from v2; > + > +let $q1= > +select v1.a, > + sum(v1.a) over (partition by v1.a order by v1.a) as k, > + avg(v1.a) over (partition by v1.a order by v1.a) as m > +from v1; > + > +eval create procedure sp1() $q1; > +call sp1(); > +call sp1(); > + > +eval prepare stmt from "$q1"; > +execute stmt; > +execute stmt; > +deallocate prepare stmt; > + > +let $q2= > +select * from > + ( select dt1.a, > + sum(dt1.a) over (partition by dt1.a order by dt1.a) as k, > + avg(dt1.a) over (partition by dt1.a order by dt1.a) as m > + from (select * from v2) as dt1 > + ) as dt; > + > +eval create procedure sp2() $q2; > +call sp2(); > +call sp2(); > + > +eval prepare stmt from "$q2"; > +execute stmt; > +execute stmt; > +deallocate prepare stmt; > + > +let $q3= > +select * from > + ( select dt1.a, > + sum(dt1.a) over (partition by dt1.a order by dt1.a) as k, > + avg(dt1.a) over (partition by dt1.a order by dt1.a) as m > + from ( select * from (select * from t1 group by a) as dt2 ) as dt1 > + ) as dt; > + > +eval create procedure sp3() $q3; > +call sp3(); > +call sp3(); > + > +eval prepare stmt from "$q3"; > +execute stmt; > +execute stmt; > +deallocate prepare stmt; > + > +let $q4= > +with cte1 as (select * from (select * from t1 group by a) as dt2), > + cte as > + ( select cte1.a, > + sum(cte1.a) over (partition by cte1.a order by cte1.a) as > k, > + avg(cte1.a) over (partition by cte1.a order by cte1.a) as m > + from cte1 ) > +select * from cte; > + > +eval create procedure sp4() $q4; > +call sp4(); > +call sp4(); > + > +eval prepare stmt from "$q4"; > +execute stmt; > +execute stmt; > +deallocate prepare stmt; > + > +let $q5= > +with cte1 as (select * from v2), > + cte as > + ( select cte1.a, > + sum(cte1.a) over (partition by cte1.a order by cte1.a) as > k, > + avg(cte1.a) over (partition by cte1.a order by cte1.a) as m > + from cte1 ) > +select * from cte; > + > +eval create procedure sp5() $q5; > +call sp5(); > +call sp5(); > + > +eval prepare stmt from "$q5"; > +execute stmt; > +execute stmt; > +deallocate prepare stmt; > + > +let $q6= > +with > +cte1 as (with cte2 as (select * from t1 group by a) select * from cte2), > + cte as > + ( select cte1.a, > + sum(cte1.a) over (partition by cte1.a order by cte1.a) as > k, > + avg(cte1.a) over (partition by cte1.a order by cte1.a) as m > + from cte1 ) > +select * from cte; > + > +eval create procedure sp6() $q6; > +call sp6(); > +call sp6(); > + > +eval prepare stmt from "$q6"; > +execute stmt; > +execute stmt; > +deallocate prepare stmt; > + > +let $q7= > +with > + cte2 as (select * from v1), > + cte1 as (select * from cte2), > + cte as > + ( select cte1.a, > + sum(cte1.a) over (partition by cte1.a order by cte1.a) as > k, > + avg(cte1.a) over (partition by cte1.a order by cte1.a) as m > + from cte1 ) > +select * from cte; > + > +eval create procedure sp7() $q7; > +call sp7(); > +call sp7(); > + > +eval prepare stmt from "$q7"; > +execute stmt; > +execute stmt; > +deallocate prepare stmt; > + > + > +drop procedure sp1; > +drop procedure sp2; > +drop procedure sp3; > +drop procedure sp4; > +drop procedure sp5; > +drop procedure sp6; > +drop procedure sp7; > +drop view v1,v2; > +drop table t1; > + > +--echo # > --echo # End of 10.2 tests > --echo # > diff --git a/sql/sql_union.cc b/sql/sql_union.cc > index 7baedfb..f3c90b8 100644 > --- a/sql/sql_union.cc > +++ b/sql/sql_union.cc > @@ -30,6 +30,7 @@ > #include "filesort.h" // filesort_free_buffers > #include "sql_view.h" > #include "sql_cte.h" > +#include "item_windowfunc.h" > > bool mysql_union(THD *thd, LEX *lex, select_result *result, > SELECT_LEX_UNIT *unit, ulong setup_tables_done_option) > @@ -1550,6 +1551,29 @@ static void cleanup_order(ORDER *order) > } > > > +static void cleanup_window_funcs(List<Item_window_func> &win_funcs) > +{ > + List_iterator_fast<Item_window_func> it(win_funcs); > + Item_window_func *win_func; > + while ((win_func= it++)) > + { > + Window_spec *win_spec= win_func->window_spec; > + if (!win_spec) > + continue; > + if (win_spec->save_partition_list) > + { > + win_spec->partition_list= win_spec->save_partition_list; > + win_spec->save_partition_list= NULL; > + } > + if (win_spec->save_order_list) > + { > + win_spec->order_list= win_spec->save_order_list; > + win_spec->save_order_list= NULL; > + } > + } > +} > + > + > bool st_select_lex::cleanup() > { > bool error= FALSE; > @@ -1558,6 +1582,8 @@ bool st_select_lex::cleanup() > cleanup_order(order_list.first); > cleanup_order(group_list.first); > > + cleanup_window_funcs(window_funcs); > + > if (join) > { > List_iterator<TABLE_LIST> ti(leaf_tables); > diff --git a/sql/sql_window.cc b/sql/sql_window.cc > index 612c6e6..3ef751b 100644 > --- a/sql/sql_window.cc > +++ b/sql/sql_window.cc > @@ -479,9 +479,15 @@ int > compare_window_funcs_by_window_specs(Item_window_func *win_func1, > Let's use only one of the lists. > */ > if (!win_spec1->name() && win_spec2->name()) > + { > + win_spec1->save_partition_list= win_spec1->partition_list; > win_spec1->partition_list= win_spec2->partition_list; > + } > else > + { > + win_spec2->save_partition_list= win_spec2->partition_list; > win_spec2->partition_list= win_spec1->partition_list; > + } > > cmp= compare_order_lists(win_spec1->order_list, > win_spec2->order_list); > @@ -494,9 +500,15 @@ int > compare_window_funcs_by_window_specs(Item_window_func *win_func1, > Let's use only one of the lists. > */ > if (!win_spec1->name() && win_spec2->name()) > + { > + win_spec1->save_order_list= win_spec2->order_list; > win_spec1->order_list= win_spec2->order_list; > + } > else > + { > + win_spec1->save_order_list= win_spec2->order_list; > win_spec2->order_list= win_spec1->order_list; > + } > > cmp= compare_window_frames(win_spec1->window_frame, > win_spec2->window_frame); > diff --git a/sql/sql_window.h b/sql/sql_window.h > index e0c1563..417d0bc 100644 > --- a/sql/sql_window.h > +++ b/sql/sql_window.h > @@ -99,8 +99,10 @@ class Window_spec : public Sql_alloc > LEX_STRING *window_ref; > > SQL_I_List<ORDER> *partition_list; > + SQL_I_List<ORDER> *save_partition_list; > > SQL_I_List<ORDER> *order_list; > + SQL_I_List<ORDER> *save_order_list; > > Window_frame *window_frame; > > @@ -111,7 +113,8 @@ class Window_spec : public Sql_alloc > SQL_I_List<ORDER> *ord_list, > Window_frame *win_frame) > : window_names_are_checked(false), window_ref(win_ref), > - partition_list(part_list), order_list(ord_list), > + partition_list(part_list), save_partition_list(NULL), > + order_list(ord_list), save_order_list(NULL), > window_frame(win_frame), referenced_win_spec(NULL) {} > > virtual char *name() { return NULL; } > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits

1 0

Re: [Maria-developers] 19aebbd1b89: MDEV-16481: set global system_versioning_asof=sf() crashes in specific case
by Sergei Golubchik 19 Jul '21

19 Jul '21

Hi, Nikita! On Jul 12, Nikita Malyavin wrote: > revision-id: 19aebbd1b89 (mariadb-10.3.26-129-g19aebbd1b89) > parent(s): 10c163e4a18 > author: Nikita Malyavin <nikita.malyavin(a)mariadb.com> > committer: Nikita Malyavin <nikita.malyavin(a)mariadb.com> > timestamp: 2021-04-06 21:32:43 +0300 > message: > > MDEV-16481: set global system_versioning_asof=sf() crashes in specific case > > * sys_vars.h: add `MYSQL_TIME` field to `set_var::save_result` > * sys_vars.ic: get rid of calling `var->value->get_date()` from `Sys_var_vers_asof::update()` > * versioning.sysvars: add test; remove double warning > > diff --git a/sql/set_var.h b/sql/set_var.h > index 12e025e4696..6351fb5b666 100644 > --- a/sql/set_var.h > +++ b/sql/set_var.h > @@ -296,6 +296,7 @@ class set_var :public set_var_base > plugin_ref *plugins; ///< for Sys_var_pluginlist > Time_zone *time_zone; ///< for Sys_var_tz > LEX_STRING string_value; ///< for Sys_var_charptr and others > + MYSQL_TIME time; ///< for Sys_var_vers_asof old sizeof(save_result) is 16 bytes, sizeof(MYSQL_TIME) is 40. It's just an observation, set_var isn't a structure that's constantly in memory in large quantities. So, not a problem at all. > const void *ptr; ///< for Sys_var_struct > } save_result; > LEX_CSTRING base; /**< for structured variables, like keycache_name.variable_name */ > diff --git a/sql/sys_vars.ic b/sql/sys_vars.ic > index 8d094d3a1d1..eff6a59e65b 100644 > --- a/sql/sys_vars.ic > +++ b/sql/sys_vars.ic > @@ -2650,50 +2650,47 @@ class Sys_var_vers_asof: public Sys_var_enum > virtual bool do_check(THD *thd, set_var *var) > { > if (!Sys_var_enum::do_check(thd, var)) > + { > + var->save_result.time.time_type= MYSQL_TIMESTAMP_NONE; > return false; > - MYSQL_TIME ltime; > - bool res= var->value->get_date(&ltime, TIME_NO_ZERO_IN_DATE|TIME_NO_ZERO_DATE); > - if (!res) > + } > + bool res= var->value->get_date(&var->save_result.time, > + TIME_NO_ZERO_IN_DATE | TIME_NO_ZERO_DATE); > + if (res) > { > - var->save_result.ulonglong_value= SYSTEM_TIME_AS_OF; > + var->save_result.time.time_type= MYSQL_TIMESTAMP_ERROR; > } > return res; > } > > private: > - bool update(set_var *var, vers_asof_timestamp_t &out, system_variables& pool) > + bool update(THD *thd, set_var *var, vers_asof_timestamp_t *out) > { > - bool res= false; > - uint error; > - MYSQL_TIME ltime; > - out.type= static_cast<enum_var_type>(var->save_result.ulonglong_value); > - if (out.type == SYSTEM_TIME_AS_OF) > + MYSQL_TIME &ltime= var->save_result.time; > + uint error= 0; > + > + if (var->value && ltime.time_type >= 0) // any valid value is ok > { > - if (var->value) > - { > - res= var->value->get_date(&ltime, TIME_NO_ZERO_IN_DATE > - | TIME_NO_ZERO_DATE); > - out.unix_time= pool.time_zone->TIME_to_gmt_sec(&ltime, &error); > - out.second_part= ltime.second_part; > - res|= (error != 0); > - } > - else // set DEFAULT from global var > - { > - out.type= SYSTEM_TIME_UNSPECIFIED; > - res= false; > - } > + out->type = SYSTEM_TIME_AS_OF; > + out->unix_time = thd->variables.time_zone->TIME_to_gmt_sec(&ltime, > + &error); why are you doing it here? You could've converted in ::check(), and as a bonus you wouldn't need MYSQL_TIME in the save_result. And what would you do if TIME_to_gmt_sec would fail? ::update() isn't expected to fail, ::check() should guarantee that as much as possible. > + out->second_part= ltime.second_part; > } > - return res; > + else // DEFAULT is set > + { > + out->type= SYSTEM_TIME_UNSPECIFIED; > + } > + return (error != 0); > } > > public: > virtual bool global_update(THD *thd, set_var *var) > { > - return update(var, global_var(vers_asof_timestamp_t), thd->variables); > + return update(thd, var, &global_var(vers_asof_timestamp_t)); > } > virtual bool session_update(THD *thd, set_var *var) > { > - return update(var, session_var(thd, vers_asof_timestamp_t), thd->variables); > + return update(thd, var, &session_var(thd, vers_asof_timestamp_t)); > } > > private: > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 3

Re: [Maria-developers] a11756e4fba: MDEV-23178: Qualified asterisk not supported in INSERT .. RETURNING
by Sergei Golubchik 16 Jul '21

16 Jul '21

Hi, Rucha! On Jul 15, Rucha Deodhar wrote: > revision-id: a11756e4fba (mariadb-10.5.11-27-ga11756e4fba) > parent(s): f0f47cbca18 > author: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > committer: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > timestamp: 2021-07-05 20:14:19 +0530 > message: > > MDEV-23178: Qualified asterisk not supported in INSERT .. RETURNING > > Analysis: When we have INSERT/REPLACE returning with qualified asterisk in the > RETURNING clause, '*' is not resolved properly because of wrong context. > context->table_list is NULL or has incorrect table because context->table_list > has tables from the FROM clause. > Fix: If filling fields instead of '*' for qualified asterisk in RETURNING, > use first_name_resolution_table for correct resolution of item. please, add a couple of tests with a qualified asterisk, where it's qualified _not_ by the table you're inserting into. Like INSERT INTO t1(id1,val1) VALUES(14,'m') RETURNING t2.* not very interesting case, an obvious error ^^^ INSERT INTO t2 SELECT t1.* FROM t1 WHERE id1=2 RETURNING t1.*; but what will happen here? ^^^ > diff --git a/sql/sql_base.cc b/sql/sql_base.cc > index 1476d708d75..60d14c6dbea 100644 > --- a/sql/sql_base.cc > +++ b/sql/sql_base.cc > @@ -8032,12 +8032,13 @@ insert_fields(THD *thd, Name_resolution_context *context, const char *db_name, > else treat natural joins as leaves and do not iterate over their underlying > tables. > */ > - for (TABLE_LIST *tables= (table_name ? context->table_list : > - context->first_name_resolution_table); > + TABLE_LIST *tables= returning_field || !table_name ? context->first_name_resolution_table : > + context->table_list; > + for (; > tables; > - tables= (table_name ? tables->next_local : > - tables->next_name_resolution_table) > + tables= returning_field || !table_name ? tables->next_name_resolution_table : > + tables->next_local > ) > { > Field *field; > TABLE *table= tables->table; 1. please also update the comment to mention RETURNING. 2. Could you apply the following patch as a separate commit before your fix: ============================ --- a/sql/sql_base.cc +++ b/sql/sql_base.cc @@ -8098,12 +8098,14 @@ insert_fields(THD *thd, Name_resolution_context *context, const char *db_name, else treat natural joins as leaves and do not iterate over their underlying tables. */ - for (TABLE_LIST *tables= (table_name ? context->table_list : - context->first_name_resolution_table); - tables; - tables= (table_name ? tables->next_local : - tables->next_name_resolution_table) - ) + TABLE_LIST *first= context->first_name_resolution_table; + TABLE_LIST *TABLE_LIST::* next= &TABLE_LIST::next_name_resolution_table; + if (table_name) + { + first= context->table_list; + next= &TABLE_LIST::next_local; + } + for (TABLE_LIST *tables= first; tables; tables= tables->*next) { Field *field; TABLE *table= tables->table; ============================ it'll remove the condition from inside the loop, will put it in one place only. And will make your change simpler and clearer. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] 70252733d41: MDEV-25908: -e does not work for my_print_defaults
by Sergei Golubchik 15 Jul '21

15 Jul '21

Hi, Rucha! On Jul 15, Rucha Deodhar wrote: > revision-id: 70252733d41 (mariadb-10.5.11-35-g70252733d41) > parent(s): f88d130e710 > author: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > committer: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > timestamp: 2021-07-14 22:55:21 +0530 > message: > > MDEV-25908: -e does not work for my_print_defaults > > Analysis: get_defaults_options() rewrites the value for my_defaults_file, > my_defaults_extra_file and my_defaults_group_suffix to 0. So the config file > can't be read. > Fix: Let handle_options() handle --defaults* option and create a new command > line to pass it to load_defaults(). > > --- > extra/my_print_defaults.c | 37 +++++++++++++++++----- > mysql-test/main/my_print_defaults.result | 35 ++++++++++++++++++++- > mysql-test/main/my_print_defaults.test | 54 +++++++++++++++++++++++++++++++- > mysql-test/main/mysqldump.result | 1 + > 4 files changed, 117 insertions(+), 10 deletions(-) > > diff --git a/extra/my_print_defaults.c b/extra/my_print_defaults.c > index 1dec667949f..409ef273b44 100644 > --- a/extra/my_print_defaults.c > +++ b/extra/my_print_defaults.c > @@ -141,28 +141,49 @@ static int get_options(int *argc,char ***argv) > return 0; > } > > +static char *make_args(const char *s1, const char *s2) > +{ > + char *s= malloc(strlen(s1) + strlen(s2) + 1); > + strmov(strmov(s, s1), s2); > + return s; > +} > > int main(int argc, char **argv) > { > - int count, error, args_used; > - char **load_default_groups= 0, *tmp_arguments[6]; > + int count= 0, error, no_defaults= 0; > + char **load_default_groups= 0; > char **argument, **arguments, **org_argv; > int nargs, i= 0; > MY_INIT(argv[0]); > > org_argv= argv; > - args_used= get_defaults_options(argv); > - > - /* Copy defaults-xxx arguments & program name */ > - count=args_used; > - arguments= tmp_arguments; > - memcpy((char*) arguments, (char*) org_argv, count * sizeof(*org_argv)); > + if (*argv && !strcmp(*argv, "--no-defaults")) > + { > + argv++; > + ++count; > + no_defaults= 1; > + } > + /* Copy program name and --no-defaults if present*/ > + memcpy((char*) arguments, (char*) org_argv, (++count)*sizeof(*org_argv)); isn't this wrong? `arguments` is uninitialized at this point. > arguments[count]= 0; > > /* Check out the args */ > if (get_options(&argc,&argv)) > cleanup_and_exit(1); > > +--echo # Testing -e > +--exec $MYSQL_MY_PRINT_DEFAULTS -e $MYSQLTEST_VARDIR/tmp/tmp1.cnf --mysqld mysql.server > +--echo #Testing --defaults-extra-file > +--exec $MYSQL_MY_PRINT_DEFAULTS --defaults-extra-file=$MYSQLTEST_VARDIR/tmp/tmp1.cnf --mysqld mysql.server > + > +--echo # Testing -c option > +--exec $MYSQL_MY_PRINT_DEFAULTS -c $MYSQLTEST_VARDIR/tmp/tmp2.cnf --mysqld mysql.server > +--echo # Testing --defaults-file > +--exec $MYSQL_MY_PRINT_DEFAULTS --defaults-file=$MYSQLTEST_VARDIR/tmp/tmp2.cnf --mysqld mysql.server > + > +--echo # Testing -g option > +--exec $MYSQL_MY_PRINT_DEFAULTS -c $MYSQLTEST_VARDIR/tmp/tmp3.cnf --mysqld mysql.server -g .1 > +--echo # Testing --defaults-group-suffix > +--exec $MYSQL_MY_PRINT_DEFAULTS -c $MYSQLTEST_VARDIR/tmp/tmp3.cnf --mysqld mysql.server --defaults-group-suffix=.1 you don't seem have a --no-defaults test Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] da70f190279: MDEV-25985 Spider handle ">=" as ">" in some cases
by Sergei Golubchik 13 Jul '21

13 Jul '21

Hi, Nayuta! The code change is good, no comments about it at all. But why did you create a new suite spider/bugfix and all those files in it? I know that some later spider version has it. Still, I'd expect it'd be much simpler and a smaller change to drop the test file into the exising spider suite. On Jul 12, Nayuta Yanagisawa wrote: > revision-id: da70f190279 (mariadb-10.2.39-44-gda70f190279) > parent(s): 99f700a820e > author: Nayuta Yanagisawa <nayuta.yanagisawa(a)hey.com> > committer: Nayuta Yanagisawa <nayuta.yanagisawa(a)hey.com> > timestamp: 2021-07-08 14:12:45 +0000 > message: > > MDEV-25985 Spider handle ">=" as ">" in some cases > > The function spider_db_append_key_where_internal() converts > HA_READ_AFTER_KEY to '>'. The conversion seems to be correct for > single-column indexes because HA_READ_AFTER_KEY means "read the > key after the provided value." > > However, how about multi-column indexes? Assume that there is a > multi-column index on c1 and c2 and we search with the condition > 'c1 >= 100 AND c2 > 200'. The key_range.flag corresponds to the > search condition could be HA_READ_AFTER_KEY. In such a case, > we could not simply convert HA_READ_AFTER_KEY to '>'. > > The correct conversion is to convert HA_READ_AFTER_KEY to '>' > only for the last column in key_part_map and to convert > HA_READ_AFTER_KEY to '>=' for the other column. > > The similar discussion also applies to the conversion from > key_range.flag to a sign of inequality. > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] a6e96ce3c96: MDEV-21916: COM_STMT_BULK_EXECUTE with RETURNING insert wrong values
by Sergei Golubchik 12 Jul '21

12 Jul '21

Hi, Oleksandr! ok to push! On Jul 12, Oleksandr Byelkin wrote: > revision-id: a6e96ce3c96 (mariadb-10.5.4-666-ga6e96ce3c96) > parent(s): f4425d3a3db > author: Oleksandr Byelkin <sanja(a)mariadb.com> > committer: Oleksandr Byelkin <sanja(a)mariadb.com> > timestamp: 2021-06-09 10:43:24 +0200 > message: > > MDEV-21916: COM_STMT_BULK_EXECUTE with RETURNING insert wrong values > > The problem is that array binding uses net buffer to read parameters for each > execution while each execiting with RETURNING write in the same buffer. > > Solution is to allocate new net buffer to avoid changing buffer we are reading > from. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] b1881e07797: MDEV-20131 Assertion `!pk->has_virtual ()' failed in instant_alter_column_possible
by Sergei Golubchik 12 Jul '21

12 Jul '21

Hi, Sachin! On Jul 12, Sachin Kumar wrote: > revision-id: b1881e07797 (mariadb-10.4.20-9-gb1881e07797) > parent(s): 8a2b4d531dc > author: Sachin Kumar <sachin.setiya(a)mariadb.com> > committer: Sachin Kumar <sachin.setiya(a)mariadb.com> > timestamp: 2021-06-11 14:26:19 +0100 > message: > > MDEV-20131 Assertion `!pk->has_virtual ()' failed in instant_alter_column_possible > > Problem:- Primary key on long unique columns is not allowed but when > `CREATE TABLE t2 (a TEXT, PRIMARY KEY(a(1871))) ENGINE=InnoDB;` > is executed with innodb_page_size=8k or 4k, primary key on DB_ROW_HASH_1 column > is created. Reason of this is in mysql_prepare_create_table we check against > max_key_part_length() to see whether key_part length is more then engine supported > key_part length , and if it is true error is thrown for primary key and long > unique index is created for unique key. But in this case max_key_part_length() > returns 3072 which is more the max_key_length() , which later in code triggers > creating of long unique index. > > Solution:- max_supported_key_part_length() should return according to innodb > page size. > > diff --git a/storage/innobase/handler/ha_innodb.cc b/storage/innobase/handler/ha_innodb.cc > index ce2e4b6990f..e5e864bffc5 100644 > --- a/storage/innobase/handler/ha_innodb.cc > +++ b/storage/innobase/handler/ha_innodb.cc > @@ -6537,14 +6537,21 @@ ha_innobase::clone( > DBUG_RETURN(new_handler); > } > > - > -uint > -ha_innobase::max_supported_key_part_length() const > -/*==============================================*/ > -{ > - /* A table format specific index column length check will be performed > - at ha_innobase::add_index() and row_create_index_for_mysql() */ > - return(REC_VERSION_56_MAX_INDEX_COL_LEN); > +uint ha_innobase::max_supported_key_part_length() const > +{ > + /* A table format specific index column length check will be performed > + at ha_innobase::add_index() and row_create_index_for_mysql() */ > + /* FIXME: rewrite this as well as ha_innobase::max_supported_key_length() > + using an API that considers the PRIMARY KEY as well as secondary index > + metadata and the ROW_FORMAT and KEY_BLOCK_SIZE */ > + switch (srv_page_size) { > + case 4096: > + return 1173; > + case 8192: > + return 1536; > + default: > + return REC_VERSION_56_MAX_INDEX_COL_LEN; > + } dunno, why would you duplicate the whole switch logic? can you just - return(REC_VERSION_56_MAX_INDEX_COL_LEN); - return MY_MIN(max_supported_key_length(), REC_VERSION_56_MAX_INDEX_COL_LEN); > } > > /******************************************************************//** > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 85186734823: MDEV-24248: my_print_defaults is not taking all the values when using -e
by Sergei Golubchik 12 Jul '21

12 Jul '21

Hi, Rucha! On Jul 12, Rucha Deodhar wrote: > revision-id: 85186734823 (mariadb-10.5.4-665-g85186734823) > parent(s): bf5c050fd28 > author: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > committer: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > timestamp: 2021-06-14 10:27:13 +0530 > message: > > MDEV-24248: my_print_defaults is not taking all the values when using -e > option which is called from mysql.server (extra_args). > > Fix: change mysql.server script to use --defaults-extra-file instead of -e > > diff --git a/mysql-test/main/my_print_defaults.test b/mysql-test/main/my_print_defaults.test > --- /dev/null > +++ b/mysql-test/main/my_print_defaults.test > @@ -0,0 +1,29 @@ > +--echo # > +--echo # Beginning of 10.5 Test > +--echo # > +--echo # > +--echo # MDEV-24248: my_print_defaults is not taking all the values when using > +--echo # -e option which is called from mysql.server (extra_args). > +--echo # > + > +--echo # checking for mysql.server > + > +--write_file $MYSQLTEST_VARDIR/tmp/tmp1.cnf > + > +[mariadb-10.5] don't use the version number in the test - you're forcing some poor soul to update the test on merges for every new version. > +key_buffer_size=20M > +max_allowed_packet=250M > +table_open_cache=1000 > +table_definition_cache=2000 > +read_buffer_size=1M > +thread_cache_size=8 > +max_connections=1024 > +long_query_time=60 > +slow_query_log=1 > +EOF > + > +--exec $MYSQL_MY_PRINT_DEFAULTS --defaults-extra-file=$MYSQLTEST_VARDIR/tmp/tmp1.cnf --mysqld mysql.server > + > +--remove_file $MYSQLTEST_VARDIR/tmp/tmp1.cnf > + > +--echo # End of 10.5 test > diff --git a/support-files/mysql.server.sh b/support-files/mysql.server.sh > index cefb3225e9e..8504e4def4b 100644 > --- a/support-files/mysql.server.sh > +++ b/support-files/mysql.server.sh > @@ -200,11 +200,11 @@ su_kill() { > extra_args="" > if test -r "$basedir/my.cnf" > then > - extra_args="-e $basedir/my.cnf" > + extra_args="--defaults-file-name=$basedir/my.cnf" eh. -e is --defaults-extra-file same below. > else > if test -r "$datadir/my.cnf" > then > - extra_args="-e $datadir/my.cnf" > + extra_args="--defaults-file-name=$datadir/my.cnf" > fi > fi > > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] b29ec61724d: MDEV-25906: SIGSEGV in flush_tables_with_read_lock on FTWRL or FTFE | SIGSEGV in ha_maria::extra
by Sergei Golubchik 01 Jul '21

01 Jul '21

Hi, Oleksandr! On Jul 01, Oleksandr Byelkin wrote: > revision-id: b29ec61724d (mariadb-10.6.2-41-gb29ec61724d) > parent(s): 83684fc9a4d > author: Oleksandr Byelkin <sanja(a)mariadb.com> > committer: Oleksandr Byelkin <sanja(a)mariadb.com> > timestamp: 2021-07-01 11:20:29 +0200 > message: > > MDEV-25906: SIGSEGV in flush_tables_with_read_lock on FTWRL or FTFE | SIGSEGV in ha_maria::extra > > Fixed check of derived tables of all kinds (view, derived, information schema). > > diff --git a/sql/sql_reload.cc b/sql/sql_reload.cc > index 0fa2fa10df8..e41da9680b0 100644 > --- a/sql/sql_reload.cc > +++ b/sql/sql_reload.cc > @@ -592,10 +592,19 @@ bool flush_tables_with_read_lock(THD *thd, TABLE_LIST *all_tables) > for (TABLE_LIST *table_list= all_tables; table_list; > table_list= table_list->next_global) > { > - if (!(table_list->is_view() || > - table_list->table->file->ha_table_flags() & HA_CAN_EXPORT)) > + /* > + We allow all deriveds and view here, but prohibit information_schema > + tables (explicitly) and performance_schema tables (implicetly: > + engine do not suport export) > + */ > + if (table_list->is_non_derived() && > + (!table_list->is_base_table() || > + !(table_list->table->file->ha_table_flags() & HA_CAN_EXPORT))) > { > - my_error(ER_ILLEGAL_HA, MYF(0),table_list->table->file->table_type(), > + my_error(ER_ILLEGAL_HA, MYF(0), > + (table_list->is_base_table() ? > + table_list->table->file->table_type(): > + "information_schema"), I'd say if you want to detect I_S tables, you should test for table_list->schema_table. It's not clear what your condition actually does. And there is no such engine "information_schema", so the message is misleading. In fact, this should be access denied. For both. Like: mysqltest: At line 12: query 'flush table information_schema.collations for export' failed: ER_DBACCESS_DENIED_ERROR (1044): Access denied for user 'foo'@'localhost' to database 'information_schema' this is from a test file that only does flush table information_schema.collations for export. > table_list->db.str, table_list->table_name.str); > goto error_reset_bits; > } > @@ -607,7 +616,8 @@ bool flush_tables_with_read_lock(THD *thd, TABLE_LIST *all_tables) > for (auto table_list= all_tables; table_list; > table_list= table_list->next_global) > { > - if (!table_list->is_view() && > + if (table_list->table && why do you check for table_list->table ? Can it happen that table_list->table == NULL and table_list->is_base_table() == true? > + table_list->is_base_table() && > table_list->table->file->extra(HA_EXTRA_FLUSH)) > goto error_reset_bits; > } Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

[Maria-developers] MDEV-25325 built-in documentation for performance_schema tables
by Haidong Ji 24 Jun '21

24 Jun '21

Hi, I've grabbed this JIRA story, https://jira.mariadb.org/browse/MDEV-25325 . The coding and test cases creation were all done. I've also submitted a PR here: https://github.com/MariaDB/server/pull/1830 The PR was submitted more than 1 month ago. I'm curious about next steps. Thanks, Haidong

1 0

[Maria-developers] JSON_TABLE: on default values
by Sergey Petrunia 16 Jun '21

16 Jun '21

Hi Alexey, At the moment MariaDB requires that the values in DEFAULT clauses are quoted. Example: select * from json_table( '{"intval": 1000}', '$' columns( col1 int path '$.intval_' default '100' on empty ) ) as T; here, "100" must be quoted, otherwise one gets a parse error. However, the quoted value is interpreted as an SQL literal. This looks puzzling. MySQL-8 also requires that the default value is quoted, but they have a (very odd) reason for it: they interpret the default value as JSON: https://docs.oracle.com/cd/E17952_01/mysql-8.0-en/json-table-functions.html says: DEFAULT json_string ON EMPTY: the provided json_string is parsed as JSON, as long as it is valid, and stored instead of the missing value. Column type rules also apply to the default value. I am not sure why MySQL chose to do this. Looking into the SQL Standard, one can see: <JSON table regular column definition> ::= <column name> <data type> [ PATH <JSON table column path specification> ] [ <JSON table column empty behavior> ON EMPTY ] [ <JSON table column error behavior> ON ERROR ] <JSON table column empty behavior> ::= ERROR | NULL | DEFAULT <value expression> ... This doesn't say whether the <value expression> should be interepreted as JSON or just as a value. But one can find this passage: <quote> Without Feature T826, “General value expression in ON ERROR or ON EMPTY clauses”, the <value expression> contained in <JSON table column empty behavior> or <JSON table column error behavior> contained in a <JSON table regular column definition> JTRCD shall be a <literal> that can be cast to the data type specified by the <data type> contained in JTRCD without raising an exception condition according to the General Rules of Subclause 6.13, “<cast specification>”. </quote> The important part is: ... shall be a <literal> that can be cast to the data type specified ... which means it is not JSON. It is just a literal, and literal can be a string literal (in quotes, 'string') or an integer literal (without quotes) or other kind of literal. Btw, Oracle Database allows non-string literals in the default clause: https://dbfiddle.uk/?rdbms=oracle_18&fiddle=9af7e43ede77ee285e1a65f1f419d3bd What are your thoughts on this? Is MariaDB's behavior intentional? Should we follow the standard and allow all kinds of literals? What was the reason for the limitation that default values are quoted? BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://petrunia.net

2 4

[Maria-developers] MDEV-25875: RE 02469b: MDEV-17399 JSON_TABLE Accept JSON values for the JSON fields
by Sergey Petrunia 16 Jun '21

16 Jun '21

Hi Alexey, This is review input re https://github.com/MariaDB/server/commit/02469bdead5753eccb5d70c98a158a0702…. First, our workflow dictates that this patch should have its own MDEV entry. I've filed it https://jira.mariadb.org/browse/MDEV-25875. Please use it in commit descriptions. == Updates mixed with another patch == The patch has .result changes like: > @@ -551,10 +551,12 @@ JSON_TABLE('{}', '$' COLUMNS (x INT PATH '$.x' DEFAULT NULL ON ERROR)) jt; > ERROR 42000: You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'NULL ON ERROR)) jt' at line 2 > SELECT * FROM > JSON_TABLE('{}', '$' COLUMNS (x INT PATH '$.x' DEFAULT 0 ON EMPTY)) jt; > -ERROR 42000: You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near '0 ON EMPTY)) jt' at line 2 > +x > +0 > SELECT * FROM > JSON_TABLE('{}', '$' COLUMNS (x INT PATH '$.x' DEFAULT 0 ON ERROR)) jt; > -ERROR 42000: You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near '0 ON ERROR)) jt' at line 2 > +x > +NULL > SELECT * FROM > JSON_TABLE('{}', '$' COLUMNS (x DATE > PATH '$.x' These are obviously from another patch (the one about "accepting non-string literals as default"). Please move them to that patch. == Comments == > @@ -377,6 +378,25 @@ static void store_json_in_field(Field *f, const json_engine_t *je) > } > > > +static int store_json_in_json(Field *f, json_engine_t *je) > +{ Please add a comment for the function. Something like "Store the current JSON element pointed by je into field f, as a separate JSON document" The name "store_json_in_json" doesn't look good to me. Do you think store_element_as_json_doc() would be better? > + const uchar *from= je->value_begin; > + const uchar *to; > + > + if (json_value_scalar(je)) > + to= je->value_end; > + else > + { > + int error; > + if ((error= json_skip_level(je))) > + return error; > + to= je->s.c_str; > + } > + f->store((const char *) from, (uint32) (to - from), je->s.cs); > + return 0; > +} > + > + > bool Json_table_nested_path::check_error(const char *str) > { > if (m_engine.s.error) > @@ -541,7 +561,12 @@ int ha_json_table::fill_column_values(THD *thd, uchar * buf, uchar *pos) > } > else > { > - if (!(error= !json_value_scalar(&je))) > + if (jc->m_format_json) > + { > + if (!(error= store_json_in_json(*f, &je))) > + error= er_handler.errors; Can there be a case when this assignment assigns any value other than 0? (My logic - er_handler.errors already had some error status, why did we call store_json_in_json? If there was an error inside store_json_in_json, we do not get to this assignment, as store_json_in_json() returned non-zero error. Is this correct?) > + } > + else if (!(error= !json_value_scalar(&je))) > { > store_json_in_field(*f, &je); > error= er_handler.errors; > @@ -868,6 +893,10 @@ int Json_table_column::set(THD *thd, enum_type ctype, const LEX_CSTRING &path) > anctual content. Not sure though if we should. > */ > m_path.s.c_str= (const uchar *) path.str; > + > + if (ctype == PATH) > + m_format_json= m_field->type_handler() == &type_handler_json_longtext; This will probably cause warnings on some compiler. Please wrap into "MY_TEST(...)" or just brackets, "z= (x == y)". == A discrepancy between this and MySQL == I've found this case: select * from json_table('{"foo": null}', '$' columns( jscol json path '$.foo') ) as T; with the above patch, it produces a JSON null value: +-------+ | jscol | +-------+ | null | +-------+ while in MySQL-8 it produces an SQL NULL: +-------+ | jscol | +-------+ | NULL | +-------+ I'm not yet sure what we should do about this. Any thoughts? BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://petrunia.net

2 1

[Maria-developers] New Member (GSoC MDEV-25015)
by Alan Cueva 15 Jun '21

15 Jun '21

Hello everyone, I am Alan Cueva, during the summer I will be working on the Custom formatting of strings in MariaDB queries (MDEV-25015 <https://jira.mariadb.org/browse/MDEV-25015>) project, but I would like to continue contributing in this community. I hope to be useful and learn from everyone. Thanks!

1 0

Re: [Maria-developers] 27060eb6ba5: MDEV-21916: COM_STMT_BULK_EXECUTE with RETURNING insert wrong values
by Sergei Golubchik 10 Jun '21

10 Jun '21

Hi, Oleksandr! On Oct 12, Oleksandr Byelkin wrote: > revision-id: 27060eb6ba5 (mariadb-10.5.4-220-g27060eb6ba5) > parent(s): 861cd4ce286 > author: Oleksandr Byelkin <sanja(a)mariadb.com> > committer: Oleksandr Byelkin <sanja(a)mariadb.com> > timestamp: 2020-10-07 15:22:41 +0200 > message: > > MDEV-21916: COM_STMT_BULK_EXECUTE with RETURNING insert wrong values > > To allocate new net buffer to avoid changing bufer we are reading. You still didn't clarify the commit comment > diff --git a/sql/sql_delete.cc b/sql/sql_delete.cc > index 7280236e43f..5aaff3cf623 100644 > --- a/sql/sql_delete.cc > +++ b/sql/sql_delete.cc > @@ -685,8 +685,14 @@ bool mysql_delete(THD *thd, TABLE_LIST *table_list, COND *conds, > !table->prepare_triggers_for_delete_stmt_or_event()) > will_batch= !table->file->start_bulk_delete(); > > - if (returning) > + /* > + thd->get_stmt_da()->is_set() means first iteration of prepared statement > + with array binding operation execution (non optimized so it is not > + INSERT) > + */ > + if (returning && !thd->get_stmt_da()->is_set()) > { > + DBUG_ASSERT(thd->lex->sql_command != SQLCOM_INSERT); a strange assert to see in sql_delete.cc :) can one even reach mysql_delete() with SQLCOM_INSERT? not just with returning and !thd->get_stmt_da()->is_set(), anywhere? > if (result->send_result_set_metadata(returning->item_list, > Protocol::SEND_NUM_ROWS | Protocol::SEND_EOF)) > goto cleanup; > diff --git a/sql/sql_error.cc b/sql/sql_error.cc > index b3ef0d89a98..a753af2b34d 100644 > --- a/sql/sql_error.cc > +++ b/sql/sql_error.cc > @@ -380,16 +380,33 @@ Diagnostics_area::set_eof_status(THD *thd) > if (unlikely(is_error() || is_disabled())) > return; > > + if (m_status == DA_EOF_BULK) > + { > /* > If inside a stored procedure, do not return the total > number of warnings, since they are not available to the client > anyway. > */ > + if (!thd->spcont) > + m_statement_warn_count+= current_statement_warn_count(); > + } > + else > + { > + /* > + If inside a stored procedure, do not return the total > + number of warnings, since they are not available to the client > + anyway. > + */ I don't think it helps to duplicate the comment. You could just put it once before the if. > if (thd->spcont) > { > m_statement_warn_count= 0; > + m_affected_rows= 0; why do you reset m_affected_rows too? > } > else > m_statement_warn_count= current_statement_warn_count(); > + m_status= (is_bulk_op() ? DA_EOF_BULK : DA_EOF); > + } do we have tests for bulk operations and CALL? > > - m_status= DA_EOF; > DBUG_VOID_RETURN; > } > > diff --git a/sql/sql_prepare.cc b/sql/sql_prepare.cc > index ecb56e70f88..c144d3a8d7e 100644 > --- a/sql/sql_prepare.cc > +++ b/sql/sql_prepare.cc > @@ -4357,24 +4361,37 @@ Prepared_statement::execute_bulk_loop(String *expanded_query, > > + /* > + Here second buffer for not optimized commands, > + optimized commands do it inside thier internal loop. > + */ > + if (!(sql_command_flags[lex->sql_command] & CF_SP_BULK_OPTIMIZED) && why "SP" and not "PS" ? > + this->lex->has_returning()) what about CALL? It won't have lex->has_returning(). What about SELECT? Can you bind an array to a parameter in SELECT or CALL? > + { > + // Above check can be true for SELECT in future > + DBUG_ASSERT(lex->sql_command != SQLCOM_SELECT); how can SQLCOM_SELECT have lex->has_returning() ? > + readbuff= thd->net.buff; // old buffer > + if (net_allocate_new_packet(&thd->net, thd, MYF(MY_THREAD_SPECIFIC))) > + { > + readbuff= NULL; // failure, net_allocate_new_packet keeps old buffer > + goto err; > + } > } > > #ifndef EMBEDDED_LIBRARY Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 2

Re: [Maria-developers] 557ab2992dd: MDEV-21117 Incremental patch for a few review notes addressed.
by Sergei Golubchik 09 Jun '21

09 Jun '21

Hi, Andrei! Again, this is a review of the git diff f09d33f521b 557ab2992dd, not only 557ab2992dd. On Jun 02, Andrei Elkin wrote: > revision-id: 557ab2992dd (mariadb-10.6.0-67-g557ab2992dd) > parent(s): 4cc5a3074da > author: Andrei Elkin <andrei.elkin(a)mariadb.com> > committer: Andrei Elkin <andrei.elkin(a)mariadb.com> > timestamp: 2021-05-27 17:37:22 +0300 > message: > > MDEV-21117 Incremental patch for a few review notes addressed. > diff --git a/mysql-test/suite/binlog/t/binlog_truncate_active_log.inc b/mysql-test/suite/binlog/t/binlog_truncate_active_log.inc > new file mode 100644 > --- /dev/null > +++ b/mysql-test/suite/binlog/t/binlog_truncate_active_log.inc > @@ -0,0 +1,65 @@ > +connect(master1,localhost,root,,); > +connect(master2,localhost,root,,); > +connect(master3,localhost,root,,); > + > +--connection default > + > +# First to commit few transactions > +INSERT INTO t VALUES (10); > +INSERT INTO tm VALUES (10); > + > +--connection master1 > +# Hold insert after write to binlog and before "run_commit_ordered" in engine > +SET DEBUG_SYNC= "commit_before_get_LOCK_commit_ordered SIGNAL master1_ready WAIT_FOR signal_never_arrives"; > +--send_eval $query1 > + > +--connection master2 > +SET DEBUG_SYNC= "now WAIT_FOR master1_ready"; > +SET DEBUG_SYNC= "commit_before_get_LOCK_after_binlog_sync SIGNAL master2_ready"; > +if ($pre_q2) > +{ > + --send_eval $pre_q2; > +} > +# To binlog non-xid transactional group which will be truncated all right > +if (!$pre_q2) > +{ > + --send_eval $query2 > +} eh, this makes no sense. In the previous version of the patch you've run pre_q2 before sending query2. now you send either pre_q2 or query2. so you don't need pre_q2 anymore now, just set query2 to the what used to be pre_q2. > + > +--connection master3 > +SET DEBUG_SYNC= "now WAIT_FOR master2_ready"; > +SELECT @@global.gtid_binlog_pos as 'Before the crash'; > + > +--connection default > +--source include/kill_mysqld.inc > +--disconnect master1 > +--disconnect master2 > +--disconnect master3 > + > +# > +# Server restart > +# > +--let $restart_parameters= --rpl-semi-sync-slave-enabled=1 > +--source include/start_mysqld.inc > + > +# Check error log for a successful truncate message. > +--let $log_error_ = $MYSQLTEST_VARDIR/log/mysqld.1.err > + > +--let SEARCH_FILE=$log_error_ > +--let SEARCH_PATTERN=Successfully truncated.*to remove transactions starting from GTID $truncate_gtid_pos > +--replace_regex /FOUND [0-9]+/FOUND #/ can it be found an unpredictable number of times? Why would binlog be truncated more than once? > +--source include/search_pattern_in_file.inc > + > +--echo Pre-crash binlog file content: > +--let $binlog_file= query_get_value(show binary logs, Log_name, $binlog_file_index) > +--source include/show_binlog_events.inc > + > +SELECT @@global.gtid_binlog_pos as 'After the crash'; > +--echo "One row should be present in table 't'" > +SELECT * FROM t; > + > +# prepare binlog file index for the next test > +--inc $binlog_file_index > + > +# Local cleanup > +DELETE FROM t; > diff --git a/mysql-test/suite/binlog/t/binlog_truncate_active_log.test b/mysql-test/suite/binlog/t/binlog_truncate_active_log.test > new file mode 100644 > --- /dev/null > +++ b/mysql-test/suite/binlog/t/binlog_truncate_active_log.test > @@ -0,0 +1,104 @@ > +# ==== Purpose ==== > +# > +# Test verifies the truncation of single binary log file. > +# > +# ==== References ==== > +# > +# MDEV-21117: recovery for --rpl-semi-sync-slave-enabled server > + > +--source include/have_innodb.inc > +--source include/have_aria.inc > +# File: binlog_truncate_active_log.inc included in test makes use of > +# 'debug_sync' facility. > +--source include/have_debug_sync.inc > +--source include/have_binlog_format_statement.inc > + > +call mtr.add_suppression("Can.t init tc log"); > +call mtr.add_suppression("Aborting"); > + > +# The following cases are tested: > +# A. 2pc transaction is followed by a blank "zero-engines" one > +# B. 2pc transaction follows the blank one > +# C. Similarly to A, with the XA blank transaction > + > +RESET MASTER; > +CREATE TABLE t (f INT) ENGINE=INNODB; > +CREATE TABLE t2 (f INT) ENGINE=INNODB; > +CREATE TABLE tm (f INT) ENGINE=Aria; > + > +# Old (pre-crash) binlog file index initial value. > +# It keeps incremented at the end of each case. > +--let $binlog_file_index=1 > + > +--echo # Case A. > +# Using 'debug_sync' hold 'query1' execution after 'query1' is flushed and > +# synced to binary log but not yet committed. In an another connection hold > +# 'query2' execution after 'query2' is flushed and synced to binlog. > +# Crash and restart server with --rpl-semi-sync-slave-enabled=1 > +# > +# During recovery of binary log 'query1' status is checked with InnoDB engine, > +# it will be in prepared but not yet commited. All transactions starting from > +# 'query1' onwards will be removed from the binary log. > +# Show-binlog-events is to prove that. > + > +--let $truncate_gtid_pos = 0-1-6 > +--let $query1 = INSERT INTO t VALUES (20) > +--let $query2 = DELETE FROM t2 WHERE f = 0 /* no such record */ > +--source binlog_truncate_active_log.inc > + > +--echo # Case B. > +# The inverted sequence ends up to truncate starting from $query2 > +--let $truncate_gtid_pos = 0-1-10 > +--let $query1 = DELETE FROM t2 WHERE f = 0 > +--let $query2 = INSERT INTO t VALUES (20) > +--source binlog_truncate_active_log.inc > + > + > +--echo # Case C. > +delimiter |; > +CREATE PROCEDURE sp_blank_xa() > +BEGIN > + XA START 'blank'; > + DELETE FROM t2 WHERE f = 0 /* no such record */; > + XA END 'blank'; > + XA PREPARE 'blank'; > +END| > +delimiter ;| > + > +# The same as in A with $query2 being the zero-engine XA transaction. > +# Both $query1 and $query2 are going to be truncated. > +--let $truncate_gtid_pos = 0-1-14 > +--let $query1 = INSERT INTO t VALUES (20) > +--let $pre_q2 = CALL sp_blank_xa > +--let $query2 = XA COMMIT 'blank' hm.. now your binlog_truncate_active_log.inc only runs query2 is pre_q2 is empty. So this XA COMMIT isn't done at all. Very confusing. > +--source binlog_truncate_active_log.inc > + > +DROP PROCEDURE sp_blank_xa; > + > + > +--echo # Case D. > +delimiter |; > +CREATE PROCEDURE sp_xa() > +BEGIN > + XA START 'xid'; > + DELETE FROM t WHERE f = 10; > + XA END 'xid'; > + XA PREPARE 'xid'; > +END| > +delimiter ;| > + > +# The same as in B with $query1 being the prepared XA transaction. > +# Truncation must occurs at $query2. > +--let $truncate_gtid_pos = 0-1-20 > +--let $query1 = CALL sp_xa > +--let $pre_q2 = > +--let $query2 = INSERT INTO t2 VALUES (20) > +--source binlog_truncate_active_log.inc > + > +DROP PROCEDURE sp_xa; > + > + > +--echo # Cleanup > +DROP TABLE t,t2,tm; > + > +--echo # End of the tests > diff --git a/sql/handler.h b/sql/handler.h > --- a/sql/handler.h > +++ b/sql/handler.h > @@ -931,6 +932,32 @@ struct xid_t { > }; > typedef struct xid_t XID; > > +/* > + Enumerates a sequence in the order of > + their creation that is in the top-down order of the index file. > + Ranges from zero through MAX_binlog_id. > + Not confuse the value with the binlog file numerical suffix, > + neither with the binlog file line in the binlog index file. Ah! Good, thanks. I thought it was a binlog file numerical suffix, indeed. > +*/ > +typedef uint Binlog_file_id; > +const Binlog_file_id MAX_binlog_id= UINT_MAX; > +/* > + Compound binlog-id and byte offset of transaction's first event > + in a sequence (e.g the recovery sequence) of binlog files. > + Binlog_offset(0,0) is the minimum value to mean > + the first byte of the first binlog file. > +*/ > +typedef std::pair<Binlog_file_id, my_off_t> Binlog_offset; > + > +/* binlog-based recovery transaction descriptor */ > +struct xid_recovery_member > +{ > + my_xid xid; > + uint in_engine_prepare; // number of engines that have xid prepared > + bool decided_to_commit; > + Binlog_offset binlog_coord; // semisync recovery binlog offset > +}; > + > /* for recover() handlerton call */ > #define MIN_XID_LIST_SIZE 128 > #define MAX_XID_LIST_SIZE (1024*128) > diff --git a/sql/log.cc b/sql/log.cc > --- a/sql/log.cc > +++ b/sql/log.cc > @@ -1692,17 +1698,13 @@ int binlog_init(void *p) > binlog_hton->savepoint_rollback= binlog_savepoint_rollback; > binlog_hton->savepoint_rollback_can_release_mdl= > binlog_savepoint_rollback_can_release_mdl; > - binlog_hton->commit= binlog_commit; > + binlog_hton->commit= binlog_commit_dummy; you could use a lambda for that too, [](handlerton*,THD*,bool) { return 0; } > binlog_hton->rollback= binlog_rollback; > binlog_hton->drop_table= [](handlerton *, const char*) { return -1; }; Oh did you actually need it? Or it's for completeness only? I mean, was there some code that tried to drop a "binlog" table? > if (WSREP_ON || opt_bin_log) > { > binlog_hton->prepare= binlog_prepare; > binlog_hton->start_consistent_snapshot= binlog_start_consistent_snapshot; > - binlog_hton->commit_by_xid= binlog_commit_by_xid; > - binlog_hton->rollback_by_xid= binlog_rollback_by_xid; > - // recover needs to be set to make xa{commit,rollback}_handlerton effective > - binlog_hton->recover= binlog_xa_recover_dummy; > } > binlog_hton->flags= HTON_NOT_USER_SELECTABLE | HTON_HIDDEN | HTON_NO_ROLLBACK; > return 0; > @@ -9975,6 +9977,154 @@ int TC_LOG::using_heuristic_recover() > /****** transaction coordinator log for 2pc - binlog() based solution ******/ > #define TC_LOG_BINLOG MYSQL_BIN_LOG > > +/** > + Truncates the current binlog to specified position. Removes the rest of binlogs > + which are present after this binlog file. > + > + @param truncate_file Holds the binlog name to be truncated > + @param truncate_pos Position within binlog from where it needs to > + truncated. > + > + @retval true ok > + @retval false error > + > +*/ > +bool MYSQL_BIN_LOG::truncate_and_remove_binlogs(const char *file_name, > + my_off_t pos, > + rpl_gtid *ptr_gtid) > +{ > + int error= 0; > +#ifdef HAVE_REPLICATION > + LOG_INFO log_info; > + THD *thd= current_thd; > + my_off_t index_file_offset= 0; > + File file= -1; > + MY_STAT s; > + my_off_t old_size; > + > + if ((error= find_log_pos(&log_info, file_name, 1))) > + { > + sql_print_error("Failed to locate binary log file:%s." > + "Error:%d", file_name, error); > + goto end; > + } > + > + while (!(error= find_next_log(&log_info, 1))) > + { > + if (!index_file_offset) > + { > + index_file_offset= log_info.index_file_start_offset; > + if ((error= open_purge_index_file(TRUE))) > + { > + sql_print_error("Failed to open purge index " > + "file:%s. Error:%d", purge_index_file_name, error); > + goto end; > + } > + } > + if ((error= register_purge_index_entry(log_info.log_file_name))) > + { > + sql_print_error("Failed to copy %s to purge index" > + " file. Error:%d", log_info.log_file_name, error); > + goto end; > + } > + } > + > + if (error != LOG_INFO_EOF) > + { > + sql_print_error("Failed to find the next binlog to " > + "add to purge index register. Error:%d", error); > + goto end; > + } > + > + if (is_inited_purge_index_file()) > + { > + if (!index_file_offset) > + index_file_offset= log_info.index_file_start_offset; > + > + if ((error= sync_purge_index_file())) > + { > + sql_print_error("Failed to flush purge index " > + "file. Error:%d", error); > + goto end; > + } > + > + // Trim index file > + if ((error= > + mysql_file_chsize(index_file.file, index_file_offset, '\n', > + MYF(MY_WME))) || > + (error= > + mysql_file_sync(index_file.file, MYF(MY_WME|MY_SYNC_FILESIZE)))) > + { > + sql_print_error("Failed to trim binlog index " > + "file:%s to offset:%llu. Error:%d", index_file_name, > + index_file_offset, error); > + goto end; > + } > + > + /* Reset data in old index cache */ > + if ((error= reinit_io_cache(&index_file, READ_CACHE, (my_off_t) 0, 0, 1))) > + { > + sql_print_error("Failed to reinit binlog index " > + "file. Error:%d", error); > + goto end; > + } > + > + /* Read each entry from purge_index_file and delete the file. */ > + if ((error= purge_index_entry(thd, NULL, TRUE))) > + { > + sql_print_error("Failed to process registered " > + "files that would be purged."); > + goto end; > + } > + } > + > + DBUG_ASSERT(pos); > + > + if ((file= mysql_file_open(key_file_binlog, file_name, > + O_RDWR | O_BINARY, MYF(MY_WME))) < 0) > + { > + error= 1; > + sql_print_error("Failed to open binlog file:%s for " > + "truncation.", file_name); > + goto end; > + } > + my_stat(file_name, &s, MYF(0)); > + old_size= s.st_size; > + clear_inuse_flag_when_closing(file); > + /* Change binlog file size to truncate_pos */ > + if ((error= > + mysql_file_chsize(file, pos, 0, MYF(MY_WME))) || > + (error= mysql_file_sync(file, MYF(MY_WME|MY_SYNC_FILESIZE)))) I'd rather not to overload if() conditions with function calls like this, they're rather annoying to single-step through and impossible to put a breakpoint on a specific function call. > + { > + sql_print_error("Failed to trim the " > + "binlog file:%s to size:%llu. Error:%d", > + file_name, pos, error); > + goto end; > + } > + else > + { > + char buf[21]; > + my_stat(file_name, &s, MYF(0)); > + my_off_t new_size= s.st_size; do you need it? new_size == pos, isn't it? You yourself use the word "size" to describe pos in sql_print_error just few lines above :) > + > + longlong10_to_str(ptr_gtid->seq_no, buf, 10); > + sql_print_information("Successfully truncated binlog file:%s " > + "to pos:%llu to remove transactions starting from " > + "GTID %u-%u-%s; " > + "former file size shrunk from %llu to %llu.", This isn't quite English. Better just "previous (or old) file size: %llu", that is not "former" and no need to specify the new size twice. Also, you say that you "truncated binlog file" here, but on failure you fail to "trim the binlog file". I suggest to use consistent wording, either always "trim" or always "truncate" (and always with or without "the") - both on success and on failure. Personally, I favor "truncate the binlog file" > + file_name, pos, > + ptr_gtid->domain_id, ptr_gtid->server_id, buf, > + old_size, new_size); > + } > + > +end: > + if (file >= 0) > + mysql_file_close(file, MYF(MY_WME)); > + > + error= error || close_purge_index_file(); > +#endif > + return error > 0; > +} > int TC_LOG_BINLOG::open(const char *opt_name) > { > int error= 1; > @@ -10449,7 +10599,528 @@ start_binlog_background_thread() ... > + > +Recovery_context::Recovery_context() : > + prev_event_pos(0), > + last_gtid_standalone(false), last_gtid_valid(false), last_gtid_no2pc(false), > + last_gtid_engines(0), > + do_truncate(rpl_semi_sync_slave_enabled), > + truncate_validated(false), truncate_reset_done(false), > + truncate_set_in_1st(false), id_binlog(MAX_binlog_id), > + checksum_alg(BINLOG_CHECKSUM_ALG_UNDEF), gtid_maybe_to_truncate(NULL) > +{ > + last_gtid_coord= Binlog_offset(0,0); > + binlog_truncate_coord= binlog_truncate_coord_1st_round= Binlog_offset(0,0); > + binlog_unsafe_coord= Binlog_offset(0,0); Okay, I admit, just typedef'ing std::pair is a huge improvement already > + binlog_truncate_file_name[0]= 0; > + binlog_unsafe_file_name [0]= 0; > + binlog_unsafe_gtid= truncate_gtid= truncate_gtid_1st_round= rpl_gtid(); > + if (do_truncate) > + gtid_maybe_to_truncate= new Dynamic_array<rpl_gtid>(16, 16); > +} > + Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 1

[Maria-developers] Tips on Writing/Reading JSON
by Michael Okoko 09 Jun '21

09 Jun '21

Hi Alexey, I'm Michael Okoko, a GSoC '21 student working on Using JSON as the on-disk format for MariaDB histograms(https://jira.mariadb.org/browse/MDEV-21130). The project requires writing and reading/parsing JSON, and I was wondering if you could provide some tips and pointers regarding a proper JSON writer/JSON parser to use. Regards, Michael.

2 1

Re: [Maria-developers] ef2519fee4e: MDEV-16546 System versioning setting to allow history modification
by Aleksey Midenkov 08 Jun '21

08 Jun '21

Hello, Sergei! On Fri, May 3, 2019 at 8:43 PM Sergei Golubchik <serg(a)mariadb.com> wrote: > > Hi, Aleksey! > > On May 03, Aleksey Midenkov wrote: > > revision-id: ef2519fee4e (versioning-1.0.5-17-gef2519fee4e) > > parent(s): 56145be2951 > > author: Aleksey Midenkov <midenok(a)gmail.com> > > committer: Aleksey Midenkov <midenok(a)gmail.com> > > timestamp: 2018-06-28 13:42:09 +0300 > > message: > > > > MDEV-16546 System versioning setting to allow history modification > > > > 1. Add server variable system_versioning_modify_history which will > > allow to set values for row_start, row_end in DML operations. > > > > 2. If secure_timestamp is YES or REPLICATION, > > system_versioning_modify_history does not have effect. If > > secure_timestamp is SUPER, system_versioning_modify_history requires > > special privilege (same as for setting current timestamp). > > I thought more about this idea. We don't really want to have the history > editable, do we? Well, I'm thinking about rollback table data to specific point in time. That could be a useful feature. > But it's needed for replication, to keep the master and > slave identical. That's what secure_timestamp is for. > > The idea was that this new variable, system_versioning_modify_history, > will be just a convenience feature, it will not allow history editing > any more than one can do without it. > > But now I suspect that even with secure_timestamp=NO one cannot truly > edit history. One can only insert new rows with arbitrary timestamps. > For example, to insert a row with row_start=1000 and row_end=2000, one > needs to do (if secure_timestamp=NO): > > set timestamp=1000; > insert row; > set timestamp=2000; > delete row; > > But I don't see how one can update or delete a history row with > secure_timestamp=NO. > > Now, with a SUPER privilege and secure_timestamp=NO or SUPER, one can > use the BINLOG command and truly edit the history arbitrarily, by faking > row events. I don't really get it why this is so important: since there is some limitation by configuration and privilege, we are just fine. Everything can be changed at filesystem level after all. > > The conclusion, I believe, is that system_versioning_modify_history > should allow INSERTs when secure_timestamp=NO, and it should allow > UPDATE/DELETE only for a SUPER user when secure_timestamp=NO or SUPER. I don't see a reason to argue on that. The only thing that is not clear, why we don't allow INSERTs when secure_timestamp=SUPER? > > The second thing I don't like at all, is when a table is created like > > CREATE TABLE t1 (a int) WITH SYSTEM VERSIONING > > with row_start/row_end implicit. You don't have it in the test, but > anyway one should be able to load history into such a table, while the > table does not have row_start and row_end columns. From the user point > of view these columns don't exist, they're pseudo-columns, like ROWID. > They just cannot be insertable-into, conceptually. But a user will want > to restore the history, right? I don't have a solution for this yet :( > Any ideas? We don't have to follow the conception if it doesn't help us. Since we have physical row_start/row_end, we don't have to pretend they don't exist. Who will win from that? > > See below a couple of minor comments about the patch itself. > > ... These are going to be fixed. > > Regards, > Sergei > Chief Architect MariaDB > and security(a)mariadb.org -- All the best, Aleksey Midenkov @midenok

2 15

[Maria-developers] Request for Discussion: Structure for JSON histogram objects
by Michael Okoko 04 Jun '21

04 Jun '21

Hi, MariaDB community! I'm implementing the patch for Using JSON as the on-disk format for MariaDB histograms(https://jira.mariadb.org/browse/MDEV-21130). as part of GSoC '21 and hoping to get insights and ideas from the community on how the JSON format for histograms should be structured. We are planning to change the columns of `mysql.column_stats` into something like: CREATE TABLE `column_stats` ( `db_name` varchar(64) COLLATE utf8mb3_bin NOT NULL, `table_name` varchar(64) COLLATE utf8mb3_bin NOT NULL, `column_name` varchar(64) COLLATE utf8mb3_bin NOT NULL, `histogram` blob ); The histogram column above will then be responsible for storing the JSON document for that row. Currently, the JSON structure we are looking at is based on the existing columns in `column_stats` table as well MySQL's JSON histogram implementation( https://dev.mysql.com/doc/refman/8.0/en/optimizer-statistics.html) Here's an example: { "min_value": 0, "max_value": 0, "hist_type": "singleton", "bucket": [ [1,0.3333333333333333], [2, 0.6666666666666666], [3,1] ], "nulls_ratios": 0, "last_updated": "2021-06-02 13:32:40.000000", "sampling_rate": 1, } - The min_value and max_values retain the meanings described at ( https://mariadb.com/kb/en/mysqlcolumn_stats-table/) - hist_type can be one of `SINGLETON` (where the number of buckets is >= the number of distinct values in the specified column) or `EQUI-HEIGHT` (where the number of buckets is < the number of distinct values found in the column). - nulls_ratio is a value between 0.0 and 1.0 indicating the fraction of column values that are SQL NULL values. If 0, the column contains no NULL values. - `last_updated` indicates when the histogram was generated, as a UTC value in YYYY-MM-DD hh:mm:ss.uuuuuu format. - `sampling_rate` is also a value between 0.0 and 1.0 indicating the fraction of column values that were sampled to create the histogram. A value of 1 means that all of the data was read (no sampling).

3 2

Re: [Maria-developers] 557ab2992dd: MDEV-21117 Incremental patch for a few review notes addressed.
by Sergei Golubchik 03 Jun '21

03 Jun '21

Hi, Andrei! On Jun 03, Andrei Elkin wrote: > >> - binlog_hton->commit= binlog_commit; > >> + binlog_hton->commit= binlog_commit_dummy; > >> binlog_hton->rollback= binlog_rollback; > >> binlog_hton->drop_table= [](handlerton *, const char*) { return -1; }; > > > > Oh did you actually need it? Or it's for completeness only? > > I am confused a bit, > 6c52931680a4 (Sergei Golubchik 2020-06-22 1703) binlog_hton->drop_table= [](handlerton *, const char*) { return -1; }; > > And I never reviewed the patch. Grrr. Sorry! When reviewing I diffed this patch with a previous version that I reviewed (a useful speedup technique at the late phase, when commits only change slightly) and this drop_table line was in the new patch and not in the old. Of course, it was because the new patch was in 10.6 and the old in 10.2, there were quite a few changes in the diff because of that. I tried to ignore them, but somehow missed this one. Note, there's no + before the drop_table line, it's not yours, it's indeed something that already was in 10.6, I should've noticed, my mistake :( Why I added it in 6c52931680a4? It was for DROP TABLE FORCE, that simply does plugin_foreach() and tries to hton->drop_table() in every hton available. In binlog_hton too. The goal was to solve cases like a table in InnoDB data dictionary but with no frm, or Aria table where MAD file exists, but no MAI - cases when DROP TABLE doesn't see the table, but CREATE TABLE still fails as some remnants of a table still exist. > >> @@ -9975,6 +9977,154 @@ int TC_LOG::using_heuristic_recover() > >> + else > >> + { > >> + char buf[21]; > >> + my_stat(file_name, &s, MYF(0)); > >> + my_off_t new_size= s.st_size; > > > > do you need it? new_size == pos, isn't it? You yourself use the word > > "size" to describe pos in sql_print_error just few lines above :) > > Sure. Let turn it into an > > assert( pos == new_size ) > > [x] and this is, what I think, you'd call "too much checks" :) I'd personally would've dropped my_stat and new_size, especially as I also suggest to remove them from the sql_print_information below. > >> + > >> + longlong10_to_str(ptr_gtid->seq_no, buf, 10); > >> + sql_print_information("Successfully truncated binlog file:%s " > >> + "to pos:%llu to remove transactions starting from " > >> + "GTID %u-%u-%s; " > >> + "former file size shrunk from %llu to %llu.", > > > > This isn't quite English. Better just "previous (or old) file > > size: %llu", that is not "former" and no need to specify the new size twice. > > > > Also, you say that you "truncated binlog file" here, but on failure > > you fail to "trim the binlog file". I suggest to use consistent > > wording, either always "trim" or always "truncate" (and always with > > or without "the") - both on success and on failure. Personally, I favor > > "truncate the binlog file" > > some two styles of two writers of these blocks mixed up :-). (Partly > relates to the size in above). > > [x] > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 8e79d168614: MDEV-25672 table alias from previous statement interferes later commands.
by Sergei Golubchik 02 Jun '21

02 Jun '21

Hi, Aleksey! On Jun 02, Aleksey Midenkov wrote: > revision-id: 8e79d168614 (mariadb-10.2.31-989-g8e79d168614) > parent(s): 433dd490d33 > author: Aleksey Midenkov <midenok(a)gmail.com> > committer: Aleksey Midenkov <midenok(a)gmail.com> > timestamp: 2021-05-31 16:20:38 +0300 > message: > > MDEV-25672 table alias from previous statement interferes later commands. I'm sorry. I don't understand what all this new code for. To make sure that vcol's field->table_name is refreshed for every statement? What is vcol's field->table_name used for besides your CREATE TABLE check? If it's not used for anything then a much simpler fix would be narrow the check - it only needs to be run for new vcols in CREATE TABLE, disabling it for old vcols in ALTER TABLE will do the trick. Like - if (p.table_name.length && table_name) + if (!field && p.table_name.length && table_name) Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] e92037989f7: MDEV-21117: refine the server binlog-based recovery for semisync
by Sergei Golubchik 01 Jun '21

01 Jun '21

Hi, Andrei! On Jun 01, Andrei Elkin wrote: > >>>> > >>>> It's actually turns out not to be easy. A sequence of execution events > >>>> > >>>> E1.prepare(trx), E2.prepare(trx), E1.commit(trx), *crash* > >>>> > >>>> does not make E1 such as Innodb committed persistently in file. > >>>> At recovery in E1 trx may be found in prepared state, unless > >>>> before the crash I'd do something like Binlog CheckPoint (BCP) > >>>> notification request and wait of it. > >>> > The committed state is done asynchronously (by virtue of MDEV-232 that > implements a part of BCP). Oh. Indeed. Sorry, I've missed that. Still a prepare of another transaction can force this commit to disk, as far as I understand. So what about this: you stop the thread on a debug sync after the first commit then another connection starts committing in InnoDB, it stops on a debug sync after the prepare - before anything is written into binlog then you crash. looks like it'll create the case you need > >> No doubt it's technically possible to wait for the flushing and > >> crash, but still does not look easy. > >> Consider if we defer this sort of mtr test to MDEV-18959? > > Sujatha just contributed `74a13b4de2` with an mtr test that crashes > after the asynchronous fsync is done to the partially (2 engine case) > committed state. ... > Aslo now that `74a13b4de2` provides with a method to crash after the > "commmit" fsync we might also employ it for RQG. No, this is very different. This is a crash happening where Sujatha expects it to happen, there's no value in that now, it's needed for regression testing, to make sure the recovery won't be broken in the future. But it makes no sense to keep testing now how recovery works in case of a crash happening at exactly the place where Sujatha wanted it to happen and after she verified that crash at that exactly line does not break the recovery. The whole point of RQG tests is to try crashes where you did not expect them to happen. Adding DBUG_EXECUTE_IF() doesn't help this goal at all. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] e92037989f7: MDEV-21117: refine the server binlog-based recovery for semisync
by Sergei Golubchik 31 May '21

31 May '21

Hi, Andrei! On May 27, Andrei Elkin wrote: >>> >>> I had a rather sobering followup, slipped communicating to you >>> though, sorry. >>> >>> It's actually turns out not to be easy. A sequence of execution events >>> >>> E1.prepare(trx), E2.prepare(trx), E1.commit(trx), *crash* >>> >>> does not make E1 such as Innodb committed persistently in file. At >>> recovery in E1 trx may be found in prepared state, unless before the >>> crash I'd do something like Binlog CheckPoint (BCP) notification request >>> and wait of it. >> >> I don't understand, what a binlog checkpoint has to do with it? > > So E1=Innodb. > In the above case in order to create partially committed trx > Innodb.commit(trx) > must be followed with flushing trx to disk, which is typically > asynchronous. > > I mentioned BCP 'cos it ensures a trx (actually the last in a binlog > file) is flushed. With --innodb-flush-log-at-trx-commit=1 it should be perfectly synnchronous, should it not? > >> In this just a selected engine would be requested, unlike in BCP > >> event case that is generated when both (in this case) have > >> persistently committed. (That is we even can't involve BCP event to > >> wait for, and the involvement would require some simulation as > >> well). > >> > >> So I propose to keep the current method leaving the "honest" crash > >> to RQG that Alice has been training the patch with. > > > > Alice was loading the server with transactions and doing `kill -9` > > at some point. Many interesting cases that we discuss (for example, > > the one above) happen in a very small time window, so I don't know > > whether random killing could provide an adequate test coverage. > > No doubt it's technically possible to wait for the flushing and crash, > but still does not look easy. > Consider if we defer this sort of mtr test to MDEV-18959? But we've spent numerous hours and emails discussing and you've spend countless days and weeks implementing various tricky corner cases are are and won't be ever tested. Here's an idea of how to test it. We're only interested in crashes during commit. So * when starting a commit - atomically increment some in_commit counter * on leaving the commit - decrement it * on signal (some unused one) - crash the server if in_commit > 0 * after the crash - get the stack traces from the core to see where in the commit did it crash. I've attached a small patch to illustrate this idea. Hopefully, it'll cover everything, but if not, then after analyzing stack traces we'll see where it did not crash, and move atomic increment/decrements to be around this area only. > >> >> diff --git a/sql/handler.h b/sql/handler.h > >> >> --- a/sql/handler.h > >> >> +++ b/sql/handler.h > >> >> @@ -873,6 +874,15 @@ typedef struct xid_t XID; > >> >> /* The 'buf' has to have space for at least SQL_XIDSIZE bytes. */ > >> >> uint get_sql_xid(XID *xid, char *buf); > >> >> > >> >> +/* struct for semisync slave binlog truncate recovery */ > >> >> +struct xid_recovery_member > >> >> +{ > >> >> + my_xid xid; > >> >> + uint in_engine_prepare; // number of engines that have xid prepared > >> >> + bool decided_to_commit; > >> >> + std::pair<uint, my_off_t> binlog_coord; // semisync recovery binlog offset > >> > > >> > wouldn't it be clearer to have a struct with named members? > >> > >> I am not sure if there's a need for naming. But we do need > >> comparison that std::pair provides for granted. > >> > >> I could consider (a) new typedef(s) to represent the binlog id and > >> offset in the file to make it more readable. > >> > >> You say? > >> > >> > in fact, I'm somewhat surprised there's no such struct for binlog coords > >> > already. > >> > >> I had started that back at HB implementation actually, but that struct > >> did not attact much of attention. > > > > > I'd still suggest a structure, like > > > > struct binlog_coords_t { > > uint binlog_num; > > my_off_t offset; > > }; > > Well, with this, like I said about std:pair, we'd also have to have > bool operator < (const binlog_coords_t) like bool operator <(const binlog_coords_t c) { return binlog_num < c.binlog_num || binlog_num == c.binlog_num && offset < c.offset; } > I seriously rate refuse to reuse as a deadly sin of programming. > (Unlike normal sins it may be soften by realization though :-)). > > Let's waive it away, right :-)? > > Hoping on that `72a59e4b068` already defined the following: ... > +/* > + Compound binlog-id and byte offset of transaction's first event > + in a sequence (e.g the recovery sequence) of binlog files. > + Binlog_offset(0,0) is the minimum value to mean > + the first byte of the first binlog file. > +*/ > +typedef std::pair<Binlog_file_id, my_off_t> Binlog_offset; This is better than writing std::pair<uint, my_off_t> everywhere. Not as clear as a binlog_coords_t from above, but _perhaps_ it'll do, let's see how it'll be used. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

[Maria-developers] Using JSON_TABLE to create JSON columns?
by Sergey Petrunia 31 May '21

31 May '21

Hi Alexey and everyone, I was looking at whether it's possible use JSON_TABLE to extract portions of JSON document. Apparently it is possible in MySQL with JSON datatype: Q1: select * from json_table('[{"color": "blue", "price": { "high": 10, "low": 5}}, {"color": "red", "price": { "high": 20, "low": 8}}]', '$[*]' columns(color varchar(100) path '$.color', price json path '$.price' ) ) as T; +-------+------------------------+ | color | price | +-------+------------------------+ | blue | {"low": 5, "high": 10} | | red | {"low": 8, "high": 20} | +-------+------------------------+ Note that if one uses any datatype other than JSON, they get NULLs: Q2: select * from json_table('[{"color": "blue", "price": { "high": 10, "low": 5}}, {"color": "red", "price": { "high": 20, "low": 8}}]', '$[*]' columns(color varchar(100) path '$.color', price text path '$.price' ) ) as T; +-------+-------+ | color | price | +-------+-------+ | blue | NULL | | red | NULL | +-------+-------+ Oracle-the-database doesn't yet(*) have a JSON datatype. So I can only run Q2 and then I get NULLs in the price column. MariaDB accepts JSON as datatype so query Q1 is accepted. However the logic in MDEV-17399 code doesn't have support for dumping a portion of JSON document, so one gets empty strings in the price column. Should we support Q1 with JSON output in the price column? If yes, should we do it within the scope of MDEV-17399 or create another task for this? (*) I see this: https://docs.oracle.com/en/database/oracle/oracle-database/20/newft/new-jso… BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

3 2

[Maria-developers] GSoC Student Introduction for MDEV-24676
by Владислав Какурин 31 May '21

31 May '21

Hello, my name is Vladislav Kakurin, and this year I'm a GSOC 2021 participant for MariaDB. I will work on concurrent ring-buffer for improvement of IO_CACHE (https://jira.mariadb.org/browse/MDEV-24676) I hope I will succeed in this matter, as well as become a part of the MariaDB community

1 0

[Maria-developers] GSoC Student Introduction for MDEV-21130
by Michael Okoko 27 May '21

27 May '21

Hi, I'm Michael Okoko, a GSOC 2021 mentee for MariaDB and I'd be working on switching the on-disk storage format for MariaDB histograms to JSON ( https://jira.mariadb.org/browse/MDEV-21130) I'm looking forward to working with everyone in the community and contributing more to the MariaDB during and after GSOC.

1 0

Re: [Maria-developers] e92037989f7: MDEV-21117: refine the server binlog-based recovery for semisync
by Sergei Golubchik 26 May '21

26 May '21

Hi, Andrei! On Apr 27, Andrei Elkin wrote: > >> +--let $case = B: one engine has committed its transaction branch > >> +# Hold off after one engine has committed. > >> +--let $shutdown_timeout=0 > >> +--let $debug_sync_action = "commit_after_run_commit_ordered SIGNAL con1_ready WAIT_FOR signal_no_signal" > >> +# Both debug_sync and debug-dbug are required to make sure Engines remember the commit state > >> +# debug_sync alone will not help. > >> +--let $restart_parameters = --rpl-semi-sync-slave-enabled=1 --debug-dbug=d,binlog_truncate_partial_commit > > > > in the first review I wrote > > > > this seems to be a rather crude way of faking a partially committed > > transaction. better to crash after the first engine has committed, > > that'd be much more natural. > > > > and you replied > > > > This simulation aimed at (allows for) more complicated recovery time > > event sequences. > > In this case, indeed, crashing by demand is about of the same efforts. > > I can convert to that. > > > > [x] > > I had a rather sobering followup, slipped communicating to you though, sorry. > > It's actually turns out not to be easy. A sequence of execution events > > E1.prepare(trx), E2.prepare(trx), E1.commit(trx), *crash* > > does not make E1 such as Innodb committed persistently in file. At > recovery in E1 trx may be found in prepared state, unless before the > crash I'd do something like Binlog CheckPoint (BCP) notification request > and wait of it. I don't understand, what a binlog checkpoint has to do with it? > In this just a selected engine would be requested, unlike in BCP event > case that is generated when both (in this case) have persistently committed. > (That is we even can't involve BCP event to wait for, and the involvement > would require some simulation as well). > > So I propose to keep the current method leaving the "honest" crash to > RQG that Alice has been training the patch with. Alice was loading the server with transactions and doing `kill -9` at some point. Many interesting cases that we discuss (for example, the one above) happen in a very small time window, so I don't know whether random killing could provide an adequate test coverage. > >> diff --git a/mysql-test/suite/rpl/t/rpl_semi_sync_fail_over.test b/mysql-test/suite/rpl/t/rpl_semi_sync_fail_over.test > >> --- /dev/null > >> +++ b/mysql-test/suite/rpl/t/rpl_semi_sync_fail_over.test > >> @@ -0,0 +1,143 @@ > >> + > >> +--source include/have_innodb.inc > >> +--source include/have_debug_sync.inc > >> +--source include/have_binlog_format_row.inc > >> +--let $rpl_topology=1->2 > >> +--source include/rpl_init.inc > > > > why not to source master-slave.inc if you're using a standard master-slave > > topology anyway? > > Using numbers it's easier to manipulate with failover operatations. > Besides it's truly uncomfortable both read and write > when `master` would be serving the slave role as it happens in the test :-). I believe I already replied to that earlier. master-slave.inc creates server_1 and server_2 connections all right. > >> diff --git a/sql/handler.h b/sql/handler.h > >> --- a/sql/handler.h > >> +++ b/sql/handler.h > >> @@ -873,6 +874,15 @@ typedef struct xid_t XID; > >> /* The 'buf' has to have space for at least SQL_XIDSIZE bytes. */ > >> uint get_sql_xid(XID *xid, char *buf); > >> > >> +/* struct for semisync slave binlog truncate recovery */ > >> +struct xid_recovery_member > >> +{ > >> + my_xid xid; > >> + uint in_engine_prepare; // number of engines that have xid prepared > >> + bool decided_to_commit; > >> + std::pair<uint, my_off_t> binlog_coord; // semisync recovery binlog offset > > > > wouldn't it be clearer to have a struct with named members? > > I am not sure if there's a need for naming. But we do need > comparison that std::pair provides for granted. > > I could consider (a) new typedef(s) to represent the binlog id and > offset in the file to make it more readable. > > You say? > > > in fact, I'm somewhat surprised there's no such struct for binlog coords > > already. > > I had started that back at HB implementation actually, but that struct > did not attact much of attention. I'd still suggest a structure, like struct binlog_coords_t { uint binlog_num; my_off_t offset; }; > > > >> +}; > >> + > >> /* for recover() handlerton call */ > >> #define MIN_XID_LIST_SIZE 128 > >> #define MAX_XID_LIST_SIZE (1024*128) > >> @@ -4820,7 +4830,8 @@ int ha_commit_one_phase(THD *thd, bool all); > >> int ha_commit_trans(THD *thd, bool all); > >> int ha_rollback_trans(THD *thd, bool all); > >> int ha_prepare(THD *thd); > >> -int ha_recover(HASH *commit_list); > >> +int ha_recover(HASH *commit_list, MEM_ROOT *mem_root= NULL); > >> +uint ha_recover_complete(HASH *commit_list, std::pair<uint, my_off_t> *coord= NULL); > > > > is coord a truncation position? > > So NULL is for the normal recovery. > Non-NULL is an estimate of the 1st to rollback. > Trx with lesser offsets are to commit, greater of equal - to rollback. > > Why it's only an estimate is 'cos the function also handles > no-truncation cases, it then passes as `coord` the last committed > trx's coordinate. > The last committed can't be decided to rollback because of higher precedence of > > xid_recovery_member::decided_to_commit > > which is `true` under current circumstance, in > > +static bool xarecover_decide(xid_recovery_member* member, > + xid_t x, std::pair<uint, my_off_t> *coord_ptr) > +{ > + return > + member->decided_to_commit ? true : > + !coord_ptr ? false : > + (member->binlog_coord < *coord_ptr > > I did not reflect the subtlety of the above in function header comments. > To be done, if you're sastified here. yes, please, reflect it. thanks :) > >> diff --git a/sql/log_event.h b/sql/log_event.h > >> --- a/sql/log_event.h > >> +++ b/sql/log_event.h > >> @@ -482,6 +482,16 @@ class String; > >> */ > >> #define LOG_EVENT_IGNORABLE_F 0x80 > >> > >> +/** > >> + @def LOG_EVENT_ACCEPT_OWN_F > >> + > >> + Flag sets by the semisync slave for accepting > >> + the same server_id ("own") events which the slave must not have > >> + in its state. Typically such events were never committed by > >> + their originator (this server) and discared at its semisync-slave recovery. > >> +*/ > >> +#define LOG_EVENT_ACCEPT_OWN_F 0x4000 > > > > may be, add an assert on all received events that such a flag is not set? > > it can only be set on events in relay log. > > There's no place the flag gets logged. It'd be way too fool-proof in my > opinion :-)! > Or if you have a case I'll do. too fool-proof is when you do a=1; assert(a == 1); if you write a binlog event to disk, another thread reads it and sends over the network to another server, which also writes it to disk first, then another thread reads it again and executes - I don't think it's "too" to check that the event doesn't have this flag. Also, I personally value a lot a documentation role of asserts. Assert, like DBUG_ASSERT(ev->flags & LOG_EVENT_ACCEPT_OWN_F == 0); is basically saying /* LOG_EVENT_ACCEPT_OWN_F must be not set here */ but it's a self-fulfilling comment, it documents the code and at the same time guarantees that the documentation is up-to-date. > >> @@ -3357,6 +3367,12 @@ class Gtid_log_event: public Log_event > >> uint64 commit_id; > >> uint32 domain_id; > >> uchar flags2; > >> + uint flags_extra; // more flags area placed after the regular flags2's one > >> + /* > >> + Extra to a "base" engine recoverable engines participating > >> + in the transaction. Zero, when the base engine only is present. > > > > what's a "base engine"? > > The 1st ha_info one that is not binlog. why not to say "number of participating engines minus one" ? this won't puzzle anyone with a new concept of a "base engine" > > > >> + */ > >> + uint8 extra_engines; > >> > >> /* Flags2. */ > >> > >> diff --git a/sql/handler.cc b/sql/handler.cc > >> --- a/sql/handler.cc > >> +++ b/sql/handler.cc > >> @@ -1637,9 +1672,17 @@ commit_one_phase_2(THD *thd, bool all, THD_TRANS *trans, bool is_real_trans) > >> DEBUG_SYNC(thd, "commit_one_phase_2"); > >> if (ha_info) > >> { > >> + int err; > >> + > > >> + if (has_binlog_hton(ha_info) && > > > > can you replace has_binlog_hton() with, like, if trx cache is not empty or > > binlog enabled or something like that? > > I see you're driving at removing binlog ha_info from the list, are not > you? I'm not sure :) Yes, I'd like to get rid of binlog_hton, as we discussed. But here, perhaps, I just didn't like that you scan the list for something that most likely could be checked much easier. > I already made some experiments to have observed various complications > in the way of implementing an equivalent of > has_binlog_hton(). > The closest would be > non-empty cache. > > But `binlog_commit` may need invoking even in this case, as its branch > shows > > if (cache_mngr->trx_cache.empty()) > { > /* > we're here because cache_log was flushed in MYSQL_BIN_LOG::log_xid() > */ > cache_mngr->reset(false, true); > > > Overall, I'd stay with the current approach to defer its refinement to > time binlog_hton stops registering for transaction. > > Your comment? Okay. I just don't like scanning the list, aesthetically, when one can check a value in some fixed location or a variable. But it's not a performance concern, the list has usually only two elements and hardly ever can have more than three. So okay, let's stay with the current approach. > > @@ -1962,8 +2008,177 @@ struct xarecover_st > > >> sql_print_warning("%s transaction with xid %llu", > > > > may be not a sql_print_warning, but a sql_print_information? > > it's just an informational message > > [->] I presume, it means [x] > >> + member->decided_to_commit ? > >> + "Committed" : "Rolled back", (ulonglong) member->xid); > >> + } > >> + > >> + return false; > >> +} Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 308c28e8a60: MDEV-15888 Implement FLUSH TABLES tbl_name [, tbl_name] ... WITH READ LOCK for views.
by Sergei Golubchik 23 May '21

23 May '21

Hi, Alexey! On Mar 21, Alexey Botchkov wrote: > revision-id: 308c28e8a60 (mariadb-10.5.2-393-g308c28e8a60) > parent(s): 786bc312b85 > author: Alexey Botchkov <holyfoot(a)mariadb.com> > committer: Alexey Botchkov <holyfoot(a)mariadb.com> > timestamp: 2021-02-09 00:59:55 +0400 > message: > > MDEV-15888 Implement FLUSH TABLES tbl_name [, tbl_name] ... WITH READ LOCK for views. > > Kind of 'naive' solution that surprisingly worked so trying to > understand why. > > diff --git a/mysql-test/main/flush-innodb.result b/mysql-test/main/flush-innodb.result > index 21e5bda7785..2c886e4f9fc 100644 > --- a/mysql-test/main/flush-innodb.result > +++ b/mysql-test/main/flush-innodb.result > @@ -60,7 +60,7 @@ DROP TABLE export; > CREATE VIEW v1 AS SELECT 1; > CREATE TEMPORARY TABLE t1 (a INT); > FLUSH TABLES v1 FOR EXPORT; > -ERROR HY000: 'test.v1' is not of type 'BASE TABLE' > +UNLOCK TABLES; good, but it only tests that there's no error. may be you need to add tests to verify that tables are actually locked? > FLUSH TABLES t1 FOR EXPORT; > ERROR 42S02: Table 'test.t1' doesn't exist > FLUSH TABLES non_existent FOR EXPORT; > diff --git a/sql/sql_reload.cc b/sql/sql_reload.cc > index 8f87d633d19..c146b7693c1 100644 > --- a/sql/sql_reload.cc > +++ b/sql/sql_reload.cc > @@ -543,6 +543,7 @@ bool flush_tables_with_read_lock(THD *thd, TABLE_LIST *all_tables) > > if (thd->lex->type & REFRESH_READ_LOCK) > { > +#ifdef MDEV15887 > /* > Acquire SNW locks on tables to be flushed. Don't acquire global > IX and database-scope IX locks on the tables as this will make > @@ -559,6 +560,7 @@ bool flush_tables_with_read_lock(THD *thd, TABLE_LIST *all_tables) > for (auto table_list= all_tables; table_list; > table_list= table_list->next_global) > table_list->mdl_request.ticket= NULL; > +#endif /*MDEV15887*/ what's that? btw, the number is MDEV15888 > } > > thd->variables.option_bits|= OPTION_TABLE_LOCK; Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] Request for a discusison: A fine-grained concurrent ring buffer mode for IO_CACHE
by Nikita Malyavin 22 May '21

22 May '21

Welcome to the thread, Andrei! For everybody: we just had a productive chat with Andrei, and I'd like to outline the results. Let's take a look at binlog_commit, or more exactly MYSQL_BIN_LOG::trx_group_commit_leader called from there (follow binlog_commit_flush_trx_cache -> THD::binlog_flush_pending_rows_event -> MYSQL_BIN_LOG::flush_and_set_pending_rows_event -> MYSQL_BIN_LOG::write_transaction_to_binlog call chain). The committing is currently strongly sequenced. The transactions are organized into groups, and when the latest transaction is acknowledged on the first commit phase, they all are being committed by leader in a chosen order. However, even here we can write the transactions in parallel, preserving the order. Andrei also claims, that the order can be potentially restored on the replication side. Anyway we technically can't send the transaction to the replication slave before the binlog flush&fsync, nevertheless the data will still be preserved in the volatile append cache. There was MDEV-20925 <https://jira.mariadb.org/browse/MDEV-20925> to store the transaction length in the event, but it unfortunately was rejected: > I will be closing this issue because we have COMMIT/ROLLBACK query log event > in the end of transaction , whose size is difficult to determine , > So current plan is to do MDEv-19687 without transaction length. The replication team decided to calculate the transaction sizes during receiving the data form the io, and then to store it in the hash, and no protocol modifications would be required then. I suggested to buffer the transaction separately, and then push it into the relay log in the data frame, storing the length. Then, we have MDEV-19687 <https://jira.mariadb.org/browse/MDEV-19687>, which was the supertask for MDEV-20925 <https://jira.mariadb.org/browse/MDEV-20925>. The aim is to parallelize the replication on the slave side. They are implementing their own parallel circular buffer with single-writer, multiple-reader use case: the parallel workers are going to pick out the transaction data, and resolve the commit order later. Sergei petrunya, you've been questioning: > I see SEQ_READ_APPEND is only used for Relay Log on the slave. Afaiu, the > relay > log has only only one producer, the network io thread. Andrei clarified, that there can actually be multiple sources. However they are just going to create a working queue for each source. =============================================================== To underline, the rationale is changed for this IO_CACHE improvement: * The replication relay log is not the use-case anymore, since a separate, much simplified circular buffer is going to be implemented (AFAIU, Sachin is in the middle of the progress of MDEV-19687 <https://jira.mariadb.org/browse/MDEV-19687>). * The binlog commit is instead a good case! Most likely I have forgotten about it, when I was writing the rationale. But anyway it wasn't clear for me, can the transactions be written in parallel there. On the reader side, the network replication sender should not block the committing process by reading from the binlog. Regards, Nikita

1 0

[Maria-developers] Request for a discusison: A fine-grained concurrent ring buffer mode for IO_CACHE
by Nikita Malyavin 19 May '21

19 May '21

Hello, dear MariaDB community! I am glad to present the design for IO_CACHE's SEQ_READ_APPEND mode concurrency improvements. Jira task is MDEV-24676. A student, Vladislav Kakurin, has applied to GSoC for this task, and has passed, so he will be engaged in implementing it this summer. The discussion is very important here, since I could make the mistakes, and Vladislav can miss them, but the success of the task completion may depend on it. We're hoping to succeed in implementing the multi-producer till GSoC end. So please don't hesitate to ask for clarifications. Note that reader/writer and producer/consumer pairs are used interchangeably in the text. The text is attached as PDF, but for the sake of convenient quoting it is additionally embodied in the email with shrinked markup and no illustrations. Sincerely, Nikita Malyavin Software Engineer of MariaDB ==================================================================== A fine-grained concurrent ring buffer mode for IO_CACHE 1. Rationale. --------------------------------------------------- Current implementation of IO_CACHE’s SEQ_READ_APPEND mode behaves coarsely grained on its write buffer: every read and every write is protected by append_buffer_lock. int _my_b_seq_read(IO_CACHE *info, ...) { lock_append_buffer(info); ... // read logic unlock_append_buffer(info); return Count ? 1 : 0; } int my_b_append(IO_CACHE *info, ...) { lock_append_buffer(info); ... // append logic unlock_append_buffer(info); return 0; } Despite the separate read buffer is read-only, and therefore is accessed wait-free, the write buffer can have a contention with medium-sized transactions. The design described hereafter is going to solve this issue, and an extension for a parallel multi-producer workflow is additionally provided. Furthermore, the API extension for multi-producer approach support is proposed, and the multi-consumerness is discussed. 2. The single-producer, single-consumer case. --------------------------------------------------- Idea. The memcpy operations of consumer and producer never overlap, therefore they can be freed of locks. Overflow and emptiness We cannot begin writing in the area still involved in reading. Therefore, reader should not update the pointers before it finishes reading. This means that we should lock in the beginning to atomically read the data, and in the end, to write the new reader data. The same for vice-versa, we cannot read from the area still involved into writing, therefore a read should finish with EMPTY error (currently _my_b_seq_read just returns 1) When we reach a “buffer is full” condition, we can flip the read and write (append) buffers, if we were reading from an append buffer. Otherwise, the append buffer is flushed. The algorithm. The following pseudocode will describe the single-consumer, single-producer approach. It is assumed that reading from the read buffer is handled in the usual way. io->total_size and io->read_buffer are considered to be accessed atomically. io_err_t read(IO_CACHE *io, uchar *buffer, size_t sz) { if (sz > io->total_size) return E_IO_EMPTY; uchar *read_buffer = io->read_buffer; if (io->read_pos points to read_buffer) sz_read = read_from_read_buffer(io, buffer, sz); buffer += sz_read; sz -= sz_read; io->total_size -= sz_read; if (sz == 0) return 0; // else copy from append buffer lock(io->append_buffer_lock); // copy the local variables uchar *read_pos = io->read_pos; uchar *read_buffer = io->read_buffer; uchar *append_start_pos = i->append_start_pos; uchar *append_size = io->append_size; uchar *append_pos = io->append_pos; // etc, if needed unlock(io->append_buffer_lock); read from append buffer; lock(io->append_buffer_lock); // update the variables io->append_size -= sz; io->append_start_pos += sz; if (i->append_start_pos >= io->append_buf + io->cache_size) io->append_start_pos -= io->cache_size; unlock(io->append_buffer_lock); io->total_size -= sz; } The first read()’s part tries to read from a read-only buffer. If it’s empty, it moves the effort to a volatile append buffer. All the metadata is copied in the first critical section, before the data copying, to the stack. It is updated back in the second critical section, after the data copying. io_err_t write(IO_CACHE *io, uchar *buffer, size_t sz) { lock(io->append_buffer_lock); if (append_buffer is full and io->total_size <= io->append_size) swap(io->append_buffer, io->read_buffer); else flush the append buffer if needed; write to disk directly, if the data is too large; uchar *write_pos = io->write_pos; unlock(io->append_buffer_lock); write to append buffer; lock(io->append_buffer_lock); io->write_pos = new_write_pos; unlock(io->append_buffer_lock); io->total_size += sz; } The important note here is that we access io->read_buffer in the reader’s thread without the lock (the accesses are marked bold). However this access happens only once in the beginning and is safe: Only writer changes read_buffer. The writer can change it only once during one read() if io->read_buffer is considered reads-only, then it will not flip again, and continue to be consistent, until io->total_size is changed: io->total_size -= sz_read; Then the lock happens. It should be fine to read from a flipped buffer on that stage. 3. Multi-producer concurrency --------------------------------------------------- Idea. Writes start from io->write_start, which is to update immediately. Reads are possible only until io->read_end, which is updated, as soon as writes are finished. Medium-grained approach io->write_start is updated immediately to allow parallel writes. However, we cannot update io->read_end immediately after this thread’s write ends, because earlier writes can still be in progress. We should wait for them i.e. we wait while (io->read_end != local_read_end) Algorithm (medium-grained). Medium-grained approach will modify write() function as follows (the changed lines and locks are bolded): io_err_t write(IO_CACHE *io, uchar *buffer, size_t sz) { lock(io->append_buffer_lock); if (buffer flip of flush is needed) wait until all the writes are finished; if (append_buffer is full && io->write_total_size <= io->append_size) swap(io->append_buffer, io->read_buffer); else flush the append buffer if needed; write to disk directly, if the data is too large; uchar *local_write_start = io->write_start; io->write_total_size += sz; io->write_start += sz; if (io->write_start > io->append_buffer + io->cache_size) io->write_start -= io->cache_size; unlock(io->append_buffer_lock); write to append buffer; lock(io->write_event_lock) while(local_write_start != io->read_end) cond_wait(io->write_event, io->write_event_lock); unlock(io->write_event_lock) lock(io->append_buffer_lock); io->read_end = new_read_end; unlock(io->append_buffer_lock); cond_signal(io->write_event); io->total_size += sz; } The read function should be modified mostly cosmetically. Fine graining. The writers are still waiting for each other’s finish. The approach described here defers waiting through helping pattern by introducing progress slots. Each time a writer begins progress it allocates a slot in the dedicated (fixed size) array. When the writer finishes its job, it checks whether it is the leftmost one (relative to its read_end value. If it is, it updates read_end for itself, and for all the consecutive writers already finished. The slot allocation will be controlled by a semaphore to prevent overflow. Therefore, only a fixed number of producers can work simultaneously. The slot array is made of elements of private cache_slot_t structure: struct cache_slot_t { bool vacant: 1; bool finished: 1; uint next: size_bits(uint) - 2; uint pos; } The slot is acquired whenever a write begins by searching an array cell with vacant=1. When it’s found, vacant = 0, finished = 0 is set. The last_slot variable holds the slot index for the latest write. slots[ last_slot].next is set to a new index, and last_slot itself is updated. The following example demonstrates how the slots work: +-------------------------------------------------------------+ | | write1 | write2 | write3 | | +-------------------------------------------------------------+ A B C D +-------------------------------------------------+ | vacant=0 | vacant=1 | vacant=0 | vacant=0 | | finished=0 | | finished=1 | finished=1 | | next=2 | | next=3 | next=? | | pos=A | | pos=B | pos=C | +-------------------------------------------------+ there were three writes currently running in parallel. write2 and write3 are finished, but write1 is still running. When it finishes, it will hop through slot.next while vacant==0 and finished==1 and pos != io->write_start. Therefore, read_end will be updated to C if no other write will begin in parallel. If another write begins in parallel before write1 finishes, it allocates slots[1] and sets pos=D. slots[3].next would be set to 1, and last_slot will be updated from 3 to 1. The slot run through expected complexity is O(1). The proof for acquisition is however not that obvious to prove the same, and no effort was spent for proving it (It’s only obvious that it’s O(slots)). 4. Arbitrary data sources support --------------------------------------------------- The widely spread use-case is pouring from another IO_CACHE source (like a statement or transaction cache). The operation may require several consecutive write() calls with an external lock: lock(write_lock); uchar buffer[SIZE]; while(cache_out is not empty) { read(ceche_out, buffer, SIZE); write(cache_in, buffer, SIZE); } unlock(write_lock); This case destroys all the parallel design described. However, let’s make api changes to allow blocks of predicted size be written in parallel: /** Allocates the slot of a requested size for a writer. Returns new slot id. */ slot_id_t append_allocate(IO_CAHCE*, size_t block_size); /** Frees the slot and propogates the data to be available for reading */ void append_commit(IO_CACHE*, slot_id_t); These two functions just decompose our write() function: append_allocate would include the first critical section and append_commit would include the second one. The use-case will be changed slightly: slot_id_t slot = append_allocate(cache_out, append_tell(cache_in)); uchar buffer[SIZE]; while(cache_out is not empty) { read(ceche_out, buffer, SIZE); write(cache_in, buffer, SIZE); } append_commit(cache_out, slot); 5. Multi-consumerness --------------------------------------------------- We currently have no cases with several readers working in parallel in SEQ_READ_APPEND mode. It is only used by the replication thread to read out the log, where it is delegated to a dedicated worker. The first problem is that parallel readout would require additional coordination -- the order of event application can be important. Another problem is that a variable-sized blocks require at least two consecutive reads if the structure is not known. If the length is stored, it can be read out with exactly two reads (first reads length, second reads the body). The slot allocation strategy can be applied, and api can be added similar to a new write api: /** lock the cache and allocate the read slot */ slot_id_t read_allocate_lock(IO_CACHE*); /** Allocate a read zone of the requested size and unlock the cache */ void read_allocate_unlock(IO_CACHE*, slot_id_t, size_t size); /** Finish reading; deallocate the read slot */ viod read_commit(IO_CACHE*, slot_id_t); Reading api needs one function more than writing api -- the allocation is split on two phases: locking phase (to compute the block length), and the actual requesting phase. This approach has several disadvantages: 1. The read buffer access is no longer lock-free 2. read_allocate_lock leaves the IO_CACHE in a locked state, which can be potentially misused. Additionally, two SX locks can be used (one for readers and one for writers) for extra parallelism.

2 2

[Maria-developers] Request for a discusison: A fine-grained concurrent ring buffer mode for IO_CACHE
by Nikita Malyavin 18 May '21

18 May '21

Hello, dear MariaDB community! I am glad to present the design for IO_CACHE's SEQ_READ_APPEND mode concurrency improvements. Jira task is MDEV-24676. A student, Vladislav Kakurin, has applied to GSoC for this task, and has passed, so he will be engaged in implementing it this summer. The discussion is very important here, since I could make the mistakes, and Vladislav can miss them, but the success of the task completion may depend on it. We're hoping to succeed in implementing the multi-producer till GSoC end. So please don't hesitate to ask for clarifications. Note that reader/writer and producer/consumer pairs are used interchangeably in the text. The text is attached as PDF, but for the sake of convenient quoting it is additionally embodied in the email with shrinked markup and no illustrations. Sincerely, Nikita Malyavin Software Engineer of MariaDB ==================================================================== A fine-grained concurrent ring buffer mode for IO_CACHE 1. Rationale. --------------------------------------------------- Current implementation of IO_CACHE’s SEQ_READ_APPEND mode behaves coarsely grained on its write buffer: every read and every write is protected by append_buffer_lock. int _my_b_seq_read(IO_CACHE *info, ...) { lock_append_buffer(info); ... // read logic unlock_append_buffer(info); return Count ? 1 : 0; } int my_b_append(IO_CACHE *info, ...) { lock_append_buffer(info); ... // append logic unlock_append_buffer(info); return 0; } Despite the separate read buffer is read-only, and therefore is accessed wait-free, the write buffer can have a contention with medium-sized transactions. The design described hereafter is going to solve this issue, and an extension for a parallel multi-producer workflow is additionally provided. Furthermore, the API extension for multi-producer approach support is proposed, and the multi-consumerness is discussed. 2. The single-producer, single-consumer case. --------------------------------------------------- Idea. The memcpy operations of consumer and producer never overlap, therefore they can be freed of locks. Overflow and emptiness We cannot begin writing in the area still involved in reading. Therefore, reader should not update the pointers before it finishes reading. This means that we should lock in the beginning to atomically read the data, and in the end, to write the new reader data. The same for vice-versa, we cannot read from the area still involved into writing, therefore a read should finish with EMPTY error (currently _my_b_seq_read just returns 1) When we reach a “buffer is full” condition, we can flip the read and write (append) buffers, if we were reading from an append buffer. Otherwise, the append buffer is flushed. The algorithm. The following pseudocode will describe the single-consumer, single-producer approach. It is assumed that reading from the read buffer is handled in the usual way. io->total_size and io->read_buffer are considered to be accessed atomically. io_err_t read(IO_CACHE *io, uchar *buffer, size_t sz) { if (sz > io->total_size) return E_IO_EMPTY; uchar *read_buffer = io->read_buffer; if (io->read_pos points to read_buffer) sz_read = read_from_read_buffer(io, buffer, sz); buffer += sz_read; sz -= sz_read; io->total_size -= sz_read; if (sz == 0) return 0; // else copy from append buffer lock(io->append_buffer_lock); // copy the local variables uchar *read_pos = io->read_pos; uchar *read_buffer = io->read_buffer; uchar *append_start_pos = i->append_start_pos; uchar *append_size = io->append_size; uchar *append_pos = io->append_pos; // etc, if needed unlock(io->append_buffer_lock); read from append buffer; lock(io->append_buffer_lock); // update the variables io->append_size -= sz; io->append_start_pos += sz; if (i->append_start_pos >= io->append_buf + io->cache_size) io->append_start_pos -= io->cache_size; unlock(io->append_buffer_lock); io->total_size -= sz; } The first read()’s part tries to read from a read-only buffer. If it’s empty, it moves the effort to a volatile append buffer. All the metadata is copied in the first critical section, before the data copying, to the stack. It is updated back in the second critical section, after the data copying. io_err_t write(IO_CACHE *io, uchar *buffer, size_t sz) { lock(io->append_buffer_lock); if (append_buffer is full and io->total_size <= io->append_size) swap(io->append_buffer, io->read_buffer); else flush the append buffer if needed; write to disk directly, if the data is too large; uchar *write_pos = io->write_pos; unlock(io->append_buffer_lock); write to append buffer; lock(io->append_buffer_lock); io->write_pos = new_write_pos; unlock(io->append_buffer_lock); io->total_size += sz; } The important note here is that we access io->read_buffer in the reader’s thread without the lock (the accesses are marked bold). However this access happens only once in the beginning and is safe: Only writer changes read_buffer. The writer can change it only once during one read() if io->read_buffer is considered reads-only, then it will not flip again, and continue to be consistent, until io->total_size is changed: io->total_size -= sz_read; Then the lock happens. It should be fine to read from a flipped buffer on that stage. 3. Multi-producer concurrency --------------------------------------------------- Idea. Writes start from io->write_start, which is to update immediately. Reads are possible only until io->read_end, which is updated, as soon as writes are finished. Medium-grained approach io->write_start is updated immediately to allow parallel writes. However, we cannot update io->read_end immediately after this thread’s write ends, because earlier writes can still be in progress. We should wait for them i.e. we wait while (io->read_end != local_read_end) Algorithm (medium-grained). Medium-grained approach will modify write() function as follows (the changed lines and locks are bolded): io_err_t write(IO_CACHE *io, uchar *buffer, size_t sz) { lock(io->append_buffer_lock); if (buffer flip of flush is needed) wait until all the writes are finished; if (append_buffer is full && io->write_total_size <= io->append_size) swap(io->append_buffer, io->read_buffer); else flush the append buffer if needed; write to disk directly, if the data is too large; uchar *local_write_start = io->write_start; io->write_total_size += sz; io->write_start += sz; if (io->write_start > io->append_buffer + io->cache_size) io->write_start -= io->cache_size; unlock(io->append_buffer_lock); write to append buffer; lock(io->write_event_lock) while(local_write_start != io->read_end) cond_wait(io->write_event, io->write_event_lock); unlock(io->write_event_lock) lock(io->append_buffer_lock); io->read_end = new_read_end; unlock(io->append_buffer_lock); cond_signal(io->write_event); io->total_size += sz; } The read function should be modified mostly cosmetically. Fine graining. The writers are still waiting for each other’s finish. The approach described here defers waiting through helping pattern by introducing progress slots. Each time a writer begins progress it allocates a slot in the dedicated (fixed size) array. When the writer finishes its job, it checks whether it is the leftmost one (relative to its read_end value. If it is, it updates read_end for itself, and for all the consecutive writers already finished. The slot allocation will be controlled by a semaphore to prevent overflow. Therefore, only a fixed number of producers can work simultaneously. The slot array is made of elements of private cache_slot_t structure: struct cache_slot_t { bool vacant: 1; bool finished: 1; uint next: size_bits(uint) - 2; uint pos; } The slot is acquired whenever a write begins by searching an array cell with vacant=1. When it’s found, vacant = 0, finished = 0 is set. The last_slot variable holds the slot index for the latest write. slots[ last_slot].next is set to a new index, and last_slot itself is updated. The following example demonstrates how the slots work: +-------------------------------------------------------------+ | | write1 | write2 | write3 | | +-------------------------------------------------------------+ A B C D +-------------------------------------------------+ | vacant=0 | vacant=1 | vacant=0 | vacant=0 | | finished=0 | | finished=1 | finished=1 | | next=2 | | next=3 | next=? | | pos=A | | pos=B | pos=C | +-------------------------------------------------+ there were three writes currently running in parallel. write2 and write3 are finished, but write1 is still running. When it finishes, it will hop through slot.next while vacant==0 and finished==1 and pos != io->write_start. Therefore, read_end will be updated to C if no other write will begin in parallel. If another write begins in parallel before write1 finishes, it allocates slots[1] and sets pos=D. slots[3].next would be set to 1, and last_slot will be updated from 3 to 1. The slot run through expected complexity is O(1). The proof for acquisition is however not that obvious to prove the same, and no effort was spent for proving it (It’s only obvious that it’s O(slots)). 4. Arbitrary data sources support --------------------------------------------------- The widely spread use-case is pouring from another IO_CACHE source (like a statement or transaction cache). The operation may require several consecutive write() calls with an external lock: lock(write_lock); uchar buffer[SIZE]; while(cache_out is not empty) { read(ceche_out, buffer, SIZE); write(cache_in, buffer, SIZE); } unlock(write_lock); This case destroys all the parallel design described. However, let’s make api changes to allow blocks of predicted size be written in parallel: /** Allocates the slot of a requested size for a writer. Returns new slot id. */ slot_id_t append_allocate(IO_CAHCE*, size_t block_size); /** Frees the slot and propogates the data to be available for reading */ void append_commit(IO_CACHE*, slot_id_t); These two functions just decompose our write() function: append_allocate would include the first critical section and append_commit would include the second one. The use-case will be changed slightly: slot_id_t slot = append_allocate(cache_out, append_tell(cache_in)); uchar buffer[SIZE]; while(cache_out is not empty) { read(ceche_out, buffer, SIZE); write(cache_in, buffer, SIZE); } append_commit(cache_out, slot); 5. Multi-consumerness --------------------------------------------------- We currently have no cases with several readers working in parallel in SEQ_READ_APPEND mode. It is only used by the replication thread to read out the log, where it is delegated to a dedicated worker. The first problem is that parallel readout would require additional coordination -- the order of event application can be important. Another problem is that a variable-sized blocks require at least two consecutive reads if the structure is not known. If the length is stored, it can be read out with exactly two reads (first reads length, second reads the body). The slot allocation strategy can be applied, and api can be added similar to a new write api: /** lock the cache and allocate the read slot */ slot_id_t read_allocate_lock(IO_CACHE*); /** Allocate a read zone of the requested size and unlock the cache */ void read_allocate_unlock(IO_CACHE*, slot_id_t, size_t size); /** Finish reading; deallocate the read slot */ viod read_commit(IO_CACHE*, slot_id_t); Reading api needs one function more than writing api -- the allocation is split on two phases: locking phase (to compute the block length), and the actual requesting phase. This approach has several disadvantages: 1. The read buffer access is no longer lock-free 2. read_allocate_lock leaves the IO_CACHE in a locked state, which can be potentially misused. Additionally, two SX locks can be used (one for readers and one for writers) for extra parallelism.

1 0

Re: [Maria-developers] eebebe8ef75: MDEV-23842 Atomic RENAME TABLE
by Sergei Golubchik 16 May '21

16 May '21

Hi, Monty! On May 10, Michael Widenius wrote: > revision-id: eebebe8ef75 (mariadb-10.6.0-77-geebebe8ef75) > parent(s): 5a602e2ce7f > author: Michael Widenius <michael.widenius(a)gmail.com> > committer: Michael Widenius <michael.widenius(a)gmail.com> > timestamp: 2021-05-04 21:09:10 +0300 > message: > > MDEV-23842 Atomic RENAME TABLE > diff --git a/client/mysqltest.cc b/client/mysqltest.cc > index 350c55edee2..133d1f76839 100644 > --- a/client/mysqltest.cc > +++ b/client/mysqltest.cc > @@ -8095,9 +8095,10 @@ void handle_error(struct st_command *command, > const char *err_sqlstate, DYNAMIC_STRING *ds) > { > int i; > - > DBUG_ENTER("handle_error"); > > + var_set_int("$errno", err_errno); there was $mysql_errno already, was is not something you could've used? > + > command->used_replace= 1; > if (command->require_file) > { > diff --git a/mysql-test/suite/atomic/rename_case.result b/mysql-test/suite/atomic/rename_case.result > new file mode 100644 > index 00000000000..4b58c555cf2 > --- /dev/null > +++ b/mysql-test/suite/atomic/rename_case.result > @@ -0,0 +1,52 @@ > +create database test2; > +# > +# Testing rename error in different places > +# > +create table t1 (a int); > +create table T2 (b int); > +create table t3 (c int); > +create table T4 (d int); > +insert into t1 values(1); > +insert into T2 values(2); > +insert into t3 values(3); > +insert into T4 values(4); > +create temporary table tmp1 (a int); > +create temporary table tmp2 (b int); > +create temporary table tmp3 (c int); > +create temporary table tmp4 (d int); > +insert into tmp1 values(11); > +insert into tmp2 values(22); > +insert into tmp3 values(33); > +insert into tmp4 values(44); > +rename table t3 to T4, t1 to t5, T2 to t1, t5 to T2; > +ERROR 42S01: Table 'T4' already exists > +rename table t1 to t5, t3 to T4, T2 to t1, t5 to T2; > +ERROR 42S01: Table 'T4' already exists > +rename table t1 to t5, T2 to t1, t3 to T4, t5 to T2; > +ERROR 42S01: Table 'T4' already exists > +rename table t1 to t5, T2 to t1, t5 to T2, t3 to T4; > +ERROR 42S01: Table 'T4' already exists > +# Try failed rename using two databases > +rename table test.t1 to test2.t5, test.T2 to test.t1, t5 to test.T2; > +ERROR 42S02: Table 'test.t5' doesn't exist > +select t1.a+T2.b+t3.c+T4.d from t1,T2,t3,T4; better do select t1.a,T2.b,t3.c,T4.d from t1,T2,t3,T4; same one row, but it can detect when tables are swapped. or if you need one scalar value (you do below) it could ne select CONCAT(t1.a,T2.b,t3.c,T4.d) from t1,T2,t3,T4; > +t1.a+T2.b+t3.c+T4.d > +10 > +select * from t5; > +ERROR 42S02: Table 'test.t5' doesn't exist > +T2.MYD > +T2.MYI > +T2.frm > +T4.MYD > +T4.MYI > +T4.frm > +db.opt > +t1.MYD > +t1.MYI > +t1.frm > +t3.MYD > +t3.MYI > +t3.frm > +# Cleanup > +drop table t1,T2,t3,T4; > +drop database test2; > diff --git a/sql/sql_class.h b/sql/sql_class.h > index 0dcb2e04ba4..5ae0866672d 100644 > --- a/sql/sql_class.h > +++ b/sql/sql_class.h > @@ -2835,6 +2835,7 @@ class THD: public THD_count, /* this must be first */ > > #ifndef MYSQL_CLIENT > binlog_cache_mngr * binlog_setup_trx_data(); > + ulonglong binlog_xid; /* binlog request storing of xid */ "Tell binlog to store thd->query_id in the next Query_log_event" > > /* > Public interface to write RBR events to the binlog > diff --git a/sql/log_event_client.cc b/sql/log_event_client.cc > index 6b5d71348e1..733fc14c697 100644 > --- a/sql/log_event_client.cc > +++ b/sql/log_event_client.cc > @@ -1822,7 +1822,7 @@ bool Query_log_event::print_query_header(IO_CACHE* file, > my_b_printf(file, > "\t%s\tthread_id=%lu\texec_time=%lu\terror_code=%d\n", > get_type_str(), (ulong) thread_id, (ulong) exec_time, > - error_code)) > + error_code, (ulong) xid)) huh? you didn't add "\nxid=%lu" to the format string > goto err; > } > > diff --git a/sql/debug_sync.h b/sql/debug_sync.h > index 3b8aa8815e1..4e3e10fcc51 100644 > --- a/sql/debug_sync.h > +++ b/sql/debug_sync.h > @@ -53,4 +53,12 @@ static inline bool debug_sync_set_action(THD *, const char *, size_t) > { return false; } > #endif /* defined(ENABLED_DEBUG_SYNC) */ > > +#ifndef DBUG_OFF > +void debug_crash_here(const char *keyword); > +bool debug_simulate_error(const char *keyword, uint error); > +#else > +#define debug_crash_here(A) do { } while(0) > +#define debug_simulate_error(A, B) 0 > +#endif this doesn't belong to debug_sync.h or debug_sync.cc put it, hmm, may be in sql_priv.h ? or sql_class.h/sql_class.cc > + > #endif /* DEBUG_SYNC_INCLUDED */ > diff --git a/sql/debug_sync.cc b/sql/debug_sync.cc > index 1fbb15592a4..f523e22f6ce 100644 > --- a/sql/debug_sync.cc > +++ b/sql/debug_sync.cc > @@ -1628,3 +1628,74 @@ bool debug_sync_set_action(THD *thd, const char *action_str, size_t len) > /* prevent linker/lib warning about file without public symbols */ > int debug_sync_dummy; > #endif /* defined(ENABLED_DEBUG_SYNC) */ > + > + > +/** > + Debug utility to do crash after a set number of executions > + > + The user variable, either @debug_crash_counter or @debug_error_counter, this is very hackish. better to use @@debug_dbug_counter, something like that. also it could be global for an extra benefit if you'd need to crash a non-user thread. > + is decremented each time debug_crash() or debug_simulate_error is called > + if the keyword is set with @@debug_push, like > + @@debug_push="d+frm_data_type_info_emulate" > + > + If the variable is not set or is not an integer it will be ignored. > +*/ > + > +#ifndef DBUG_OFF > + > +static const LEX_CSTRING debug_crash_counter= > +{ STRING_WITH_LEN("debug_crash_counter") }; > +static const LEX_CSTRING debug_error_counter= > +{ STRING_WITH_LEN("debug_error_counter") }; > + > +static bool debug_decrement_counter(const LEX_CSTRING *name) > +{ > + THD *thd= current_thd; > + user_var_entry *entry= (user_var_entry*) > + my_hash_search(&thd->user_vars, (uchar*) name->str, name->length); > + if (!entry || entry->type != INT_RESULT || ! entry->value) > + return 0; > + (*(ulonglong*) entry->value)= (*(ulonglong*) entry->value)-1; > + return !*(ulonglong*) entry->value; > +} > + > +void debug_crash_here(const char *keyword) > +{ > + DBUG_ENTER("debug_crash_here"); > + DBUG_PRINT("enter", ("keyword: %s", keyword)); > + > + DBUG_EXECUTE_IF(keyword, > + if (debug_decrement_counter(&debug_crash_counter)) > + { > + my_printf_error(ER_INTERNAL_ERROR, > + "Crashing at %s", > + MYF(ME_ERROR_LOG | ME_NOTE), keyword); > + DBUG_SUICIDE(); > + }); > + DBUG_VOID_RETURN; > +} > + > +/* > + This can be used as debug_counter to simulate an error at a specific > + position. > + > + Typical usage would be > + if (debug_simualte_error("keyword")) > + error= 1; > +*/ > + > +bool debug_simulate_error(const char *keyword, uint error) > +{ > + DBUG_ENTER("debug_crash_here"); > + DBUG_PRINT("enter", ("keyword: %s", keyword)); > + DBUG_EXECUTE_IF(keyword, > + if (debug_decrement_counter(&debug_error_counter)) > + { > + my_printf_error(error, > + "Simulating error for '%s'", > + MYF(ME_ERROR_LOG), keyword); > + DBUG_RETURN(1); > + }); > + DBUG_RETURN(0); > +} > +#endif /* DBUG_OFF */ > diff --git a/sql/log_event_server.cc b/sql/log_event_server.cc > index cca05f1fcca..0cf037e3a6f 100644 > --- a/sql/log_event_server.cc > +++ b/sql/log_event_server.cc > @@ -1294,6 +1294,15 @@ bool Query_log_event::write() > int3store(start, when_sec_part); > start+= 3; > } > + > + /* xid's is used with ddl_log handling */ it's not really an XID, it's a query_id or the DDL statetent. XID is a combination of a 64-byte global transaction identifier and a 64-byte branch qualifier. It would be much less confusing not to pretend that the query_id is an "xid" is some shape or form. Better to rename it to, say ddl_query_id. Or even ddl_id, if you like it short. > + if (thd && thd->binlog_xid) > + { > + *start++= Q_XID; > + int8store(start, thd->query_id); > + start+= 8; > + } > + > /* > NOTE: When adding new status vars, please don't forget to update > the MAX_SIZE_LOG_EVENT_STATUS in log_event.h and update the function > diff --git a/sql/log.cc b/sql/log.cc > index 17c599627cb..6e0a9706ec8 100644 > --- a/sql/log.cc > +++ b/sql/log.cc > @@ -10654,6 +10669,8 @@ int TC_LOG_BINLOG::recover(LOG_INFO *linfo, const char *last_log_name, > { > if (ha_recover(&xids)) > goto err2; > + if (ddl_log_close_binlogged_events(&xids)) your xids and normal Xid_log_event xids are from two different non-intersecting sets and have even different semantics. It'd be safer not to have them in the same hash. > + goto err2; > > free_root(&mem_root, MYF(0)); > my_hash_free(&xids); > diff --git a/sql/sql_view.cc b/sql/sql_view.cc > index 5df5ecb76a9..e6d726b30d7 100644 > --- a/sql/sql_view.cc > +++ b/sql/sql_view.cc > @@ -2231,16 +2234,25 @@ mysql_rename_view(THD *thd, > (uchar*)&view_def, view_parameters)) > { > /* restore renamed view in case of error */ > - rename_in_schema_file(thd, new_db->str, new_name->str, view->db.str, > - view->table_name.str); > + rename_in_schema_file(thd, new_db->str, new_name->str, old_db->str, > + old_name->str); > goto err; > } > - } else > + } > + else > DBUG_RETURN(1); > > /* remove cache entries */ > - query_cache_invalidate3(thd, view, 0); > - sp_cache_invalidate(); > + { > + char key[NAME_LEN*2+1], *ptr; > + memcpy(key, old_db->str, old_db->length); > + ptr= key+ old_db->length; > + *ptr++= 0; > + memcpy(key, old_name->str, old_name->length); > + ptr= key+ old_db->length; > + *ptr++= 0; there's tdc_create_key for that. > + query_cache.invalidate(thd, key, (size_t) (ptr-key), 0); > + } > error= FALSE; > > err: > diff --git a/sql/ddl_log.h b/sql/ddl_log.h > index 0747699cd85..316e6708f22 100644 > --- a/sql/ddl_log.h > +++ b/sql/ddl_log.h > @@ -24,6 +24,9 @@ > enum ddl_log_entry_code > { > /* > + DDL_LOG_UNKOWN s/UNKOWN/UNKNOWN/ > + Here mainly to detect blocks that are all zero > + > DDL_LOG_EXECUTE_CODE: > This is a code that indicates that this is a log entry to > be executed, from this entry a linked list of log entries > @@ -34,48 +37,95 @@ enum ddl_log_entry_code > DDL_IGNORE_LOG_ENTRY_CODE: > An entry that is to be ignored > */ > - DDL_LOG_EXECUTE_CODE = 'e', > - DDL_LOG_ENTRY_CODE = 'l', > - DDL_IGNORE_LOG_ENTRY_CODE = 'i' > + DDL_LOG_UNKNOWN= 0, > + DDL_LOG_EXECUTE_CODE= 1, > + DDL_LOG_ENTRY_CODE= 2, > + DDL_IGNORE_LOG_ENTRY_CODE= 3, > + DDL_LOG_ENTRY_CODE_LAST= 4 > }; > > + > +/* > + When adding things below, also add an entry to ddl_log_entry_phases in > + ddl_log.cc > +*/ > + > enum ddl_log_action_code > { > /* > The type of action that a DDL_LOG_ENTRY_CODE entry is to > perform. > - DDL_LOG_DELETE_ACTION: > - Delete an entity > - DDL_LOG_RENAME_ACTION: > - Rename an entity > - DDL_LOG_REPLACE_ACTION: > - Rename an entity after removing the previous entry with the > - new name, that is replace this entry. > - DDL_LOG_EXCHANGE_ACTION: > - Exchange two entities by renaming them a -> tmp, b -> a, tmp -> b. > */ > - DDL_LOG_DELETE_ACTION = 'd', > - DDL_LOG_RENAME_ACTION = 'r', > - DDL_LOG_REPLACE_ACTION = 's', > - DDL_LOG_EXCHANGE_ACTION = 'e' > + DDL_LOG_UNKNOWN_ACTION= 0, > + > + /* Delete a .frm file or a table in the partition engine */ > + DDL_LOG_DELETE_ACTION= 1, > + > + /* Rename a .frm fire a table in the partition engine */ > + DDL_LOG_RENAME_ACTION= 2, > + > + /* > + Rename an entity after removing the previous entry with the > + new name, that is replace this entry. > + */ > + DDL_LOG_REPLACE_ACTION= 3, > + > + /* Exchange two entities by renaming them a -> tmp, b -> a, tmp -> b */ > + DDL_LOG_EXCHANGE_ACTION= 4, > + /* > + log do_rename(): Rename of .frm file, table, stat_tables and triggers > + */ > + DDL_LOG_RENAME_TABLE_ACTION= 5, > + DDL_LOG_RENAME_VIEW_ACTION= 6, > + DDL_LOG_LAST_ACTION /* End marker */ > }; > > + > +/* Number of phases for each ddl_log_action_code */ > +extern const uchar ddl_log_entry_phases[DDL_LOG_LAST_ACTION]; > + > + > enum enum_ddl_log_exchange_phase { > EXCH_PHASE_NAME_TO_TEMP= 0, > EXCH_PHASE_FROM_TO_NAME= 1, > EXCH_PHASE_TEMP_TO_FROM= 2 > }; > > +enum enum_ddl_log_rename_table_phase { > + DDL_RENAME_PHASE_TRIGGER= 0, > + DDL_RENAME_PHASE_STAT, > + DDL_RENAME_PHASE_TABLE, > +}; > + > +/* > + Setting ddl_log_entry.phase to this has the same effect as setting > + action_type to DDL_IGNORE_LOG_ENTRY_CODE this doesn't make much sense. May be you can add a line or two about why it's needed and one cannot simply set action_type to DDL_IGNORE_LOG_ENTRY_CODE instead? > +*/ > + > +#define DDL_LOG_FINAL_PHASE ((uchar) 0xff) > > typedef struct st_ddl_log_entry > { > - const char *name; > - const char *from_name; > - const char *handler_name; > - const char *tmp_name; > + LEX_CSTRING name; > + LEX_CSTRING from_name; > + LEX_CSTRING handler_name; > + LEX_CSTRING tmp_name; > + LEX_CSTRING db; > + LEX_CSTRING from_db; > + LEX_CSTRING from_handler_name; > + uchar uuid[MY_UUID_SIZE]; // UUID for new frm file > + > + ulonglong xid; // Xid stored in the binary log > uint next_entry; > - uint entry_pos; > - enum ddl_log_entry_code entry_type; > + uint entry_pos; // Set by write_dll_log_entry() > + /* > + unique_id can be used to store a unique number to check current state. > + Currently it is used to store new size of frm file or link to a ddl log > + entry. is that so? I don't see it storing a file version, a new size of the frm or a link, I see it stores some kind of a counter to abort recovery when it was repeated more than DDL_LOG_MAX_RETRY times > + */ > + uint32 unique_id; // Unique id for file version > + uint16 flags; // Flags unique for each command > + enum ddl_log_entry_code entry_type; // Set automatically > enum ddl_log_action_code action_type; > /* > Most actions have only one phase. REPLACE does however have two > @@ -95,17 +145,52 @@ typedef struct st_ddl_log_memory_entry > } DDL_LOG_MEMORY_ENTRY; > > > +typedef struct st_ddl_log_state > +{ > + /* List of ddl log entries */ > + DDL_LOG_MEMORY_ENTRY *list; > + /* One execute entry per list */ > + DDL_LOG_MEMORY_ENTRY *execute_entry; > +} DDL_LOG_STATE; would be nice to have a small comment, explaining what is a "state" and how two lists can represent it. is execute_entry a pointer to a some element from the list? > + > + > +/* These functions are for recovery */ > +bool ddl_log_initialize(); > +void ddl_log_release(); > +bool ddl_log_close_binlogged_events(HASH *xids); > +int ddl_log_execute_recovery(); > + > +/* functions for updating the ddl log */ > bool ddl_log_write_entry(DDL_LOG_ENTRY *ddl_log_entry, > DDL_LOG_MEMORY_ENTRY **active_entry); > + > bool ddl_log_write_execute_entry(uint first_entry, > - bool complete, > - DDL_LOG_MEMORY_ENTRY **active_entry); > -bool ddl_log_increment_phase(uint entry_no); > + DDL_LOG_MEMORY_ENTRY **active_entry); > +bool ddl_log_disable_execute_entry(DDL_LOG_MEMORY_ENTRY **active_entry); > + > +void ddl_log_complete(DDL_LOG_STATE *ddl_log_state); > +void ddl_log_revert(THD *thd, DDL_LOG_STATE *ddl_log_state); > + > +bool ddl_log_update_phase(DDL_LOG_STATE *entry, uchar phase); > +bool ddl_log_update_xid(DDL_LOG_STATE *state, ulonglong xid); > +bool ddl_log_disable_entry(DDL_LOG_STATE *state); > +bool ddl_log_increment_phase(uint entry_pos); > void ddl_log_release_memory_entry(DDL_LOG_MEMORY_ENTRY *log_entry); > bool ddl_log_sync(); > -void ddl_log_release(); > -void ddl_log_execute_recovery(); > bool ddl_log_execute_entry(THD *thd, uint first_entry); > > +void ddl_log_release_entries(DDL_LOG_STATE *ddl_log_state); > +bool ddl_log_rename_table(THD *thd, DDL_LOG_STATE *ddl_state, > + handlerton *hton, > + const LEX_CSTRING *org_db, > + const LEX_CSTRING *org_alias, > + const LEX_CSTRING *new_db, > + const LEX_CSTRING *new_alias); > +bool ddl_log_rename_view(THD *thd, DDL_LOG_STATE *ddl_state, > + const LEX_CSTRING *org_db, > + const LEX_CSTRING *org_alias, > + const LEX_CSTRING *new_db, > + const LEX_CSTRING *new_alias); > + > extern mysql_mutex_t LOCK_gdl; > #endif /* DDL_LOG_INCLUDED */ > diff --git a/sql/sql_rename.cc b/sql/sql_rename.cc > index b7aed97a8a2..2b2f32cf126 100644 > --- a/sql/sql_rename.cc > +++ b/sql/sql_rename.cc > @@ -186,49 +172,43 @@ bool mysql_rename_tables(THD *thd, TABLE_LIST *table_list, bool silent, > /* Add IF EXISTS to binary log */ > thd->variables.option_bits|= OPTION_IF_EXISTS; > } > + > + debug_crash_here("ddl_log_rename_before_binlog"); > + /* > + Store xid in ddl log and binary log so that we can check on ddl recovery > + if the item is in the binary log (and thus the operation was complete > + */ > + thd->binlog_xid= thd->query_id; > + ddl_log_update_xid(&ddl_log_state, thd->binlog_xid); > binlog_error= write_bin_log(thd, TRUE, thd->query(), thd->query_length()); I'd prefer to avoid blowing up THD with arbitrary flags that are only needed rarely in one specific place in the code to pass some argument few stack frames down. Perhaps you can do instead something like bool Query_log_event::write() { ... /* xid's is used with ddl_log handling */ if (ddl_log_entry_incomplete()) { *start++= Q_XID; int8store(start, thd->query_id); start+= 8; } or at least you could replace ulonglong binlog_xid with a one bit state, like thd->server_status & SERVER_STATUS_IN_DDL or with some other bitmap of flags. > + if (binlog_error) > + error= 1; > + thd->binlog_xid= 0; > thd->variables.option_bits= save_option_bits; > + debug_crash_here("ddl_log_rename_after_binlog"); > > if (likely(!binlog_error)) > my_ok(thd); > } > > if (likely(!error)) > + { > query_cache_invalidate3(thd, table_list, 0); > + ddl_log_complete(&ddl_log_state); > + } > + else > + { > + /* Revert the renames of normal tables with the help of the ddl log */ > + ddl_log_revert(thd, &ddl_log_state); > + } > > err: > DBUG_RETURN(error || binlog_error); > } > > > -/* > - reverse table list > - > - SYNOPSIS > - reverse_table_list() > - table_list pointer to table _list > - > - RETURN > - pointer to new (reversed) list > -*/ > -static TABLE_LIST *reverse_table_list(TABLE_LIST *table_list) > -{ > - TABLE_LIST *prev= 0; > - > - while (table_list) > - { > - TABLE_LIST *next= table_list->next_local; > - table_list->next_local= prev; > - prev= table_list; > - table_list= next; > - } > - return (prev); > -} > - > - > static bool > -do_rename_temporary(THD *thd, TABLE_LIST *ren_table, TABLE_LIST *new_table, > - bool skip_error) > +do_rename_temporary(THD *thd, TABLE_LIST *ren_table, TABLE_LIST *new_table) I wonder why we never got any warnings about it > { > LEX_CSTRING *new_alias; > DBUG_ENTER("do_rename_temporary"); > diff --git a/sql/ddl_log.cc b/sql/ddl_log.cc > index bad786b3799..777dfdddbc7 100644 > --- a/sql/ddl_log.cc > +++ b/sql/ddl_log.cc > @@ -22,176 +22,333 @@ > #include "log.h" // sql_print_error() > #include "ddl_log.h" > #include "ha_partition.h" // PAR_EXT > +#include "sql_table.h" // build_table_filename > +#include "sql_statistics.h" // rename_table_in_stats_tables > +#include "sql_view.h" // mysql_rename_view() > +#include "strfunc.h" // strconvert > +#include <mysys_err.h> // EE_LINK > > > /*-------------------------------------------------------------------------- > > MODULE: DDL log > ----------------- > > - This module is used to ensure that we can recover from crashes that occur > - in the middle of a meta-data operation in MySQL. E.g. DROP TABLE t1, t2; > - We need to ensure that both t1 and t2 are dropped and not only t1 and > - also that each table drop is entirely done and not "half-baked". > + This module is used to ensure that we can recover from crashes that > + occur in the middle of a meta-data operation in MySQL. E.g. DROP > + TABLE t1, t2; We need to ensure that both t1 and t2 are dropped and > + not only t1 and also that each table drop is entirely done and not > + "half-baked". > > - To support this we create log entries for each meta-data statement in the > - ddl log while we are executing. These entries are dropped when the > - operation is completed. > + To support this we create log entries for each meta-data statement > + in the ddl log while we are executing. These entries are dropped > + when the operation is completed. > > At recovery those entries that were not completed will be executed. > > There is only one ddl log in the system and it is protected by a mutex > and there is a global struct that contains information about its current > state. > > + DDl recovery after a crash works the following way: DDL or ddl > + > + - ddl_log_initialize() initializes the global global_ddl_log variable > + and opens the binary log if it exists. If it doesn't exists a new one > + is created. it doesn't seem to be opening a binary log. Did you mean ddl log? > + - ddl_log_close_binlogged_events() loops over all log events and checks if > + their xid (stored in the EXECUTE_CODE event) is in the binary log. If xid > + exists in the binary log the entry is marked as finished in the ddl log. > + - After a new binary log is created and is open for new entries, > + ddl_log_execute_recovery() is executed on remaining open events: > + - Loop over all events > + - For each entry with DDL_LOG_ENTRY_CODE execute the remaining phases > + in ddl_log_execute_entry_no_lock() > + > + The ddl_log.log file is created at startup and deleted when server goes down. > + After the final recovery phase is done, the file is truncated. > + > History: > First version written in 2006 by Mikael Ronstrom > - Second version written in 2020 by Monty > + Second version in 2020 by Monty > --------------------------------------------------------------------------*/ > > +#define DDL_LOG_MAGIC_LENGTH 4 > +/* How many times to try to execute a ddl log entry that causes crashes */ > +#define DDL_LOG_MAX_RETRY 3 > + > +uchar ddl_log_file_magic[]= > +{ (uchar) 254, (uchar) 254, (uchar) 11, (uchar) 2 }; > + > +/* Action names for ddl_log_action_code */ > + > +const char *ddl_log_action_name[DDL_LOG_LAST_ACTION]= > +{ > + "Unknown", "partitioning delete", "partitioning rename", > + "partitioning replace", "partitioning exchange", > + "rename table", "rename view" > +}; > + > +/* Number of phases per entry */ > +const uchar ddl_log_entry_phases[DDL_LOG_LAST_ACTION]= > +{ > + 1, 1, 2, 3, 4, 1 > +}; > + > + > struct st_global_ddl_log > { > - /* > - We need to adjust buffer size to be able to handle downgrades/upgrades > - where IO_SIZE has changed. We'll set the buffer size such that we can > - handle that the buffer size was upto 4 times bigger in the version > - that wrote the DDL log. > - */ > - char file_entry_buf[4*IO_SIZE]; > - char file_name_str[FN_REFLEN]; > - char *file_name; > + uchar *file_entry_buf; > DDL_LOG_MEMORY_ENTRY *first_free; > DDL_LOG_MEMORY_ENTRY *first_used; > File file_id; > - uint name_len; > uint num_entries; > + uint name_pos; > uint io_size; > - bool inited; > - bool do_release; > - bool recovery_phase; > - st_global_ddl_log() : inited(false), do_release(false) {} > + bool initialized; > + bool open; > }; > > st_global_ddl_log global_ddl_log; static? > > mysql_mutex_t LOCK_gdl; > > +/* Positions to different data in a ddl log block */ > #define DDL_LOG_ENTRY_TYPE_POS 0 > +/* > + Note that ACTION_TYPE and PHASE_POS must be after each other. > + See update_phase() > +*/ > #define DDL_LOG_ACTION_TYPE_POS 1 > #define DDL_LOG_PHASE_POS 2 > #define DDL_LOG_NEXT_ENTRY_POS 4 > -#define DDL_LOG_NAME_POS 8 > +/* Flags to remember something unique about the query, like if .frm was used */ > +#define DDL_LOG_FLAG_POS 8 > +/* Used to store XID entry that was written to binary log */ > +#define DDL_LOG_XID_POS 10 > +/* Used to store unique uuid from the .frm file */ "unique uuid" is a weird choice of words > +#define DDL_LOG_UUID_POS 18 > +/* ID_POS can be used to store something unique, like file size (4 bytes) */ > +#define DDL_LOG_ID_POS DDL_LOG_UUID_POS + MY_UUID_SIZE > +#define DDL_LOG_END_POS DDL_LOG_ID_POS + 8 > > -#define DDL_LOG_NUM_ENTRY_POS 0 > -#define DDL_LOG_NAME_LEN_POS 4 > -#define DDL_LOG_IO_SIZE_POS 8 > +/* > + Position to where names are stored in the ddl log blocks. The current > + value is stored in the header and can thus be changed if we need more > + space for constants in the header than what is between DDL_LOG_ID_POS and > + DDL_LOG_TMP_NAME_POS. > +*/ > +#define DDL_LOG_TMP_NAME_POS 56 > > +/* Definitions for the ddl log header, the first block in the file */ > +/* IO_SIZE is stored in the header and can thus be changed */ > +#define DDL_LOG_IO_SIZE IO_SIZE > + > +/* Header is stored in positions 0-3 */ > +#define DDL_LOG_IO_SIZE_POS 4 > +#define DDL_LOG_NAME_OFFSET_POS 6 > +/* Sum of the above variables */ > +#define DDL_LOG_HEADER_SIZE 4+2+2 > + > /** > + Sync the ddl log file. > + > + @return Operation status > + @retval FALSE Success > + @retval TRUE Error > +*/ > + > +static bool ddl_log_sync_file() > +{ > + DBUG_ENTER("ddl_log_sync_file"); > + DBUG_RETURN(mysql_file_sync(global_ddl_log.file_id, MYF(MY_WME))); > +} I don't quite understand that, are there cases when you want to sync the ddl log and you *don't know* whether LOCK_gdl is locked or not? because if you know there should be no lock, you'd better use mysql_mutex_assert_not_owner here. > + > +/* Same as above, but ensure we have the LOCK_gdb locked */ that's an interesting Freudian slip > + > +static bool ddl_log_sync_no_lock() > +{ > + DBUG_ENTER("ddl_log_sync_no_lock"); > + > + mysql_mutex_assert_owner(&LOCK_gdl); > + DBUG_RETURN(ddl_log_sync_file()); > +} ... > +/* > + Store and read strings in ddl log buffers > + > + Format is: > + 2 byte: length (not counting end \0) > + X byte: string value of length 'length' > + 1 byte: \0 a bit redundant, don't you think? Why did you do it? > +*/ > + > +static uchar *store_string(uchar *pos, uchar *end, const LEX_CSTRING *str) > { > + uint32 length= (uint32) str->length; > + if (unlikely(pos + 2 + length + 1 > end)) > + { > + DBUG_ASSERT(0); > + return end; // Overflow > } > + > + int2store(pos, length); > + if (likely(length)) > + memcpy(pos+2, str->str, length); > + pos[2+length]= 0; // Store end \0 > + return pos + 2 + length +1; > +} > + > + > +static LEX_CSTRING get_string(uchar **pos, const uchar *end) > { > + LEX_CSTRING tmp; > + uint32 length; > + if (likely(*pos + 3 <= end)) > + { > + length= uint2korr(*pos); > + if (likely(*pos + 2 + length + 1 <= end)) > + { > + char *str= (char*) *pos+2; > + *pos= *pos + 2 + length + 1; > + tmp.str= str; > + tmp.length= length; > + return tmp; > } > + } > + /* > + Overflow on read, should never happen > + Set *pos to end to ensure any future calls also returns empty string > + */ > + DBUG_ASSERT(0); > + *pos= (uchar*) end; > + tmp.str= ""; > + tmp.length= 0; > + return tmp; > } > > > @@ -256,38 +484,32 @@ static uint read_ddl_log_header() > > static void set_global_from_ddl_log_entry(const DDL_LOG_ENTRY *ddl_log_entry) it'd be good to have some comment explaining why you need this function. Couldn't you write directly from the ddl_log_entry? or use a local buffer? why to serialize into a buffer inside global_ddl_log? > { > + uchar *file_entry_buf= global_ddl_log.file_entry_buf, *pos, *end; > + > mysql_mutex_assert_owner(&LOCK_gdl); it would be logical to have LOCK_gdl inside the global_ddl_log struct > - global_ddl_log.file_entry_buf[DDL_LOG_ENTRY_TYPE_POS]= > - (char)DDL_LOG_ENTRY_CODE; > - global_ddl_log.file_entry_buf[DDL_LOG_ACTION_TYPE_POS]= > - (char)ddl_log_entry->action_type; > - global_ddl_log.file_entry_buf[DDL_LOG_PHASE_POS]= 0; > - int4store(&global_ddl_log.file_entry_buf[DDL_LOG_NEXT_ENTRY_POS], > - ddl_log_entry->next_entry); > - DBUG_ASSERT(strlen(ddl_log_entry->name) < FN_REFLEN); > - strmake(&global_ddl_log.file_entry_buf[DDL_LOG_NAME_POS], > - ddl_log_entry->name, FN_REFLEN - 1); > - if (ddl_log_entry->action_type == DDL_LOG_RENAME_ACTION || > - ddl_log_entry->action_type == DDL_LOG_REPLACE_ACTION || > - ddl_log_entry->action_type == DDL_LOG_EXCHANGE_ACTION) > - { > - DBUG_ASSERT(strlen(ddl_log_entry->from_name) < FN_REFLEN); > - strmake(&global_ddl_log.file_entry_buf[DDL_LOG_NAME_POS + FN_REFLEN], > - ddl_log_entry->from_name, FN_REFLEN - 1); > - } > - else > - global_ddl_log.file_entry_buf[DDL_LOG_NAME_POS + FN_REFLEN]= 0; > - DBUG_ASSERT(strlen(ddl_log_entry->handler_name) < FN_REFLEN); > - strmake(&global_ddl_log.file_entry_buf[DDL_LOG_NAME_POS + (2*FN_REFLEN)], > - ddl_log_entry->handler_name, FN_REFLEN - 1); > - if (ddl_log_entry->action_type == DDL_LOG_EXCHANGE_ACTION) > - { > - DBUG_ASSERT(strlen(ddl_log_entry->tmp_name) < FN_REFLEN); > - strmake(&global_ddl_log.file_entry_buf[DDL_LOG_NAME_POS + (3*FN_REFLEN)], > - ddl_log_entry->tmp_name, FN_REFLEN - 1); > - } > - else > - global_ddl_log.file_entry_buf[DDL_LOG_NAME_POS + (3*FN_REFLEN)]= 0; > + > + file_entry_buf[DDL_LOG_ENTRY_TYPE_POS]= (uchar) ddl_log_entry->entry_type; > + file_entry_buf[DDL_LOG_ACTION_TYPE_POS]= (uchar) ddl_log_entry->action_type; > + file_entry_buf[DDL_LOG_PHASE_POS]= (uchar) ddl_log_entry->phase; > + int4store(file_entry_buf+DDL_LOG_NEXT_ENTRY_POS, ddl_log_entry->next_entry); > + int2store(file_entry_buf+DDL_LOG_FLAG_POS, ddl_log_entry->flags); > + int8store(file_entry_buf+DDL_LOG_XID_POS, ddl_log_entry->xid); > + memcpy(file_entry_buf+DDL_LOG_UUID_POS, ddl_log_entry->uuid, MY_UUID_SIZE); > + int8store(file_entry_buf+DDL_LOG_ID_POS, ddl_log_entry->unique_id); > + bzero(file_entry_buf+DDL_LOG_END_POS, > + global_ddl_log.name_pos - DDL_LOG_END_POS); > + > + pos= file_entry_buf + global_ddl_log.name_pos; > + end= file_entry_buf + global_ddl_log.io_size; > + > + pos= store_string(pos, end, &ddl_log_entry->handler_name); > + pos= store_string(pos, end, &ddl_log_entry->db); > + pos= store_string(pos, end, &ddl_log_entry->name); > + pos= store_string(pos, end, &ddl_log_entry->from_handler_name); > + pos= store_string(pos, end, &ddl_log_entry->from_db); > + pos= store_string(pos, end, &ddl_log_entry->from_name); > + pos= store_string(pos, end, &ddl_log_entry->tmp_name); you have DBUG_ASSERT(0) that all those strings won't overflow io_size. How can you guarantee that? > + bzero(pos, global_ddl_log.io_size - (pos - file_entry_buf)); > } > > > @@ -442,63 +691,120 @@ static bool ddl_log_sync_no_lock() ... > + > + > +/* > + Build a filename for a table, trigger file or .frm > + Delete also any temporary file suffixed with ~ Where do you get these *~ files from? What creates them? > +*/ > + > +static void build_filename_and_delete_tmp_file(char *path, size_t path_length, > + const LEX_CSTRING *db, > + const LEX_CSTRING *name, > + const char *ext, > + PSI_file_key psi_key) > +{ > + uint length= build_table_filename(path, path_length-1, > + db->str, name->str, ext, 0); > + > + path[length]= '~'; > + path[length+1]= 0; > + (void) mysql_file_delete(psi_key, path, MYF(0)); > + path[length]= 0; > } > > > @@ -806,154 +1296,132 @@ static bool ddl_log_execute_entry_no_lock(THD *thd, uint first_entry) > bool ddl_log_write_entry(DDL_LOG_ENTRY *ddl_log_entry, > DDL_LOG_MEMORY_ENTRY **active_entry) > { > - bool error, write_header; > + bool error; > + uchar *pos, *end; > DBUG_ENTER("ddl_log_write_entry"); > > mysql_mutex_assert_owner(&LOCK_gdl); > - if (init_ddl_log()) > + if (!global_ddl_log.open) > DBUG_RETURN(TRUE); > + > + ddl_log_entry->entry_type= DDL_LOG_ENTRY_CODE; > set_global_from_ddl_log_entry(ddl_log_entry); > - if (get_free_ddl_log_entry(active_entry, &write_header)) > + if (ddl_log_get_free_entry(active_entry)) > DBUG_RETURN(TRUE); > + > error= FALSE; > + pos= global_ddl_log.file_entry_buf + global_ddl_log.name_pos; > + end= global_ddl_log.file_entry_buf + global_ddl_log.io_size; > DBUG_PRINT("ddl_log", > - ("write type %c next %u name '%s' from_name '%s' handler '%s'" > - " tmp_name '%s'", > + ("type: %c next: %u handler: %s " > + "to_name: '%s.%s' from_name: '%s.%s' " > + "tmp_name: '%s'", > (char) global_ddl_log.file_entry_buf[DDL_LOG_ACTION_TYPE_POS], > ddl_log_entry->next_entry, > - (char*) &global_ddl_log.file_entry_buf[DDL_LOG_NAME_POS], > - (char*) &global_ddl_log.file_entry_buf[DDL_LOG_NAME_POS > - + FN_REFLEN], > - (char*) &global_ddl_log.file_entry_buf[DDL_LOG_NAME_POS > - + (2*FN_REFLEN)], > - (char*) &global_ddl_log.file_entry_buf[DDL_LOG_NAME_POS > - + (3*FN_REFLEN)])); > + get_string(&pos, end).str, // Handler > + get_string(&pos, end).str, // to db.table > + get_string(&pos, end).str, > + get_string(&pos, end).str, // From db.table > + get_string(&pos, end).str, > + get_string(&pos, end).str)); // Tmp name This is not a critical bug, because it's in a DBUG_PRINT, but it's still a bug. C++ (and C) standard both say that the order of evaluation of function arguments is *unspecified*. > + > if (unlikely(write_ddl_log_file_entry((*active_entry)->entry_pos))) > { > + ddl_log_release_memory_entry(*active_entry); > + *active_entry= 0; > error= TRUE; > - sql_print_error("Failed to write entry_no = %u", > + sql_print_error("DDL_LOG: Failed to write entry %u", > (*active_entry)->entry_pos); > } > - if (write_header && likely(!error)) > - { > - (void) ddl_log_sync_no_lock(); > - if (write_ddl_log_header()) > - error= TRUE; > - } > - if (unlikely(error)) > - ddl_log_release_memory_entry(*active_entry); > DBUG_RETURN(error); > } > > > /** > - @brief Write final entry in the ddl log. > + @brief Write or update execute entry in the ddl log. > > - @details This is the last write in the ddl log. The previous log entries > - have already been written but not yet synched to disk. > - We write a couple of log entries that describes action to perform. > - This entries are set-up in a linked list, however only when a first > - execute entry is put as the first entry these will be executed. > - This routine writes this first. > + @details An execute entry points to the first entry that should > + be excuted during recovery. In some cases it's only written once, > + in other cases it's updated for each log entry to point to the new > + header for the list. > > + When called, the previous log entries have already been written but not yet > + synched to disk. We write a couple of log entries that describes > + action to perform. This entries are set-up in a linked list, > + however only when an execute entry is put as the first entry these will be > + executed during recovery. > + > @param first_entry First entry in linked list of entries > - to execute, if 0 = NULL it means that > - the entry is removed and the entries > - are put into the free list. > - @param complete Flag indicating we are simply writing > - info about that entry has been completed > + to execute. > @param[in,out] active_entry Entry to execute, 0 = NULL if the entry > is written first time and needs to be > returned. In this case the entry written > is returned in this parameter > - > @return Operation status > @retval TRUE Error > @retval FALSE Success > */ > > bool ddl_log_write_execute_entry(uint first_entry, > - bool complete, > DDL_LOG_MEMORY_ENTRY **active_entry) > { > - bool write_header= FALSE; > - char *file_entry_buf= (char*)global_ddl_log.file_entry_buf; > + uchar *file_entry_buf= global_ddl_log.file_entry_buf; > + bool got_free_entry= 0; > DBUG_ENTER("ddl_log_write_execute_entry"); > > mysql_mutex_assert_owner(&LOCK_gdl); > - if (init_ddl_log()) > - { > - DBUG_RETURN(TRUE); > - } > - if (!complete) > - { > /* > We haven't synched the log entries yet, we synch them now before > writing the execute entry. If complete is true we haven't written there's no 'complete' anymore > any log entries before, we are only here to write the execute > entry to indicate it is done. > */ > (void) ddl_log_sync_no_lock(); Why do you need to sync previous log entries? As far as I understand, they're useless without DDL_LOG_EXECUTE_CODE entry, so why would it matter whether they're on disk or not after a crash? > - file_entry_buf[DDL_LOG_ENTRY_TYPE_POS]= (char)DDL_LOG_EXECUTE_CODE; > - } > - else > - file_entry_buf[DDL_LOG_ENTRY_TYPE_POS]= (char)DDL_IGNORE_LOG_ENTRY_CODE; > - file_entry_buf[DDL_LOG_ACTION_TYPE_POS]= 0; /* Ignored for execute entries */ > - file_entry_buf[DDL_LOG_PHASE_POS]= 0; > - int4store(&file_entry_buf[DDL_LOG_NEXT_ENTRY_POS], first_entry); > - file_entry_buf[DDL_LOG_NAME_POS]= 0; > - file_entry_buf[DDL_LOG_NAME_POS + FN_REFLEN]= 0; > - file_entry_buf[DDL_LOG_NAME_POS + 2*FN_REFLEN]= 0; > + bzero(file_entry_buf, global_ddl_log.io_size); > + > + file_entry_buf[DDL_LOG_ENTRY_TYPE_POS]= (uchar)DDL_LOG_EXECUTE_CODE; > + int4store(file_entry_buf + DDL_LOG_NEXT_ENTRY_POS, first_entry); > + > if (!(*active_entry)) > { > - if (get_free_ddl_log_entry(active_entry, &write_header)) > - { > + if (ddl_log_get_free_entry(active_entry)) > DBUG_RETURN(TRUE); > + got_free_entry= TRUE; > } > - write_header= TRUE; > - } > if (write_ddl_log_file_entry((*active_entry)->entry_pos)) > { > - sql_print_error("Error writing execute entry in ddl log"); > - ddl_log_release_memory_entry(*active_entry); > - DBUG_RETURN(TRUE); > - } > - (void) ddl_log_sync_no_lock(); > - if (write_header) > + if (got_free_entry) > { > - if (write_ddl_log_header()) > - { > ddl_log_release_memory_entry(*active_entry); > + *active_entry= 0; > + } > + sql_print_error("DDL_LOG: Error writing execute entry %u", > + (*active_entry)->entry_pos); > DBUG_RETURN(TRUE); > } > - } > + (void) ddl_log_sync_no_lock(); > DBUG_RETURN(FALSE); > } > > - > /** > - Deactivate an individual entry. > + Increment phase for enty. Will deactivate entry after all phases are done > > @details see ddl_log_increment_phase_no_lock. > > - @param entry_no Entry position of record to change > + @param entry_pos Entry position of record to change > > @return Operation status > @retval TRUE Error > @retval FALSE Success > */ > > -bool ddl_log_increment_phase(uint entry_no) > +bool ddl_log_increment_phase(uint entry_pos) > { > bool error; > DBUG_ENTER("ddl_log_increment_phase"); > > mysql_mutex_lock(&LOCK_gdl); > - error= ddl_log_increment_phase_no_lock(entry_no); > + error= ddl_log_increment_phase_no_lock(entry_pos); > mysql_mutex_unlock(&LOCK_gdl); > DBUG_RETURN(error); > } > @@ -1042,87 +1484,157 @@ static void close_ddl_log() > (void) mysql_file_close(global_ddl_log.file_id, MYF(MY_WME)); > global_ddl_log.file_id= (File) -1; > } > + global_ddl_log.open= 0; > DBUG_VOID_RETURN; > } > > > /** > + Loop over ddl log excute entries and mark those that are already stored > + in the binary log as completed > + > + @return > + @retval 0 ok > + @return 1 fail (write error) > + > +*/ > + > +bool ddl_log_close_binlogged_events(HASH *xids) > +{ > + uint i; > + DDL_LOG_ENTRY ddl_log_entry; > + DBUG_ENTER("ddl_log_close_binlogged_events"); > + > + if (global_ddl_log.num_entries == 0 || xids->records == 0) > + DBUG_RETURN(0); > + > + mysql_mutex_lock(&LOCK_gdl); > + for (i= 1; i <= global_ddl_log.num_entries; i++) > + { > + if (read_ddl_log_entry(i, &ddl_log_entry)) > + break; // Read error. Ignore it's not really "ignore", it's more like "stop reading" > + if (ddl_log_entry.entry_type == DDL_LOG_EXECUTE_CODE && > + ddl_log_entry.xid != 0 && > + my_hash_search(xids, (uchar*) &ddl_log_entry.xid, > + sizeof(ddl_log_entry.xid))) > + { > + if (disable_execute_entry(i)) > + { > + mysql_mutex_unlock(&LOCK_gdl); > + DBUG_RETURN(1); // Write error. Fatal! > + } > + } > + } > + (void) ddl_log_sync_no_lock(); > + mysql_mutex_unlock(&LOCK_gdl); > + DBUG_RETURN(0); > +} > + > + > +/** > Execute the ddl log at recovery of MySQL Server. > + > + @return > + @retval 0 Ok. > + @retval > 0 Fatal error. We have to abort (can't create ddl log) > + @return < -1 Recovery failed, but new log exists and is usable > + > */ > > -void ddl_log_execute_recovery() > +int ddl_log_execute_recovery() > { > - uint num_entries, i; > + uint i, count= 0; > + int error= 0; > THD *thd; > DDL_LOG_ENTRY ddl_log_entry; > - char file_name[FN_REFLEN]; > static char recover_query_string[]= "INTERNAL DDL LOG RECOVER IN PROGRESS"; > DBUG_ENTER("ddl_log_execute_recovery"); > > - /* > - Initialise global_ddl_log struct > - */ > - bzero(global_ddl_log.file_entry_buf, sizeof(global_ddl_log.file_entry_buf)); > - global_ddl_log.inited= FALSE; > - global_ddl_log.recovery_phase= TRUE; > - global_ddl_log.io_size= IO_SIZE; > - global_ddl_log.file_id= (File) -1; > + if (global_ddl_log.num_entries == 0) > + DBUG_RETURN(0); > > /* > To be able to run this from boot, we allocate a temporary THD > */ > if (!(thd=new THD(0))) > - DBUG_VOID_RETURN; > + { > + DBUG_ASSERT(0); // Fatal error > + DBUG_RETURN(1); > + } > thd->thread_stack= (char*) &thd; > thd->store_globals(); > + thd->log_all_errors= (global_system_variables.log_warnings >= 3); > > thd->set_query(recover_query_string, strlen(recover_query_string)); > > - /* this also initialize LOCK_gdl */ > - num_entries= read_ddl_log_header(); > mysql_mutex_lock(&LOCK_gdl); > - for (i= 1; i < num_entries + 1; i++) > + for (i= 1; i <= global_ddl_log.num_entries; i++) > { > if (read_ddl_log_entry(i, &ddl_log_entry)) > { > - sql_print_error("Failed to read entry no = %u from ddl log", > - i); > + error= -1; > continue; > } > if (ddl_log_entry.entry_type == DDL_LOG_EXECUTE_CODE) > { > + /* purecov: begin tested */ really? you tested with purify? > + if (ddl_log_entry.unique_id > DDL_LOG_MAX_RETRY) > + { > + error= -1; > + continue; > + } > + update_unique_id(i, ++ddl_log_entry.unique_id); > + if (ddl_log_entry.unique_id > DDL_LOG_MAX_RETRY) > + { > + sql_print_error("DDL_LOG: Aborting executing entry %u after %llu " > + "retries", i, ddl_log_entry.unique_id); > + error= -1; > + continue; > + } > + /* purecov: end tested */ > if (ddl_log_execute_entry_no_lock(thd, ddl_log_entry.next_entry)) > { > - /* Real unpleasant scenario but we continue anyways. */ > + /* Real unpleasant scenario but we have to continue anyway */ why can this happen? do you have an example? besides hardware failure, like a disk cannot be read > + error= -1; > continue; > } > + count++; > } > } > close_ddl_log(); > - create_ddl_log_file_name(file_name); > - (void) mysql_file_delete(key_file_global_ddl_log, file_name, MYF(0)); > - global_ddl_log.recovery_phase= FALSE; > mysql_mutex_unlock(&LOCK_gdl); > thd->reset_query(); > delete thd; > - DBUG_VOID_RETURN; > + > + /* > + Create a new ddl_log to get rid of old stuff and ensure that header matches > + the current source version > + */ because here you delete/overwrite the old ddl log, it'd make sense to log all details of every entry that you failed to execute but decided to continue anyway. Now you log only the entry number on failure, which is rather useless, because it's a number in a ddl log that you overwrite. > + if (create_ddl_log()) > + DBUG_RETURN(1); > + if (count > 0) > + sql_print_information("DDL_LOG: Crash recovery executed %u entries", > + count); > + DBUG_RETURN(error); > } > > > /** > - Release all memory allocated to the ddl log. > + Release all memory allocated to the ddl log and delete the ddl log > */ > > void ddl_log_release() > { > + char file_name[FN_REFLEN]; > DDL_LOG_MEMORY_ENTRY *free_list; > DDL_LOG_MEMORY_ENTRY *used_list; > DBUG_ENTER("ddl_log_release"); > > - if (!global_ddl_log.do_release) > + if (!global_ddl_log.initialized) > DBUG_VOID_RETURN; > > - mysql_mutex_lock(&LOCK_gdl); > + global_ddl_log.initialized= 0; > + > free_list= global_ddl_log.first_free; > used_list= global_ddl_log.first_used; > while (used_list) Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] c80f867b0df: MDEV-23844 Atomic DROP TABLE
by Sergei Golubchik 16 May '21

16 May '21

Hi, Michael! On May 12, Michael Widenius wrote: > revision-id: c80f867b0df (mariadb-10.6.0-78-gc80f867b0df) > parent(s): eebebe8ef75 > author: Michael Widenius <michael.widenius(a)gmail.com> > committer: Michael Widenius <michael.widenius(a)gmail.com> > timestamp: 2021-05-04 21:09:11 +0300 > message: > > MDEV-23844 Atomic DROP TABLE ... > diff --git a/sql/sql_view.cc b/sql/sql_view.cc > index e6d726b30d7..ff8aa55adef 100644 > --- a/sql/sql_view.cc > +++ b/sql/sql_view.cc > @@ -1861,8 +1868,18 @@ bool mysql_drop_view(THD *thd, TABLE_LIST *views, enum_drop_mode drop_mode) > not_exists_count++; > continue; > } > + if (!first_table++) yuck. so your "first_table" actually means "not_first_view", because when it's FALSE, it's the first view. why not to rename it to "not_first_view" ? > + { > + if (ddl_log_drop_view_init(thd, &ddl_log_state)) > + DBUG_RETURN(TRUE); what about a crash here? the first entry is written, the second is not. > + } > + if (ddl_log_drop_view(thd, &ddl_log_state, &cpath, &view->db, > + &view->table_name)) > + DBUG_RETURN(TRUE); > + debug_crash_here("ddl_log_drop_before_delete_view"); > if (unlikely(mysql_file_delete(key_file_frm, path, MYF(MY_WME)))) > delete_error= TRUE; > + debug_crash_here("ddl_log_drop_after_delete_view"); > > some_views_deleted= TRUE; > > diff --git a/sql/log.cc b/sql/log.cc > index 6e0a9706ec8..dba999054e4 100644 > --- a/sql/log.cc > +++ b/sql/log.cc > @@ -10471,16 +10483,16 @@ int TC_LOG_BINLOG::recover(LOG_INFO *linfo, const char *last_log_name, > bool last_gtid_standalone= false; > bool last_gtid_valid= false; > #endif > + DBUG_ENTER("TC_LOG_BINLOG::recover"); > > if (! fdle->is_valid() || > - (do_xa && my_hash_init(key_memory_binlog_recover_exec, &xids, > - &my_charset_bin, TC_LOG_PAGE_SIZE/3, 0, > - sizeof(my_xid), 0, 0, MYF(0)))) > + (my_hash_init(key_memory_binlog_recover_exec, &xids, > + &my_charset_bin, TC_LOG_PAGE_SIZE/3, 0, > + sizeof(my_xid), 0, 0, MYF(0)))) > goto err1; > > - if (do_xa) > - init_alloc_root(key_memory_binlog_recover_exec, &mem_root, > - TC_LOG_PAGE_SIZE, TC_LOG_PAGE_SIZE, MYF(0)); > + init_alloc_root(key_memory_binlog_recover_exec, &mem_root, > + TC_LOG_PAGE_SIZE, TC_LOG_PAGE_SIZE, MYF(0)); I don't think these two changes are correct. do_xa is TRUE on crash recovery and FALSE on gtid state recovery on 5.5->10.0 upgade or if gtid state file was deleted. There is no point to initialize the xid hash or mem_root, as it's not a crash recovery and there can be no incomplete transactions or ddl statements. Yes, I've seen the commit comment: > - TC_LOG_BINLOG::recover() was changed to always collect Xid's from the > binary log and always call ddl_log_close_binlogged_events(). This was > needed to be able to collect DROP TABLE events with embedded Xid's > (used by ddl log). but I cannot understand how it applies here. > fdle->flags&= ~LOG_EVENT_BINLOG_IN_USE_F; // abort on the first error > > diff --git a/sql/sql_table.cc b/sql/sql_table.cc > index 7a88ffdfd1b..0a83396ae14 100644 > --- a/sql/sql_table.cc > +++ b/sql/sql_table.cc > @@ -1363,6 +1373,17 @@ int mysql_rm_table_no_locks(THD *thd, TABLE_LIST *tables, bool if_exists, > > thd->replication_flags= 0; > was_view= table_type == TABLE_TYPE_VIEW; > + > + if (!first_table++) first_table -> not_first_table > + { > + LEX_CSTRING comment= {comment_start, (size_t) comment_len}; > + if (ddl_log_drop_table_init(thd, &ddl_log_state, &comment)) > + { > + error= 1; > + goto err; > + } > + } > + > if ((table_type == TABLE_TYPE_UNKNOWN) || (was_view && !drop_view) || > (table_type != TABLE_TYPE_SEQUENCE && drop_sequence)) > { > @@ -1565,6 +1611,7 @@ int mysql_rm_table_no_locks(THD *thd, TABLE_LIST *tables, bool if_exists, > table_name.str, (uint)table_name.length); > mysql_audit_drop_table(thd, table); > } > + ddl_log_update_phase(&ddl_log_state, DDL_DROP_PHASE_COLLECT); better call it DDL_DROP_PHASE_BINLOG. It would've saved me ~10 minutes reading the code back and forth, as ""DDL_DROP_PHASE_BINLOG" immediately explains what this phase is for. > > if (!dont_log_query && > (!error || table_dropped || non_existing_table_error(error))) > diff --git a/sql/ddl_log.h b/sql/ddl_log.h > index 316e6708f22..b843404fc3c 100644 > --- a/sql/ddl_log.h > +++ b/sql/ddl_log.h > @@ -97,6 +101,14 @@ enum enum_ddl_log_rename_table_phase { > DDL_RENAME_PHASE_TABLE, > }; > > +enum enum_ddl_log_drop_table_phase { > + DDL_DROP_PHASE_TABLE=0, > + DDL_DROP_PHASE_TRIGGER, no phase for deleting data from stat tables? > + DDL_DROP_PHASE_COLLECT, > + DDL_DROP_PHASE_RESET, /* Reset found list of dropped tables */ this one is unused (also unused in further patches, I've checked) > + DDL_DROP_PHASE_END > +}; > + > /* > Setting ddl_log_entry.phase to this has the same effect as setting > action_type to DDL_IGNORE_LOG_ENTRY_CODE > diff --git a/sql/ddl_log.cc b/sql/ddl_log.cc > index 777dfdddbc7..6145b2f7318 100644 > --- a/sql/ddl_log.cc > +++ b/sql/ddl_log.cc > @@ -109,6 +112,7 @@ struct st_global_ddl_log > }; > > st_global_ddl_log global_ddl_log; > +String ddl_drop_query; // Used during startup recovery static > > mysql_mutex_t LOCK_gdl; > > @@ -1132,6 +1149,122 @@ static int ddl_log_execute_action(THD *thd, MEM_ROOT *mem_root, > (void) update_phase(entry_pos, DDL_LOG_FINAL_PHASE); > } > break; > + case DDL_LOG_DROP_TABLE_INIT_ACTION: > + { > + LEX_CSTRING *comment= &ddl_log_entry->tmp_name; > + ddl_drop_query.length(0); > + ddl_drop_query.set_charset(system_charset_info); > + ddl_drop_query.append(STRING_WITH_LEN("DROP TABLE IF EXISTS ")); > + if (comment->length) > + { > + ddl_drop_query.append(comment); > + ddl_drop_query.append(' '); > + } > + /* We don't increment phase as we want to retry this in case of crash */ > + break; > + } > + case DDL_LOG_DROP_TABLE_ACTION: > + { > + LEX_CSTRING db, table, path; > + db= ddl_log_entry->db; > + table= ddl_log_entry->name; > + /* Note that path is without .frm extension */ > + path= ddl_log_entry->tmp_name; > + > + switch (ddl_log_entry->phase) { > + case DDL_DROP_PHASE_TABLE: > + if (hton) > + { > + if ((error= hton->drop_table(hton, path.str))) > + { > + if (!non_existing_table_error(error)) > + break; > + error= -1; > + } > + } > + else > + error= ha_delete_table_force(thd, path.str, &db, &table); > + if (error <= 0) > + { > + /* Not found or already deleted. Delete .frm if it exists */ > + strxnmov(to_path, sizeof(to_path)-1, path.str, reg_ext, NullS); > + mysql_file_delete(key_file_frm, to_path, MYF(MY_WME|MY_IGNORE_ENOENT)); this should rather be inside the previous if(). no need to do it for ha_delete_table_force(), as it doesn't leave any traces and removes all remnants of a table. > + } > + if (ddl_log_increment_phase_no_lock(entry_pos)) > + break; > + (void) ddl_log_sync_no_lock(); No need to, ddl_log_increment_phase_no_lock() syncs internally > + /* Fall through */ > + case DDL_DROP_PHASE_TRIGGER: > + Table_triggers_list::drop_all_triggers(thd, &db, &table, > + MYF(MY_WME | MY_IGNORE_ENOENT)); > + if (ddl_log_increment_phase_no_lock(entry_pos)) > + break; > + (void) ddl_log_sync_no_lock(); same. and in other places too, I'd expect. > + /* Fall through */ > + > + case DDL_DROP_PHASE_COLLECT: indentation? > + append_identifier(thd, &ddl_drop_query, &db); > + ddl_drop_query.append('.'); > + append_identifier(thd, &ddl_drop_query, &table); > + ddl_drop_query.append(','); > + /* We don't increment phase as we want to retry this in case of crash */ > + > + if (!ddl_log_entry->next_entry && mysql_bin_log.is_open()) > + { > + /* Last drop table. Write query to binlog */ > + LEX_CSTRING end_comment= > + { STRING_WITH_LEN(" /* generated by ddl log */")}; "generated by ddl log" ? a "log" cannot "generate" a statement, may be "generated during ddl recovery" ? > + ddl_drop_query.length(ddl_drop_query.length()-1); > + ddl_drop_query.append(&end_comment); > + > + mysql_mutex_unlock(&LOCK_gdl); > + (void) thd->binlog_query(THD::STMT_QUERY_TYPE, ddl_drop_query.ptr(), > + ddl_drop_query.length(), TRUE, FALSE, > + FALSE, 0); > + mysql_mutex_lock(&LOCK_gdl); > + } > + break; > + } > + break; > + } > + case DDL_LOG_DROP_VIEW_INIT_ACTION: > + { > + ddl_drop_query.length(0); > + ddl_drop_query.set_charset(system_charset_info); > + ddl_drop_query.append(STRING_WITH_LEN("DROP VIEW IF EXISTS ")); > + /* We don't increment phase as we want to retry this in case of crash */ > + break; > + } > + case DDL_LOG_DROP_VIEW_ACTION: > + { > + LEX_CSTRING db, table, path; > + db= ddl_log_entry->db; > + table= ddl_log_entry->name; > + /* Note that for views path is WITH .frm extension */ > + path= ddl_log_entry->tmp_name; > + > + mysql_file_delete(key_file_frm, path.str, MYF(MY_WME|MY_IGNORE_ENOENT)); > + append_identifier(thd, &ddl_drop_query, &db); > + ddl_drop_query.append('.'); > + append_identifier(thd, &ddl_drop_query, &table); > + ddl_drop_query.append(','); > + > + if (!ddl_log_entry->next_entry) > + { > + /* Last drop view. Write query to binlog */ > + LEX_CSTRING end_comment= > + { STRING_WITH_LEN(" /* generated by ddl log */")}; > + ddl_drop_query.length(ddl_drop_query.length()-1); > + ddl_drop_query.append(&end_comment); > + > + mysql_mutex_unlock(&LOCK_gdl); > + (void) thd->binlog_query(THD::STMT_QUERY_TYPE, ddl_drop_query.ptr(), > + ddl_drop_query.length(), TRUE, FALSE, > + FALSE, 0); > + mysql_mutex_lock(&LOCK_gdl); > + } > + break; > + } > default: > DBUG_ASSERT(0); > break; > @@ -1682,6 +1823,7 @@ void ddl_log_release_entries(DDL_LOG_STATE *ddl_log_state) > next= log_entry->next_active_log_entry; > ddl_log_release_memory_entry(log_entry); > } > + ddl_log_state->list= 0; may be you should also clear ddl_log_state->execute_entry below? > > if (ddl_log_state->execute_entry) > ddl_log_release_memory_entry(ddl_log_state->execute_entry); > @@ -1776,6 +1918,33 @@ bool ddl_log_update_xid(DDL_LOG_STATE *state, ulonglong xid) > } > > > +/* > + Write ddl_log_entry and write or update ddl_execute_entry > +*/ > + > +static bool ddl_log_write(DDL_LOG_STATE *ddl_state, > + DDL_LOG_ENTRY *ddl_log_entry) could you use a more descriptive name, please? > +{ > + int error; > + DDL_LOG_MEMORY_ENTRY *log_entry; > + DBUG_ENTER("ddl_log_write"); > + > + mysql_mutex_lock(&LOCK_gdl); > + error= ((ddl_log_write_entry(ddl_log_entry, &log_entry)) || > + ddl_log_write_execute_entry(log_entry->entry_pos, > + &ddl_state->execute_entry)); > + mysql_mutex_unlock(&LOCK_gdl); > + if (error) > + { > + if (log_entry) > + ddl_log_release_memory_entry(log_entry); > + DBUG_RETURN(1); > + } > + add_log_entry(ddl_state, log_entry); > + DBUG_RETURN(0); > +} > + > + > /** > Logging of rename table > */ Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] 2be9b69f4ff: MDEV-23001 Precreate static Item_bool() to simplify code
by Sergei Golubchik 14 May '21

14 May '21

Hi, Monty! On Mar 25, Michael Widenius wrote: > revision-id: 2be9b69f4ff (mariadb-10.5.2-510-g2be9b69f4ff) > parent(s): cb545f11169 > author: Michael Widenius <michael.widenius(a)gmail.com> > committer: Michael Widenius <michael.widenius(a)gmail.com> > timestamp: 2021-03-24 14:05:15 +0200 > message: > > MDEV-23001 Precreate static Item_bool() to simplify code > > The following changes where done: > - Create global Item: Item_false and Item_true > - Replace all creation if 'FALSE' and 'TRUE' top level items used for > WHERE/HAVING/ON clauses to use Item_false and Item_true. > > The benefit are: > - No test needed if we where able to create the new item. > - Fixed possible errors if 'new' would have failed for the Item_bool's agree > - Less and faster code I don't quite agree with that, it's not less code: > 7 files changed, 65 insertions(+), 43 deletions(-) and I doubt that speedup can be measured in any benchmarks whatsoever. > diff --git a/sql/item.cc b/sql/item.cc > index 1a86b8b3114..0f160fa257b 100644 > --- a/sql/item.cc > +++ b/sql/item.cc > @@ -55,7 +55,8 @@ const char *item_empty_name=""; > const char *item_used_name= "\0"; > > static int save_field_in_field(Field *, bool *, Field *, bool); > - > +const Item_bool_static Item_false("FALSE", 0); > +const Item_bool_static Item_true("TRUE", 1); The main question - we have tons of modifiable Item members. How do you know none of them will modified in your Item_bool_static objects? > @@ -421,6 +421,9 @@ Item::Item(THD *thd): > /* Initially this item is not attached to any JOIN_TAB. */ > join_tab_idx= MAX_TABLES; > > + /* thd is NULL in case of static items like Item_true */ > + if (!thd) > + return; No, please, don't. thd can be NULL only twice, for statically initialized Item_true and Item_false. For that you removed a useful assert and added an if() that is completely unnecessary for billions of items created runtime. Better to have special contructor just for these two very special items. > /* Put item in free list so that we can free all items at end */ > next= thd->free_list; > thd->free_list= this; > @@ -3637,6 +3641,19 @@ void Item_int::print(String *str, enum_query_type query_type) > } > > > +/* > + This function is needed to ensure that Item_bool_static doesn't change > + the value of the member str_value. > +*/ > + > +void Item_bool::print(String *str, enum_query_type query_type) > +{ > + // my_charset_bin is good enough for numbers > + String tmp(value ? (char*) "1" : (char*) "0" , 1, &my_charset_bin); > + str->append(tmp); > +} This is not needed anymore. Item_bool inherits from Item_int, and Item_int::print is void Item_int::print(String *str, enum_query_type query_type) { StringBuffer<LONGLONG_BUFFER_SIZE> buf; // my_charset_bin is good enough for numbers buf.set_int(value, unsigned_flag, &my_charset_bin); str->append(buf); } You removed str_value from Item_int::print in July. > Item *Item_bool::neg_transformer(THD *thd) > { > value= !value; > diff --git a/sql/item.h b/sql/item.h > index a1c288ab1f0..1087c08869e 100644 > --- a/sql/item.h > +++ b/sql/item.h > @@ -1976,7 +1976,7 @@ class Item: public Value_source, > virtual bool limit_index_condition_pushdown_processor(void *arg) { return 0; } > virtual bool exists2in_processor(void *arg) { return 0; } > virtual bool find_selective_predicates_list_processor(void *arg) { return 0; } > - bool cleanup_is_expensive_cache_processor(void *arg) > + virtual bool cleanup_is_expensive_cache_processor(void *arg) You've added a new virtual method? First you try so hard to remove virtual methods, then you add a new one specifically for these two static items. Why does it matter what cleanup_is_expensive_cache_processor will do with them? You overwrite is_expensive() anyway. > { > is_expensive_cache= (int8)(-1); > return 0; > @@ -3220,9 +3220,11 @@ class Item_literal: public Item_basic_constant > Item_literal(THD *thd): Item_basic_constant(thd) > { } > Type type() const override { return CONST_ITEM; } > - bool check_partition_func_processor(void *) override { return false;} > + bool check_partition_func_processor(void *int_arg) override { return false;} > bool const_item() const override { return true; } > bool basic_const_item() const override { return true; } > + bool is_expensive() override { return false; } > + bool cleanup_is_expensive_cache_processor(void *arg) override { return 0; } > }; > > > diff --git a/sql/opt_index_cond_pushdown.cc b/sql/opt_index_cond_pushdown.cc > index 15bc2074e1f..e950ad1b7ca 100644 > --- a/sql/opt_index_cond_pushdown.cc > +++ b/sql/opt_index_cond_pushdown.cc > @@ -185,8 +185,8 @@ bool uses_index_fields_only(Item *item, TABLE *tbl, uint keyno, > static Item *make_cond_for_index(THD *thd, Item *cond, TABLE *table, uint keyno, > bool other_tbls_ok) > { > - if (!cond) > - return NULL; > + if (!cond || cond->basic_const_item()) > + return cond; The effect of this change is that you don't set cond->marker= ICP_COND_USES_INDEX_ONLY for basic_const_item's anymore. This looked suspicious at first. But then I noticed (and Sergey Petrunia confirmed) that ICP_COND_USES_INDEX_ONLY is not used at all. Since 5.3. So here I'd suggest to revert this your change and completely remove any ICP_COND_USES_INDEX_ONLY manipulations together with the ICP_COND_USES_INDEX_ONLY definition. > if (cond->type() == Item::COND_ITEM) > { > uint n_marked= 0; > diff --git a/sql/sql_parse.cc b/sql/sql_parse.cc > index 119c7360f07..e63e120a02b 100644 > --- a/sql/sql_parse.cc > +++ b/sql/sql_parse.cc > @@ -2395,6 +2395,7 @@ dispatch_command_return dispatch_command(enum enum_server_command command, THD * > thd->update_server_status(); > thd->protocol->end_statement(); > query_cache_end_of_result(thd); > + /* Check that the query didn't change const items */ what does that mean? Is it a TODO comment? Or it documents what happens below? > } > if (drop_more_results) > thd->server_status&= ~SERVER_MORE_RESULTS_EXISTS; > diff --git a/sql/sql_select.cc b/sql/sql_select.cc > index 93f5d3591ed..894d511adf4 100644 > --- a/sql/sql_select.cc > +++ b/sql/sql_select.cc > @@ -2364,7 +2364,7 @@ int JOIN::optimize_stage2() > if (!conds && outer_join) > { > /* Handle the case where we have an OUTER JOIN without a WHERE */ > - conds= new (thd->mem_root) Item_bool(thd, true); // Always true > + conds= (Item*) &Item_true; Hmm, does casting const away even work in the Intel compiler? > } > > if (impossible_where) > @@ -22743,6 +22743,8 @@ make_cond_for_table_from_pred(THD *thd, Item *root_cond, Item *cond, > return new_cond; > } > } > + else if (cond->basic_const_item()) > + return cond; > > if (is_top_and_level && used_table == rand_table_bit && > (cond->used_tables() & ~OUTER_REF_TABLE_BIT) != rand_table_bit) this is rather fragile, isn't it? You skip item->set_join_tab_idx(join_tab_idx_arg), quite understandably. And currently, as far as I can see, item->get_join_tab_idx() is always used under if (!item->const_item()), so what you did is safe. But can it change in the future? Or is join_tab_idx something that conceptually doesn't make sense for const items? Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 2

Re: [Maria-developers] 9083627c50f: Make rename atomic/repeatable in MyISAM and Aria
by Sergei Golubchik 14 May '21

14 May '21

Hi, Michael! On May 04, Michael Widenius wrote: > revision-id: 9083627c50f (mariadb-10.5.2-562-g9083627c50f) > parent(s): 105cda3b1cb > author: Michael Widenius <michael.widenius(a)gmail.com> > committer: Michael Widenius <michael.widenius(a)gmail.com> > timestamp: 2021-03-25 12:06:34 +0200 > message: > > Make rename atomic/repeatable in MyISAM and Aria > > This is required to make Atomic RENAME TABLE work for these engines > > The requirement is that if we have a server crash in the middle of a > storage engine rename call, the upcoming ddl log recovery should be able > to finalize it by re-execute the rename. > > diff --git a/storage/maria/ma_rename.c b/storage/maria/ma_rename.c > index a4388596f6b..64dac32a51c 100644 > --- a/storage/maria/ma_rename.c > +++ b/storage/maria/ma_rename.c > @@ -106,28 +106,37 @@ int maria_rename(const char *old_name, const char *new_name) > _ma_reset_state(info); > maria_close(info); > > + /* > + This code is written so that it should be possible to re-run a > + failed rename (even if there is a server crash in between the > + renames) and complete it. > + */ > fn_format(from,old_name,"",MARIA_NAME_IEXT,MY_UNPACK_FILENAME|MY_APPEND_EXT); > fn_format(to,new_name,"",MARIA_NAME_IEXT,MY_UNPACK_FILENAME|MY_APPEND_EXT); > if (mysql_file_rename_with_symlink(key_file_kfile, from, to, > MYF(MY_WME | sync_dir))) > - DBUG_RETURN(my_errno); > + index_file_rename_error= my_errno; Shouldn't you continue only if index_file_rename_error == ENOENT ? also, I only very quickly looked at mysql_file_rename_with_symlink, but it didn't look atomic to me when symlinks are present. > fn_format(from,old_name,"",MARIA_NAME_DEXT,MY_UNPACK_FILENAME|MY_APPEND_EXT); > fn_format(to,new_name,"",MARIA_NAME_DEXT,MY_UNPACK_FILENAME|MY_APPEND_EXT); ... > diff --git a/storage/myisam/mi_rename.c b/storage/myisam/mi_rename.c > index 19df2e54213..13cf8c60897 100644 > --- a/storage/myisam/mi_rename.c > +++ b/storage/myisam/mi_rename.c > @@ -22,6 +22,7 @@ > int mi_rename(const char *old_name, const char *new_name) > { > char from[FN_REFLEN],to[FN_REFLEN]; > + int save_errno= 0; > DBUG_ENTER("mi_rename"); > > #ifdef EXTRA_DEBUG > @@ -32,10 +33,13 @@ int mi_rename(const char *old_name, const char *new_name) > fn_format(from,old_name,"",MI_NAME_IEXT,MY_UNPACK_FILENAME|MY_APPEND_EXT); > fn_format(to,new_name,"",MI_NAME_IEXT,MY_UNPACK_FILENAME|MY_APPEND_EXT); > if (mysql_file_rename_with_symlink(mi_key_file_kfile, from, to, MYF(MY_WME))) > - DBUG_RETURN(my_errno); > + save_errno= my_errno; > fn_format(from,old_name,"",MI_NAME_DEXT,MY_UNPACK_FILENAME|MY_APPEND_EXT); > fn_format(to,new_name,"",MI_NAME_DEXT,MY_UNPACK_FILENAME|MY_APPEND_EXT); > - DBUG_RETURN(mysql_file_rename_with_symlink(mi_key_file_dfile, > - from, to, > - MYF(MY_WME)) ? my_errno : 0); > + if (mysql_file_rename_with_symlink(mi_key_file_dfile, > + from, to, > + MYF(MY_WME))) > + if (save_errno) > + save_errno= my_errno; > + DBUG_RETURN(save_errno); Why mi_rename isn't trying to undo a half-failed rename like ma_rename does? Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] 32c73138ff4: Remove some usage of Check_level_instant_set and Sql_mode_save
by Sergei Golubchik 14 May '21

14 May '21

Hi, Monty! On Apr 19, Michael Widenius wrote: > revision-id: 32c73138ff4 (mariadb-10.5.2-559-g32c73138ff4) > parent(s): 07708eb9b28 > author: Michael Widenius <michael.widenius(a)gmail.com> > committer: Michael Widenius <michael.widenius(a)gmail.com> > timestamp: 2021-03-25 12:06:34 +0200 > message: > > Remove some usage of Check_level_instant_set and Sql_mode_save > > The reason for the removal are: > - Generates more code > - Storing and retreving THD > - Causes extra code and daata to be generated to handle possible throw > exceptions (which never happens in MariaDB code) > - Uses more stack space No, Monty. We've discussed it already, this makes code more complex, fragile and bug prone. And it does *not* generates extra code or data or stack - before writing this email I've compiled mariadbd both ways and compared the generated code for the function in question, it was *identical*. > Other things: > - Changed convert_const_to_int() to use item->save_in_field_no_warnings(), > which made the code shorter and simpler. > - Added thd as argument to store_key.copy() to make function simpler > - Added thd as argument to some subselect* constructor that inherites > from Item_subselect. these changes are good, thanks Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] 6b06a99e4a4: Avoid creating the .frm file twice in some cases
by Sergei Golubchik 14 May '21

14 May '21

Hi, Monty! On Apr 19, Michael Widenius wrote: > revision-id: 6b06a99e4a4 (mariadb-10.5.2-557-g6b06a99e4a4) > parent(s): 9c1e36696f3 > author: Michael Widenius <michael.widenius(a)gmail.com> > committer: Michael Widenius <michael.widenius(a)gmail.com> > timestamp: 2021-03-25 12:06:34 +0200 > message: > > Avoid creating the .frm file twice in some cases > > diff --git a/sql/handler.cc b/sql/handler.cc > index c286aef3a9e..fe2fe6e9426 100644 > --- a/sql/handler.cc > +++ b/sql/handler.cc > @@ -5536,8 +5536,9 @@ int ha_create_table(THD *thd, const char *path, > > if (frm) > { > - bool write_frm_now= !create_info->db_type->discover_table && > - !create_info->tmp_table(); > + bool write_frm_now= (!create_info->db_type->discover_table && > + !create_info->tmp_table() && > + !create_info->frm_is_created); Is this the real problem? frm created twice? Why ha_create_table() is invoked twice at all? > > share.frm_image= frm; > > diff --git a/sql/sql_table.cc b/sql/sql_table.cc > index e00d3ee1ed2..73642a5bd97 100644 > --- a/sql/sql_table.cc > +++ b/sql/sql_table.cc > @@ -547,8 +547,13 @@ uint build_table_filename(char *buff, size_t bufflen, const char *db, > > (void) tablename_to_filename(db, dbbuff, sizeof(dbbuff)); > > - /* Check if this is a temporary table name. Allow it if a corresponding .frm file exists */ > - if (is_prefix(table_name, tmp_file_prefix) && strlen(table_name) < NAME_CHAR_LEN && > + /* > + Check if this is a temporary table name. Allow it if a corresponding .frm > + file exists. > + */ > + if (!(flags & FN_IS_TMP) && > + is_prefix(table_name, tmp_file_prefix) && > + strlen(table_name) < NAME_CHAR_LEN && good idea! > check_if_frm_exists(tbbuff, dbbuff, table_name)) > flags|= FN_IS_TMP; > > @@ -9817,7 +9827,7 @@ do_continue:; > > if (table->s->tmp_table != NO_TMP_TABLE) > { > - /* Close lock if this is a transactional table */ > + /* Unlock lock if this is a transactional temporary table */ strictly speaking, you don't "unlock lock", you "unlock a resource" (e.g. a table) or you "release a lock" > if (thd->lock) > { > if (thd->locked_tables_mode != LTM_LOCK_TABLES && Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] 9c1e36696f3: MDEV-20025: ADD_MONTHS() Oracle function
by Sergei Golubchik 14 May '21

14 May '21

Hi, Monty! On Apr 16, Michael Widenius wrote: > commit 9c1e36696f3 > Author: Michael Widenius <michael.widenius(a)gmail.com> > Date: Fri Feb 5 17:10:43 2021 +0200 > > diff --git a/mysql-test/suite/compat/oracle/r/func_add_months.result b/mysql-test/suite/compat/oracle/r/func_add_months.result > new file mode 100644 > index 00000000000..0e8ca1eef94 > --- /dev/null > +++ b/mysql-test/suite/compat/oracle/r/func_add_months.result > @@ -0,0 +1,85 @@ > +Test for ADD_MONTHS > +CREATE TABLE t1(c1 int, c2 datetime, c3 date, c4 time, c5 timestamp); > +INSERT INTO t1 VALUES (1, '2011-11-12 12:10:11', '2011-11-12', '12:10:11', '2011-11-12 12:10:11'); > +INSERT INTO t1 VALUES (2, '2021-11-12 00:23:12', '2021-11-12', '00:23:12', '2021-11-12 00:23:12'); > +INSERT INTO t1 VALUES (3, '2011-01-22 16:45:45', '2011-01-22', '16:45:45', '2011-01-22 16:45:45'); > +INSERT INTO t1 VALUES (4, '2031-05-12 04:11:34', '2031-05-12', '04:11:34', '2031-05-12 04:11:34'); > +INSERT INTO t1 VALUES (5, '2031-09-02 08:15:22', '2031-09-02', '08:15:22', '2031-09-02 08:15:22'); > +INSERT INTO t1 VALUES (6, '0000-09-02 00:00:00', '0000-09-02', '00:00:00', '1980-09-02 00:00:00'); > +INSERT INTO t1 VALUES (7, '9999-09-02', '9999-09-02', '00:00:00', '1980-09-02'); there's no last-day test here. Like ADD_MONTHS('2012-01-31', 1) = '2012-02-29' should be here, as Oracle manual specifically documents this case. Regards, Sergei

2 1

Re: [Maria-developers] fbe1dad15a1: Ensure that we do not allocate strings bigger than 4G in String objects.
by Sergei Golubchik 14 May '21

14 May '21

Hi, Monty! On Apr 16, Michael Widenius wrote: > revision-id: fbe1dad15a1 (mariadb-10.5.2-555-gfbe1dad15a1) > parent(s): 57c19902326 > author: Michael Widenius <michael.widenius(a)gmail.com> > committer: Michael Widenius <michael.widenius(a)gmail.com> > timestamp: 2021-03-25 12:06:34 +0200 > message: > > Ensure that we do not allocate strings bigger than 4G in String objects. > > This is needed as we are using uint32 for allocated and current length. > > diff --git a/sql/sql_string.cc b/sql/sql_string.cc > index 9c57bb22085..fedf5f4a48a 100644 > --- a/sql/sql_string.cc > +++ b/sql/sql_string.cc > @@ -37,6 +37,8 @@ bool Binary_string::real_alloc(size_t length) > DBUG_ASSERT(arg_length > length); > if (arg_length <= length) > return TRUE; /* Overflow */ > + if (arg_length >= UINT_MAX32) > + return FALSE; > str_length=0; > if (Alloced_length < arg_length) > { > @@ -45,7 +47,6 @@ bool Binary_string::real_alloc(size_t length) > arg_length,MYF(MY_WME | (thread_specific ? > MY_THREAD_SPECIFIC : 0))))) > return TRUE; > - DBUG_ASSERT(length < UINT_MAX32); > Alloced_length=(uint32) arg_length; > alloced=1; I liked assert more. It meant that we should never under any circumstances request more than 4G from Binary_string::real_alloc. You're saying that it's ok to do it and Binary_string::real_alloc should be ready to deal with it. I'd say the first approach was better, >4G should just never happen. Regards, Sergei

2 1

Re: [Maria-developers] c157c285db9: Optimize Sql_alloc
by Sergei Golubchik 14 May '21

14 May '21

Hi, Michael! On Mar 27, Michael Widenius wrote: > revision-id: c157c285db9 (mariadb-10.5.2-514-gc157c285db9) > parent(s): 16e38888c06 > author: Michael Widenius <michael.widenius(a)gmail.com> > committer: Michael Widenius <michael.widenius(a)gmail.com> > timestamp: 2021-03-24 14:31:53 +0200 > message: > > Optimize Sql_alloc > > - Remove 'dummy_for_valgrind' overrun marker as this doesn't help much. > The element also distorts the sizes of objects a bit, which makes it > harder to calculate gain in object sizes when doing size optimizations. > - Avoid one extra call indirection when using thd_get_current_thd(), which > is used by Sql_alloc. > > diff --git a/sql/mysqld.cc b/sql/mysqld.cc > index 4f522d96a9f..1ab52401705 100644 > --- a/sql/mysqld.cc > +++ b/sql/mysqld.cc > @@ -662,7 +662,15 @@ static std::atomic<char*> shutdown_user; > > static thread_local THD *THR_THD; > > +/** > + Get current THD object from thread local data > + > + @retval The THD object for the thread, NULL if not connection thread > +*/ > + > MYSQL_THD _current_thd() { return THR_THD; } > +THD *thd_get_current_thd() { return THR_THD; } I'd rather remove thd_get_current_thd() completely, why do we need two identical functions? Sql_alloc can use _current_thd() just fine, I've tried (also rocksdb uses thd_get_current_thd in one place, but it can use current_thd like the rest of the server code). Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] 21eb8969ce9: Improved storage size for Item, Field and some other classes
by Sergei Golubchik 14 May '21

14 May '21

Hi, Monty! Looks ok, but again it doesn't seem you've squashed intermediate commits as you said you did. Bit fields and non-existent commits in columnstore - it's is clearly an intermediate work-in-progress state, all fixed in your later commits. On Sep 08, Michael Widenius wrote: > revision-id: 21eb8969ce9 (mariadb-10.5.2-270-g21eb8969ce9) > parent(s): c3ecf0d6243 > author: Michael Widenius <michael.widenius(a)gmail.com> > committer: Michael Widenius <michael.widenius(a)gmail.com> > timestamp: 2020-09-02 20:58:34 +0300 > message: > > Improved storage size for Item, Field and some other classes > > - Changed order of class fields to remove dead alignment space. > - Changed bool fields in Item to bit fields. > - Used packed enum's for some fields in common classes > - Removed not used Item::rsize. > - Changed some class variables from uint/int to smaller type int's. > - Ensured that field_index is uint16 in all classes and functions. Fixed > also that we proparly compare with NO_CACHED_FIELD_INDEX when checking > if variable is not set. > - Removed checking of highest bit of unireg_check (has not been used in > a long time) > - Fixed wrong arguments to make_cond_for_table() for join_tab_idx_arg > from false to 0. > > One of the result was reducing the size if class Item with ~24 bytes ... > @@ -929,18 +904,48 @@ class Item: public Value_source, > + > + /* > + str_values's main purpose is to be used to cache the value in > + save_in_field > + */ > + String str_value; > + > + LEX_CSTRING name; /* Name of item */ > + /* Original item name (if it was renamed)*/ > + const char *orig_name; > + > + uint32 /* All common bool variables for Item stored here */ > + maybe_null:1, /* If item may be null */ > + in_rollup:1, /* If used in GROUP BY list of a query with ROLLUP */ > + with_param:1, /* True if Item contains an SP parameter */ > + with_window_func:1, /* True if item contains a window func */ > + with_field:1, /* True if any item except Item_sum contains a field. > + Set during parsing. */ > + fixed:1; /* If item was fixed with fix_fields */ > + > + int16 marker; > + > diff --git a/storage/columnstore/columnstore b/storage/columnstore/columnstore > index b6b02ed516f..f606e76fb77 160000 > --- a/storage/columnstore/columnstore > +++ b/storage/columnstore/columnstore > @@ -1 +1 @@ > -Subproject commit b6b02ed516f92055127d416370799d91a82754ea > +Subproject commit f606e76fb779e40f3376693fff9969e4f2b7669a Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 3

Re: [Maria-developers] 4088269c578d: Improved storage size for Item, Field and some other classes
by Sergei Golubchik 14 May '21

14 May '21

Hi, Michael! On Dec 03, Michael Widenius wrote: > revision-id: 4088269c578d (mariadb-10.5.2-270-g4088269c578d) > parent(s): 994ea2af3973 > author: Michael Widenius <michael.widenius(a)gmail.com> > committer: Michael Widenius <michael.widenius(a)gmail.com> > timestamp: 2020-09-17 12:25:44 +0300 > message: > > Improved storage size for Item, Field and some other classes > > diff --git a/storage/columnstore/columnstore b/storage/columnstore/columnstore > index b6b02ed516f9..f606e76fb779 160000 > --- a/storage/columnstore/columnstore > +++ b/storage/columnstore/columnstore > @@ -1 +1 @@ > -Subproject commit b6b02ed516f92055127d416370799d91a82754ea > +Subproject commit f606e76fb779e40f3376693fff9969e4f2b7669a Again. You should not push invalid commit hashes into the main branch: storage/columnstore/columnstore $ git fetch origin storage/columnstore/columnstore $ git show f606e76fb779e40f3376693fff9969e4f2b7669a fatal: bad object f606e76fb779e40f3376693fff9969e4f2b7669a Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] 16e38888c06: Improved storage size for Item, Field and some other classes
by Sergei Golubchik 14 May '21

14 May '21

Hi, Michael! Did you consider trying -Wpadded? '-Wpadded' Warn if padding is included in a structure, either to align an element of the structure or to align the whole structure. Sometimes when this happens it is possible to rearrange the fields of the structure to reduce the padding and so make the structure smaller. Might reveal more useful cases where fields could be rearranged. On Mar 27, Michael Widenius wrote: > revision-id: 16e38888c06 (mariadb-10.5.2-513-g16e38888c06) > parent(s): 4cfb680b685 > author: Michael Widenius <michael.widenius(a)gmail.com> > committer: Michael Widenius <michael.widenius(a)gmail.com> > timestamp: 2021-03-24 14:31:53 +0200 > message: > > Improved storage size for Item, Field and some other classes > > - Changed order of class fields to remove dead alignment space. > - Changed bool fields in Item to bit fields. > - Used packed enum's for some fields in common classes > - Removed not used Item::rsize. > - Changed some class variables from uint/int to smaller type int's. > - Ensured that field_index is uint16 in all classes and functions. Fixed > also that we proparly compare with NO_CACHED_FIELD_INDEX when checking > if variable is not set. > - Removed checking of highest bit of unireg_check (has not been used in > a long time) > - Fixed wrong arguments to make_cond_for_table() for join_tab_idx_arg > from false to 0. > > One of the result was reducing the size if class Item with ~24 bytes Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] 094ae6bbcc0: MDEV-24395 Atomic DROP TRIGGER
by Sergei Golubchik 13 May '21

13 May '21

Hi, Monty! On May 13, Michael Widenius wrote: > revision-id: 094ae6bbcc0 (mariadb-10.6.0-79-g094ae6bbcc0) > parent(s): c80f867b0df > author: Michael Widenius <michael.widenius(a)gmail.com> > committer: Michael Widenius <michael.widenius(a)gmail.com> > timestamp: 2021-05-04 21:09:11 +0300 > message: > > MDEV-24395 Atomic DROP TRIGGER > diff --git a/sql/ddl_log.cc b/sql/ddl_log.cc > index 6145b2f7318..cf65f8ad876 100644 > --- a/sql/ddl_log.cc > +++ b/sql/ddl_log.cc > @@ -1265,6 +1290,70 @@ static int ddl_log_execute_action(THD *thd, MEM_ROOT *mem_root, > } > break; > } > + case DDL_LOG_DROP_TRIGGER_ACTION: > + { > + MY_STAT stat_info; > + off_t frm_length= 1; // Impossible length > + LEX_CSTRING thd_db= thd->db; > + > + /* Delete trigger temporary file if it still exists */ > + if (!build_filename_and_delete_tmp_file(to_path, sizeof(to_path) - 1, > + &ddl_log_entry->db, > + &ddl_log_entry->name, > + TRG_EXT, > + key_file_fileparser)) > + { > + /* Temporary file existed and was deleted, nothing left to do */ you didn't answer my question in one of the previous reviews, what is a temporary file *~ ? What creates it and where? > + (void) update_phase(entry_pos, DDL_LOG_FINAL_PHASE); > + break; > + } > + /* > + We can use length of TRG file as an indication if trigger was removed. > + If there is no file, then it means that this was the last trigger > + and the file was removed. > + */ > + if (my_stat(to_path, &stat_info, MYF(0))) > + frm_length= (off_t) stat_info.st_size; > + if (frm_length != (off_t) ddl_log_entry->unique_id && > + mysql_bin_log.is_open()) > + { > + /* > + File size changed and it was not binlogged (as this entry was > + executed) > + */ > + (void) rm_trigname_file(to_path, &ddl_log_entry->db, > + &ddl_log_entry->from_name, > + MYF(0)); > + > + ddl_drop_query.length(0); > + ddl_drop_query.set_charset(system_charset_info); > + if (ddl_log_entry->tmp_name.length) I'm not sure it's a good idea. It'll have some rather confusing conditional behavior when sometimes a query is logged with the original user's comment. And adding a couple of# characters to the comment makes the comment to disappear because the query will be replaced with a generated one. It might've been more predictable and less confusing, if the query would be always generated. After all, it's only for the ddl recovery case, it's reasonable to see a generated query for redo actions that happen during the crash recovery. > + { > + /* We can use the original query */ > + ddl_drop_query.append(&ddl_log_entry->tmp_name); > + } > + else > + { > + /* Generate new query */ > + ddl_drop_query.append(STRING_WITH_LEN("DROP TRIGGER IF EXISTS ")); > + append_identifier(thd, &ddl_drop_query, &ddl_log_entry->from_name); > + ddl_drop_query.append(&end_comment); > + } > + if (mysql_bin_log.is_open()) > + { > + mysql_mutex_unlock(&LOCK_gdl); > + thd->db= ddl_log_entry->db; > + (void) thd->binlog_query(THD::STMT_QUERY_TYPE, > + ddl_drop_query.ptr(), > + ddl_drop_query.length(), TRUE, FALSE, > + FALSE, 0); > + thd->db= thd_db; > + mysql_mutex_lock(&LOCK_gdl); > + } > + } > + (void) update_phase(entry_pos, DDL_LOG_FINAL_PHASE); > + break; > + } > default: > DBUG_ASSERT(0); > break; Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] c76ac1e6de8: MDEV-24089 support oracle syntax: rownum
by Sergei Golubchik 13 May '21

13 May '21

Hi, Michael! On Apr 12, Michael Widenius wrote: > revision-id: c76ac1e6de8 (mariadb-10.5.2-551-gc76ac1e6de8) > parent(s): a507eb17708 > author: Michael Widenius <michael.widenius(a)gmail.com> > committer: Michael Widenius <michael.widenius(a)gmail.com> > timestamp: 2021-03-24 19:42:24 +0200 > message: > diff --git a/mysql-test/main/rownum.test b/mysql-test/main/rownum.test > new file mode 100644 > index 00000000000..8d89eed2f86 > --- /dev/null > +++ b/mysql-test/main/rownum.test > @@ -0,0 +1,542 @@ > +--source include/have_sequence.inc > +# > +# Test of basic rownum() functionallity > +# Test are done with Aria to ensure that row order is stable what does that mean? > +# > + > +CREATE OR REPLACE TABLE t1(a int, b int) engine=aria; > +CREATE OR REPLACE TABLE t2(a int, b int) engine=aria; > +insert into t1 values (1,10),(2,20),(3,30); > +insert into t2 values (2,21),(3,31),(4,41); > + > +--echo # > +--echo # Simple selects > +--echo # > + > +select a,b,rownum() from t1; > +select a,b,rownum() from t1 where rownum() < 2; > +select a,b from t1 where rownum() <= 2; > +select a,b from t1 where rownum() > 2; > + > +--echo # > +--echo # Subqueries > +--echo # > + > +select t1.a,rownum(),t3.a,t3.t2_rownum from t1, (select t2.a,rownum() as t2_rownum from t2 where rownum() <=2) t3; > +select t1.a, (select t2.b from t2 where t1.a=t2.a and rownum() <= 1) 'b' from t1; > +select t1.a, t3.a from t1, (select * from t2 where rownum() <= 2) t3; > +select * from (select tt.*,rownum() as id from (select * from t1) tt) t3 where id>2; > + > +--echo # > +--echo # Joins > +--echo # > + > +select t1.a,t1.b,t2.a,t2.b,rownum() from t1,t2 where rownum() <= 4; > +select *,rownum() from t1,t2 where t1.a=t2.a and rownum()<=2; > +select * from t1 left join t2 on (t2.a=t1.a and rownum()=0); > +select * from t1 left join t2 on (t2.a=t1.a and rownum()>1); > +select * from t1 left join t2 on (t2.a=t1.a and rownum()<1); > +select * from t1 left join t2 on (t2.a=t1.a and rownum()=1); > +select * from t1 left join t2 on (t2.a=t1.a and rownum()>=1); > + > +--echo # > +--echo # Union > +--echo # > + > +select * from t1 where rownum() <=2 union select * from t2 where rownum()<= 1; > + > +--echo # > +--echo # Order by > +--echo # > + > +select * from t1 where rownum() <= 2 order by a desc; > +explain select * from t1 where rownum() <= 2 order by a desc; > +select t1.a,t1.b,rownum() from t1 where rownum() <= 2 order by a desc; > +explain select t1.a,t1.b,rownum() from t1 where rownum() <= 2 order by a desc; > +select *,rownum() from t1,t2; > +select *,rownum() from t1,t2 order by t2.a desc, t1.a desc; > +select * from (select * from t1 order by a desc) as t where rownum() <= 2; > +select * from t1,t2 where t1.a=t2.a and rownum()<=2 order by t1.a,t2.a; > + > +--echo # > +--echo # Having > +--echo # > + > +select t1.a, sum(t1.b), rownum() as 'r' from t1 group by t1.a having r <= 2; > +--error ER_FUNCTION_CANNOT_BE_USED_IN_CLAUSE > +select * from t1 having rownum() <= 2; > +select t1.a, sum(t1.b), rownum() from t1 group by t1.a; > +--error ER_FUNCTION_CANNOT_BE_USED_IN_CLAUSE > +select t1.a, sum(t1.b), rownum() as 'r' from t1 group by t1.a having rownum() <= 2; > + > +--echo # > +--echo # Sum functions > +--echo # > + > +select max(rownum()),min(rownum()) from t1; > +select sum(rownum()),avg(rownum()) from t1; > + > +--echo # > +--echo # Group by > +--echo # > + > +select t1.a,sum(t1.b) from t1 where rownum() < 2 group by t1.a; > +select t1.a,sum(t2.b) from t1 JOIN t2 ON (t1.a=t2.a) where rownum() <= 2 group by t1.a; > +select * from (select t1.a,sum(t2.b) from t1 JOIN t2 ON (t1.a=t2.a) group by t1.a) as t where rownum() <= 1; > +select t1.a,sum(rownum()),count(*) from t1 where rownum() <= 2 group by t1.a; > +select * from (select t1.a,sum(t1.b) from t1 group by t1.a) as t3 where rownum() < 2; > + > +create table t3 (a int) engine=myisam; > +insert into t3 values (3),(5),(5),(3); > +select a, max(rownum()) from t3 group by a; > +drop table t3; > + > +CREATE TABLE t3 ( > + a int(11) DEFAULT NULL, > + b varchar(1024) DEFAULT NULL > +); > +insert into t3 select mod(seq*3,20)+1, repeat(char(33+mod(seq,90)),mod(seq,10)*100) from seq_1_to_23; > +SELECT sq.a,length(sq.f) FROM (SELECT a, GROUP_CONCAT(b,b) AS f FROM t3 GROUP BY a ORDER BY a desc) as sq WHERE ROWNUM() <= 10; > +drop table t3; > + > +--echo # > +--echo # Prepared statements > +--echo # > + > +PREPARE stmt1 from "select a,b,rownum() from t1 where rownum() <= 2"; > +execute stmt1; > +execute stmt1; > +deallocate prepare stmt1; > + > +--echo # > +--echo # Views > +--echo # > + > +create view v1 as select t1.a,rownum() from t1; > +select * from v1; > +select t1.a,v1.* from t1,v1 where t1.a=v1.a; > +drop view v1; > + > +CREATE TABLE t3 (a INT); > +INSERT INTO t3 VALUES (1),(2),(3); > +CREATE VIEW v1 AS SELECT a FROM t3 WHERE ROWNUM() <= 2; > +SELECT * FROM v1; > +drop view v1; > +drop table t3; > + > +--echo # > +--echo # Reserved words > +--echo # > + > +create table t4 (a int, rownum int); > +insert into t4 (a,rownum) values (1,2); > +select t4.a,t4.rownum from t4; > +drop table t4; > + > +--echo # > +--echo # Test Oracle mode > +--echo # > + > +set SQL_MODE=ORACLE; > +select t1.a,rownum from t1 where rownum<=2; > +select t1.a,rownum() from t1 where rownum()<=2; > +--error ER_PARSE_ERROR > +create table t4 (a int, rownum int); > + > +DELIMITER |; > +DECLARE > + CURSOR c_cursor > + IS select a,b,rownum from t1 where rownum <= 2; > + v_a t1.a%TYPE; > + v_b t1.b%TYPE; > + v_rn t1.a%TYPE; > +BEGIN > + OPEN c_cursor; > + FETCH c_cursor INTO v_a, v_b, v_rn; > + WHILE c_cursor%FOUND LOOP > + SELECT concat(v_a,'--',v_b,'--',v_rn); > + FETCH c_cursor INTO v_a, v_b, v_rn; > + END LOOP; > + CLOSE c_cursor; > +END;| > +DELIMITER ;| > + > +--error ER_FUNCTION_CANNOT_BE_USED_IN_CLAUSE > +select a, rownum from t1 group by a, rownum having rownum < 3; > +select a, rownum from t1 group by a, rownum having "rownum" < 3; what's the difference between these two? Why the first is an error and the second is not? > +select a, rownum as r from t1 group by a, rownum having r < 3; And the this one. Why is it not an error either? > + > +set SQL_MODE=DEFAULT; > + > +--echo # Cleanup > + > +drop table t1,t2; > + > +--echo # > +--echo # INSERT > +--echo # > + > +create table t1 (a int not null primary key, b int); > +insert into t1 values (1,rownum()),(2,rownum()),(3,rownum()); > +select * from t1; > +drop table t1; > + > +--echo # > +--echo # INSERT DELAYED > +--echo # > + > +create table t1 (a int not null primary key, b int); > +insert delayed into t1 values (1,rownum()),(2,rownum()),(3,rownum()); > +select * from t1; > +drop table t1; > + > +--echo # > +--echo # INSERT ... RETURNING > +--echo # > + > +create or replace table t1 (a int); > +insert into t1 values (1),(2) returning a, rownum(); > +drop table t1; > + > +--echo # > +--echo # UPDATE > +--echo # > + > +create table t1 (a int not null primary key, b int); > +insert into t1 values (1,1),(2,2),(3,3); > +update t1 set b=0; > +update t1 set b=rownum()+1; > +select * from t1; > + > +update t1 set b=0; > +update t1 set b=rownum() where a < 10 and rownum() < 2; > +select * from t1; > +drop table t1; > + > +create table t1 (a int); > +insert into t1 values (10),(20),(30); > +update t1 set a = rownum(); > +select * from t1; > +update t1 set a = rownum(); > +select * from t1; > +drop table t1; > + > +--echo # > +--echo # DELETE > +--echo # > + > +create table t1 (a int not null primary key, b int); > +insert into t1 values (1,1),(2,0),(3,0); > +delete from t1 where a < 10 and rownum() < 2; > +select * from t1; > +drop table t1; > + > +--echo # > +--echo # MULTI-TABLE-DELETE > +--echo # > + > +create table t1 (a int not null primary key); > +insert into t1 values (1),(2),(3); > +create table t2 (a int not null primary key); > +insert into t2 values (1),(2),(3); > + > +delete t1,t2 from t1,t2 where t1.a=t2.a and rownum() <= 2; > +select * from t1; > +select * from t2; > +drop table t1,t2; > + > +--echo # > +--echo # MULTI-TABLE-UPDATE > + > +CREATE TABLE t1 (ID INT); > +CREATE TABLE t2 (ID INT, > + s1 TEXT, s2 TEXT, s3 VARCHAR(10), s4 TEXT, s5 VARCHAR(10)); > + > +INSERT INTO t1 VALUES (1),(2); > +INSERT INTO t2 VALUES (1,'test', 'test', 'test', 'test', 'test'), > + (2,'test', 'test', 'test', 'test', 'test'); > + > +SELECT * FROM t1 LEFT JOIN t2 USING(ID); > +UPDATE t1 LEFT JOIN t2 USING(ID) SET s1 = 'changed'; > +select * from t2; > +update t2 set s1=""; > +UPDATE t1 LEFT JOIN t2 USING(ID) SET s1 = 'changed' where rownum() <=1; > +select * from t2; > +drop table t1,t2; > + > +--echo # > +--echo # LOAD DATA > +--echo # > + > +create table t1 (a int, b int, c int); > +load data infile '../../std_data/loaddata7.dat' into table t1 fields terminated by ',' lines terminated by "\r\n" (a,b) set c=rownum(); > +select * from t1; > +drop table t1; > + > +--echo # > +--echo # LIMIT OPTIMIZATION > +--echo # > + > +create table t1 (a int); > +insert into t1 select seq from seq_1_to_100; > + > +flush status; > +select * from t1 where rownum() <= 3; > +show status like "Rows_read"; > +flush status; > +flush status; > +select * from t1 where rownum() <= 4 and rownum() <= 3; > +show status like "Rows_read"; > +flush status; > +select * from t1 where rownum() < 4 and a > 10; > +show status like "Rows_read"; > +flush status; > +select * from t1 where 3 >= rownum(); > +show status like "Rows_read"; > +flush status; > +select * from t1 where 4 > rownum() and a > 20; > +show status like "Rows_read"; > +flush status; > +select * from t1 where rownum() = 1 and a > 10; > +show status like "Rows_read"; > +flush status; > +select * from t1 where a > 30 && 1 = rownum(); > +show status like "Rows_read"; > +flush status; > + > +--echo # No limit optimization > + > +select * from t1 where rownum() > 10; > +show status like "Rows_read"; > +flush status; > +select * from t1 where 10 < rownum(); > +show status like "Rows_read"; > +flush status; > +select * from t1 where rownum() >= 10; > +show status like "Rows_read"; > +flush status; > +select * from t1 where 10 < rownum(); > +show status like "Rows_read"; > +flush status; > +select * from t1 where 10 <= rownum(); > +show status like "Rows_read"; > +flush status; > +select * from t1 where 2 = rownum(); > +show status like "Rows_read"; > +flush status; > +select * from t1 where rownum() = 2; > +show status like "Rows_read"; > +flush status; > +select * from t1 where rownum() <= 0; > +show status like "Rows_read"; > +flush status; > +select * from t1 where rownum() < 10 limit 4, 4; > +show status like "Rows_read"; > +flush status; > +select * from t1 where rownum() < 10 order by a; > +show status like "Rows_read"; > +flush status; > + > + > +--echo # rownum and limit > + > +select * from t1 where rownum() < 4 limit 10; > +show status like "Rows_read"; > +flush status; > +select * from t1 where rownum() < 10 limit 4; > +show status like "Rows_read"; > + > +drop table t1; > + > +--echo # > +--echo # Rownum examples from Woqutech > +--echo # > + > +set SQL_MODE=ORACLE; > +create table t1 (c1 int ,c2 varchar(20)) engine=myisam; > +insert into t1 values (1, 'aaa'),(2, 'bbb'),(3, 'ccc'),(4, 'ddd'),(5, 'eee'); > +update t1 set c2 = 'xxx' where rownum = 2; > +select * from t1 where c2='xxx'; > +update t1 set c2 = 'xxx' where rownum < 3; > +select * from t1 where c2='xxx'; > +delete from t1 where rownum = 2; > +select count(*) from t1; > +delete from t1 where rownum < 3; > +select count(*) from t1; > +delete from t1 where c1=rownum ; > +select count(*) from t1; > +delete from t1 where c1=rownum+2 ; > +select count(*) from t1; > +set SQL_MODE=DEFAULT; > +drop table t1; > + > +--echo # > +--echo # Rownum() used in not supported places (returns 0 or gives an error) > +--echo # > + > +set @a=rownum(); > +select @a; > + > +--error ER_GENERATED_COLUMN_FUNCTION_IS_NOT_ALLOWED > +create or replace table t (a int, b int as (rownum()) virtual); > + > +create table t1 (a int); > +insert into t1 values (3),(1),(5),(8),(4); > +handler t1 open; > +--error ER_GENERATED_COLUMN_FUNCTION_IS_NOT_ALLOWED > +handler t1 read next where rownum() < 1; > +handler t1 close; > +drop table t1; > + > +# rownum() executed in a function will be run in the function context. > + > +create table t1 (a int not null primary key, b int); > +insert into t1 values (1,1),(2,2),(3,3); > +create function f() returns int return rownum(); > +select a, rownum(), f() from t1; > +drop function f; > +drop table t1; > + > +# rownum() executed in a trigger will be run in the function context. > + > +create or replace table t1 (a int, r int); > +create trigger tr before update on t1 for each row set NEW.r = rownum(); > +insert into t1 (a) values (1),(2); > +select * from t1; > +update t1 set a=a+10; > +select * from t1; > +drop trigger tr; > +drop table t1; > + > +--echo # > +--echo # LIMIT optimisation > +--echo # > + > +create table t1 (a int); > +insert into t1 values (1),(2),(3),(4),(5); > + > +let $query= > +select * from (select a from t1 where a < 1000) as tt where rownum() <= 2; > +flush status; > +eval $query; > +show status like "Rows_read"; > +eval explain extended $query; > +eval prepare stmt from "$query"; > +flush status; > +execute stmt; > +show status like "Rows_read"; > +flush status; > +execute stmt; > +show status like "Rows_read"; > +deallocate prepare stmt; > + > + > +let $query= > +select * from (select a from t1 where a < 1000 group by a) as tt where rownum() <= 2; > +flush status; > +eval $query; > +show status like "Rows_read"; > +eval explain extended $query; > +eval prepare stmt from "$query"; > +execute stmt; > +execute stmt; > +deallocate prepare stmt; > + > +let $query= > +select * from (select a from t1 where a < 1000 group by a order by 1) as tt where rownum() <= 2; > +flush status; > +eval $query; > +show status like "Rows_read"; > +eval explain extended $query; > +eval prepare stmt from "$query"; > +execute stmt; > +execute stmt; > +deallocate prepare stmt; > + > +let $query= > +select * from (select a from t1 where a < 1000 union select 10) as tt where rownum() <= 2; > +flush status; > +eval $query; > +show status like "Rows_read"; > +eval explain extended $query; > +eval prepare stmt from "$query"; > +execute stmt; > +execute stmt; > +deallocate prepare stmt; > + > +--echo # Other limit > + > +let $query= > +select * from (select a from t1 where a < 1000 group by a order by 1 limit 1) as tt where rownum() <= 2; > +eval $query; > +eval explain extended $query; > +eval prepare stmt from "$query"; > +execute stmt; > +execute stmt; > +deallocate prepare stmt; > + > +--echo # Other limit less > + > +let $query= > +select * from (select a from t1 where a < 1000 group by a order by 1 limit 10) as tt where rownum() <= 2; > +eval $query; > +eval explain extended $query; > +eval prepare stmt from "$query"; > +execute stmt; > +execute stmt; > +deallocate prepare stmt; > + > +let $query= > +select * from (select a from t1 where a < 1000 union select 10 limit 1) as tt where rownum() <= 2; > +eval $query; > +eval explain extended $query; > +eval prepare stmt from "$query"; > +execute stmt; > +execute stmt; > +deallocate prepare stmt; > + > +--echo # < rownum > + > +let $query= > +select * from (select a from t1 where a < 1000) as tt where rownum() < 2; > +eval $query; > +eval explain extended $query; > +eval prepare stmt from "$query"; > +execute stmt; > +execute stmt; > +deallocate prepare stmt; > + > +--echo # Simple expression > + > +let $query= > +select * from (select a from t1 where a < 1000) as tt where rownum() <= 1+1; > +eval $query; > +eval explain extended $query; > +eval prepare stmt from "$query"; > +execute stmt; > +execute stmt; > +deallocate prepare stmt; > + > +--echo # Simple expression reversed > + > +let $query= > +select * from (select a from t1 where a < 1000) as tt where 1+1 >= rownum(); > +eval $query; > +eval explain extended $query; > +eval prepare stmt from "$query"; > +execute stmt; > +execute stmt; > +deallocate prepare stmt; > + > +--echo # expensive (no opt) > + > +let $query= > +select * from (select a from t1 where a < 1000) as tt where (select max(a) from t1) >= rownum(); > +eval $query; > +eval explain extended $query; > +eval prepare stmt from "$query"; > +execute stmt; > +execute stmt; > +deallocate prepare stmt; > + > +drop table t1; > diff --git a/mysql-test/main/rownum.result b/mysql-test/main/rownum.result > new file mode 100644 > index 00000000000..e7ea92e5dde > --- /dev/null > +++ b/mysql-test/main/rownum.result > @@ -0,0 +1,925 @@ > +CREATE OR REPLACE TABLE t1(a int, b int) engine=aria; > +CREATE OR REPLACE TABLE t2(a int, b int) engine=aria; > +insert into t1 values (1,10),(2,20),(3,30); > +insert into t2 values (2,21),(3,31),(4,41); > +# > +# Simple selects > +# > +select a,b,rownum() from t1; > +a b rownum() > +1 10 1 > +2 20 2 > +3 30 3 > +select a,b,rownum() from t1 where rownum() < 2; > +a b rownum() > +1 10 1 > +select a,b from t1 where rownum() <= 2; > +a b > +1 10 > +2 20 > +select a,b from t1 where rownum() > 2; > +a b > +# > +# Subqueries > +# > +select t1.a,rownum(),t3.a,t3.t2_rownum from t1, (select t2.a,rownum() as t2_rownum from t2 where rownum() <=2) t3; > +a rownum() a t2_rownum > +1 1 2 1 > +1 2 3 2 > +2 3 2 1 > +2 4 3 2 > +3 5 2 1 > +3 6 3 2 > +select t1.a, (select t2.b from t2 where t1.a=t2.a and rownum() <= 1) 'b' from t1; > +a b > +1 NULL > +2 21 > +3 31 > +select t1.a, t3.a from t1, (select * from t2 where rownum() <= 2) t3; > +a a > +1 2 > +1 3 > +2 2 > +2 3 > +3 2 > +3 3 > +select * from (select tt.*,rownum() as id from (select * from t1) tt) t3 where id>2; > +a b id > +3 30 3 > +# > +# Joins > +# > +select t1.a,t1.b,t2.a,t2.b,rownum() from t1,t2 where rownum() <= 4; > +a b a b rownum() > +1 10 2 21 1 > +2 20 2 21 2 > +3 30 2 21 3 > +1 10 3 31 4 > +select *,rownum() from t1,t2 where t1.a=t2.a and rownum()<=2; > +a b a b rownum() > +2 20 2 21 1 > +3 30 3 31 2 > +select * from t1 left join t2 on (t2.a=t1.a and rownum()=0); > +a b a b > +1 10 NULL NULL > +2 20 NULL NULL > +3 30 NULL NULL > +select * from t1 left join t2 on (t2.a=t1.a and rownum()>1); > +a b a b > +1 10 NULL NULL > +2 20 NULL NULL > +3 30 NULL NULL > +select * from t1 left join t2 on (t2.a=t1.a and rownum()<1); > +a b a b > +1 10 NULL NULL > +2 20 NULL NULL > +3 30 NULL NULL > +select * from t1 left join t2 on (t2.a=t1.a and rownum()=1); > +a b a b > +2 20 2 21 > +1 10 NULL NULL > +3 30 NULL NULL > +select * from t1 left join t2 on (t2.a=t1.a and rownum()>=1); > +a b a b > +2 20 2 21 > +3 30 3 31 > +1 10 NULL NULL > +# > +# Union > +# > +select * from t1 where rownum() <=2 union select * from t2 where rownum()<= 1; > +a b > +1 10 > +2 20 > +2 21 > +# > +# Order by > +# > +select * from t1 where rownum() <= 2 order by a desc; > +a b > +2 20 > +1 10 > +explain select * from t1 where rownum() <= 2 order by a desc; > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 ALL NULL NULL NULL NULL 3 Using where; Using filesort > +select t1.a,t1.b,rownum() from t1 where rownum() <= 2 order by a desc; > +a b rownum() > +2 20 2 > +1 10 1 > +explain select t1.a,t1.b,rownum() from t1 where rownum() <= 2 order by a desc; > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 ALL NULL NULL NULL NULL 3 Using where; Using temporary; Using filesort > +select *,rownum() from t1,t2; > +a b a b rownum() > +1 10 2 21 1 > +2 20 2 21 2 > +3 30 2 21 3 > +1 10 3 31 4 > +2 20 3 31 5 > +3 30 3 31 6 > +1 10 4 41 7 > +2 20 4 41 8 > +3 30 4 41 9 > +select *,rownum() from t1,t2 order by t2.a desc, t1.a desc; > +a b a b rownum() > +3 30 4 41 9 > +2 20 4 41 8 > +1 10 4 41 7 > +3 30 3 31 6 > +2 20 3 31 5 > +1 10 3 31 4 > +3 30 2 21 3 > +2 20 2 21 2 > +1 10 2 21 1 > +select * from (select * from t1 order by a desc) as t where rownum() <= 2; > +a b > +3 30 > +2 20 > +select * from t1,t2 where t1.a=t2.a and rownum()<=2 order by t1.a,t2.a; > +a b a b > +2 20 2 21 > +3 30 3 31 > +# > +# Having > +# > +select t1.a, sum(t1.b), rownum() as 'r' from t1 group by t1.a having r <= 2; > +a sum(t1.b) r > +1 10 1 > +2 20 2 > +select * from t1 having rownum() <= 2; > +ERROR HY000: Function 'rownum()' cannot be used in the HAVING clause > +select t1.a, sum(t1.b), rownum() from t1 group by t1.a; > +a sum(t1.b) rownum() > +1 10 1 > +2 20 2 > +3 30 3 > +select t1.a, sum(t1.b), rownum() as 'r' from t1 group by t1.a having rownum() <= 2; > +ERROR HY000: Function 'rownum()' cannot be used in the HAVING clause > +# > +# Sum functions > +# > +select max(rownum()),min(rownum()) from t1; > +max(rownum()) min(rownum()) > +3 1 > +select sum(rownum()),avg(rownum()) from t1; > +sum(rownum()) avg(rownum()) > +6 2.0000 > +# > +# Group by > +# > +select t1.a,sum(t1.b) from t1 where rownum() < 2 group by t1.a; > +a sum(t1.b) > +1 10 > +select t1.a,sum(t2.b) from t1 JOIN t2 ON (t1.a=t2.a) where rownum() <= 2 group by t1.a; > +a sum(t2.b) > +2 21 > +3 31 > +select * from (select t1.a,sum(t2.b) from t1 JOIN t2 ON (t1.a=t2.a) group by t1.a) as t where rownum() <= 1; > +a sum(t2.b) > +2 21 > +select t1.a,sum(rownum()),count(*) from t1 where rownum() <= 2 group by t1.a; > +a sum(rownum()) count(*) > +1 1 1 > +2 2 1 > +select * from (select t1.a,sum(t1.b) from t1 group by t1.a) as t3 where rownum() < 2; > +a sum(t1.b) > +1 10 > +create table t3 (a int) engine=myisam; > +insert into t3 values (3),(5),(5),(3); > +select a, max(rownum()) from t3 group by a; > +a max(rownum()) > +3 4 > +5 3 > +drop table t3; > +CREATE TABLE t3 ( > +a int(11) DEFAULT NULL, > +b varchar(1024) DEFAULT NULL > +); > +insert into t3 select mod(seq*3,20)+1, repeat(char(33+mod(seq,90)),mod(seq,10)*100) from seq_1_to_23; > +SELECT sq.a,length(sq.f) FROM (SELECT a, GROUP_CONCAT(b,b) AS f FROM t3 GROUP BY a ORDER BY a desc) as sq WHERE ROWNUM() <= 10; > +a length(sq.f) > +20 600 > +19 1200 > +18 1800 > +17 400 > +16 1000 > +15 1600 > +14 200 > +13 800 > +12 1400 > +11 0 > +drop table t3; > +# > +# Prepared statements > +# > +PREPARE stmt1 from "select a,b,rownum() from t1 where rownum() <= 2"; > +execute stmt1; > +a b rownum() > +1 10 1 > +2 20 2 > +execute stmt1; > +a b rownum() > +1 10 1 > +2 20 2 > +deallocate prepare stmt1; > +# > +# Views > +# > +create view v1 as select t1.a,rownum() from t1; > +select * from v1; > +a rownum() > +1 1 > +2 2 > +3 3 > +select t1.a,v1.* from t1,v1 where t1.a=v1.a; > +a a rownum() > +1 1 1 > +2 2 2 > +3 3 3 > +drop view v1; > +CREATE TABLE t3 (a INT); > +INSERT INTO t3 VALUES (1),(2),(3); > +CREATE VIEW v1 AS SELECT a FROM t3 WHERE ROWNUM() <= 2; > +SELECT * FROM v1; > +a > +1 > +2 > +drop view v1; > +drop table t3; > +# > +# Reserved words > +# > +create table t4 (a int, rownum int); > +insert into t4 (a,rownum) values (1,2); > +select t4.a,t4.rownum from t4; > +a rownum > +1 2 > +drop table t4; > +# > +# Test Oracle mode > +# > +set SQL_MODE=ORACLE; > +select t1.a,rownum from t1 where rownum<=2; > +a rownum > +1 1 > +2 2 > +select t1.a,rownum() from t1 where rownum()<=2; > +a rownum() > +1 1 > +2 2 > +create table t4 (a int, rownum int); > +ERROR 42000: You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'rownum int)' at line 1 > +DECLARE > +CURSOR c_cursor > +IS select a,b,rownum from t1 where rownum <= 2; > +v_a t1.a%TYPE; > +v_b t1.b%TYPE; > +v_rn t1.a%TYPE; > +BEGIN > +OPEN c_cursor; > +FETCH c_cursor INTO v_a, v_b, v_rn; > +WHILE c_cursor%FOUND LOOP > +SELECT concat(v_a,'--',v_b,'--',v_rn); > +FETCH c_cursor INTO v_a, v_b, v_rn; > +END LOOP; > +CLOSE c_cursor; > +END;| > +concat(v_a,'--',v_b,'--',v_rn) > +1--10--1 > +concat(v_a,'--',v_b,'--',v_rn) > +2--20--2 > +select a, rownum from t1 group by a, rownum having rownum < 3; > +ERROR HY000: Function 'rownum()' cannot be used in the HAVING clause > +select a, rownum as r from t1 group by a, rownum having r < 3; > +a r > +1 1 > +2 2 > +select a, rownum from t1 group by a, rownum having "rownum" < 3; > +a rownum > +1 1 > +2 2 > +set SQL_MODE=DEFAULT; > +# Cleanup > +drop table t1,t2; > +# > +# INSERT > +# > +create table t1 (a int not null primary key, b int); > +insert into t1 values (1,rownum()),(2,rownum()),(3,rownum()); > +select * from t1; > +a b > +1 1 > +2 2 > +3 3 > +drop table t1; > +# > +# INSERT DELAYED > +# > +create table t1 (a int not null primary key, b int); > +insert delayed into t1 values (1,rownum()),(2,rownum()),(3,rownum()); > +select * from t1; > +a b > +1 1 > +2 2 > +3 3 > +drop table t1; > +# > +# INSERT ... RETURNING > +# > +create or replace table t1 (a int); > +insert into t1 values (1),(2) returning a, rownum(); > +a rownum() > +1 1 > +2 2 > +drop table t1; > +# > +# UPDATE > +# > +create table t1 (a int not null primary key, b int); > +insert into t1 values (1,1),(2,2),(3,3); > +update t1 set b=0; > +update t1 set b=rownum()+1; > +select * from t1; > +a b > +1 2 > +2 3 > +3 4 > +update t1 set b=0; > +update t1 set b=rownum() where a < 10 and rownum() < 2; > +select * from t1; > +a b > +1 1 > +2 0 > +3 0 > +drop table t1; > +create table t1 (a int); > +insert into t1 values (10),(20),(30); > +update t1 set a = rownum(); > +select * from t1; > +a > +1 > +2 > +3 > +update t1 set a = rownum(); > +select * from t1; > +a > +1 > +2 > +3 > +drop table t1; > +# > +# DELETE > +# > +create table t1 (a int not null primary key, b int); > +insert into t1 values (1,1),(2,0),(3,0); > +delete from t1 where a < 10 and rownum() < 2; > +select * from t1; > +a b > +2 0 > +3 0 > +drop table t1; > +# > +# MULTI-TABLE-DELETE > +# > +create table t1 (a int not null primary key); > +insert into t1 values (1),(2),(3); > +create table t2 (a int not null primary key); > +insert into t2 values (1),(2),(3); > +delete t1,t2 from t1,t2 where t1.a=t2.a and rownum() <= 2; > +select * from t1; > +a > +3 > +select * from t2; > +a > +3 > +drop table t1,t2; > +# > +# MULTI-TABLE-UPDATE > +CREATE TABLE t1 (ID INT); > +CREATE TABLE t2 (ID INT, > +s1 TEXT, s2 TEXT, s3 VARCHAR(10), s4 TEXT, s5 VARCHAR(10)); > +INSERT INTO t1 VALUES (1),(2); > +INSERT INTO t2 VALUES (1,'test', 'test', 'test', 'test', 'test'), > +(2,'test', 'test', 'test', 'test', 'test'); > +SELECT * FROM t1 LEFT JOIN t2 USING(ID); > +ID s1 s2 s3 s4 s5 > +1 test test test test test > +2 test test test test test > +UPDATE t1 LEFT JOIN t2 USING(ID) SET s1 = 'changed'; > +select * from t2; > +ID s1 s2 s3 s4 s5 > +1 changed test test test test > +2 changed test test test test > +update t2 set s1=""; > +UPDATE t1 LEFT JOIN t2 USING(ID) SET s1 = 'changed' where rownum() <=1; > +select * from t2; > +ID s1 s2 s3 s4 s5 > +1 changed test test test test > +2 test test test test > +drop table t1,t2; > +# > +# LOAD DATA > +# > +create table t1 (a int, b int, c int); > +load data infile '../../std_data/loaddata7.dat' into table t1 fields terminated by ',' lines terminated by "\r\n" (a,b) set c=rownum(); > +select * from t1; > +a b c > +2 2 1 > +3 3 2 > +4 4 3 > +5 5 4 > +6 6 5 > +drop table t1; > +# > +# LIMIT OPTIMIZATION > +# > +create table t1 (a int); > +insert into t1 select seq from seq_1_to_100; > +flush status; > +select * from t1 where rownum() <= 3; > +a > +1 > +2 > +3 > +show status like "Rows_read"; > +Variable_name Value > +Rows_read 3 > +flush status; > +flush status; why twice? > +select * from t1 where rownum() <= 4 and rownum() <= 3; > +a > +1 > +2 > +3 > +show status like "Rows_read"; > +Variable_name Value > +Rows_read 3 > +flush status; > +select * from t1 where rownum() < 4 and a > 10; > +a > +11 > +12 > +13 > +show status like "Rows_read"; > +Variable_name Value > +Rows_read 13 > +flush status; > +select * from t1 where 3 >= rownum(); > +a > +1 > +2 > +3 > +show status like "Rows_read"; > +Variable_name Value > +Rows_read 3 > +flush status; > +select * from t1 where 4 > rownum() and a > 20; > +a > +21 > +22 > +23 > +show status like "Rows_read"; > +Variable_name Value > +Rows_read 23 > +flush status; > +select * from t1 where rownum() = 1 and a > 10; > +a > +11 > +show status like "Rows_read"; > +Variable_name Value > +Rows_read 11 > +flush status; > +select * from t1 where a > 30 && 1 = rownum(); > +a > +31 > +show status like "Rows_read"; > +Variable_name Value > +Rows_read 31 > +flush status; > +# No limit optimization > +select * from t1 where rownum() > 10; > +a > +show status like "Rows_read"; > +Variable_name Value > +Rows_read 100 You could've optimized it to `limit 0` but I guess it's not a particularly common use case > +flush status; > +select * from t1 where 10 < rownum(); > +a > +show status like "Rows_read"; > +Variable_name Value > +Rows_read 100 > +flush status; > +select * from t1 where rownum() >= 10; > +a > +show status like "Rows_read"; > +Variable_name Value > +Rows_read 100 > +flush status; > +select * from t1 where 10 < rownum(); > +a > +show status like "Rows_read"; > +Variable_name Value > +Rows_read 100 > +flush status; > +select * from t1 where 10 <= rownum(); > +a > +show status like "Rows_read"; > +Variable_name Value > +Rows_read 100 > +flush status; > +select * from t1 where 2 = rownum(); > +a > +show status like "Rows_read"; > +Variable_name Value > +Rows_read 100 > +flush status; > +select * from t1 where rownum() = 2; > +a > +show status like "Rows_read"; > +Variable_name Value > +Rows_read 100 > +flush status; > +select * from t1 where rownum() <= 0; > +a > +show status like "Rows_read"; > +Variable_name Value > +Rows_read 100 > +flush status; > +select * from t1 where rownum() < 10 limit 4, 4; can you add a test for `select a, rownum() from t1 limit 4, 4` please ? > +a > +5 > +6 > +7 > +8 > +show status like "Rows_read"; > +Variable_name Value > +Rows_read 8 > +flush status; > +select * from t1 where rownum() < 10 order by a; > +a > +1 > +2 > +3 > +4 > +5 > +6 > +7 > +8 > +9 > +show status like "Rows_read"; > +Variable_name Value > +Rows_read 100 > +flush status; > +# rownum and limit > +select * from t1 where rownum() < 4 limit 10; > +a > +1 > +2 > +3 > +show status like "Rows_read"; > +Variable_name Value > +Rows_read 3 > +flush status; > +select * from t1 where rownum() < 10 limit 4; > +a > +1 > +2 > +3 > +4 > +show status like "Rows_read"; > +Variable_name Value > +Rows_read 4 > +drop table t1; > +# > +# Rownum examples from Woqutech > +# > +set SQL_MODE=ORACLE; > +create table t1 (c1 int ,c2 varchar(20)) engine=myisam; > +insert into t1 values (1, 'aaa'),(2, 'bbb'),(3, 'ccc'),(4, 'ddd'),(5, 'eee'); > +update t1 set c2 = 'xxx' where rownum = 2; > +select * from t1 where c2='xxx'; > +c1 c2 > +update t1 set c2 = 'xxx' where rownum < 3; > +select * from t1 where c2='xxx'; > +c1 c2 > +1 xxx > +2 xxx > +delete from t1 where rownum = 2; > +select count(*) from t1; > +count(*) > +5 > +delete from t1 where rownum < 3; > +select count(*) from t1; > +count(*) > +3 > +delete from t1 where c1=rownum ; > +select count(*) from t1; > +count(*) > +3 > +delete from t1 where c1=rownum+2 ; > +select count(*) from t1; > +count(*) > +0 > +set SQL_MODE=DEFAULT; > +drop table t1; > +# > +# Rownum() used in not supported places (returns 0 or gives an error) > +# > +set @a=rownum(); > +select @a; > +@a > +0 > +create or replace table t (a int, b int as (rownum()) virtual); > +ERROR HY000: Function or expression 'rownum()' cannot be used in the GENERATED ALWAYS AS clause of `b` > +create table t1 (a int); > +insert into t1 values (3),(1),(5),(8),(4); > +handler t1 open; > +handler t1 read next where rownum() < 1; > +ERROR HY000: Function or expression 'rownum()' cannot be used in the WHERE clause of `HANDLER` why is that? `handler t1 read next limit 2` works just fine. > +handler t1 close; > +drop table t1; > +create table t1 (a int not null primary key, b int); > +insert into t1 values (1,1),(2,2),(3,3); > +create function f() returns int return rownum(); > +select a, rownum(), f() from t1; > +a rownum() f() > +1 1 0 > +2 2 0 > +3 3 0 > +drop function f; > +drop table t1; > +create or replace table t1 (a int, r int); > +create trigger tr before update on t1 for each row set NEW.r = rownum(); > +insert into t1 (a) values (1),(2); > +select * from t1; > +a r > +1 NULL > +2 NULL > +update t1 set a=a+10; > +select * from t1; > +a r > +11 0 > +12 0 > +drop trigger tr; > +drop table t1; > +# > +# LIMIT optimisation > +# > +create table t1 (a int); > +insert into t1 values (1),(2),(3),(4),(5); > +flush status; > +select * from (select a from t1 where a < 1000) as tt where rownum() <= 2; > +a > +1 > +2 > +show status like "Rows_read"; > +Variable_name Value > +Rows_read 2 > +explain extended select * from (select a from t1 where a < 1000) as tt where rownum() <= 2; > +id select_type table type possible_keys key key_len ref rows filtered Extra > +1 PRIMARY <derived2> ALL NULL NULL NULL NULL 2 100.00 Using where > +2 DERIVED t1 ALL NULL NULL NULL NULL 5 100.00 Using where > +Warnings: > +Note 1003 /* select#1 */ select `tt`.`a` AS `a` from (/* select#2 */ select `test`.`t1`.`a` AS `a` from `test`.`t1` where `test`.`t1`.`a` < 1000 limit 2) `tt` where rownum() <= 2 > +prepare stmt from "select * from (select a from t1 where a < 1000) as tt where rownum() <= 2"; > +flush status; > +execute stmt; > +a > +1 > +2 > +show status like "Rows_read"; > +Variable_name Value > +Rows_read 2 > +flush status; > +execute stmt; > +a > +1 > +2 > +show status like "Rows_read"; > +Variable_name Value > +Rows_read 2 > +deallocate prepare stmt; > +flush status; > +select * from (select a from t1 where a < 1000 group by a) as tt where rownum() <= 2; > +a > +1 > +2 > +show status like "Rows_read"; > +Variable_name Value > +Rows_read 5 > +explain extended select * from (select a from t1 where a < 1000 group by a) as tt where rownum() <= 2; > +id select_type table type possible_keys key key_len ref rows filtered Extra > +1 PRIMARY <derived2> ALL NULL NULL NULL NULL 2 100.00 Using where > +2 DERIVED t1 ALL NULL NULL NULL NULL 5 100.00 Using where; Using temporary; Using filesort > +Warnings: > +Note 1003 /* select#1 */ select `tt`.`a` AS `a` from (/* select#2 */ select `test`.`t1`.`a` AS `a` from `test`.`t1` where `test`.`t1`.`a` < 1000 group by `test`.`t1`.`a` limit 2) `tt` where rownum() <= 2 > +prepare stmt from "select * from (select a from t1 where a < 1000 group by a) as tt where rownum() <= 2"; > +execute stmt; > +a > +1 > +2 > +execute stmt; > +a > +1 > +2 > +deallocate prepare stmt; > +flush status; > +select * from (select a from t1 where a < 1000 group by a order by 1) as tt where rownum() <= 2; > +a > +1 > +2 > +show status like "Rows_read"; > +Variable_name Value > +Rows_read 5 > +explain extended select * from (select a from t1 where a < 1000 group by a order by 1) as tt where rownum() <= 2; > +id select_type table type possible_keys key key_len ref rows filtered Extra > +1 PRIMARY <derived2> ALL NULL NULL NULL NULL 2 100.00 Using where > +2 DERIVED t1 ALL NULL NULL NULL NULL 5 100.00 Using where; Using temporary; Using filesort > +Warnings: > +Note 1003 /* select#1 */ select `tt`.`a` AS `a` from (/* select#2 */ select `test`.`t1`.`a` AS `a` from `test`.`t1` where `test`.`t1`.`a` < 1000 group by `test`.`t1`.`a` order by 1 limit 2) `tt` where rownum() <= 2 > +prepare stmt from "select * from (select a from t1 where a < 1000 group by a order by 1) as tt where rownum() <= 2"; > +execute stmt; > +a > +1 > +2 > +execute stmt; > +a > +1 > +2 > +deallocate prepare stmt; > +flush status; > +select * from (select a from t1 where a < 1000 union select 10) as tt where rownum() <= 2; > +a > +1 > +2 > +show status like "Rows_read"; > +Variable_name Value > +Rows_read 5 > +explain extended select * from (select a from t1 where a < 1000 union select 10) as tt where rownum() <= 2; > +id select_type table type possible_keys key key_len ref rows filtered Extra > +1 PRIMARY <derived2> ALL NULL NULL NULL NULL 2 100.00 Using where > +2 DERIVED t1 ALL NULL NULL NULL NULL 5 100.00 Using where > +3 UNION NULL NULL NULL NULL NULL NULL NULL NULL No tables used > +NULL UNION RESULT <union2,3> ALL NULL NULL NULL NULL NULL NULL > +Warnings: > +Note 1003 /* select#1 */ select `tt`.`a` AS `a` from (/* select#2 */ select `test`.`t1`.`a` AS `a` from `test`.`t1` where `test`.`t1`.`a` < 1000 union /* select#3 */ select 10 AS `10` limit 2) `tt` where rownum() <= 2 > +prepare stmt from "select * from (select a from t1 where a < 1000 union select 10) as tt where rownum() <= 2"; > +execute stmt; > +a > +1 > +2 > +execute stmt; > +a > +1 > +2 > +deallocate prepare stmt; > +# Other limit > +select * from (select a from t1 where a < 1000 group by a order by 1 limit 1) as tt where rownum() <= 2; > +a > +1 > +explain extended select * from (select a from t1 where a < 1000 group by a order by 1 limit 1) as tt where rownum() <= 2; > +id select_type table type possible_keys key key_len ref rows filtered Extra > +1 PRIMARY <derived2> ALL NULL NULL NULL NULL 2 100.00 Using where > +2 DERIVED t1 ALL NULL NULL NULL NULL 5 100.00 Using where; Using temporary; Using filesort > +Warnings: > +Note 1003 /* select#1 */ select `tt`.`a` AS `a` from (/* select#2 */ select `test`.`t1`.`a` AS `a` from `test`.`t1` where `test`.`t1`.`a` < 1000 group by `test`.`t1`.`a` order by 1 limit 1) `tt` where rownum() <= 2 > +prepare stmt from "select * from (select a from t1 where a < 1000 group by a order by 1 limit 1) as tt where rownum() <= 2"; > +execute stmt; > +a > +1 > +execute stmt; > +a > +1 > +deallocate prepare stmt; > +# Other limit less > +select * from (select a from t1 where a < 1000 group by a order by 1 limit 10) as tt where rownum() <= 2; > +a > +1 > +2 > +explain extended select * from (select a from t1 where a < 1000 group by a order by 1 limit 10) as tt where rownum() <= 2; > +id select_type table type possible_keys key key_len ref rows filtered Extra > +1 PRIMARY <derived2> ALL NULL NULL NULL NULL 2 100.00 Using where > +2 DERIVED t1 ALL NULL NULL NULL NULL 5 100.00 Using where; Using temporary; Using filesort > +Warnings: > +Note 1003 /* select#1 */ select `tt`.`a` AS `a` from (/* select#2 */ select `test`.`t1`.`a` AS `a` from `test`.`t1` where `test`.`t1`.`a` < 1000 group by `test`.`t1`.`a` order by 1 limit 2) `tt` where rownum() <= 2 > +prepare stmt from "select * from (select a from t1 where a < 1000 group by a order by 1 limit 10) as tt where rownum() <= 2"; > +execute stmt; > +a > +1 > +2 > +execute stmt; > +a > +1 > +2 > +deallocate prepare stmt; > +select * from (select a from t1 where a < 1000 union select 10 limit 1) as tt where rownum() <= 2; > +a > +1 > +explain extended select * from (select a from t1 where a < 1000 union select 10 limit 1) as tt where rownum() <= 2; > +id select_type table type possible_keys key key_len ref rows filtered Extra > +1 PRIMARY <derived2> ALL NULL NULL NULL NULL 5 100.00 Using where > +2 DERIVED t1 ALL NULL NULL NULL NULL 5 100.00 Using where > +3 UNION NULL NULL NULL NULL NULL NULL NULL NULL No tables used > +NULL UNION RESULT <union2,3> ALL NULL NULL NULL NULL NULL NULL > +Warnings: > +Note 1003 /* select#1 */ select `tt`.`a` AS `a` from (/* select#2 */ select `test`.`t1`.`a` AS `a` from `test`.`t1` where `test`.`t1`.`a` < 1000 union /* select#3 */ select 10 AS `10` limit 1) `tt` where rownum() <= 2 > +prepare stmt from "select * from (select a from t1 where a < 1000 union select 10 limit 1) as tt where rownum() <= 2"; > +execute stmt; > +a > +1 > +execute stmt; > +a > +1 > +deallocate prepare stmt; > +# < rownum > +select * from (select a from t1 where a < 1000) as tt where rownum() < 2; > +a > +1 > +explain extended select * from (select a from t1 where a < 1000) as tt where rownum() < 2; > +id select_type table type possible_keys key key_len ref rows filtered Extra > +1 PRIMARY <derived2> ALL NULL NULL NULL NULL 2 100.00 Using where > +2 DERIVED t1 ALL NULL NULL NULL NULL 5 100.00 Using where > +Warnings: > +Note 1003 /* select#1 */ select `tt`.`a` AS `a` from (/* select#2 */ select `test`.`t1`.`a` AS `a` from `test`.`t1` where `test`.`t1`.`a` < 1000 limit 1) `tt` where rownum() < 2 > +prepare stmt from "select * from (select a from t1 where a < 1000) as tt where rownum() < 2"; > +execute stmt; > +a > +1 > +execute stmt; > +a > +1 > +deallocate prepare stmt; > +# Simple expression > +select * from (select a from t1 where a < 1000) as tt where rownum() <= 1+1; > +a > +1 > +2 > +explain extended select * from (select a from t1 where a < 1000) as tt where rownum() <= 1+1; > +id select_type table type possible_keys key key_len ref rows filtered Extra > +1 PRIMARY <derived2> ALL NULL NULL NULL NULL 2 100.00 Using where > +2 DERIVED t1 ALL NULL NULL NULL NULL 5 100.00 Using where > +Warnings: > +Note 1003 /* select#1 */ select `tt`.`a` AS `a` from (/* select#2 */ select `test`.`t1`.`a` AS `a` from `test`.`t1` where `test`.`t1`.`a` < 1000 limit 2) `tt` where rownum() <= <cache>(1 + 1) > +prepare stmt from "select * from (select a from t1 where a < 1000) as tt where rownum() <= 1+1"; > +execute stmt; > +a > +1 > +2 > +execute stmt; > +a > +1 > +2 > +deallocate prepare stmt; > +# Simple expression reversed > +select * from (select a from t1 where a < 1000) as tt where 1+1 >= rownum(); > +a > +1 > +2 > +explain extended select * from (select a from t1 where a < 1000) as tt where 1+1 >= rownum(); > +id select_type table type possible_keys key key_len ref rows filtered Extra > +1 PRIMARY <derived2> ALL NULL NULL NULL NULL 2 100.00 Using where > +2 DERIVED t1 ALL NULL NULL NULL NULL 5 100.00 Using where > +Warnings: > +Note 1003 /* select#1 */ select `tt`.`a` AS `a` from (/* select#2 */ select `test`.`t1`.`a` AS `a` from `test`.`t1` where `test`.`t1`.`a` < 1000 limit 2) `tt` where <cache>(1 + 1) >= rownum() > +prepare stmt from "select * from (select a from t1 where a < 1000) as tt where 1+1 >= rownum()"; > +execute stmt; > +a > +1 > +2 > +execute stmt; > +a > +1 > +2 > +deallocate prepare stmt; > +# expensive (no opt) > +select * from (select a from t1 where a < 1000) as tt where (select max(a) from t1) >= rownum(); > +a > +1 > +2 > +3 > +4 > +5 > +explain extended select * from (select a from t1 where a < 1000) as tt where (select max(a) from t1) >= rownum(); > +id select_type table type possible_keys key key_len ref rows filtered Extra > +1 PRIMARY <derived2> ALL NULL NULL NULL NULL 5 100.00 Using where > +3 SUBQUERY t1 ALL NULL NULL NULL NULL 5 100.00 > +2 DERIVED t1 ALL NULL NULL NULL NULL 5 100.00 Using where > +Warnings: > +Note 1003 /* select#1 */ select `tt`.`a` AS `a` from (/* select#2 */ select `test`.`t1`.`a` AS `a` from `test`.`t1` where `test`.`t1`.`a` < 1000) `tt` where (/* select#3 */ select max(`test`.`t1`.`a`) from `test`.`t1`) >= rownum() > +prepare stmt from "select * from (select a from t1 where a < 1000) as tt where (select max(a) from t1) >= rownum()"; > +execute stmt; > +a > +1 > +2 > +3 > +4 > +5 > +execute stmt; > +a > +1 > +2 > +3 > +4 > +5 > +deallocate prepare stmt; > +drop table t1; > diff --git a/sql/filesort.h b/sql/filesort.h > index 29ae5e20cc6..806a4a8b027 100644 > --- a/sql/filesort.h > +++ b/sql/filesort.h > @@ -44,24 +44,21 @@ class Filesort: public Sql_alloc > /** Number of records to return */ > ha_rows limit; > /** ORDER BY list with some precalculated info for filesort */ > + ha_rows *accepted_rows; /* For ROWNUM */ > SORT_FIELD *sortorder; first, I suspect that you want to put your accepted_rows declaration before the comment that applies to sortorder. second, could you do a bit more than just "For ROWNUM"? why do you need something "for ROWNUM" inside Filesort? > /** select to use for getting records */ > SQL_SELECT *select; > + > /** TRUE <=> free select on destruction */ > bool own_select; > - /** true means we are using Priority Queue for order by with limit. */ > + /** TRUE means we are using Priority Queue for order by with limit. */ > bool using_pq; > - > /* > TRUE means sort operation must produce table rowids. > FALSE means that it halso has an option of producing {sort_key, > addon_fields} pairs. > */ > bool sort_positions; > - > - Filesort_tracker *tracker; > - Sort_keys *sort_keys; > - > /* > TRUE means all the fields of table of whose bitmap read_set is set > need to be read while reading records in the sort buffer. > diff --git a/sql/item_func.h b/sql/item_func.h > index 53c236bbce7..c237fcd9ecc 100644 > --- a/sql/item_func.h > +++ b/sql/item_func.h > @@ -2125,6 +2125,63 @@ class Item_func_rand :public Item_real_func > }; > > > +class Item_func_rownum :public Item_longlong_func > +{ > + /* > + This points to a variable that contains the number of rows > + accpted so far in the result set > + */ > + ha_rows *accepted_rows; > + SELECT_LEX *select; > +public: > + Item_func_rownum(THD *thd); > + longlong val_int() override; > + LEX_CSTRING func_name_cstring() const override > + { > + static LEX_CSTRING name= {STRING_WITH_LEN("rownum") }; > + return name; > + } > + enum Functype functype() const override { return ROWNUM_FUNC; } > + void update_used_tables() override {} > + bool const_item() const override { return 0; } > + void fix_after_optimize(THD *thd) override; > + bool fix_fields(THD* thd, Item **ref) override; > + bool fix_length_and_dec() override > + { > + unsigned_flag= 1; > + used_tables_cache= RAND_TABLE_BIT; > + const_item_cache=0; > + set_maybe_null(); how can rownum be NULL? > + return FALSE; > + } > + void cleanup() override > + { > + Item_longlong_func::cleanup(); > + /* Ensure we don't point to freed memory */ > + accepted_rows= 0; > + } > + bool check_vcol_func_processor(void *arg) override > + { > + return mark_unsupported_function(func_name(), "()", arg, > + VCOL_IMPOSSIBLE); > + } > + bool check_handler_func_processor(void *arg) override > + { > + return mark_unsupported_function(func_name(), "()", arg, > + VCOL_IMPOSSIBLE); first, as I wrote above, I think that there's no logical reason why rownum should not with with handler. but even if it doesn't, it's much better to check for SQLCOM_HA_READ from Item_func_rownum::fix_fields instead of traversing the whole item tree for every single HANDLER READ command looking for Item_func_rownum that nobody (literally, not a single user out of all our millions) will use there. > + } > + Item *get_copy(THD *thd) override > + { > + return this; It looks wrong. We don't have any other Item that returns `this` here. I think rownum cannot be pushed down because of RAND_TABLE_BIT, So better return NULL here, meaning "cannot be pushed down". Instead of `this` meaning "if you push it down, it'll surreptitiously start corrupting the memory" > + } > + /* This function is used in insert, update and delete */ > + void store_pointer_to_row_counter(ha_rows *row_counter) So this isn't a virtual method, and to use it you downcast after checking type() == FUNC_ITEM and functype() == ROWNUM_FUNC. To avoid adding a new virtual method thus blowing up the vtable, I presume. And fix_after_optimize() is virtual. To avoid error-prone downcasting, would be my guess. Can you explain when you use what approach, please? > + { > + accepted_rows= row_counter; > + } > +}; > + > + > class Item_func_sign :public Item_long_func > { > bool check_arguments() const override > diff --git a/sql/item_func.cc b/sql/item_func.cc > index 25714dda03c..1165b87356a 100644 > --- a/sql/item_func.cc > +++ b/sql/item_func.cc > @@ -7206,3 +7206,72 @@ void Item_func_setval::print(String *str, enum_query_type query_type) > str->append_ulonglong(round); > str->append(')'); > } > + > + > +/* > + Return how many row combinations has accepted so far + 1 > + > + The + 1 is to ensure that 'WHERE ROWNUM <=1' returns one row this is a very weird explanation, it sounds as if it's a heuristical a hack for ROWNUM <=1 to work Why not to write "+1 takes into account the currently computed row" > +*/ > + > +longlong Item_func_rownum::val_int() > +{ > + if (!accepted_rows) > + { > + /* > + Rownum is not properly set up. Probably used in wrong context when > + it should not be used. In this case returning 0 is probably the best > + solution. > + */ DBUG_ASSERT(0); here? > + return 0; > + } > + return (longlong) *accepted_rows+1; > +} > + > + > +Item_func_rownum::Item_func_rownum(THD *thd): > + Item_longlong_func(thd),accepted_rows(0) > +{ > + /* > + Remember the select context. > + Add the function to the list fix_after_optimize in the select context > + so that we can easily initializef all rownum functions with the pointers > + to the row counters. > + */ > + select= thd->lex->current_select; > + select->fix_after_optimize.push_back(this, thd->mem_root); > + > + /* > + Mark that query is using rownum() and ensure that this select is > + not merged with other selects > + */ > + select->with_rownum= 1; > + thd->lex->with_rownum= 1; > + thd->lex->uncacheable(UNCACHEABLE_RAND); Why? ROWNUM doesn't make a query uncacheable, it's perfectly deterministic and repeatable. > + > + /* If this command changes data, mark it as unsafe for statement logging */ > + if (sql_command_flags[thd->lex->sql_command] & > + (CF_UPDATES_DATA | CF_DELETES_DATA)) > + thd->lex->set_stmt_unsafe(LEX::BINLOG_STMT_UNSAFE_SYSTEM_FUNCTION); Why? It's the same as LIMIT. unsafe without ORDER BY and perfectly safe with. > +} > + > + > +bool Item_func_rownum::fix_fields(THD* thd, Item **ref) > +{ > + if (select->having_fix_field) > + { > + my_error(ER_FUNCTION_CANNOT_BE_USED_IN_CLAUSE, MYF(0), > + "rownum()", "HAVING"); > + return 1; > + } > + return Item_longlong_func::fix_fields(thd, ref); > +} > + > +/* > + Store a reference to the variable that contains number of accepted rows > +*/ > + > +void Item_func_rownum::fix_after_optimize(THD *thd) > +{ > + accepted_rows= &select->join->accepted_rows; why do you need both that and fix_rownum_pointers()? I'd think that the latter alone would be sufficient > +} > diff --git a/sql/share/errmsg-utf8.txt b/sql/share/errmsg-utf8.txt > index badaaa17716..aecb00563f7 100644 > --- a/sql/share/errmsg-utf8.txt > +++ b/sql/share/errmsg-utf8.txt > @@ -7973,3 +7973,5 @@ ER_DATA_WAS_COMMITED_UNDER_ROLLBACK > eng "Engine %s does not support rollback. Changes were committed during rollback call" > ER_PK_INDEX_CANT_BE_IGNORED > eng "A primary key cannot be marked as IGNORE" > +ER_FUNCTION_CANNOT_BE_USED_IN_CLAUSE > + eng "Function '%s' cannot be used in the %s clause" Is it the first function we've got that cannot be used in HAVING? > diff --git a/sql/sql_derived.cc b/sql/sql_derived.cc > index 35b4294707f..da6c567b87c 100644 > --- a/sql/sql_derived.cc > +++ b/sql/sql_derived.cc > @@ -656,14 +656,22 @@ static > bool mysql_derived_prepare(THD *thd, LEX *lex, TABLE_LIST *derived) > { > SELECT_LEX_UNIT *unit= derived->get_unit(); > - bool res= FALSE; > + SELECT_LEX *first_select; > + bool res= FALSE, keep_row_order; > DBUG_ENTER("mysql_derived_prepare"); > DBUG_PRINT("enter", ("unit: %p table_list: %p alias: '%s'", > unit, derived, derived->alias.str)); > if (!unit) > DBUG_RETURN(FALSE); > > - SELECT_LEX *first_select= unit->first_select(); > + first_select= unit->first_select(); > + /* > + If rownum() is used we have to preserve the insert row order > + to make group by with filesort to work. it'd help if you'd add an example of a SELECT here > + */ > + keep_row_order= (thd->lex->with_rownum && > + (first_select->group_list.elements || > + first_select->order_list.elements)); > > if (derived->is_recursive_with_table() && > !derived->is_with_table_recursive_reference() && > diff --git a/sql/sql_insert.cc b/sql/sql_insert.cc > index 806049b4465..5c4dff33914 100644 > --- a/sql/sql_insert.cc > +++ b/sql/sql_insert.cc > @@ -746,6 +746,9 @@ bool mysql_insert(THD *thd, TABLE_LIST *table_list, > DBUG_RETURN(TRUE); > } > + > + if (thd->lex->with_rownum && table_list->lock_type == TL_WRITE_DELAYED) > + table_list->lock_type= TL_WRITE; why? do you have a test case when it's needed? > > if (table_list->lock_type == TL_WRITE_DELAYED) > { > if (open_and_lock_for_insert_delayed(thd, table_list)) > @@ -968,11 +971,16 @@ bool mysql_insert(THD *thd, TABLE_LIST *table_list, > */ > if (returning && > result->send_result_set_metadata(returning->item_list, > - Protocol::SEND_NUM_ROWS | Protocol::SEND_EOF)) > + Protocol::SEND_NUM_ROWS | > + Protocol::SEND_EOF)) > goto values_loop_end; > > THD_STAGE_INFO(thd, stage_update); > thd->decide_logging_format_low(table); > + fix_rownum_pointers(thd, thd->lex->current_select, &info.copied); > + if (returning) > + fix_rownum_pointers(thd, thd->lex->returning(), &info.copied); > + Note that info.copied can be incremented more than once per row (in INSERT...ON DUPLICATE KEY UPDATE, for example). This will probably break rownum logic. > do > { > DBUG_PRINT("info", ("iteration %llu", iteration)); > @@ -2133,8 +2141,14 @@ int write_record(THD *thd, TABLE *table, COPY_INFO *info, select_result *sink) > autoinc values (generated inside the handler::ha_write()) and > values updated in ON DUPLICATE KEY UPDATE. > */ > - if (sink && sink->send_data(thd->lex->returning()->item_list) < 0) > - trg_error= 1; > + if (sink) > + { > + /* We have to adjust 'copied' to make ROWNUM() work in RETURNING */ > + info->copied--; > + if (sink->send_data(thd->lex->returning()->item_list) < 0) > + trg_error= 1; > + info->copied++; this is quite silly. Can you simply increment info->copied after this sink->send_data call? > + } > > after_trg_or_ignored_err: > if (key) > diff --git a/sql/sql_limit.h b/sql/sql_limit.h > index a4fcedac14a..62fd0546af0 100644 > --- a/sql/sql_limit.h > +++ b/sql/sql_limit.h > @@ -67,6 +67,15 @@ class Select_limit_counters > { return select_limit_cnt; } > ha_rows get_offset_limit() > { return offset_limit_cnt; } > + > + void set_limit_if_lower(ha_rows limit) > + { > + if (!offset_limit_cnt) I suppose, alternatively, you could've had if (offset_limit_cnt + select_limit_cnt > limit) select_limit_cnt= limit - offset_limit_cnt; > + { > + if (select_limit_cnt > limit) > + select_limit_cnt= limit; > + } > + } > }; > > #endif // INCLUDES_MARIADB_SQL_LIMIT_H > diff --git a/sql/sql_yacc.yy b/sql/sql_yacc.yy > index 2bd2575b882..57ba9df42c0 100644 > --- a/sql/sql_yacc.yy > +++ b/sql/sql_yacc.yy > @@ -10286,6 +10287,22 @@ function_call_nonkeyword: > if (unlikely($$ == NULL)) > MYSQL_YYABORT; > } > +/* Start SQL_MODE_ORACLE_SPECIFIC > + | ROWNUM_SYM optional_braces > + { > + $$= new (thd->mem_root) Item_func_rownum(thd); > + if (unlikely($$ == NULL)) > + MYSQL_YYABORT; > + } > +End SQL_MODE_ORACLE_SPECIFIC */ > +/* Start SQL_MODE_DEFAULT_SPECIFIC */ > + | ROWNUM_SYM '(' ')' > + { > + $$= new (thd->mem_root) Item_func_rownum(thd); > + if (unlikely($$ == NULL)) > + MYSQL_YYABORT; > + } > +/* End SQL_MODE_DEFAULT_SPECIFIC */ You have two nearly identical rules compiled conditionally. I can pretty much guarantee that they'll go out of sync eventually. Better do something like /* Start SQL_MODE_ORACLE_SPECIFIC oracle_optional_braces: optional_braces ; End SQL_MODE_ORACLE_SPECIFIC */ /* Start SQL_MODE_DEFAULT_SPECIFIC */ oracle_optional_braces: '(' ')' ; /* End SQL_MODE_DEFAULT_SPECIFIC */ or even | ROWNUM_SYM /* Start SQL_MODE_ORACLE_SPECIFIC optional_braces End SQL_MODE_ORACLE_SPECIFIC */ /* Start SQL_MODE_DEFAULT_SPECIFIC */ '(' ')' /* End SQL_MODE_DEFAULT_SPECIFIC */ { $$= new (thd->mem_root) Item_func_rownum(thd); if (unlikely($$ == NULL)) MYSQL_YYABORT; } > | SUBDATE_SYM '(' expr ',' expr ')' > { > $$= new (thd->mem_root) Item_date_add_interval(thd, $3, $5, > diff --git a/sql/table.cc b/sql/table.cc > index 7f3a6468aa2..100813e1f95 100644 > --- a/sql/table.cc > +++ b/sql/table.cc > @@ -9331,6 +9331,10 @@ bool TABLE_LIST::init_derived(THD *thd, bool init_view) > { > /* A subquery might be forced to be materialized due to a side-effect. */ > if (!is_materialized_derived() && first_select->is_mergeable() && > + (unit->outer_select() && !unit->outer_select()->with_rownum) && no merging if *outer select* uses rownum? why? example? > + (!thd->lex->with_rownum || > + (!first_select->group_list.elements && > + !first_select->order_list.elements)) && no merging if no rownum but there's group by or order by? group by - this is obvious, and is already checked earlier in this if() order by - why? > optimizer_flag(thd, OPTIMIZER_SWITCH_DERIVED_MERGE) && > !thd->lex->can_not_use_merged(1) && > !is_recursive_with_table()) > diff --git a/sql/sql_select.cc b/sql/sql_select.cc > index 4bb0bc661bd..b002e27799b 100644 > --- a/sql/sql_select.cc > +++ b/sql/sql_select.cc > @@ -1789,10 +1803,18 @@ JOIN::optimize_inner() > { > DBUG_ENTER("JOIN::optimize_inner"); > subq_exit_fl= false; > - do_send_rows = (unit->lim.get_select_limit()) ? 1 : 0; > > DEBUG_SYNC(thd, "before_join_optimize"); > > + // rownum used somewhere in query, no limits and it is derived > + if (unlikely(thd->lex->with_rownum && > + select_lex->first_cond_optimization && > + select_lex->master_unit()->derived)) > + optimize_upper_rownum_func(); > + > + do_send_rows = (unit->lim.get_select_limit()) ? 1 : 0; Is there any particular reason why you put that between DEBUG_SYNC(thd, "before_join_optimize"); and THD_STAGE_INFO(thd, stage_optimizing); ? I'd expect them logically to go directly one after the other > + > + > THD_STAGE_INFO(thd, stage_optimizing); > > set_allowed_join_cache_types(); > @@ -14210,7 +14236,24 @@ static ORDER * > remove_const(JOIN *join,ORDER *first_order, COND *cond, > bool change_list, bool *simple_order) > { > - *simple_order= join->rollup.state == ROLLUP::STATE_NONE; > + /* > + We can't do ORDER BY using filesort if the select list contains a non > + deterministic value like RAND() or ROWNUM(). > + For example: > + SELECT RAND() as 'r' from t1 order by r; This one uses temporary for me. Can you put here a correct example please? > + > + If we would first sort the table 't1', the RAND() column would be generated > + during end_send() and the order would be wrong. > + > + Previously we had here also a test of ROLLUP: > + 'join->rollup.state == ROLLUP::STATE_NONE' > + I deleted this because > + the ROLLUP was never enforced because of a bug where the inital value > + of simple_order was ignored. Having ROLLUP tested now when the code > + is fixed causes many test failure and some wrong results so better to > + leave the code as it was related to ROLLUP. No need to explain in a comment why something that is no longer here was removed. Use git for that, commit comments. The code comment should explain the code as it is. > + */ > + *simple_order= !join->rand_table_in_field_list; why do you need rand_table_in_field_list instead of writing *simple_order= join->select_lex->select_list_tables & RAND_TABLE_BIT; ? > if (join->only_const_tables()) > return change_list ? 0 : first_order; // No need to sort > > @@ -29587,6 +29640,273 @@ void unpack_to_base_table_fields(TABLE *table) > (*cp->do_copy)(cp); > } > > +/* > + Call item->fix_after_optimize for all items registered in > + lex->fix_after_optimize > + > + This is needed for items like ROWNUM(), which needs access to structures > + created by the early optimizer pass, like JOIN > +*/ > + > +static void fix_items_after_optimize(THD *thd, SELECT_LEX *select_lex) > +{ > + List_iterator<Item> li(select_lex->fix_after_optimize); > + > + while (Item *item= li++) > + item->fix_after_optimize(thd); > +} > + > + here you could've had a comment, like /* callbacks for process_direct_rownum_comparison() */ > +static bool is_const(Item *item) > +{ > + /* > + Constant expression in WHERE can be also subquery or other expensive > + thing we shoud not evaluate on OPTIMIZATION phase > + */ > + return item->const_item() && !item->is_expensive(); > +} > + > + > +// Check if this is an unexpensive functional expression over basic constants > + > +static bool is_basic_const(Item *item) > +{ > + if (item->basic_const_item()) > + return true; > + if (!item->const_item() || item->is_expensive()) > + return false; > + if (item->type() == Item::FUNC_ITEM) > + { > + Item_func *func= ((Item_func *)item); > + if (func->argument_count() == 0) > + return false; > + bool result= true; > + Item **args= func->arguments(); > + for (uint i= 0; i < func->argument_count() && result; i++) > + result= result && is_basic_const(args[i]); > + return result; > + } > + return false; > +} is_basic_const() is a misleading name, a function of basic consts isn't a basic const itself. what's an example of an expression has is_const() == true, but is_basic_const() == false? > + > + > +inline bool is_simple_rownum_exp(Item *rownum, Item *constant) unused > +{ > + return (constant->basic_const_item() && > + rownum->type() == Item::FUNC_ITEM && > + ((Item_func*) rownum)->functype() == Item_func::ROWNUM_FUNC); > +} > + > + > +static bool temporary_set_limit(THD *thd __attribute__((unused)), > + SELECT_LEX_UNIT *unit, > + ha_rows lim) > +{ > + unit->set_limit_if_lower(lim); why is it temporary? in the PS/SP sense, just for one execution? > + return false; > +} > + > + > +static bool permanent_limit(THD *thd, SELECT_LEX_UNIT *unit, ha_rows lim) > +{ > + SELECT_LEX *gpar= unit->global_parameters(); > + if (gpar->select_limit != 0 && > + // limit can not be an expression but can be parameter > + (!gpar->select_limit->basic_const_item() || > + ((ha_rows)gpar->select_limit->val_int()) < lim)) > + return false; > + > + Query_arena *arena, backup; > + arena= thd->activate_stmt_arena_if_needed(&backup); > + > + gpar->select_limit= > + new (thd->mem_root)Item_int(thd, lim, 20 /* max 64 bit length*/ ); we have defines for that, MAX_BIGINT_WIDTH, MY_INT64_NUM_DECIMAL_DIGITS, LONGLONG_BUFFER_SIZE - whatever you prefer. > + if (gpar->select_limit == 0) > + return true; // EOM > + > + unit->set_limit(gpar); > + > + gpar->explicit_limit= true; // to show in EXPLAIN > + > + if (arena) > + thd->restore_active_arena(arena, &backup); > + > + return false; > +} > + > + > +/** > + Check posibility of LIMIT setting by rownum() of upper SELECT and do it > + > + @note Ideal is to convert something like > + SELECT ... > + FROM (SELECT ...) table > + WHERE rownum() < <CONSTANT>; > + to > + SELECT ... > + FROM (SELECT ... LIMIT <CONSTANT>) table > + WHERE rownum() < <CONSTANT>; 1. what if the outer select uses LIMIT, not ROWNUM? why not to push that down too? 2. why are you not removing rownum from the outer select? if you remove rownum and reset with_rownum=false, it'll generally give optimizer more freedom. > + > + @retval true EOM > + @retval false no errors > +*/ > + > +bool JOIN::optimize_upper_rownum_func() > +{ > + DBUG_ASSERT(select_lex->master_unit()->derived); > + > + if (select_lex->master_unit()->first_select() != select_lex) > + return false; // first will set parameter > + > + if (select_lex->master_unit()->global_parameters()->offset_limit != NULL) > + return false; // offset is set, we cannot set limit > + > + SELECT_LEX *outer_select= select_lex->master_unit()->outer_select(); > + /* > + Check that the outer query (the one that has 'WHERE rownum()...') is using > + to see if the can be use it to limit this sub query > + */ > + if (outer_select == NULL || > + !outer_select->with_rownum || > + (outer_select->options & SELECT_DISTINCT) || > + outer_select->table_list.elements != 1 || > + outer_select->where == NULL || > + outer_select->where->type() != Item::FUNC_ITEM) > + return false; > + > + return process_direct_rownum_comparison(thd, unit, outer_select->where, > + &is_basic_const, > + &permanent_limit); > +} > + > +/** > + Test if the predicate compares rownum() with a constant > + > + @return 1 No or invalid rownum() compare > + @return 0 rownum() is compared with a constant. > + In this case *args contains the constant and > + *inv_order constains 1 if the rownum() was the right > + argument, like in 'WHERE 2 >= rownum()'. > +*/ > + > +static bool check_rownum_usage(Item_func *func_item, longlong *limit, > + bool *inv_order, const_probe probe) > +{ > + Item *arg1, *arg2; > + *inv_order= 0; > + DBUG_ASSERT(func_item->argument_count() == 2); > + > + /* 'rownum op const' or 'const op field' */ > + arg1= func_item->arguments()[0]->real_item(); > + if (arg1->type() == Item::FUNC_ITEM && > + ((Item_func*) arg1)->functype() == Item_func::ROWNUM_FUNC) > + { > + arg2= func_item->arguments()[1]->real_item(); > + if ((*probe)(arg2)) > + { > + *limit= arg2->val_int(); > + return *limit <= 0 || (ulonglong) *limit >= HA_POS_ERROR; > + } > + } > + else if ((*probe)(arg1)) > + { > + arg2= func_item->arguments()[1]->real_item(); why not to assign arg2 outside of the if() ? > + if (arg2->type() == Item::FUNC_ITEM && > + ((Item_func*) arg2)->functype() == Item_func::ROWNUM_FUNC) > + { > + *limit= arg1->val_int(); > + *inv_order= 1; > + return *limit <= 0 || (ulonglong) *limit >= HA_POS_ERROR; > + } > + } > + return 1; > +} > + > + > +/* > + Limit optimization for ROWNUM() > + > + Go through the WHERE clause and find out if there are any of the following > + constructs on the top level: > + rownum() <= integer_constant > + rownum() < integer_constant > + rownum() = 1 > + > + If yes, then threat the select as if 'LIMIT integer_constant' would > + have been used > +*/ > + > +static void optimize_rownum(THD *thd, SELECT_LEX_UNIT *unit, Item *cond) > +{ > + DBUG_ENTER("optimize_rownum"); > + > + if (cond->type() == Item::COND_ITEM) > + { > + if (((Item_cond*) cond)->functype() == Item_func::COND_AND_FUNC) > + { > + List_iterator<Item> li(*((Item_cond*) cond)->argument_list()); > + Item *item; > + while ((item= li++)) > + optimize_rownum(thd, unit, item); > + } > + DBUG_VOID_RETURN; > + } > + > + process_direct_rownum_comparison(thd, unit, cond, &is_const, > + &temporary_set_limit); please, explain why it's is_const/temporary_set_limit here and is_basic_const/permanent_limit in optimize_upper_rownum_func(). > + DBUG_VOID_RETURN; > +} > + > + > +static bool process_direct_rownum_comparison(THD *thd, > + SELECT_LEX_UNIT *unit, > + Item *cond, > + const_probe probe, > + set_limit_callback set_limit) > +{ > + DBUG_ENTER("process_direct_rownum_comparison"); > + if (cond->real_type() == Item::FUNC_ITEM) > + { > + Item_func *pred= (Item_func*) cond; > + longlong limit; > + bool inv; > + > + if (pred->argument_count() != 2) > + DBUG_RETURN(false); // Not a compare functions > + if (check_rownum_usage(pred, &limit, &inv, probe)) > + DBUG_RETURN(false); > + > + Item_func::Functype pred_type= pred->functype(); > + > + if (inv && pred_type != Item_func::EQ_FUNC) > + { > + if (pred_type == Item_func::GT_FUNC) // # > rownum() > + pred_type= Item_func::LT_FUNC; > + else if (pred_type == Item_func::GE_FUNC) // # >= rownum() > + pred_type= Item_func::LE_FUNC; > + else > + DBUG_RETURN(false); > + } > + switch (pred_type) { > + case Item_func::LT_FUNC: // rownum() < # > + { > + if (limit <= 0) > + DBUG_RETURN(false); > + DBUG_RETURN((*set_limit)(thd, unit, limit - 1)); > + case Item_func::LE_FUNC: > + DBUG_RETURN((*set_limit)(thd, unit, limit)); > + case Item_func::EQ_FUNC: > + if (limit == 1) > + DBUG_RETURN((*set_limit)(thd, unit, limit)); > + break; > + default: > + break; > + } > + } what if it's rownum() > 5 ? why not to mark it as always false right away? > + } > + DBUG_RETURN(false); > +} > + > > /** > @} (end of group Query_Optimizer) Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 4

Re: [Maria-developers] a94502ab674: Added typedef decimal_digits_t (uint16) for number of decimals
by Sergei Golubchik 11 May '21

11 May '21

Hi, Michael! On Mar 26, Michael Widenius wrote: > revision-id: a94502ab674 (mariadb-10.5.2-511-ga94502ab674) > parent(s): 2be9b69f4ff > author: Michael Widenius <michael.widenius(a)gmail.com> > committer: Michael Widenius <michael.widenius(a)gmail.com> > timestamp: 2021-03-24 14:19:55 +0200 > message: > > Added typedef decimal_digits_t (uint16) for number of decimals > For fields and Item's uint8 should be good enough. After > discussions with Alexander Barkov we choose uint16 (for now) > as some format functions may accept +256 digits. > > The reason for this patch was to make the storage of decimal > digits simlar. Before this patch decimals was stored/used as > uint8, int and uint. > > Changed most decimal variables and functions to use the new > typedef. > > diff --git a/include/decimal.h b/include/decimal.h > index cab18f99348..f713409ede1 100644 > --- a/include/decimal.h > +++ b/include/decimal.h > @@ -53,10 +53,11 @@ int decimal2double(const decimal_t *from, double *to); > int double2decimal(double from, decimal_t *to); > int decimal_actual_fraction(const decimal_t *from); > int decimal2bin(const decimal_t *from, uchar *to, int precision, int scale); > -int bin2decimal(const uchar *from, decimal_t *to, int precision, int scale); > +int bin2decimal(const uchar *from, decimal_t *to, int precision, > + decimal_digits_t scale); > > -int decimal_size(int precision, int scale); > -int decimal_bin_size(int precision, int scale); > +int decimal_size(int precision, decimal_digits_t scale); > +int decimal_bin_size(int precision, decimal_digits_t scale); this is *very* confusing. "decimal" in the context of an Item mean "number of digits after a dot" "decimal" in decimal.h means the whole number, and after a dot is "scale" decimal_digits_t in decimal.h does *not* mean what you want, I'm reviewing this your patch for the third time (first was in Sept, then in Dec). And only now I finally understood what you mean (I think). Please, please, don't use decimal_digits_t in decimal.h FYI, more confusing terminology thay I might need below "precision" for decimal numbers is the total number of digits in a number "precision" for temporal types means the number of digits after a dot > int decimal_result_size(decimal_t *from1, decimal_t *from2, char op, > int param); > > diff --git a/include/m_ctype.h b/include/m_ctype.h > index 5fa8f28ff7a..66ca2bf4537 100644 > --- a/include/m_ctype.h > +++ b/include/m_ctype.h > @@ -79,6 +79,7 @@ typedef const struct my_collation_handler_st MY_COLLATION_HANDLER; > typedef const struct unicase_info_st MY_UNICASE_INFO; > typedef const struct uni_ctype_st MY_UNI_CTYPE; > typedef const struct my_uni_idx_st MY_UNI_IDX; > +typedef uint16 decimal_digits_t; This is Item (and Field) specific declaration, please put it in field.h or item.h > typedef struct unicase_info_char_st > { > diff --git a/sql/field.cc b/sql/field.cc > index 52074417046..08c168e0e21 100644 > --- a/sql/field.cc > +++ b/sql/field.cc > @@ -3281,10 +3281,11 @@ Field *Field_decimal::make_new_field(MEM_ROOT *root, TABLE *new_table, > ** Field_new_decimal > ****************************************************************************/ > > -static uint get_decimal_precision(uint len, uint8 dec, bool unsigned_val) > +static decimal_digits_t get_decimal_precision(uint len, decimal_digits_t dec, > + bool unsigned_val) > { > uint precision= my_decimal_length_to_precision(len, dec, unsigned_val); > - return MY_MIN(precision, DECIMAL_MAX_PRECISION); > + return (decimal_digits_t) MY_MIN(precision, DECIMAL_MAX_PRECISION); No, this is wrong (or, rather, inconsistent with your other changes). "precision" for DECIMAL is the total number of digits, not a number of digits after the dot. Judging by your edits in decimal.h you don't want that to be decimal_digits_t. > } > > Field_new_decimal::Field_new_decimal(uchar *ptr_arg, > @@ -10390,7 +10391,8 @@ void Column_definition::create_length_to_internal_length_bit() > void Column_definition::create_length_to_internal_length_newdecimal() > { > DBUG_ASSERT(length < UINT_MAX32); > - uint prec= get_decimal_precision((uint)length, decimals, flags & UNSIGNED_FLAG); > + decimal_digit_t prec= get_decimal_precision((uint)length, decimals, > + flags & UNSIGNED_FLAG); same as above, you should decide whether you want decimal precision to be decimal_digit_t or not. Currently this change contraditcs your decimal.h changes. > pack_length= my_decimal_get_binary_size(prec, decimals); > } > > diff --git a/sql/field.h b/sql/field.h > index 4a4f7cee2a5..5b6a69d0075 100644 > --- a/sql/field.h > +++ b/sql/field.h > @@ -2331,7 +2332,7 @@ class Field_decimal final :public Field_real { > class Field_new_decimal final :public Field_num { > public: > /* The maximum number of decimal digits can be stored */ > - uint precision; > + decimal_digits_t precision; oops. And here again you use decimal_digits_t for decimal precision. > uint bin_size; > /* > Constructors take max_length of the field as a parameter - not the > diff --git a/sql/item.h b/sql/item.h > index 1087c08869e..6753474f2dd 100644 > --- a/sql/item.h > +++ b/sql/item.h > @@ -1649,14 +1649,14 @@ class Item: public Value_source, > return type_handler()->Item_decimal_precision(this); > } > /* Returns the number of integer part digits only */ > - inline int decimal_int_part() const > - { return my_decimal_int_part(decimal_precision(), decimals); } > + inline decimal_digits_t decimal_int_part() const > + { return (decimal_digits_t) my_decimal_int_part(decimal_precision(), decimals); } and here you use decimal_digits_t for precision-scale. > /* > Returns the number of fractional digits only. > NOT_FIXED_DEC is replaced to the maximum possible number > of fractional digits, taking into account the data type. > */ > - uint decimal_scale() const > + decimal_digits_t decimal_scale() const for example, this is consistent with your other changes. scale is decimal_digits_t, it's clear. > { > return type_handler()->Item_decimal_scale(this); > } > @@ -4853,7 +4853,7 @@ class Item_temporal_literal :public Item_literal > Item_literal(thd) > { > collation= DTCollation_numeric(); > - decimals= dec_arg; > + decimals= (decimal_digits_t) dec_arg; why do you cast? I'd expect dec_arg to be decimal_digits_t already > } > > int save_in_field(Field *field, bool no_conversions) override > diff --git a/sql/item_func.cc b/sql/item_func.cc > index a0ef4020aae..db51639a5af 100644 > --- a/sql/item_func.cc > +++ b/sql/item_func.cc > @@ -2710,7 +2710,7 @@ my_decimal *Item_func_round::decimal_op(my_decimal *decimal_value) > dec= INT_MIN; > > if (!(null_value= (value.is_null() || args[1]->null_value || > - value.round_to(decimal_value, (uint) dec, > + value.round_to(decimal_value, (int) dec, > truncate ? TRUNCATE : HALF_UP) > 1))) I don't think you need to cast decimal_digits_t (aka uint16) to int explicitly. A compiler can handle it, it's not lossy. > return decimal_value; > return 0; > diff --git a/sql/item_func.h b/sql/item_func.h > index e774d9c53bd..ae94698ff96 100644 > --- a/sql/item_func.h > +++ b/sql/item_func.h > @@ -1383,10 +1383,10 @@ class Item_decimal_typecast :public Item_func > { > my_decimal decimal_value; > public: > - Item_decimal_typecast(THD *thd, Item *a, uint len, uint dec) > + Item_decimal_typecast(THD *thd, Item *a, uint len, decimal_digits_t dec) > :Item_func(thd, a) > { > - decimals= (uint8) dec; > + decimals= (decimal_digits_t) dec; not needed, dec is decimal_digits_t already > collation= DTCollation_numeric(); > fix_char_length(my_decimal_precision_to_length_no_truncation(len, dec, > unsigned_flag)); > diff --git a/sql/sql_type.cc b/sql/sql_type.cc > index 5a31b39c7b6..1c5e03f34bd 100644 > --- a/sql/sql_type.cc > +++ b/sql/sql_type.cc > @@ -1218,9 +1218,10 @@ uint32 Type_numeric_attributes::find_max_octet_length(Item **item, uint nitems) > } > > > -int Type_numeric_attributes::find_max_decimal_int_part(Item **item, uint nitems) > +decimal_digits_t Type_numeric_attributes:: > +find_max_decimal_int_part(Item **item, uint nitems) > { > - int max_int_part= 0; > + decimal_digits_t max_int_part= 0; again, "int parts" = precision-scale. it's not clear from your patch whether it should be decimal_digits_t > for (uint i=0 ; i < nitems ; i++) > set_if_bigger(max_int_part, item[i]->decimal_int_part()); > return max_int_part; > @@ -1237,11 +1238,12 @@ Type_numeric_attributes::aggregate_numeric_attributes_decimal(Item **item, > uint nitems, > bool unsigned_arg) > { > - int max_int_part= find_max_decimal_int_part(item, nitems); > + decimal_digits_t max_int_part= find_max_decimal_int_part(item, nitems); same here > decimals= find_max_decimals(item, nitems); > - int precision= MY_MIN(max_int_part + decimals, DECIMAL_MAX_PRECISION); > + decimal_digits_t precision= (decimal_digits_t) > + MY_MIN(max_int_part + decimals, DECIMAL_MAX_PRECISION); and here > max_length= my_decimal_precision_to_length_no_truncation(precision, > - (uint8) decimals, > + decimals, > unsigned_flag); > } > > @@ -6955,20 +6957,20 @@ const Vers_type_handler* Type_handler_blob_common::vers() const > > /***************************************************************************/ > > -uint Type_handler::Item_time_precision(THD *thd, Item *item) const > +decimal_digits_t Type_handler::Item_time_precision(THD *thd, Item *item) const But here it's correct. "precision" in the temporal context is the same as "scale" in the decimal context and "decimals" in the Item/Field context. So, decimal_digits_t, all right. > { > return MY_MIN(item->decimals, TIME_SECOND_PART_DIGITS); > } > > > -uint Type_handler::Item_datetime_precision(THD *thd, Item *item) const > +decimal_digits_t Type_handler::Item_datetime_precision(THD *thd, Item *item) const > { > return MY_MIN(item->decimals, TIME_SECOND_PART_DIGITS); > } > > > -uint Type_handler_string_result::Item_temporal_precision(THD *thd, Item *item, > - bool is_time) const > +decimal_digits_t Type_handler_string_result:: > +Item_temporal_precision(THD *thd, Item *item, bool is_time) const > { > StringBuffer<64> buf; > String *tmp; > @@ -7020,7 +7022,7 @@ uint Type_handler_temporal_result:: > > /***************************************************************************/ > > -uint Type_handler_string_result::Item_decimal_precision(const Item *item) const > +decimal_digits_t Type_handler_string_result::Item_decimal_precision(const Item *item) const but here it's "decimal precision", so wrong/inconsistent again. > { > uint res= item->max_char_length(); > /* Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] 58df9e3446b: MDEV-22010: use executables MariaDB named in scripts
by Sergei Golubchik 11 May '21

11 May '21

Hi, Rucha! It's ok to push, with a couple of changes, see below On May 11, Rucha Deodhar wrote: > revision-id: 58df9e3446b (mariadb-10.5.2-386-g58df9e3446b) > parent(s): de407e7cb4d > author: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > committer: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > timestamp: 2021-04-06 16:40:42 +0530 > message: > > MDEV-22010: use executables MariaDB named in scripts > diff --git a/scripts/CMakeLists.txt b/scripts/CMakeLists.txt > index 8d6a486a29d..6f3393a9c22 100644 > --- a/scripts/CMakeLists.txt > +++ b/scripts/CMakeLists.txt > @@ -142,11 +142,11 @@ IF(UNIX) > # FIND_PROC and CHECK_PID are used by mysqld_safe > IF(CMAKE_SYSTEM_NAME MATCHES "Linux") > SET (FIND_PROC > - "ps wwwp $PID | grep -v mysqld_safe | grep -- $MYSQLD > /dev/null") > + "ps wwwp $PID | grep -v mariadbd-safe | grep -- $MYSQLD > /dev/null") I think you need to filter out both mysqld_safe and mariadbd-safe here. because the user can start this script under either name, you need to ignore both. Like grep -v '$mysqld_safe\|mariadbd-safe$' > ENDIF() > IF(NOT FIND_PROC AND CMAKE_SYSTEM_NAME MATCHES "SunOS") > SET (FIND_PROC > - "ps -p $PID | grep -v mysqld_safe | grep -- $MYSQLD > /dev/null") > + "ps -p $PID | grep -v mariadbd-safe | grep -- $MYSQLD > /dev/null") > ENDIF() > > IF(NOT FIND_PROC) > diff --git a/scripts/mysql_install_db.sh b/scripts/mysql_install_db.sh > index 5f183afe8fc..05ce4f31a69 100644 > --- a/scripts/mysql_install_db.sh > +++ b/scripts/mysql_install_db.sh > @@ -613,8 +613,8 @@ then > echo "PLEASE REMEMBER TO SET A PASSWORD FOR THE MariaDB root USER !" > echo "To do so, start the server, then issue the following commands:" > echo > - echo "'$bindir/mysqladmin' -u root password 'new-password'" > - echo "'$bindir/mysqladmin' -u root -h $hostname password 'new-password'" > + echo "'$bindir/mariadb-admin' -u root password 'new-password'" > + echo "'$bindir/mariadb-admin' -u root -h $hostname password 'new-password'" > echo > echo "Alternatively you can run:" > echo "'$bindir/mysql_secure_installation'" ^^^ mariadb-secure-installation Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] e7762b3a725: MDEV-8334: Rename utf8 to utf8mb3
by Sergei Golubchik 05 May '21

05 May '21

Hi, Rucha! Looks great! Just one question below re. using global vs session old_behavior variable. On May 05, Rucha Deodhar wrote: > revision-id: e7762b3a725 (mariadb-10.5.2-583-ge7762b3a725) > parent(s): bfedf1eb4b6 > author: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > committer: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > timestamp: 2021-04-20 12:50:32 +0530 > message: > > MDEV-8334: Rename utf8 to utf8mb3 > > This patch changes the main name of 3 byte character set from utf8 to > utf8mb3. New old_mode UTF8_IS_UTF8MB3 is added and set TRUE by default, > so that utf8 would mean utf8mb3. If not set, utf8 would mean utf8mb4. > diff --git a/libmariadb b/libmariadb > --- a/libmariadb > +++ b/libmariadb > @@ -1 +1 @@ > -Subproject commit b6f8883d9687936a50a7ed79bd9e5af2340efccd > +Subproject commit 03d983b287f8a1fe855cb5ed479a3f7ab4f922ab when rebasing and pushing into 10.6 take care not to rollback C/C changes. That is, only update C/C submodule reference if it's earlier than your 03d983b287f8a1fe855cb5ed479a3f7ab4f922ab > Binary files a/mysql-test/suite/sys_vars/r/character_set_results_basic.result and b/mysql-test/suite/sys_vars/r/character_set_results_basic.result differ > diff --git a/sql/item.cc b/sql/item.cc > --- a/sql/item.cc > +++ b/sql/item.cc > @@ -2359,6 +2359,9 @@ left_is_superset(const DTCollation *left, const DTCollation *right) > > bool DTCollation::aggregate(const DTCollation &dt, uint flags) > { > + > + myf utf8_flag= global_system_variables.old_behavior & > + OLD_MODE_UTF8_IS_UTF8MB3 ? MY_UTF8_IS_UTF8MB3 : 0; if old_behavior is a session variable, then you should use session value here. > if (!my_charset_same(collation, dt.collation)) > { > /* > diff --git a/sql/mysqld.cc b/sql/mysqld.cc > --- a/sql/mysqld.cc > +++ b/sql/mysqld.cc > @@ -4099,6 +4098,8 @@ static int init_common_variables() > test purposes, to be able to start "mysqld" even if > the requested character set is not available (see bug#18743). > */ > + myf utf8_flag= global_system_variables.old_behavior & > + OLD_MODE_UTF8_IS_UTF8MB3 ? MY_UTF8_IS_UTF8MB3 : 0; ok, here there's no "session" yet, so global value is correct > for (;;) > { > char *next_character_set_name= strchr(default_character_set_name, ','); > diff --git a/sql/set_var.cc b/sql/set_var.cc > --- a/sql/set_var.cc > +++ b/sql/set_var.cc > @@ -533,11 +533,12 @@ static my_old_conv old_conv[]= > CHARSET_INFO *get_old_charset_by_name(const char *name) > { > my_old_conv *conv; > - > + myf utf8_flag= global_system_variables.old_behavior & > + OLD_MODE_UTF8_IS_UTF8MB3 ? MY_UTF8_IS_UTF8MB3 : 0; technically, you should use a session value here too. but see the old_conv array, it doesn't have "utf8" anywhere, so it doesn't matter what the flag is, you can use 0 there. > for (conv= old_conv; conv->old_name; conv++) > { > if (!my_strcasecmp(&my_charset_latin1, name, conv->old_name)) > - return get_charset_by_csname(conv->new_name, MY_CS_PRIMARY, MYF(0)); > + return get_charset_by_csname(conv->new_name, MY_CS_PRIMARY, MYF(utf8_flag)); > } > return NULL; > } > diff --git a/sql/sp.cc b/sql/sp.cc > --- a/sql/sp.cc > +++ b/sql/sp.cc > @@ -291,7 +291,8 @@ bool load_charset(MEM_ROOT *mem_root, > CHARSET_INFO **cs) > { > LEX_CSTRING cs_name; > - > + myf utf8_flag= global_system_variables.old_behavior & > + OLD_MODE_UTF8_IS_UTF8MB3 ? MY_UTF8_IS_UTF8MB3 : 0; but this needs a session value, I suppose > if (field->val_str_nopad(mem_root, &cs_name)) > { > *cs= dflt_cs; > @@ -324,9 +325,10 @@ bool load_collation(MEM_ROOT *mem_root, > *cl= dflt_cl; > return TRUE; > } > + myf utf8_flag= thd->get_utf8_flag(); Hmm, here you do use a session value. So, I suppose your using global value above was not an oversight, but you intentionally did it. What were your reasons? > > DBUG_ASSERT(cl_name.str[cl_name.length] == 0); > - *cl= get_charset_by_name(cl_name.str, MYF(0)); > + *cl= get_charset_by_name(cl_name.str, MYF(utf8_flag)); > > if (*cl == NULL) > { > diff --git a/sql/sql_class.h b/sql/sql_class.h > --- a/sql/sql_class.h > +++ b/sql/sql_class.h > @@ -1051,14 +1052,16 @@ static inline void update_global_memory_status(int64 size) > @retval NULL on error > @retval Pointter to CHARSET_INFO with the given name on success > */ > -static inline CHARSET_INFO * > -mysqld_collation_get_by_name(const char *name, > +inline CHARSET_INFO * static inline, please > +mysqld_collation_get_by_name(const char *name, bool utf8_is_utf8mb3, up to you, but wouldn't it be more convenient to pass the flag here? Then you can invoke it with thd->utf8_flag() > CHARSET_INFO *name_cs= system_charset_info) > { > CHARSET_INFO *cs; > MY_CHARSET_LOADER loader; > + myf utf8_flag= utf8_is_utf8mb3 ? MY_UTF8_IS_UTF8MB3 : 0; > my_charset_loader_init_mysys(&loader); > - if (!(cs= my_collation_get_by_name(&loader, name, MYF(0)))) > + > + if (!(cs= my_collation_get_by_name(&loader, name, MYF(utf8_flag)))) > { > ErrConvString err(name, name_cs); > my_error(ER_UNKNOWN_COLLATION, MYF(0), err.ptr()); > diff --git a/sql/sql_parse.cc b/sql/sql_parse.cc > --- a/sql/sql_parse.cc > +++ b/sql/sql_parse.cc > @@ -10449,7 +10449,10 @@ merge_charset_and_collation(CHARSET_INFO *cs, CHARSET_INFO *cl) > CHARSET_INFO *find_bin_collation(CHARSET_INFO *cs) > { > const char *csname= cs->csname; > - cs= get_charset_by_csname(csname, MY_CS_BINSORT, MYF(0)); > + myf utf8_flag= global_system_variables.old_behavior & > + OLD_MODE_UTF8_IS_UTF8MB3 ? > + MY_UTF8_IS_UTF8MB3 : 0; Why not a session value here? > + cs= get_charset_by_csname(csname, MY_CS_BINSORT, MYF(utf8_flag)); > if (!cs) > { > char tmp[65]; Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 2bceae199bb: MDEV-14974: --port ignored for --host=localhost
by Sergei Golubchik 05 May '21

05 May '21

Hi, Brandon! Just a couple of minor issues: On May 05, Brandon Nesterenko wrote: > revision-id: 2bceae199bb (mariadb-10.6.0-24-g2bceae199bb) > parent(s): 4ff4df3232f > author: Brandon Nesterenko <brandon.nesterenko(a)mariadb.com> > committer: Brandon Nesterenko <brandon.nesterenko(a)mariadb.com> > timestamp: 2021-05-05 01:01:01 +0000 > message: > > MDEV-14974: --port ignored for --host=localhost > > diff --git a/client/client_priv.h b/client/client_priv.h > index 64818d2ab8d..606b629a4d5 100644 > --- a/client/client_priv.h > +++ b/client/client_priv.h > @@ -136,3 +136,37 @@ enum options_client > Name of the sys schema database. > */ > #define SYS_SCHEMA_DB_NAME "sys" > + > + > +/** > + Utility function to implicitly change the connection protocol to a > + consistent value given the command line arguments. Additionally, > + warns the user that the protocol has been changed. > + > + Arguments: if you do `git show 2bceae199bb` you'll see git highlighting invisible spaces at line ends. Could you, please, remove them? (everywhere in your commit, not only in the comment above) > + @param [in] host Name of the host to connect to > + @param [in, out] opt_protocol Location of the protocol option > + variable to update > + @param [in] new_protocol New protocol to force > diff --git a/client/mysqlcheck.c b/client/mysqlcheck.c > index fb3103a318d..4f8891817e3 100644 > --- a/client/mysqlcheck.c > +++ b/client/mysqlcheck.c > @@ -285,10 +287,14 @@ static void usage(void) > > static my_bool > get_one_option(const struct my_option *opt, > - const char *argument, > - const char *filename __attribute__((unused))) > + const char *argument, > + const char *filename) indentation went wrong here. (only here. in other files where you removed the __attribute__ it was fine) > { > int orig_what_to_do= what_to_do; > + > + /* Track when protocol is set via CLI to not force overrides */ > + static my_bool ignore_protocol_override = FALSE; > + > DBUG_ENTER("get_one_option"); > > switch(opt->id) { > diff --git a/man/mysql.1 b/man/mysql.1 > index 03f23df3660..27a7e4d4d70 100644 > --- a/man/mysql.1 > +++ b/man/mysql.1 > @@ -1199,7 +1199,8 @@ Do not write line numbers for errors\&. Useful when you want to compare result f > \fB\-S \fR\fB\fIpath\fR\fR > .sp > For connections to > -localhost, the Unix socket file to use, or, on Windows, the name of the named pipe to use\&. > +localhost, the Unix socket file to use, or, on Windows, the name of the named pipe to use\&. > +Forces --protocol=socket when specified without other connection properties\&. here and everywhere, for -P and -S: I'd clarify that "... when specified on the command line ..." > .RE > .sp > .RS 4 > diff --git a/mysql-test/main/cli_options_force_protocol.result b/mysql-test/main/cli_options_force_protocol.result > new file mode 100644 > index 00000000000..c69a2b4f578 > --- /dev/null > +++ b/mysql-test/main/cli_options_force_protocol.result > @@ -0,0 +1,25 @@ > +# > +# MDEV-14974: --port ignored for --host=localhost > +# > +# > +# The following group of tests should produce no warnings > +# > +# exec MYSQL --host=localhost -e "status" 2>&1 | grep "Connection:\|WARNING:" > +Connection: Localhost via UNIX socket > +# exec MYSQL --port=MASTER_MYPORT --protocol=tcp -e "status" 2>&1 | grep "Connection:\|WARNING:" > +Connection: localhost via TCP/IP > +# exec MYSQL --host=localhost --port=MASTER_MYPORT --protocol=socket -e "status" 2>&1 | grep "Connection:\|WARNING:" > +Connection: Localhost via UNIX socket > +# exec MYSQL --host=127.0.0.1 --port=MASTER_MYPORT -e "status" 2>&1 | grep "Connection:\|WARNING:" > +Connection: 127.0.0.1 via TCP/IP > +# exec MYSQL --host=localhost --socket=MASTER_MYSOCK --port=MASTER_MYPORT -e "status" 2>&1 | grep "Connection:\|WARNING:" > +Connection: Localhost via UNIX socket > +# > +# The remaining tests should produce warnings > +# I now have some reservations about it, see below: > +# exec MYSQL --host=localhost --port=MASTER_MYPORT -e "status" 2>&1 | grep "Connection:\|WARNING:" > +WARNING: Forcing protocol to TCP due to option specification. Please explicitly state intended protocol. > +Connection: localhost via TCP/IP old behavior was "localhost via UNIX socket", you've changed it to TCP/IP and issued a warning. Good so far. > +# exec MYSQL --host=localhost --socket=MASTER_MYSOCK -e "status" 2>&1 | grep "Connection:\|WARNING:" > +WARNING: Forcing protocol to SOCKET due to option specification. Please explicitly state intended protocol. > +Connection: Localhost via UNIX socket here the behavior isn't changed, but you still issue a warning. Is it justified? may be it'd be better only to issue a warning when the behavior changes? Regards, Sergei

1 0

Re: [Maria-developers] 2a5663ae524: MDEV-14974: --port ignored for --host=localhost
by Sergei Golubchik 04 May '21

04 May '21

Hi, Brandon! On May 04, Brandon Nesterenko wrote: > revision-id: 2a5663ae524 (mariadb-10.6.0-18-g2a5663ae524) > parent(s): 9fe681c9e4d > author: Brandon Nesterenko <brandon.nesterenko(a)mariadb.com> > committer: Brandon Nesterenko <brandon.nesterenko(a)mariadb.com> > timestamp: 2021-05-03 19:47:07 +0000 > message: > > MDEV-14974: --port ignored for --host=localhost > > Problem: > ======= > MariaDB's command line utilities (e.g., mysql, > mysqldump, etc) silently ignore connection > property options (e.g., --port and --socket) > when protocol is not explicitly set via the > command-line for localhost connections. > > Fix: > === > If connection properties are specified without a > protocol, override the protocol to be consistent. > For example, if --port is specified, automatically > set protocol=tcp. > > Caveats: > ======= > * When multiple connection properties are > specified, nothing is overridden > * If protocol is is set via the command-line, > its value is used > > diff --git a/client/client_priv.h b/client/client_priv.h > index 64818d2ab8d..94e1fc8bb08 100644 > --- a/client/client_priv.h > +++ b/client/client_priv.h > @@ -136,3 +136,57 @@ enum options_client > Name of the sys schema database. > */ > #define SYS_SCHEMA_DB_NAME "sys" > + > + > +/** > + Utility function to implicitly change the connection protocol to a > + consistent value given the command line arguments. Additionally, > + warns the user that the protocol has been changed. > + > + Arguments: > + @param [in] warn_to The file to write the warning to > + @param [in] host Name of the host to connect to > + @param [in, out] opt_protocol Location of the protocol option > + variable to update > + @param [in] new_protocol New protocol to force > +*/ > +static inline void warn_protocol_override(FILE *warn_to, it's always stderr, isn't it? Why did you want it as an argument? > + char *host, > + uint *opt_protocol, > + uint new_protocol) > +{ > + if ((host == NULL > + || strncmp(host, LOCAL_HOST, sizeof(LOCAL_HOST)-1) == 0)) > + { > + const char *TCP_NAME = "TCP"; > + const char *SOCKET_NAME = "SOCKET"; > + char *protocol_name; > + > + if(new_protocol == MYSQL_PROTOCOL_TCP) > + protocol_name = (char *) TCP_NAME; better not to hard-code protocol names here, you can do protocol_name= sql_protocol_typelib.type_names[new_protocol-1]; > + else if(new_protocol == MYSQL_PROTOCOL_SOCKET) > + protocol_name = (char *) SOCKET_NAME; > + else > + { > + /* > + This should never be entered, but just in case we are called incorrectly > + */ > + > + fprintf(warn_to, "%s %d %s %d\n", > + "WARNING: Protocol ID ", > + new_protocol, > + " cannot override connection type. " > + "Using the configuration value of ", > + *opt_protocol); > + return; for things that cannot happen code-wise (some code guarantees that it can never happen), better add an assert (DBUG_ASSERT, in fact) and there's no need to do an warning or anything. An assert is like a self-maintaining documentation. Its main purpose is to document that something cannot happen. But unlike your comment it also checks that it doesn't become outdated. so, this could become DBUG_ASSERT(new_protocol == MYSQL_PROTOCOL_SOCKET || new_protocol == MYSQL_PROTOCOL_TCP); protocol_name= sql_protocol_typelib.type_names[new_protocol-1]; > + } > + > + fprintf(warn_to, "%s %s %s\n", > + "WARNING: Forcing protocol to ", > + protocol_name, > + " due to option specification. " > + "Please explicitly state intended protocol."); > + > + *opt_protocol = new_protocol; > + } > +} > diff --git a/client/mysql.cc b/client/mysql.cc > index 433fbd281b9..5ca9b4393ec 100644 > --- a/client/mysql.cc > +++ b/client/mysql.cc > @@ -206,6 +206,8 @@ static uint opt_protocol=0; > static const char *opt_protocol_type= ""; > static CHARSET_INFO *charset_info= &my_charset_latin1; > > +static uint protocol_to_force= MYSQL_PROTOCOL_DEFAULT; > + > #include "sslopt-vars.h" > > const char *default_dbug_option="d:t:o,/tmp/mariadb.trace"; > @@ -1162,6 +1164,9 @@ int main(int argc,char *argv[]) > close(stdout_fileno_copy); /* Clean up dup(). */ > } > > + /* We need to know if protocol-related options originate from CLI args */ > + my_defaults_mark_files = TRUE; > + > load_defaults_or_exit("my", load_default_groups, &argc, &argv); > defaults_argv=argv; > if ((status.exit_status= get_options(argc, (char **) argv))) > @@ -1171,6 +1176,14 @@ int main(int argc,char *argv[]) > exit(status.exit_status); > } > > + /* Command line options override configured protocol */ > + if (protocol_to_force > MYSQL_PROTOCOL_DEFAULT > + && protocol_to_force != opt_protocol) > + { > + warn_protocol_override(stderr, current_host, &opt_protocol, protocol_to_force); > + } > + > + > if (status.batch && !status.line_buff && > !(status.line_buff= batch_readline_init(MAX_BATCH_BUFFER_SIZE, stdin))) > { > @@ -1715,8 +1728,11 @@ static void usage(int version) > > > my_bool > -get_one_option(const struct my_option *opt, const char *argument, const char *) > +get_one_option(const struct my_option *opt, const char *argument, const char *filename) > { > + /* Track when protocol is set via CLI to not force port TCP protocol override */ > + static my_bool ignore_protocol_override = FALSE; > + > switch(opt->id) { > case OPT_CHARSETS_DIR: > strmake_buf(mysql_charsets_dir, argument); > @@ -1781,6 +1797,14 @@ get_one_option(const struct my_option *opt, const char *argument, const char *) > opt->name)) <= 0) > exit(1); > #endif > + > + /* Specification of protocol via CLI trumps implicit overrides */ > + if (filename[0] == '\0') > + { > + ignore_protocol_override = TRUE; > + protocol_to_force = MYSQL_PROTOCOL_DEFAULT; > + } > + > break; > case OPT_SERVER_ARG: > #ifdef EMBEDDED_LIBRARY > @@ -1872,6 +1896,13 @@ get_one_option(const struct my_option *opt, const char *argument, const char *) > #ifdef __WIN__ > opt_protocol = MYSQL_PROTOCOL_PIPE; > opt_protocol_type= "pipe"; > + > + /* Prioritize pipe if explicit via command line */ > + if (filename[0] == '\0') > + { > + ignore_protocol_override = TRUE; > + protocol_to_force = MYSQL_PROTOCOL_DEFAULT; > + } > #endif > break; > #include <sslopt-case.h> > @@ -1883,6 +1914,40 @@ get_one_option(const struct my_option *opt, const char *argument, const char *) > status.exit_status= 0; > mysql_end(-1); > break; > + case 'P': > + /* If port and socket are set, fall back to default behavior */ > + if (protocol_to_force == MYSQL_PROTOCOL_SOCKET) > + { > + ignore_protocol_override = TRUE; > + protocol_to_force = MYSQL_PROTOCOL_DEFAULT; > + } > + > + /* If port is set via CLI, try to force protocol to TCP */ > + if (filename[0] == '\0' && > + !ignore_protocol_override && > + MYSQL_PROTOCOL_TCP > protocol_to_force) that's a strange condition. I believe it's practically identical to protocol_to_force == MYSQL_PROTOCOL_DEFAULT but a lot more confusing. Why not to write it as protocol_to_force == MYSQL_PROTOCOL_DEFAULT ? > + { > + protocol_to_force = MYSQL_PROTOCOL_TCP; > + } > + break; > + case 'S': > +#ifndef __WIN__ > + /* If port and socket are set, fall back to default behavior */ > + if (protocol_to_force == MYSQL_PROTOCOL_TCP) > + { > + ignore_protocol_override = TRUE; > + protocol_to_force = MYSQL_PROTOCOL_DEFAULT; > + } > + > + /* Prioritize socket if set via command line */ > + if (filename[0] == '\0' && > + !ignore_protocol_override && > + MYSQL_PROTOCOL_SOCKET > protocol_to_force) same here > + { > + protocol_to_force = MYSQL_PROTOCOL_SOCKET; > + } > +#endif > + break; > case 'I': > case '?': > usage(0); > diff --git a/client/mysqladmin.cc b/client/mysqladmin.cc > index e40e82f8038..08ec14815bd 100644 > --- a/client/mysqladmin.cc > +++ b/client/mysqladmin.cc > @@ -54,6 +54,8 @@ static bool sql_log_bin_off= false; > static uint opt_protocol=0; > static myf error_flags; /* flags to pass to my_printf_error, like ME_BELL */ > > +static uint protocol_to_force= MYSQL_PROTOCOL_DEFAULT; > + > /* > When using extended-status relatively, ex_val_max_len is the estimated > maximum length for any relative value printed by extended-status. The > @@ -241,8 +243,12 @@ static const char *load_default_groups[]= > 0 }; > > my_bool > -get_one_option(const struct my_option *opt, const char *argument, const char *) > +get_one_option(const struct my_option *opt, const char *argument, const char *filename) > { > + > + /* Track when protocol is set via CLI to not force overrides */ > + static my_bool ignore_protocol_override = FALSE; > + > switch(opt->id) { > case 'c': > opt_count_iterations= 1; > @@ -274,6 +280,13 @@ get_one_option(const struct my_option *opt, const char *argument, const char *) > case 'W': > #ifdef __WIN__ > opt_protocol = MYSQL_PROTOCOL_PIPE; > + > + /* Prioritize pipe if explicit via command line */ > + if (filename[0] == '\0') > + { > + ignore_protocol_override = TRUE; > + protocol_to_force = MYSQL_PROTOCOL_DEFAULT; > + } > #endif > break; > case '#': > @@ -309,6 +322,48 @@ get_one_option(const struct my_option *opt, const char *argument, const char *) > sf_leaking_memory= 1; /* no memory leak reports here */ > exit(1); > } > + > + /* Specification of protocol via CLI trumps implicit overrides */ > + if (filename[0] == '\0') > + { > + ignore_protocol_override = TRUE; > + protocol_to_force = MYSQL_PROTOCOL_DEFAULT; > + } > + > + break; > + case 'P': > + /* If port and socket are set, fall back to default behavior */ > + if (protocol_to_force == MYSQL_PROTOCOL_SOCKET) > + { > + ignore_protocol_override = TRUE; > + protocol_to_force = MYSQL_PROTOCOL_DEFAULT; > + } > + > + /* If port is set via CLI, try to force protocol to TCP */ > + if (filename[0] == '\0' && > + !ignore_protocol_override && > + MYSQL_PROTOCOL_TCP > protocol_to_force) > + { > + protocol_to_force = MYSQL_PROTOCOL_TCP; > + } > + break; > + case 'S': > +#ifndef __WIN__ > + /* If port and socket are set, fall back to default behavior */ > + if (protocol_to_force == MYSQL_PROTOCOL_TCP) > + { > + ignore_protocol_override = TRUE; > + protocol_to_force = MYSQL_PROTOCOL_DEFAULT; > + } > + > + /* Prioritize socket if set via command line */ > + if (filename[0] == '\0' && > + !ignore_protocol_override && > + MYSQL_PROTOCOL_SOCKET > protocol_to_force) > + { > + protocol_to_force = MYSQL_PROTOCOL_SOCKET; > + } > +#endif > break; > } > return 0; > @@ -323,6 +378,10 @@ int main(int argc,char *argv[]) > > MY_INIT(argv[0]); > sf_leaking_memory=1; /* don't report memory leaks on early exits */ > + > + /* We need to know if protocol-related options originate from CLI args */ > + my_defaults_mark_files = TRUE; > + > load_defaults_or_exit("my", load_default_groups, &argc, &argv); > save_argv = argv; /* Save for free_defaults */ > > @@ -331,6 +390,13 @@ int main(int argc,char *argv[]) > temp_argv= mask_password(argc, &argv); > temp_argc= argc; > > + /* Command line options override configured protocol */ > + if (protocol_to_force > MYSQL_PROTOCOL_DEFAULT > + && protocol_to_force != opt_protocol) > + { > + warn_protocol_override(stderr, host, &opt_protocol, protocol_to_force); > + } > + > if (debug_info_flag) > my_end_arg= MY_CHECK_ERROR | MY_GIVE_INFO; > if (debug_check_flag) > diff --git a/client/mysqlbinlog.cc b/client/mysqlbinlog.cc > index fd31ab6694e..1ff6048e5ad 100644 > --- a/client/mysqlbinlog.cc > +++ b/client/mysqlbinlog.cc > @@ -98,6 +98,8 @@ static const char *output_prefix= ""; > static char **defaults_argv= 0; > static MEM_ROOT glob_root; > > +static uint protocol_to_force= MYSQL_PROTOCOL_DEFAULT; > + > #ifndef DBUG_OFF > static const char *default_dbug_option = "d:t:o,/tmp/mariadb-binlog.trace"; > const char *current_dbug_option= default_dbug_option; > @@ -1959,9 +1961,13 @@ static my_time_t convert_str_to_timestamp(const char* str) > > > extern "C" my_bool > -get_one_option(const struct my_option *opt, const char *argument, const char *) > +get_one_option(const struct my_option *opt, const char *argument, const char *filename) > { > bool tty_password=0; > + > + /* Track when protocol is set via CLI to not force overrides */ > + static my_bool ignore_protocol_override = FALSE; > + > switch (opt->id) { > #ifndef DBUG_OFF > case '#': > @@ -2011,6 +2017,14 @@ get_one_option(const struct my_option *opt, const char *argument, const char *) > sf_leaking_memory= 1; /* no memory leak reports here */ > die(); > } > + > + /* Specification of protocol via CLI trumps implicit overrides */ > + if (filename[0] == '\0') > + { > + ignore_protocol_override = TRUE; > + protocol_to_force = MYSQL_PROTOCOL_DEFAULT; > + } > + > break; > #ifdef WHEN_FLASHBACK_REVIEW_READY > case opt_flashback_review: > @@ -2092,6 +2106,40 @@ get_one_option(const struct my_option *opt, const char *argument, const char *) > case OPT_PRINT_ROW_EVENT_POSITIONS: > print_row_event_positions_used= 1; > break; > + case 'P': > + /* If port and socket are set, fall back to default behavior */ > + if (protocol_to_force == MYSQL_PROTOCOL_SOCKET) > + { > + ignore_protocol_override = TRUE; > + protocol_to_force = MYSQL_PROTOCOL_DEFAULT; > + } > + > + /* If port is set via CLI, try to force protocol to TCP */ > + if (filename[0] == '\0' && > + !ignore_protocol_override && > + MYSQL_PROTOCOL_TCP > protocol_to_force) > + { > + protocol_to_force = MYSQL_PROTOCOL_TCP; > + } > + break; > + case 'S': > +#ifndef __WIN__ > + /* If port and socket are set, fall back to default behavior */ > + if (protocol_to_force == MYSQL_PROTOCOL_TCP) > + { > + ignore_protocol_override = TRUE; > + protocol_to_force = MYSQL_PROTOCOL_DEFAULT; > + } > + > + /* Prioritize socket if set via command line */ > + if (filename[0] == '\0' && > + !ignore_protocol_override && > + MYSQL_PROTOCOL_SOCKET > protocol_to_force) > + { > + protocol_to_force = MYSQL_PROTOCOL_SOCKET; > + } > +#endif > + break; > case 'v': > if (argument == disabled_my_option) > verbose= 0; > @@ -3049,6 +3097,9 @@ int main(int argc, char** argv) > my_init_time(); // for time functions > tzset(); // set tzname > > + /* We need to know if protocol-related options originate from CLI args */ > + my_defaults_mark_files = TRUE; > + > load_defaults_or_exit("my", load_groups, &argc, &argv); > defaults_argv= argv; > > @@ -3062,6 +3113,13 @@ int main(int argc, char** argv) > > parse_args(&argc, (char***)&argv); > > + /* Command line options override configured protocol */ > + if (protocol_to_force > MYSQL_PROTOCOL_DEFAULT > + && protocol_to_force != opt_protocol) > + { > + warn_protocol_override(stderr, host, &opt_protocol, protocol_to_force); > + } > + > if (!argc || opt_version) > { > if (!opt_version) > diff --git a/client/mysqlcheck.c b/client/mysqlcheck.c > index fb3103a318d..54f8d2eb4f3 100644 > --- a/client/mysqlcheck.c > +++ b/client/mysqlcheck.c > @@ -56,6 +56,8 @@ static char *opt_skip_database; > DYNAMIC_ARRAY tables4repair, tables4rebuild, alter_table_cmds; > DYNAMIC_ARRAY views4repair; > static uint opt_protocol=0; > + > +static uint protocol_to_force= MYSQL_PROTOCOL_DEFAULT; > > enum operations { DO_CHECK=1, DO_REPAIR, DO_ANALYZE, DO_OPTIMIZE, DO_FIX_NAMES }; > const char *operation_name[]= > @@ -289,6 +291,10 @@ get_one_option(const struct my_option *opt, > const char *filename __attribute__((unused))) not __attribute__((unused)) anymore here and in other files too, I won't repeat this comment for every file > { > int orig_what_to_do= what_to_do; > + > + /* Track when protocol is set via CLI to not force overrides */ > + static my_bool ignore_protocol_override = FALSE; > + > DBUG_ENTER("get_one_option"); > > switch(opt->id) { > diff --git a/mysql-test/main/cli_options_force_protocol.result b/mysql-test/main/cli_options_force_protocol.result > --- /dev/null > +++ b/mysql-test/main/cli_options_force_protocol.result > @@ -0,0 +1,27 @@ > +# > +# MDEV-14974: --port ignored for --host=localhost > +# > +# > +# The following tests until the first cat_file should produce no warnings > +# > +# /home/buildbot/workspace/server/build-mariadb-server-debug/client/mysql --defaults-file=/home/buildbot/workspace/server/build-mariadb-server-debug/mysql-test/var/my.cnf --host=localhost -e "status" | grep "Connection:" oh-oh. You cannot hard-code your local path into the result file. Not the port number either. That's why in my previous review I wrote --echo # --host=localhost --port=MASTER_MYPORT and not --echo # $MYSQL --host=localhost --port=$MASTER_MYPORT -e "status" | grep "Connection:" (actually I had --port=$MASTER_MYPORT in my email, but that was wrong, should be no $-sign in --echo :) > +Connection: Localhost via UNIX socket > +# /home/buildbot/workspace/server/build-mariadb-server-debug/client/mysql --defaults-file=/home/buildbot/workspace/server/build-mariadb-server-debug/mysql-test/var/my.cnf --port=16000 --protocol=tcp -e "status" | grep "Connection:" > +Connection: localhost via TCP/IP > +# /home/buildbot/workspace/server/build-mariadb-server-debug/client/mysql --defaults-file=/home/buildbot/workspace/server/build-mariadb-server-debug/mysql-test/var/my.cnf --host=localhost --port=16000 --protocol=socket -e "status" | grep "Connection:" > +Connection: Localhost via UNIX socket > +# /home/buildbot/workspace/server/build-mariadb-server-debug/client/mysql --defaults-file=/home/buildbot/workspace/server/build-mariadb-server-debug/mysql-test/var/my.cnf --host=127.0.0.1 --port=16000 -e "status" | grep "Connection:" > +Connection: 127.0.0.1 via TCP/IP > +# /home/buildbot/workspace/server/build-mariadb-server-debug/client/mysql --defaults-file=/home/buildbot/workspace/server/build-mariadb-server-debug/mysql-test/var/my.cnf --host=localhost --socket=/home/buildbot/workspace/server/build-mariadb-server-debug/mysql-test/var/tmp/mysqld.1.sock --port=16000 -e "status" | grep "Connection:" > +Connection: Localhost via UNIX socket > +CURRENT_TEST: main.cli_options_force_protocol > +# > +# The remaining tests should produce warnings > +# > +# /home/buildbot/workspace/server/build-mariadb-server-debug/client/mysql --defaults-file=/home/buildbot/workspace/server/build-mariadb-server-debug/mysql-test/var/my.cnf --host=localhost --port=16000 -e "status" | grep "Connection:" > +Connection: localhost via TCP/IP > +# /home/buildbot/workspace/server/build-mariadb-server-debug/client/mysql --defaults-file=/home/buildbot/workspace/server/build-mariadb-server-debug/mysql-test/var/my.cnf --host=localhost --socket=/home/buildbot/workspace/server/build-mariadb-server-debug/mysql-test/var/tmp/mysqld.1.sock -e "status" | grep "Connection:" > +Connection: Localhost via UNIX socket > +CURRENT_TEST: main.cli_options_force_protocol > +WARNING: Forcing protocol to TCP due to option specification. Please explicitly state intended protocol. > +WARNING: Forcing protocol to SOCKET due to option specification. Please explicitly state intended protocol. > diff --git a/mysql-test/main/cli_options_force_protocol.test b/mysql-test/main/cli_options_force_protocol.test > new file mode 100644 > index 00000000000..f91d7833a0d > --- /dev/null > +++ b/mysql-test/main/cli_options_force_protocol.test > @@ -0,0 +1,41 @@ > +--echo # > +--echo # MDEV-14974: --port ignored for --host=localhost > +--echo # > + > +--source include/not_embedded.inc > +--source include/not_windows.inc > + > +--echo # > +--echo # The following tests until the first cat_file should produce no warnings > +--echo # > + > +--echo # $MYSQL --host=localhost -e "status" | grep "Connection:" > +--exec $MYSQL --host=localhost -e "status" | grep "Connection:" > + > +--echo # $MYSQL --port=$MASTER_MYPORT --protocol=tcp -e "status" | grep "Connection:" > +--exec $MYSQL --port=$MASTER_MYPORT --protocol=tcp -e "status" | grep "Connection:" > + > +--echo # $MYSQL --host=localhost --port=$MASTER_MYPORT --protocol=socket -e "status" | grep "Connection:" > +--exec $MYSQL --host=localhost --port=$MASTER_MYPORT --protocol=socket -e "status" | grep "Connection:" > + > +--echo # $MYSQL --host=127.0.0.1 --port=$MASTER_MYPORT -e "status" | grep "Connection:" > +--exec $MYSQL --host=127.0.0.1 --port=$MASTER_MYPORT -e "status" | grep "Connection:" > + > +--echo # $MYSQL --host=localhost --socket=$MASTER_MYSOCK --port=$MASTER_MYPORT -e "status" | grep "Connection:" > +--exec $MYSQL --host=localhost --socket=$MASTER_MYSOCK --port=$MASTER_MYPORT -e "status" | grep "Connection:" > + > +--cat_file $MYSQLTEST_VARDIR/log/current_test > + > + > +--echo # > +--echo # The remaining tests should produce warnings > +--echo # > + > +--echo # $MYSQL --host=localhost --port=$MASTER_MYPORT -e "status" | grep "Connection:" > +--exec $MYSQL --host=localhost --port=$MASTER_MYPORT -e "status" | grep "Connection:" > + > +--echo # $MYSQL --host=localhost --socket=$MASTER_MYSOCK -e "status" | grep "Connection:" > +--exec $MYSQL --host=localhost --socket=$MASTER_MYSOCK -e "status" | grep "Connection:" > + > +--cat_file $MYSQLTEST_VARDIR/log/current_test I still don't understand why you include current test name into the result file. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 242cc3fc41f: MDEV-14974: --port ignored for --host=localhost
by Sergei Golubchik 03 May '21

03 May '21

Hi, Brandon! On May 03, Brandon Nesterenko wrote: > revision-id: 242cc3fc41f (mariadb-10.6.0-17-g242cc3fc41f) > parent(s): fd8c68c7fe6 > author: Brandon Nesterenko <brandon.nesterenko(a)mariadb.com> > committer: Brandon Nesterenko <brandon.nesterenko(a)mariadb.com> > timestamp: 2021-04-30 23:17:37 +0000 > message: > > MDEV-14974: --port ignored for --host=localhost > > Problem: > ======= > MariaDB's command line utilities (e.g., mysql, mysqldump, etc) silently ignore the --port option if no host is given or it is localhost. please, try to avoid very long lines in commit comments, they're difficult to read in git console tools. > > Fix: > === > During configuration processing, force protocol to TCP if port was specified via the command line. However, if protocol is additionally specified via the command line, it will be prioritized. > > diff --git a/client/client_priv.h b/client/client_priv.h > index 64818d2ab8d..6cb1f02ba33 100644 > --- a/client/client_priv.h > +++ b/client/client_priv.h > @@ -136,3 +136,85 @@ enum options_client > Name of the sys schema database. > */ > #define SYS_SCHEMA_DB_NAME "sys" > + > + > + > +/** > + Changes flags to prepare for (and revert to the previous state) > + from handle_options. > + > + When creating the state, the original state is saved and can be later > + resumed by calling this function and setting do_revert_values to TRUE. > + > + Flags: > + my_defaults_mark_files: propagates the source of a set option > + > + @param [in] do_revert_values behavior should revert when TRUE, prepare > + when FALSE > + */ > +static inline void set_flags_for_option_handling(my_bool do_revert_values) > +{ > + static my_bool prev_mark_files; > + > + if (do_revert_values) > + { > + my_defaults_mark_files = prev_mark_files; > + } > + else > + { > + prev_mark_files = my_defaults_mark_files; > + my_defaults_mark_files = TRUE; > + } > +} this isn't very helpful. my_defaults_mark_files is part of the my_getopt API. It's supposed to be used directly by whatever tool uses my_getopt. Here you've turned one line my_defaults_mark_files= TRUE; into 27 lines, so when I (or somebody reading the code) will see set_flags_for_option_handling(...); he'd have to go to the function definition, read the comment and the function, all 27 lines, understand what they do, and then jump back. I'd rather prefer to read one line with a very clear semantics and no conditionals instead. > + > +/** > + Helper function to prepare state for handle_options (current state is saved) > + */ > +static inline void prepare_option_handling_flags() > +{ > + set_flags_for_option_handling(FALSE); > +} > + > +/** > + Helper function to revert state after handle_options to that from before > + prepare_option_handling_flags was called > + */ > +static inline void revert_option_handling_flags() > +{ > + set_flags_for_option_handling(TRUE); > +} and it keeps going on. if you remove the first function, you won't need these two functions either. Saving old value, restoring it. my_defaults_mark_files affects only my_load_defaults(). It makes no sense to restore it. > +/** > + > + Utility function to force the connection protocol to TCP when > + just the port is specified via the command line. Additionally, > + warns the user that the protocol has been changed (MDEV-14974). > + > + Notes: > + 1) This only takes effect when connecting to localhost > + 2) Windows uses TCP by default > + > + Arguments: > + @param [in] warn_to The file to write the warning to > + @param [in] host Name of the host to connect to > + @param [in, out] protocol_loc Location of the protocol option > + variable to update > +*/ > +static inline void warn_tcp_protocol_override(FILE *warn_to, > + char *host, > + uint *protocol_loc) > +{ > +#ifndef _WIN32 > + if ((host == NULL > + || strncmp(host, LOCAL_HOST, sizeof(LOCAL_HOST)-1) == 0)) > + { > + fprintf(warn_to, "WARNING: " > + "Forcing protocol to TCP due to port specification. " > + "Please explicitly state TCP protocol or remove " > + "port if unintended.\n"); > + *protocol_loc = MYSQL_PROTOCOL_TCP; > + } > +#endif > +} > diff --git a/client/mysql.cc b/client/mysql.cc > index 433fbd281b9..980daa14365 100644 > --- a/client/mysql.cc > +++ b/client/mysql.cc > @@ -206,6 +206,10 @@ static uint opt_protocol=0; > static const char *opt_protocol_type= ""; > static CHARSET_INFO *charset_info= &my_charset_latin1; > > +#ifndef _WIN32 > +static my_bool port_forcing_tcp_proto = FALSE; > +#endif > + > #include "sslopt-vars.h" > > const char *default_dbug_option="d:t:o,/tmp/mariadb.trace"; > @@ -1162,6 +1166,8 @@ int main(int argc,char *argv[]) > close(stdout_fileno_copy); /* Clean up dup(). */ > } > > + prepare_option_handling_flags(); Just do my_defaults_mark_files= TRUE; here > load_defaults_or_exit("my", load_default_groups, &argc, &argv); > defaults_argv=argv; > if ((status.exit_status= get_options(argc, (char **) argv))) > @@ -1171,6 +1177,14 @@ int main(int argc,char *argv[]) > exit(status.exit_status); > } > > + revert_option_handling_flags(); and don't revert it. > + if (port_forcing_tcp_proto) > + { > + warn_tcp_protocol_override(stderr, current_host, &opt_protocol); I'm not sure it needs a warning. But I don't have strong arguments against the warning either, so let's keep a warning, if you like it that way. Why everything is not on _WIN32 ? It has named pipes and tcp. Wouldn't it have the same issue with --port? > + } > + > + > if (status.batch && !status.line_buff && > !(status.line_buff= batch_readline_init(MAX_BATCH_BUFFER_SIZE, stdin))) > { > @@ -1715,8 +1729,11 @@ static void usage(int version) > > > my_bool > -get_one_option(const struct my_option *opt, const char *argument, const char *) > +get_one_option(const struct my_option *opt, const char *argument, const char *filename) > { > + /* Track when protocol is set via CLI to not force port TCP protocol override */ > + static my_bool ignore_port_override = FALSE; > + > switch(opt->id) { > case OPT_CHARSETS_DIR: > strmake_buf(mysql_charsets_dir, argument); > @@ -1780,6 +1797,19 @@ get_one_option(const struct my_option *opt, const char *argument, const char *) > else if ((opt_protocol= find_type_with_warning(argument, &sql_protocol_typelib, > opt->name)) <= 0) > exit(1); > +#ifndef _WIN32 > + /* MDEV-14974 > + * > + * Where specifying port implicitly will set the protocol to use TCP, if the > + * protocol is explicitly set after the port, prioritize protocol. > + */ please, see how multi-comments are usually formatted, and follow the same style. Generally there's no need to specify the MDEV, and there's no vertical line of asterisks. > + if(filename[0] == '\0') > + { > + /* Protocol is specified via command line */ > + ignore_port_override = TRUE; > + port_forcing_tcp_proto = FALSE; > + } > +#endif > #endif > break; > case OPT_SERVER_ARG: > @@ -1883,6 +1913,20 @@ get_one_option(const struct my_option *opt, const char *argument, const char *) > status.exit_status= 0; > mysql_end(-1); > break; > + case 'P': > +#ifndef _WIN32 > + /* MDEV-14974: > + * > + * If port is set via CLI, change protocol to TCP unless protocol has already > + * been specified via CLI or TCP is already used > + */ > + if (filename[0] == '\0' && !ignore_port_override > + && opt_protocol != MYSQL_PROTOCOL_TCP) > + { > + port_forcing_tcp_proto = TRUE; > + } let's also do the same treatment for --socket. If it's specified on the command line - force the socket protocol. What could be the least surprising behavior? I'd think the most intuitive would be, like, WYSIWIG thing - * if port is specified on the command line (but no socket or protocol) - it forces protocol=TCP * if socket is specified on the command line (but no port of protocol) - it forces protocol=SOCKET * if protocol is specified explicitly anywhere on the command line - port and socket lose their magic behavior * if there's no protocol, but both socket and port are specified? - I don't know, perhaps, let's keep the old behavior, no magic? hmm. How does --host affect that? > +#endif > + break; > case 'I': > case '?': > usage(0); > @@ -1916,8 +1960,10 @@ static int get_options(int argc, char **argv) > opt_max_allowed_packet= *mysql_params->p_max_allowed_packet; > opt_net_buffer_length= *mysql_params->p_net_buffer_length; > > + my_defaults_mark_files = TRUE; > if ((ho_error=handle_options(&argc, &argv, my_long_options, get_one_option))) > return(ho_error); > + my_defaults_mark_files = FALSE; my_defaults_mark_files has no effect on handle_options() > *mysql_params->p_max_allowed_packet= opt_max_allowed_packet; > *mysql_params->p_net_buffer_length= opt_net_buffer_length; > > diff --git a/mysql-test/main/port_force_tcp.result b/mysql-test/main/port_force_tcp.result > new file mode 100644 > index 00000000000..ae976bda6cc > --- /dev/null > +++ b/mysql-test/main/port_force_tcp.result > @@ -0,0 +1,14 @@ > +# > +# MDEV-14974: --port ignored for --host=localhost > +# > +Connection: Localhost via UNIX socket > +CURRENT_TEST: main.port_force_tcp > +Connection: localhost via TCP/IP > +CURRENT_TEST: main.port_force_tcp > +Connection: Localhost via UNIX socket > +CURRENT_TEST: main.port_force_tcp > +Connection: 127.0.0.1 via TCP/IP > +CURRENT_TEST: main.port_force_tcp > +Connection: localhost via TCP/IP > +CURRENT_TEST: main.port_force_tcp > +WARNING: Forcing protocol to TCP due to port specification. Please explicitly state TCP protocol or remove port if unintended. > diff --git a/mysql-test/main/port_force_tcp.test b/mysql-test/main/port_force_tcp.test > new file mode 100644 > index 00000000000..607670d16b0 > --- /dev/null > +++ b/mysql-test/main/port_force_tcp.test > @@ -0,0 +1,20 @@ > +--echo # > +--echo # MDEV-14974: --port ignored for --host=localhost > +--echo # > + > +--source include/not_embedded.inc > + > +--exec $MYSQL --host=localhost -e "status" | grep "Connection:" > +--cat_file $MYSQLTEST_VARDIR/log/current_test > + > +--exec $MYSQL --port=$MASTER_MYPORT --protocol=tcp -e "status" | grep "Connection:" > +--cat_file $MYSQLTEST_VARDIR/log/current_test > + > +--exec $MYSQL --host=localhost --port=$MASTER_MYPORT --protocol=socket -e "status" | grep "Connection:" > +--cat_file $MYSQLTEST_VARDIR/log/current_test > + > +--exec $MYSQL --host=127.0.0.1 --port=$MASTER_MYPORT -e "status" | grep "Connection:" > +--cat_file $MYSQLTEST_VARDIR/log/current_test > + > +--exec $MYSQL --host=localhost --port=$MASTER_MYPORT -e "status" | grep "Connection:" > +--cat_file $MYSQLTEST_VARDIR/log/current_test this will likely fail on windows, you need to source not_windows.inc too. why do you cat current_test after every --exec? better add --echo before every exec, like --echo # --host=localhost --port=$MASTER_MYPORT --exec $MYSQL --host=localhost --port=$MASTER_MYPORT -e "status" | grep "Connection:" Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 242cc3fc41f: MDEV-14974: --port ignored for --host=localhost
by Sergei Golubchik 03 May '21

03 May '21

Hi, Brandon! On May 03, Brandon Nesterenko wrote: > revision-id: 242cc3fc41f (mariadb-10.6.0-17-g242cc3fc41f) > parent(s): fd8c68c7fe6 > author: Brandon Nesterenko <brandon.nesterenko(a)mariadb.com> > committer: Brandon Nesterenko <brandon.nesterenko(a)mariadb.com> > timestamp: 2021-04-30 23:17:37 +0000 > message: > > MDEV-14974: --port ignored for --host=localhost > > Problem: > ======= > MariaDB's command line utilities (e.g., mysql, mysqldump, etc) silently ignore the --port option if no host is given or it is localhost. please, try to avoid very long lines in commit comments, they're difficult to read in git console tools. > > Fix: > === > During configuration processing, force protocol to TCP if port was specified via the command line. However, if protocol is additionally specified via the command line, it will be prioritized. > > diff --git a/client/client_priv.h b/client/client_priv.h > index 64818d2ab8d..6cb1f02ba33 100644 > --- a/client/client_priv.h > +++ b/client/client_priv.h > @@ -136,3 +136,85 @@ enum options_client > Name of the sys schema database. > */ > #define SYS_SCHEMA_DB_NAME "sys" > + > + > + > +/** > + Changes flags to prepare for (and revert to the previous state) > + from handle_options. > + > + When creating the state, the original state is saved and can be later > + resumed by calling this function and setting do_revert_values to TRUE. > + > + Flags: > + my_defaults_mark_files: propagates the source of a set option > + > + @param [in] do_revert_values behavior should revert when TRUE, prepare > + when FALSE > + */ > +static inline void set_flags_for_option_handling(my_bool do_revert_values) > +{ > + static my_bool prev_mark_files; > + > + if (do_revert_values) > + { > + my_defaults_mark_files = prev_mark_files; > + } > + else > + { > + prev_mark_files = my_defaults_mark_files; > + my_defaults_mark_files = TRUE; > + } > +} this isn't very helpful. my_defaults_mark_files is part of the my_getopt API. It's supposed to be used directly by whatever tool uses my_getopt. Here you've turned one line my_defaults_mark_files= TRUE; into 27 lines, so when I (or somebody reading the code) will see set_flags_for_option_handling(...); he'd have to go to the function definition, read the comment and the function, all 27 lines, understand what they do, and then jump back. I'd rather prefer to read one line with a very clear semantics and no conditionals instead. > + > +/** > + Helper function to prepare state for handle_options (current state is saved) > + */ > +static inline void prepare_option_handling_flags() > +{ > + set_flags_for_option_handling(FALSE); > +} > + > +/** > + Helper function to revert state after handle_options to that from before > + prepare_option_handling_flags was called > + */ > +static inline void revert_option_handling_flags() > +{ > + set_flags_for_option_handling(TRUE); > +} and it keeps going on. if you remove the first function, you won't need these two functions either. Saving old value, restoring it. my_defaults_mark_files affects only my_load_defaults(). It makes no sense to restore it. > +/** > + > + Utility function to force the connection protocol to TCP when > + just the port is specified via the command line. Additionally, > + warns the user that the protocol has been changed (MDEV-14974). > + > + Notes: > + 1) This only takes effect when connecting to localhost > + 2) Windows uses TCP by default > + > + Arguments: > + @param [in] warn_to The file to write the warning to > + @param [in] host Name of the host to connect to > + @param [in, out] protocol_loc Location of the protocol option > + variable to update > +*/ > +static inline void warn_tcp_protocol_override(FILE *warn_to, > + char *host, > + uint *protocol_loc) > +{ > +#ifndef _WIN32 > + if ((host == NULL > + || strncmp(host, LOCAL_HOST, sizeof(LOCAL_HOST)-1) == 0)) > + { > + fprintf(warn_to, "WARNING: " > + "Forcing protocol to TCP due to port specification. " > + "Please explicitly state TCP protocol or remove " > + "port if unintended.\n"); > + *protocol_loc = MYSQL_PROTOCOL_TCP; > + } > +#endif > +} > diff --git a/client/mysql.cc b/client/mysql.cc > index 433fbd281b9..980daa14365 100644 > --- a/client/mysql.cc > +++ b/client/mysql.cc > @@ -206,6 +206,10 @@ static uint opt_protocol=0; > static const char *opt_protocol_type= ""; > static CHARSET_INFO *charset_info= &my_charset_latin1; > > +#ifndef _WIN32 > +static my_bool port_forcing_tcp_proto = FALSE; > +#endif > + > #include "sslopt-vars.h" > > const char *default_dbug_option="d:t:o,/tmp/mariadb.trace"; > @@ -1162,6 +1166,8 @@ int main(int argc,char *argv[]) > close(stdout_fileno_copy); /* Clean up dup(). */ > } > > + prepare_option_handling_flags(); Just do my_defaults_mark_files= TRUE; here > load_defaults_or_exit("my", load_default_groups, &argc, &argv); > defaults_argv=argv; > if ((status.exit_status= get_options(argc, (char **) argv))) > @@ -1171,6 +1177,14 @@ int main(int argc,char *argv[]) > exit(status.exit_status); > } > > + revert_option_handling_flags(); and don't revert it. > + if (port_forcing_tcp_proto) > + { > + warn_tcp_protocol_override(stderr, current_host, &opt_protocol); I'm not sure it needs a warning. But I don't have strong arguments against the warning either, so let's keep a warning, if you like it that way. Why everything is not on _WIN32 ? It has named pipes and tcp. Wouldn't it have the same issue with --port? > + } > + > + > if (status.batch && !status.line_buff && > !(status.line_buff= batch_readline_init(MAX_BATCH_BUFFER_SIZE, stdin))) > { > @@ -1715,8 +1729,11 @@ static void usage(int version) > > > my_bool > -get_one_option(const struct my_option *opt, const char *argument, const char *) > +get_one_option(const struct my_option *opt, const char *argument, const char *filename) > { > + /* Track when protocol is set via CLI to not force port TCP protocol override */ > + static my_bool ignore_port_override = FALSE; > + > switch(opt->id) { > case OPT_CHARSETS_DIR: > strmake_buf(mysql_charsets_dir, argument); > @@ -1780,6 +1797,19 @@ get_one_option(const struct my_option *opt, const char *argument, const char *) > else if ((opt_protocol= find_type_with_warning(argument, &sql_protocol_typelib, > opt->name)) <= 0) > exit(1); > +#ifndef _WIN32 > + /* MDEV-14974 > + * > + * Where specifying port implicitly will set the protocol to use TCP, if the > + * protocol is explicitly set after the port, prioritize protocol. > + */ please, see how multi-comments are usually formatted, and follow the same style. Generally there's no need to specify the MDEV, and there's no vertical line of asterisks. > + if(filename[0] == '\0') > + { > + /* Protocol is specified via command line */ > + ignore_port_override = TRUE; > + port_forcing_tcp_proto = FALSE; > + } > +#endif > #endif > break; > case OPT_SERVER_ARG: > @@ -1883,6 +1913,20 @@ get_one_option(const struct my_option *opt, const char *argument, const char *) > status.exit_status= 0; > mysql_end(-1); > break; > + case 'P': > +#ifndef _WIN32 > + /* MDEV-14974: > + * > + * If port is set via CLI, change protocol to TCP unless protocol has already > + * been specified via CLI or TCP is already used > + */ > + if (filename[0] == '\0' && !ignore_port_override > + && opt_protocol != MYSQL_PROTOCOL_TCP) > + { > + port_forcing_tcp_proto = TRUE; > + } let's also do the same treatment for --socket. If it's specified on the command line - force the socket protocol. What could be the least surprising behavior? I'd think the most intuitive would be, like, WYSIWIG thing - * if port is specified on the command line (but no socket or protocol) - it forces protocol=TCP * if socket is specified on the command line (but no port of protocol) - it forces protocol=SOCKET * if protocol is specified explicitly anywhere on the command line - port and socket lose their magic behavior * if there's no protocol, but both socket and port are specified? - I don't know, perhaps, let's keep the old behavior, no magic? hmm. How does --host affect that? > +#endif > + break; > case 'I': > case '?': > usage(0); > @@ -1916,8 +1960,10 @@ static int get_options(int argc, char **argv) > opt_max_allowed_packet= *mysql_params->p_max_allowed_packet; > opt_net_buffer_length= *mysql_params->p_net_buffer_length; > > + my_defaults_mark_files = TRUE; > if ((ho_error=handle_options(&argc, &argv, my_long_options, get_one_option))) > return(ho_error); > + my_defaults_mark_files = FALSE; my_defaults_mark_files has no effect on handle_options() > *mysql_params->p_max_allowed_packet= opt_max_allowed_packet; > *mysql_params->p_net_buffer_length= opt_net_buffer_length; > > diff --git a/mysql-test/main/port_force_tcp.result b/mysql-test/main/port_force_tcp.result > new file mode 100644 > index 00000000000..ae976bda6cc > --- /dev/null > +++ b/mysql-test/main/port_force_tcp.result > @@ -0,0 +1,14 @@ > +# > +# MDEV-14974: --port ignored for --host=localhost > +# > +Connection: Localhost via UNIX socket > +CURRENT_TEST: main.port_force_tcp > +Connection: localhost via TCP/IP > +CURRENT_TEST: main.port_force_tcp > +Connection: Localhost via UNIX socket > +CURRENT_TEST: main.port_force_tcp > +Connection: 127.0.0.1 via TCP/IP > +CURRENT_TEST: main.port_force_tcp > +Connection: localhost via TCP/IP > +CURRENT_TEST: main.port_force_tcp > +WARNING: Forcing protocol to TCP due to port specification. Please explicitly state TCP protocol or remove port if unintended. > diff --git a/mysql-test/main/port_force_tcp.test b/mysql-test/main/port_force_tcp.test > new file mode 100644 > index 00000000000..607670d16b0 > --- /dev/null > +++ b/mysql-test/main/port_force_tcp.test > @@ -0,0 +1,20 @@ > +--echo # > +--echo # MDEV-14974: --port ignored for --host=localhost > +--echo # > + > +--source include/not_embedded.inc > + > +--exec $MYSQL --host=localhost -e "status" | grep "Connection:" > +--cat_file $MYSQLTEST_VARDIR/log/current_test > + > +--exec $MYSQL --port=$MASTER_MYPORT --protocol=tcp -e "status" | grep "Connection:" > +--cat_file $MYSQLTEST_VARDIR/log/current_test > + > +--exec $MYSQL --host=localhost --port=$MASTER_MYPORT --protocol=socket -e "status" | grep "Connection:" > +--cat_file $MYSQLTEST_VARDIR/log/current_test > + > +--exec $MYSQL --host=127.0.0.1 --port=$MASTER_MYPORT -e "status" | grep "Connection:" > +--cat_file $MYSQLTEST_VARDIR/log/current_test > + > +--exec $MYSQL --host=localhost --port=$MASTER_MYPORT -e "status" | grep "Connection:" > +--cat_file $MYSQLTEST_VARDIR/log/current_test this will likely fail on windows, you need to source not_windows.inc too. why do you cat current_test after every --exec? better add --echo before every exec, like --echo # --host=localhost --port=$MASTER_MYPORT --exec $MYSQL --host=localhost --port=$MASTER_MYPORT -e "status" | grep "Connection:" Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] a7032fbb32d: Bug#29363867: LOST CONNECTION TO MYSQL SERVER DURING QUERY
by Sergei Golubchik 27 Apr '21

27 Apr '21

Hi, Sanja! On Apr 27, Oleksandr Byelkin wrote: > revision-id: a7032fbb32d (mariadb-10.2.31-906-ga7032fbb32d) > parent(s): b862377c3e9 > author: Oleksandr Byelkin <sanja(a)mariadb.com> > committer: Oleksandr Byelkin <sanja(a)mariadb.com> > timestamp: 2021-04-27 16:24:43 +0200 > message: > > Bug#29363867: LOST CONNECTION TO MYSQL SERVER DURING QUERY > > The problem is that sharing default expression among set instruction > leads to attempt access result field of function created in > other instruction runtime MEM_ROOT and already freed > (a bit different then MySQL problem). > > Fix is the same as in MySQL (but no optimisation for constant), turn > DECLARE a, b, c type DEFAULT expr; > to > DECLARE a type DEFAULT expr, b type DEFAULT a, c type DEFAULT a; > > diff --git a/mysql-test/t/sp.test b/mysql-test/t/sp.test > index f13b3fbc281..007dfd1a10b 100644 > --- a/mysql-test/t/sp.test > +++ b/mysql-test/t/sp.test > @@ -10025,4 +10025,22 @@ DROP PROCEDURE p1; > DROP VIEW v1; > DROP TABLE t1; > > + > +--echo # > +--echo # forgot the bug summary (the first line of the commit comment) > +--echo # > + > +delimiter |; > +create function f1() returns bigint return now()-1| > +create procedure p1() > +begin > + declare b, c bigint default f1(); > + select b-c; > +end| > +call p1()| > +drop procedure p1| > +drop function f1| > +delimiter ;| > + > + > --echo #End of 10.2 tests > diff --git a/sql/item.cc b/sql/item.cc > index 42272fe0148..ec9f4ffb993 100644 > --- a/sql/item.cc > +++ b/sql/item.cc > @@ -2933,7 +2933,7 @@ bool Item_field::eq(const Item *item, bool binary_cmp) const > > table_map Item_field::used_tables() const > { > - if (field->table->const_table) > + if (!field || !field->table || field->table->const_table) in what cases can field or field->table be NULL here? > return 0; // const item > return (get_depended_from() ? OUTER_REF_TABLE_BIT : field->table->map); > } Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] e92037989f7: MDEV-21117: refine the server binlog-based recovery for semisync
by Sergei Golubchik 25 Apr '21

25 Apr '21

Hi, Andrei! Don't be confused by the subject, this is a review of git diff 450c017c2d 2fa526e26e that is of everything, combined. Not just one e92037989f7 commit. On Apr 25, Sujatha wrote: > revision-id: e92037989f7 (mariadb-10.3.26-128-ge92037989f7) > parent(s): 450c017c2d9 > author: Sujatha <sujatha.sivakumar(a)mariadb.com> > committer: Andrei Elkin <andrei.elkin(a)mariadb.com> > timestamp: 2021-04-13 12:26:12 +0300 > message: > > MDEV-21117: refine the server binlog-based recovery for semisync > diff --git a/libmariadb b/libmariadb > index fc431a035a2..e3824422064 160000 > --- a/libmariadb > +++ b/libmariadb > @@ -1 +1 @@ > -Subproject commit fc431a035a21ac1d4ef25d9d3cd8c4d7e64a8ee7 > +Subproject commit e38244220646a7e95c9be22576460aa7a4eb715f This is clearly a mistake, you erroneously checked in old libmaridb commit, rolling back a bunch of changes. See your commit 4bc83b2749 > diff --git a/mysql-test/suite/binlog/t/binlog_truncate_multi_engine-master.opt b/mysql-test/suite/binlog/t/binlog_truncate_multi_engine-master.opt > new file mode 100644 > index 00000000000..df675545bf9 > --- /dev/null > +++ b/mysql-test/suite/binlog/t/binlog_truncate_multi_engine-master.opt please rename this file to mysql-test/include/have_rocksdb.opt > diff --git a/mysql-test/suite/binlog/t/binlog_truncate_active_log.test b/mysql-test/suite/binlog/t/binlog_truncate_active_log.test > new file mode 100644 > index 00000000000..2b794d02dd0 > --- /dev/null > +++ b/mysql-test/suite/binlog/t/binlog_truncate_active_log.test > @@ -0,0 +1,74 @@ > +# ==== Purpose ==== > +# > +# Test verifies the truncation of single binary log file. > +# > +# ==== References ==== > +# > +# MDEV-21117: recovery for --rpl-semi-sync-slave-enabled server > + > +--source include/have_innodb.inc > +--source include/have_aria.inc > +# File: binlog_truncate_active_log.inc included in test makes use of > +# 'debug_sync' facility. > +--source include/have_debug_sync.inc you wouldn't need a comment if you'd include have_debug_sync.inc directly into binlog_truncate_active_log.inc. but ok, whatever you like > +--source include/have_binlog_format_statement.inc > + > +call mtr.add_suppression("Can.t init tc log"); > +call mtr.add_suppression("Aborting"); > + > +# The following cases are tested: > +# A. 2pc transaction is followed by a blank "zero-engines" one > +# B. 2pc transaction follows the blank one > +# C. Similarly to A, with the XA blank transaction > + > +RESET MASTER; > +CREATE TABLE t (f INT) ENGINE=INNODB; > +CREATE TABLE t2 (f INT) ENGINE=INNODB; > +CREATE TABLE tm (f INT) ENGINE=Aria; could you add a comment, explaining why you're using Aria here. (you wrote that in an email, but please add a comment too) > + > +--echo # Case A. > +# Using 'debug_sync' hold 'query1' execution after 'query1' is flushed and > +# synced to binary log but not yet committed. In an another connection hold > +# 'query2' execution after 'query2' is flushed and synced to binlog. > +# Crash and restart server with --rpl-semi-sync-slave-enabled=1 > +# > +# During recovery of binary log 'query1' status is checked with InnoDB engine, > +# it will be in prepared but not yet commited. All transactions starting from > +# 'query1' onwards will be removed from the binary log. > + > +--let $truncate_gtid_pos = 0-1-6 > +--let $query1 = INSERT INTO t VALUES (20) > +--let $query2 = DELETE FROM t2 WHERE f = 0 /* no such record */ > +--source binlog_truncate_active_log.inc > + > +--echo # Case B. > +# The inverted sequence ends up to truncate only $query2 > +--let $truncate_gtid_pos = 0-1-10 > +--let $query1 = DELETE FROM t2 WHERE f = 0 > +--let $query2 = INSERT INTO t VALUES (20) > +--source binlog_truncate_active_log.inc > + > + > +delimiter |; > +CREATE PROCEDURE sp_blank_xa() > +BEGIN > + XA START 'blank'; > + DELETE FROM t2 WHERE f = 0 /* no such record */; > + XA END 'blank'; > + XA PREPARE 'blank'; > +END| > +delimiter ;| > + > + > +--echo # Case C. > +--let $truncate_gtid_pos = 0-1-14 > +--let $query1 = INSERT INTO t VALUES (20) > +--let $pre_q2 = CALL sp_blank_xa > +--let $query2 = XA COMMIT 'blank' > +--source binlog_truncate_active_log.inc what was truncated here? a comment explains it for cases A and B, but not here. may be it'd make sense to do show binlog events after every restart, just to see the state of the binlog after truncation? > +DROP PROCEDURE sp_blank_xa; > + > +--echo # Cleanup > +DROP TABLE t,t2,tm; > + > +--echo # End of the tests > diff --git a/mysql-test/suite/binlog/t/binlog_truncate_active_log.inc b/mysql-test/suite/binlog/t/binlog_truncate_active_log.inc > new file mode 100644 > index 00000000000..bbc464066fc > --- /dev/null > +++ b/mysql-test/suite/binlog/t/binlog_truncate_active_log.inc > @@ -0,0 +1,55 @@ > +connect(master1,localhost,root,,); > +connect(master2,localhost,root,,); > +connect(master3,localhost,root,,); > + > +--connection default > + > +# First to commit few transactions > +INSERT INTO t VALUES (10); > +INSERT INTO tm VALUES (10); > + > +--connection master1 > +# Hold insert after write to binlog and before "run_commit_ordered" in engine > +SET DEBUG_SYNC= "commit_before_get_LOCK_commit_ordered SIGNAL master1_ready WAIT_FOR signal_never_arrives"; > +--send_eval $query1 > + > +--connection master2 > +SET DEBUG_SYNC= "now WAIT_FOR master1_ready"; > +if ($pre_q2) > +{ > + eval $pre_q2; > +} > +SET DEBUG_SYNC= "commit_before_get_LOCK_after_binlog_sync SIGNAL master2_ready"; > +# To binlog non-xid transactional group which will be truncated all right > +--send_eval $query2 > + > +--connection master3 > +SET DEBUG_SYNC= "now WAIT_FOR master2_ready"; > +SELECT @@global.gtid_binlog_pos as 'Before the crash'; > + > +--connection default > +--source include/kill_mysqld.inc > +--disconnect master1 > +--disconnect master2 > +--disconnect master3 > + > +# > +# Server restart > +# > +--let $restart_parameters= --rpl-semi-sync-slave-enabled=1 > +--source include/start_mysqld.inc > + > +# Check error log for a successful truncate message. > +--let $log_error_ = $MYSQLTEST_VARDIR/log/mysqld.1.err > + > +--let SEARCH_FILE=$log_error_ why not to set SEARCH_FILE directly? but ok, as you like > +--let SEARCH_PATTERN=Successfully truncated.*to remove transactions starting from GTID $truncate_gtid_pos > +--replace_regex /FOUND [0-9]+/FOUND #/ can it be found multiple times? Why would binlog be truncated more than once? > +--source include/search_pattern_in_file.inc > + > +SELECT @@global.gtid_binlog_pos as 'After the crash'; > +--echo "One row should be present in table 't'" > +SELECT * FROM t; > + > +# Local cleanup > +DELETE FROM t; > diff --git a/mysql-test/suite/binlog/t/binlog_truncate_multi_engine.test b/mysql-test/suite/binlog/t/binlog_truncate_multi_engine.test > new file mode 100644 > index 00000000000..94837e3c3ea > --- /dev/null > +++ b/mysql-test/suite/binlog/t/binlog_truncate_multi_engine.test > @@ -0,0 +1,56 @@ > +# ==== Purpose ==== > +# > +# Test verifies truncation of multiple binary logs with multiple transactional > +# storage engines > +# > +# ==== References ==== > +# > +# MDEV-21117: recovery for --rpl-semi-sync-slave-enabled server > + > +--source include/have_innodb.inc > +--source include/have_rocksdb.inc > +--source include/have_debug.inc > +--source include/have_debug_sync.inc > +--source include/have_binlog_format_row.inc > + > +--let $old_max_binlog_size= `select @@global.max_binlog_size` > +call mtr.add_suppression("Can.t init tc log"); > +call mtr.add_suppression("Aborting"); > +--let $MYSQLD_DATADIR= `SELECT @@datadir` > + > +CREATE TABLE t1 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; > +CREATE TABLE t2 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=rocksdb; > + > +--let $case = A: neither engine committed => rollback & binlog truncate > +# Hold off engine commits after write to binlog and its rotation. > +# The transaction is killed along with the server after that. > +--let $shutdown_timeout=0 > +--let $debug_sync_action = "commit_after_release_LOCK_log SIGNAL con1_ready WAIT_FOR signal_no_signal" > +--let $restart_parameters = --rpl-semi-sync-slave-enabled=1 > +--let $test_outcome= 1 row should be present in both tables; binlog is truncated; number of binlogs at reconnect - 3 > +--source binlog_truncate_multi_engine.inc > +--echo Proof of the truncated binlog file is readable (two transactions must be seen): > +--exec $MYSQL_BINLOG --short-form --skip-annotate-row-events $MYSQLD_DATADIR/master-bin.000002 > + > +--let $case = B: one engine has committed its transaction branch > +# Hold off after one engine has committed. > +--let $shutdown_timeout=0 > +--let $debug_sync_action = "commit_after_run_commit_ordered SIGNAL con1_ready WAIT_FOR signal_no_signal" > +# Both debug_sync and debug-dbug are required to make sure Engines remember the commit state > +# debug_sync alone will not help. > +--let $restart_parameters = --rpl-semi-sync-slave-enabled=1 --debug-dbug=d,binlog_truncate_partial_commit in the first review I wrote this seems to be a rather crude way of faking a partially committed transaction. better to crash after the first engine has committed, that'd be much more natural. and you replied This simulation aimed at (allows for) more complicated recovery time event sequences. In this case, indeed, crashing by demand is about of the same efforts. I can convert to that. [x] > +--let $test_outcome= 2 rows should be present in both tables; no binlog truncation; one extra binlog file compare with A; number of binlogs at reconnect - 4 > +--source binlog_truncate_multi_engine.inc > + > +--let $case = C: both engines have committed its transaction branch > +# Hold off after both engines have committed. The server is shut down. > +--let $shutdown_timeout= > +--let $restart_parameters = --rpl-semi-sync-slave-enabled=1 > +--let $test_outcome= 2 rows should be present in both tables; no binlog truncation; the same # of binlog files as in B; number of binlogs at reconnect - 4 > +--source binlog_truncate_multi_engine.inc > + > + > + > +DROP TABLE t1, t2; > + > +--echo # End of the tests > diff --git a/mysql-test/suite/binlog/t/binlog_truncate_multi_engine.inc b/mysql-test/suite/binlog/t/binlog_truncate_multi_engine.inc > new file mode 100644 > index 00000000000..41ae856dd9d > --- /dev/null > +++ b/mysql-test/suite/binlog/t/binlog_truncate_multi_engine.inc > @@ -0,0 +1,54 @@ > +# > +# Loop body of binlog_truncate_multi_engine.test what do you mean "loop body"? > +# Parameters: > +# $debug_sync_action describes debug-sync actions > +# $kill_server 1 when to crash, 0 for regular restart > +# $restart_parameters the caller may simulate partial commit at recovery > +# $test_outcome summary of extected results > +# $MYSQLD_DATADIR > + > +--echo # > +--echo # > +--echo # Case $case > +--echo # > +RESET MASTER; > +FLUSH LOGS; > +SET GLOBAL max_binlog_size= 4096; > + > +connect(con1,localhost,root,,); > +--echo List of binary logs before rotation > +--source include/show_binary_logs.inc > +INSERT INTO t1 VALUES (1, REPEAT("x", 1)); > +INSERT INTO t2 VALUES (1, REPEAT("x", 1)); I'm not sure I understand the point of REPEAT(..., 1) but sure, if you like it that way... :) > +BEGIN; > + INSERT INTO t1 VALUES (2, REPEAT("x", 4100)); > + INSERT INTO t2 VALUES (2, REPEAT("x", 4100)); > + > +--eval SET DEBUG_SYNC= $debug_sync_action > +send COMMIT; > + > +--connection default > +SET DEBUG_SYNC= "now WAIT_FOR con1_ready"; > +--echo List of binary logs after rotation > +--source include/show_binary_logs.inc > + > +--echo # restart the server with $restart_parameters > +--echo # the server is restarted > +--source include/restart_mysqld.inc > + > +--connection default > +--echo # > +--echo # *** Summary: $test_outcome: > +--echo # > +SELECT COUNT(*) FROM t1; > +SELECT COUNT(*) FROM t2; > +SELECT @@GLOBAL.gtid_binlog_state; > +SELECT @@GLOBAL.gtid_binlog_pos; > +--echo List of binary logs at the end of the tests > +--source include/show_binary_logs.inc > +--echo # *** > +# cleanup > +DELETE FROM t1; > +DELETE FROM t2; > +--disconnect con1 > +--echo # > diff --git a/mysql-test/suite/binlog/t/binlog_truncate_multi_log.test b/mysql-test/suite/binlog/t/binlog_truncate_multi_log.test > new file mode 100644 > index 00000000000..3b557bc89b8 > --- /dev/null > +++ b/mysql-test/suite/binlog/t/binlog_truncate_multi_log.test > @@ -0,0 +1,78 @@ > +# ==== Purpose ==== > +# > +# Test verifies truncation of multiple binary logs. > +# > +# ==== References ==== > +# MDEV-21117: recovery for --rpl-semi-sync-slave-enabled server > + > +--source include/have_innodb.inc > +--source include/have_debug_sync.inc > +--source include/have_binlog_format_row.inc > + > +call mtr.add_suppression("Can.t init tc log"); > +call mtr.add_suppression("Aborting"); > + > +SET @@global.max_binlog_size= 4096; > + > +RESET MASTER; > +FLUSH LOGS; > +CREATE TABLE ti (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; > +CREATE TABLE tm (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=MyISAM; > + > +connect(master1,localhost,root,,); > +--echo "List of binary logs before rotation" > +--source include/show_binary_logs.inc > + > +# Some load to either non- and transactional egines > +# that should not affect the following recovery: > +INSERT INTO ti VALUES(1,"I am gonna survive"); > +INSERT INTO tm VALUES(1,"me too!"); > + > +# hold on near engine commit > +SET DEBUG_SYNC= "commit_after_release_LOCK_after_binlog_sync SIGNAL master1_ready WAIT_FOR con1_go"; > +--send_eval INSERT INTO ti VALUES (2, REPEAT("x", 4100)) > + > +connect(master2,localhost,root,,); > +# The 2nd trx for recovery, it does not rotate binlog > +SET DEBUG_SYNC= "now WAIT_FOR master1_ready"; > +SET DEBUG_SYNC= "commit_before_get_LOCK_commit_ordered SIGNAL master2_ready WAIT_FOR master2_go"; > +--send_eval INSERT INTO ti VALUES (3, "not gonna survive") send_eval? what are you evaluating here? > + > +--connection default > +SET DEBUG_SYNC= "now WAIT_FOR master2_ready"; > +--echo "List of binary logs before crash" > +--source include/show_binary_logs.inc > +--echo # The gtid binlog state prior the crash will be truncated at the end of the test > +SELECT @@global.gtid_binlog_state; > + > +--connection default > +--source include/kill_mysqld.inc > +--disconnect master1 > +--disconnect master2 > + > +# > +# Server restart > +# > +--let $restart_parameters= --rpl-semi-sync-slave-enabled=1 > +--source include/start_mysqld.inc > + > +# Check error log for a successful truncate message. > +let $log_error_ = $MYSQLTEST_VARDIR/log/mysqld.1.err; > + > +--let SEARCH_FILE=$log_error_ > +--let SEARCH_PATTERN=truncated binlog file:.*master.*000002 > +--replace_regex /FOUND [0-9]+/FOUND #/ can it be found multiple times? Why would binlog be truncated more than once? > +--source include/search_pattern_in_file.inc > + > + > +--echo "One record should be present in table" > +SELECT * FROM ti; > + > +--echo # The truncated gtid binlog state > +SELECT @@global.gtid_binlog_state; > +SELECT @@global.gtid_binlog_pos; > + > +--echo # Cleanup > +DROP TABLE ti; > + > +--echo # End of the tests > diff --git a/mysql-test/suite/binlog/t/binlog_truncate_multi_log_unsafe.test b/mysql-test/suite/binlog/t/binlog_truncate_multi_log_unsafe.test > new file mode 100644 > index 00000000000..38a9c0832f4 > --- /dev/null > +++ b/mysql-test/suite/binlog/t/binlog_truncate_multi_log_unsafe.test > @@ -0,0 +1,120 @@ > +# ==== Purpose ==== > +# The test verifies attempt to recover by the semisync slave server whose > +# binlog is unsafe for truncation. > +# > +# ==== Implementation ==== > +# 2 binlog files are created with the 1st one destined to be the binlog > +# checkpoint file for recovery. > +# The final group of events is replication unsafe (myisam INSERT). > +# Therefore the semisync slave recovery may not. > +# > +# Steps: > +# 0 - Set max_binlog_size= 4096, to help an insert into a > +# transaction table 'ti' get binlog rotated while the > +# transaction won't be committed, being stopped at > +# a prior to commit debug_sync point > +# 1 - insert into a non-transactional 'tm' table completes with > +# binary logging as well > +# 2 - kill and attempt to restart the server as semisync slave that > +# must produce an expected unsafe-to-recover error > +# 3 - complete the test with a normal restart that successfully finds and > +# commits the transaction in doubt. > +# > +# ==== References ==== > +# > +# MDEV-21117: recovery for --rpl-semi-sync-slave-enabled server > +# > + > +--source include/have_innodb.inc > +--source include/have_debug_sync.inc > +--source include/have_binlog_format_row.inc > + > +SET @@global.max_binlog_size= 4096; > + > +call mtr.add_suppression("Table '.*tm' is marked as crashed and should be repaired"); > +call mtr.add_suppression("Got an error from unknown thread"); > +call mtr.add_suppression("Checking table: '.*tm'"); > +call mtr.add_suppression("Recovering table: '.*tm'"); > +call mtr.add_suppression("Cannot trim the binary log to file"); > +call mtr.add_suppression("Crash recovery failed"); > +call mtr.add_suppression("Can.t init tc log"); > +call mtr.add_suppression("Aborting"); > +call mtr.add_suppression("Found 1 prepared transactions"); > +call mtr.add_suppression("mysqld: Table.*tm.*is marked as crashed"); > +call mtr.add_suppression("Checking table.*tm"); > + > +RESET MASTER; > +FLUSH LOGS; > +CREATE TABLE ti (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; > +CREATE TABLE tm (f INT) ENGINE=MYISAM; > + > +--let $row_count = 5 > +--let $i = `select $row_count-2` > +--disable_query_log > +while ($i) > +{ > + --eval INSERT INTO ti VALUES ($i, REPEAT("x", 1)) > + --dec $i > +} > +--enable_query_log > +INSERT INTO tm VALUES(1); > + > +connect(master1,localhost,root,,); > +connect(master2,localhost,root,,); > +connect(master3,localhost,root,,); > + > +--connection master1 > + > +# The 1st trx binlogs, rotate binlog and hold on before committing at engine > +SET DEBUG_SYNC= "commit_after_release_LOCK_after_binlog_sync SIGNAL master1_ready WAIT_FOR master1_go"; > +--send_eval INSERT INTO ti VALUES ($row_count - 1, REPEAT("x", 4100)) > + > +--connection master2 > + > +# The 2nd trx for recovery, it does not rotate binlog > +SET DEBUG_SYNC= "commit_before_get_LOCK_commit_ordered SIGNAL master2_ready WAIT_FOR master2_go"; > +--send_eval INSERT INTO ti VALUES ($row_count, REPEAT("x", 1)) > + > +--connection master3 > +SET DEBUG_SYNC= "now WAIT_FOR master2_ready"; > +SET DEBUG_SYNC= "commit_before_get_LOCK_after_binlog_sync SIGNAL master3_ready"; > +--send INSERT INTO tm VALUES (2) > + > +--connection default > +SET DEBUG_SYNC= "now WAIT_FOR master3_ready"; > +--echo # The gtid binlog state prior the crash must be restored at the end of the test; > +SELECT @@global.gtid_binlog_state; > +--source include/kill_mysqld.inc > + > +# > +# Server restarts > +# > +--echo # Failed restart as the semisync slave > +--error 1 > +--exec $MYSQLD_LAST_CMD --rpl-semi-sync-slave-enabled=1 >> $MYSQLTEST_VARDIR/log/mysqld.1.err 2>&1 > + > +--echo # Normal restart > +--source include/start_mysqld.inc > + > +# Check error log for correct messages. > +let $log_error_ = $MYSQLTEST_VARDIR/log/mysqld.1.err; > + > +--let SEARCH_FILE=$log_error_ > +--let SEARCH_PATTERN=Cannot trim the binary log to file > +--replace_regex /FOUND [0-9]+/FOUND #/ can it be found multiple times? Why would binlog be truncated more than once? > +--source include/search_pattern_in_file.inc > + > +--echo # Proof that the in-doubt transactions are recovered by the 2nd normal server restart > +--eval SELECT COUNT(*) = $row_count as 'True' FROM ti > +# myisam table may require repair (which is not tested here) > +--disable_warnings > +SELECT COUNT(*) <= 1 FROM tm; > +--enable_warnings > + > +--echo # The gtid binlog state prior the crash is restored now > +SELECT @@GLOBAL.gtid_binlog_state; > +SELECT @@GLOBAL.gtid_binlog_pos; > + > +--echo # Cleanup > +DROP TABLE ti, tm; > +--echo End of test > diff --git a/mysql-test/suite/rpl/t/rpl_semi_sync_fail_over.cnf b/mysql-test/suite/rpl/t/rpl_semi_sync_fail_over.cnf > new file mode 100644 > index 00000000000..f8312bdc5b8 > --- /dev/null > +++ b/mysql-test/suite/rpl/t/rpl_semi_sync_fail_over.cnf > @@ -0,0 +1,11 @@ > +!include suite/rpl/rpl_1slave_base.cnf > +!include include/default_client.cnf > + > + > +[mysqld.1] > +log-slave-updates > +gtid-strict-mode=1 > + > +[mysqld.2] > +log-slave-updates > +gtid-strict-mode=1 generally opt files (rpl_semi_sync_fail_over.opt in this case) are preferred, because mtr will know what options to apply, while cnf files are more opaque > diff --git a/mysql-test/suite/rpl/t/rpl_semi_sync_fail_over.test b/mysql-test/suite/rpl/t/rpl_semi_sync_fail_over.test > new file mode 100644 > index 00000000000..a8b40d6ed05 > --- /dev/null > +++ b/mysql-test/suite/rpl/t/rpl_semi_sync_fail_over.test > @@ -0,0 +1,143 @@ > +# ==== Purpose ==== > +# > +# Test verifies replication failover scenario. > +# > +# ==== Implementation ==== > +# > +# Steps: > +# 0 - Having two servers 1 and 2 enable semi-sync replication with > +# with the master wait 'after_sync'. > +# 1 - Insert a row. While inserting second row simulate > +# a server crash at once the transaction is written to binlog, flushed > +# and synced but the binlog position is not updated. > +# 2 - Post crash-recovery on the old master execute there CHANGE MASTER > +# TO command to connect to server id 2. > +# 3 - The old master new slave server 1 must connect to the new > +# master server 2. > +# 4 - repeat the above to crash the new master and restore in role the old one > +# > +# ==== References ==== > +# > +# MDEV-21117: recovery for --rpl-semi-sync-slave-enabled server > + > + > +--source include/have_innodb.inc > +--source include/have_debug_sync.inc > +--source include/have_binlog_format_row.inc > +--let $rpl_topology=1->2 > +--source include/rpl_init.inc why not to source master-slave.inc if you're using a standard master-slave topology anyway? > + > +--connection server_2 > +--source include/stop_slave.inc > + > +--connection server_1 > +RESET MASTER; > +SET @@global.max_binlog_size= 4096; > + > +--connection server_2 > +RESET MASTER; > +SET @@global.max_binlog_size= 4096; > +set @@global.rpl_semi_sync_slave_enabled = 1; > +set @@global.gtid_slave_pos = ""; > +CHANGE MASTER TO master_use_gtid= slave_pos; > +--source include/start_slave.inc > + > + > +--connection server_1 > +ALTER TABLE mysql.gtid_slave_pos ENGINE=InnoDB; here I asked "why?" and you replied Actually not need to. There's no crash in the middle of a slave transaction. So it must be a copy-paste leftover. [x] > +set @@global.rpl_semi_sync_master_enabled = 1; > +set @@global.rpl_semi_sync_master_wait_point=AFTER_SYNC; > + > +call mtr.add_suppression("Can.t init tc log"); > +call mtr.add_suppression("Aborting"); > +call mtr.add_suppression("1 client is using or hasn.t closed the table properly"); > +call mtr.add_suppression("Table './mtr/test_suppressions' is marked as crashed and should be repaired"); > + > +CREATE TABLE t1 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; > +INSERT INTO t1 VALUES (1, 'dummy1'); > + > +# > +# CRASH the original master, and FAILOVER to the new > +# > + > +# value 1 for server id 1 -> 2 failover > +--let $failover_to_slave=1 > +--let $query_to_crash= INSERT INTO t1 VALUES (2, REPEAT("x", 4100)) > +--let $log_search_pattern=truncated binlog file:.*master.*000001 > +--source rpl_semi_sync_crash.inc > + > +--connection server_2 > +--let $rows_so_far=3 > +--eval INSERT INTO t1 VALUES ($rows_so_far, 'dummy3') > +--save_master_pos > +--echo # The gtid state on current master must be equal to ... > +SHOW VARIABLES LIKE 'gtid_binlog_pos'; > + > +--connection server_1 > +--sync_with_master > +--eval SELECT COUNT(*) = $rows_so_far as 'true' FROM t1 > +--echo # ... the gtid states on the slave: > +SHOW VARIABLES LIKE 'gtid_slave_pos'; > +SHOW VARIABLES LIKE 'gtid_binlog_pos'; > + > +--connection server_2 > +# > +# CRASH the new master and FAILOVER back to the original > +# > + > +# value 0 for the reverse server id 2 -> 1 failover > +--let $failover_to_slave=0 > +--let $query_to_crash = INSERT INTO t1 VALUES (4, REPEAT("x", 4100)) > +--let $query2_to_crash= INSERT INTO t1 VALUES (5, REPEAT("x", 4100)) > +--let $log_search_pattern=truncated binlog file:.*slave.*000001 > +--source rpl_semi_sync_crash.inc > + > +--connection server_1 > +--let $rows_so_far=6 > +--eval INSERT INTO t1 VALUES ($rows_so_far, 'Done') > +--save_master_pos > +--echo # The gtid state on current master must be equal to ... > +SHOW VARIABLES LIKE 'gtid_binlog_pos'; > + > +--connection server_2 > +--sync_with_master > +--eval SELECT COUNT(*) = $rows_so_far as 'true' FROM t1 > +--echo # ... the gtid states on the slave: > +SHOW VARIABLES LIKE 'gtid_slave_pos'; > +SHOW VARIABLES LIKE 'gtid_binlog_pos'; > + > + > +--let $diff_tables=server_1:t1, server_2:t1 > +--source include/diff_tables.inc > + > +# > +--echo # Cleanup > +# > +--connection server_1 > +DROP TABLE t1; > +--save_master_pos > + > +--connection server_2 > +--sync_with_master > +--source include/stop_slave.inc > + > +--connection server_1 > +set @@global.rpl_semi_sync_master_enabled = 0; > +set @@global.rpl_semi_sync_slave_enabled = 0; > +set @@global.rpl_semi_sync_master_wait_point=default; > +RESET SLAVE; > +RESET MASTER; > + > +--connection server_2 > +set @@global.rpl_semi_sync_master_enabled = 0; > +set @@global.rpl_semi_sync_slave_enabled = 0; > +set @@global.rpl_semi_sync_master_wait_point=default; > + > +evalp CHANGE MASTER TO master_host='127.0.0.1', master_port=$SERVER_MYPORT_1, master_user='root', master_use_gtid=no; > +--source include/start_slave.inc > + > +connection default; > +--enable_reconnect > +--source include/wait_until_connected_again.inc > + > +--source include/rpl_end.inc > diff --git a/sql/handler.h b/sql/handler.h > index fc69d9423b4..05a62ed0021 100644 > --- a/sql/handler.h > +++ b/sql/handler.h > @@ -873,6 +874,15 @@ typedef struct xid_t XID; > /* The 'buf' has to have space for at least SQL_XIDSIZE bytes. */ > uint get_sql_xid(XID *xid, char *buf); > > +/* struct for semisync slave binlog truncate recovery */ > +struct xid_recovery_member > +{ > + my_xid xid; > + uint in_engine_prepare; // number of engines that have xid prepared > + bool decided_to_commit; > + std::pair<uint, my_off_t> binlog_coord; // semisync recovery binlog offset wouldn't it be clearer to have a struct with named members? in fact, I'm somewhat surprised there's no such struct for binlog coords already. > +}; > + > /* for recover() handlerton call */ > #define MIN_XID_LIST_SIZE 128 > #define MAX_XID_LIST_SIZE (1024*128) > @@ -4820,7 +4830,8 @@ int ha_commit_one_phase(THD *thd, bool all); > int ha_commit_trans(THD *thd, bool all); > int ha_rollback_trans(THD *thd, bool all); > int ha_prepare(THD *thd); > -int ha_recover(HASH *commit_list); > +int ha_recover(HASH *commit_list, MEM_ROOT *mem_root= NULL); > +uint ha_recover_complete(HASH *commit_list, std::pair<uint, my_off_t> *coord= NULL); is coord a truncation position? > > /* transactions: these functions never call handlerton functions directly */ > int ha_enable_transaction(THD *thd, bool on); > diff --git a/sql/log_event.h b/sql/log_event.h > index 8a342cb5cd3..1036e9a44d4 100644 > --- a/sql/log_event.h > +++ b/sql/log_event.h > @@ -482,6 +482,16 @@ class String; > */ > #define LOG_EVENT_IGNORABLE_F 0x80 > > +/** > + @def LOG_EVENT_ACCEPT_OWN_F > + > + Flag sets by the semisync slave for accepting > + the same server_id ("own") events which the slave must not have > + in its state. Typically such events were never committed by > + their originator (this server) and discared at its semisync-slave recovery. > +*/ > +#define LOG_EVENT_ACCEPT_OWN_F 0x4000 may be, add an assert on all received events that such a flag is not set? it can only be set on events in relay log. also, consider the case when this event is read from a relay log, applied, and then sent to further slaves. In this case this flag must be removed before sending, otherwise they'll mistakenly might apply it if the server_id will match. > + > /** > @def LOG_EVENT_SKIP_REPLICATION_F > > @@ -3357,6 +3367,12 @@ class Gtid_log_event: public Log_event > uint64 commit_id; > uint32 domain_id; > uchar flags2; > + uint flags_extra; // more flags area placed after the regular flags2's one > + /* > + Extra to a "base" engine recoverable engines participating > + in the transaction. Zero, when the base engine only is present. what's a "base engine"? > + */ > + uint8 extra_engines; > > /* Flags2. */ > > diff --git a/sql/handler.cc b/sql/handler.cc > index c0a810a72bc..a46cef6b64c 100644 > --- a/sql/handler.cc > +++ b/sql/handler.cc > @@ -1637,9 +1672,17 @@ commit_one_phase_2(THD *thd, bool all, THD_TRANS *trans, bool is_real_trans) > DEBUG_SYNC(thd, "commit_one_phase_2"); > if (ha_info) > { > + int err; > + > + if (has_binlog_hton(ha_info) && can you replace has_binlog_hton() with, like, if trx cache is not empty or binlog enabled or something like that? > + (err= binlog_commit(thd, all, > + is_ro_1pc_trans(thd, ha_info, all, is_real_trans)))) > + { > + my_error(ER_ERROR_DURING_COMMIT, MYF(0), err); > + error= 1; > + } > for (; ha_info; ha_info= ha_info_next) > { > - int err; > handlerton *ht= ha_info->ht(); > if ((err= ht->commit(ht, thd, all))) > { > @@ -1962,8 +2008,177 @@ struct xarecover_st > XID *list; > HASH *commit_list; > bool dry_run; > + MEM_ROOT *mem_root; > + bool error; > }; > > +/** > + Inserts a new hash member. > + > + returns a successfully created and inserted @c xid_recovery_member > + into hash @c hash_arg, > + or NULL. > +*/ > +static xid_recovery_member* > +xid_member_insert(HASH *hash_arg, my_xid xid_arg, MEM_ROOT *ptr_mem_root) > +{ > + xid_recovery_member *member= (xid_recovery_member*) > + alloc_root(ptr_mem_root, sizeof(xid_recovery_member)); > + if (!member) > + return NULL; > + > + member->xid= xid_arg; > + member->in_engine_prepare= 1; > + member->decided_to_commit= false; > + > + return my_hash_insert(hash_arg, (uchar*) member) ? NULL : member; > +} > + > +/* > + Inserts a new or updates an existing hash member to increment > + the member's prepare counter. > + > + returns false on success, > + true otherwise. > +*/ > +static bool xid_member_replace(HASH *hash_arg, my_xid xid_arg, > + MEM_ROOT *ptr_mem_root) > +{ > + xid_recovery_member* member; > + if ((member= (xid_recovery_member *) > + my_hash_search(hash_arg, (uchar *)& xid_arg, sizeof(xid_arg)))) > + member->in_engine_prepare++; > + else > + member= xid_member_insert(hash_arg, xid_arg, ptr_mem_root); > + > + return member == NULL; > +} > + > +/* > + Decision to commit returns true, otherwise false for rollback. > + Flagged to commit member is destined to commit. If it is in doubt in case > + A. the caller does not specify coord_ptr (always so in the normal recovery), or > + B. coord_ptr is not NULL (can only be so in the semisync slave case) and its > + offset is greater than that of the member's the decision is rollback. > + If both A,B do not hold - which is the semisync slave recovery case - > + the decision is to rollback. > +*/ > +static bool xarecover_decide(xid_recovery_member* member, > + xid_t x, std::pair<uint, my_off_t> *coord_ptr) > +{ > + return > + member->decided_to_commit ? true : > + !coord_ptr ? false : > + (member->binlog_coord < *coord_ptr ? // semisync slave recovery > + true : false); > +} > + > +struct xarecover_iterate_arg > +{ > + handlerton *hton; > + std::pair<uint, my_off_t> *binlog_coord; > +}; > + > +/* > + Hash iterate function to complete with commit or rollback as either > + has been decided already or decide now (in the semisync recovery) > + via comparison against passed offset. > + Commit when the offset is greater than that of the member. > +*/ > +static my_bool xarecover_do_commit_or_rollback(void *member_arg, > + void *iter_arg) > +{ > + xid_recovery_member *member= (xid_recovery_member*) member_arg; > + handlerton *hton= ((xarecover_iterate_arg*) iter_arg)->hton; > + std::pair<uint, my_off_t> *max_coord_ptr= > + ((xarecover_iterate_arg*) iter_arg)->binlog_coord; > + xid_t x; > + my_bool rc; > + > + x.set(member->xid); > + > + rc= xarecover_decide(member, x, max_coord_ptr) ? > + hton->commit_by_xid(hton, &x) : hton->rollback_by_xid(hton, &x); > + > + DBUG_ASSERT(rc || member->in_engine_prepare > 0); > + > + if (!rc) > + { > + /* > + This block relies on Engine to report XAER_NOTA at > + "complete"_by_xid for unknown xid. > + */ > + member->in_engine_prepare--; > + if (global_system_variables.log_warnings > 2) > + sql_print_warning("%s transaction with xid %llu", may be not a sql_print_warning, but a sql_print_information? it's just an informational message > + member->decided_to_commit ? > + "Committed" : "Rolled back", (ulonglong) member->xid); > + } > + > + return false; > +} > + > +static my_bool xarecover_do_count_in_prepare(void *member_arg, > + void *ptr_count) > +{ > + xid_recovery_member *member= (xid_recovery_member*) member_arg; > + if (member->in_engine_prepare) > + { > + (*(uint*) ptr_count)++; > + if (global_system_variables.log_warnings > 2) > + sql_print_warning("Found prepared transaction with xid %llu", > + (ulonglong) member->xid); > + } > + > + return false; > +} > + > +struct xarecover_complete_arg > +{ > + HASH *commit_list; > + std::pair<uint, my_off_t> *binlog_coord; > +}; > + > +/* > + Completes binlog recovery to invoke a decider function for > + each transaction in doubt. > +*/ > +static my_bool xarecover_binlog_handlerton(THD *unused, > + plugin_ref plugin, > + void *arg) > +{ > + handlerton *hton= plugin_hton(plugin); > + > + if (hton->state == SHOW_OPTION_YES && hton->recover) > + { > + xarecover_iterate_arg iter_arg= > + { > + hton, > + ((xarecover_complete_arg*) arg)->binlog_coord > + }; > + my_hash_iterate(((xarecover_complete_arg*) arg)->commit_list, > + xarecover_do_commit_or_rollback, &iter_arg); > + } > + > + return FALSE; > +} > + > +/* > + Completes binlog recovery to invoke decider functions for > + each handerton. > + Returns the number of transactions remained doubtful. > +*/ > +uint ha_recover_complete(HASH *commit_list, std::pair<uint, my_off_t> *coord) > +{ > + uint count= 0; > + xarecover_complete_arg complete_arg= { commit_list, coord }; > + plugin_foreach(NULL, xarecover_binlog_handlerton, > + MYSQL_STORAGE_ENGINE_PLUGIN, &complete_arg); > + my_hash_iterate(commit_list, xarecover_do_count_in_prepare, &count); wouldn't it be cleaner to do everything in one commit_list scan? for every xid_recovery_member: run plugin_foreach, commit or rollback as needed increment a counter, if still in doubt > + > + return count; > +} > + > static my_bool xarecover_handlerton(THD *unused, plugin_ref plugin, > void *arg) > { > @@ -1973,6 +2188,9 @@ static my_bool xarecover_handlerton(THD *unused, plugin_ref plugin, > > if (hton->state == SHOW_OPTION_YES && hton->recover) > { > +#ifndef DBUG_OFF > + my_xid dbug_xid_list[128] __attribute__((unused)) = {0}; > +#endif What do you use it for? > while ((got= hton->recover(hton, info->list, info->len)) > 0 ) > { > sql_print_information("Found %d prepared transaction(s) in %s", > diff --git a/sql/log.cc b/sql/log.cc > index 8073f09ab88..a90d1e757e8 100644 > --- a/sql/log.cc > +++ b/sql/log.cc > @@ -2050,14 +2058,17 @@ static int binlog_commit(handlerton *hton, THD *thd, bool all) > Otherwise, we accumulate the changes. > */ > if (likely(!error) && ending_trans(thd, all)) > + { > + cache_mngr->ro_1pc= ro_1pc; > error= binlog_commit_flush_trx_cache(thd, all, cache_mngr); > + cache_mngr->ro_1pc= false; Why do you put it in cache_mngr, instead of passing it down, like you pass `all` ? > + } > > /* > This is part of the stmt rollback. > */ > if (!all) > cache_mngr->trx_cache.set_prev_position(MY_OFF_T_UNDEF); > - > THD_STAGE_INFO(thd, org_stage); > DBUG_RETURN(error); > } > @@ -9609,6 +9626,147 @@ int TC_LOG::using_heuristic_recover() > /****** transaction coordinator log for 2pc - binlog() based solution ******/ > #define TC_LOG_BINLOG MYSQL_BIN_LOG > > +/** > + Truncates the current binlog to specified position. Removes the rest of binlogs > + which are present after this binlog file. > + > + @param truncate_file Holds the binlog name to be truncated > + @param truncate_pos Position within binlog from where it needs to > + truncated. > + > + @retval true ok > + @retval false error > + > +*/ > +bool MYSQL_BIN_LOG::truncate_and_remove_binlogs(const char *file_name, > + my_off_t pos, > + rpl_gtid *ptr_gtid) > +{ > + int error= 0; > +#ifdef HAVE_REPLICATION > + LOG_INFO log_info; > + THD *thd= current_thd; > + my_off_t index_file_offset= 0; > + File file= -1; > + MY_STAT s; > + > + if ((error= find_log_pos(&log_info, file_name, 1))) > + { > + sql_print_error("Failed to locate binary log file:%s." > + "Error:%d", file_name, error); > + goto end; > + } > + > + while (!(error= find_next_log(&log_info, 1))) > + { > + if (!index_file_offset) > + { > + index_file_offset= log_info.index_file_start_offset; > + if ((error= open_purge_index_file(TRUE))) > + { > + sql_print_error("Failed to open purge index " > + "file:%s. Error:%d", purge_index_file_name, error); > + goto end; > + } > + } > + if ((error= register_purge_index_entry(log_info.log_file_name))) > + { > + sql_print_error("Failed to copy %s to purge index" > + " file. Error:%d", log_info.log_file_name, error); > + goto end; > + } > + } > + > + if (error != LOG_INFO_EOF) > + { > + sql_print_error("Failed to find the next binlog to " > + "add to purge index register. Error:%d", error); > + goto end; > + } > + > + if (is_inited_purge_index_file()) > + { > + if (!index_file_offset) > + index_file_offset= log_info.index_file_start_offset; > + > + if ((error= sync_purge_index_file())) > + { > + sql_print_error("Failed to flush purge index " > + "file. Error:%d", error); > + goto end; > + } > + > + // Trim index file > + if ((error= > + mysql_file_chsize(index_file.file, index_file_offset, '\n', > + MYF(MY_WME))) || > + (error= > + mysql_file_sync(index_file.file, MYF(MY_WME|MY_SYNC_FILESIZE)))) > + { > + sql_print_error("Failed to trim binlog index " > + "file:%s to offset:%llu. Error:%d", index_file_name, > + index_file_offset, error); > + goto end; > + } > + > + /* Reset data in old index cache */ > + if ((error= reinit_io_cache(&index_file, READ_CACHE, (my_off_t) 0, 0, 1))) > + { > + sql_print_error("Failed to reinit binlog index " > + "file. Error:%d", error); > + goto end; > + } > + > + /* Read each entry from purge_index_file and delete the file. */ > + if ((error= purge_index_entry(thd, NULL, TRUE))) > + { > + sql_print_error("Failed to process registered " > + "files that would be purged."); > + goto end; > + } > + } > + > + DBUG_ASSERT(pos); > + > + if ((file= mysql_file_open(key_file_binlog, file_name, > + O_RDWR | O_BINARY, MYF(MY_WME))) < 0) > + { > + error= 1; > + sql_print_error("Failed to open binlog file:%s for " > + "truncation.", file_name); > + goto end; > + } > + my_stat(file_name, &s, MYF(0)); > + > + /* Change binlog file size to truncate_pos */ > + if ((error= > + mysql_file_chsize(file, pos, 0, MYF(MY_WME))) || > + (error= mysql_file_sync(file, MYF(MY_WME|MY_SYNC_FILESIZE)))) > + { > + sql_print_error("Failed to trim the " > + "binlog file:%s to size:%llu. Error:%d", > + file_name, pos, error); > + goto end; > + } > + else > + { > + char buf[21]; > + > + longlong10_to_str(ptr_gtid->seq_no, buf, 10); > + sql_print_information("Successfully truncated binlog file:%s " > + "to pos:%llu to remove transactions starting from " > + "GTID %u-%u-%s", file_name, pos, > + ptr_gtid->domain_id, ptr_gtid->server_id, buf); > + } > + > +end: > + if (file >= 0) > + mysql_file_close(file, MYF(MY_WME)); Why you don't clean inuse flag here? You used to do it in the previous version of the patch. > + > + error= error || close_purge_index_file(); > +#endif > + return error > 0; > +} > int TC_LOG_BINLOG::open(const char *opt_name) > { > int error= 1; > @@ -10215,34 +10914,50 @@ int TC_LOG_BINLOG::recover(LOG_INFO *linfo, const char *last_log_name, > end_io_cache(&log); > mysql_file_close(file, MYF(MY_WME)); > file= -1; > + /* > + NOTE: reading other binlog's FD is necessary for finding out > + the checksum status of the respective binlog file. > + */ okay, but where do you read other binlog's FD? In the previous patch I reviewed you had case FORMAT_DESCRIPTION_EVENT: read FD and replace fdle. now you don't have it anymore. > + if (find_next_log(linfo, 1)) > + { > + sql_print_error("Error reading binlog files during recovery. " > + "Aborting."); > + goto err2; > + } > } > > +#ifdef HAVE_REPLICATION > + int rc= ctx.next_binlog_or_round(round, last_log_name, > + binlog_checkpoint_name, linfo, this); > + if (rc == -1) > + goto err2; > + else if (rc == 1) > + break; // all rounds done > +#else > if (!strcmp(linfo->log_file_name, last_log_name)) > break; // No more files to do > + round++; > +#endif > + > if ((file= open_binlog(&log, linfo->log_file_name, &errmsg)) < 0) > { > sql_print_error("%s", errmsg); > goto err2; > } > - /* > - We do not need to read the Format_description_log_event of other binlog > - files. It is not possible for a binlog checkpoint to span multiple > - binlog files written by different versions of the server. So we can use > - the first one read for reading from all binlog files. > - */ > - if (find_next_log(linfo, 1)) > - { > - sql_print_error("Error reading binlog files during recovery. Aborting."); > - goto err2; > - } > fdle->reset_crypto(); > - } > + } // end of for > > if (do_xa) > { > - if (ha_recover(&xids)) > - goto err2; > - > + if (binlog_checkpoint_found) > + { > +#ifndef HAVE_REPLICATION > + if (ha_recover_complete(&xids)) so, ha_recover_complete() is for no-semisync no-replication case? basically it should be the old behavior, exactly as before? why do you need ha_recover_complete() then if it didn't exist before? > +#else > + if (ctx.complete(this, xids)) > +#endif > + goto err2; > + } > free_root(&mem_root, MYF(0)); > my_hash_free(&xids); > } Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 57c19902326: MDEV-20017 Implement TO_CHAR() Oracle compatible function
by Sergei Golubchik 22 Apr '21

22 Apr '21

Hi, Monty! On Apr 13, Michael Widenius wrote: > commit 57c19902326 > Author: Michael Widenius <michael.widenius(a)gmail.com> > Date: Sun Jan 24 23:56:43 2021 +0200 > > diff --git a/sql/share/errmsg-utf8.txt b/sql/share/errmsg-utf8.txt > index aecb00563f7..b23522ac830 100644 > --- a/sql/share/errmsg-utf8.txt > +++ b/sql/share/errmsg-utf8.txt > @@ -7975,3 +7975,5 @@ ER_PK_INDEX_CANT_BE_IGNORED > eng "A primary key cannot be marked as IGNORE" > ER_FUNCTION_CANNOT_BE_USED_IN_CLAUSE > eng "Function '%s' cannot be used in the %s clause" > +ER_ORACLE_COMPAT_FUNCTION_ERROR > + eng "Oracle compatibility function error: %s" Why? We normally just say "invalid argument" or something, in no other case we say "oracle compatibility function" or "sybase compatibility function" or "odbc compatibility function". > diff --git a/sql/sql_string.cc b/sql/sql_string.cc > index 95a57017c53..9c57bb22085 100644 > --- a/sql/sql_string.cc > +++ b/sql/sql_string.cc > @@ -1275,3 +1275,15 @@ void Binary_string::shrink(size_t arg_length) > } > } > } > + > +bool Binary_string::strfill(char fill, size_t len) > +{ > + if (len) > + { > + if (alloc(length() + len)) > + return 1; > + memset(Ptr + str_length, fill, len); > + str_length+= (uint32) len; > + } > + return 0; > +} There's Binary_string::fill() already. better use it or, at the very least, declare bool strfill(char fill, size_t len) { return fill(str_length + len, fill); } in sql_string.h. And this swapped order of arguments is confusing, please fix it too. In fact, I think it's confusing to have both: fill(max_len, fill_char) strfill(fill_char, fill_length) if you want to keep both, it'd be better to rename them to something that somehow reflects the difference, for example: strfill -> append_many or append_repeated but really, I personally would just delete strfill. > diff --git a/sql/item_timefunc.h b/sql/item_timefunc.h > index af266956b05..9b78d6c159e 100644 > --- a/sql/item_timefunc.h > +++ b/sql/item_timefunc.h > @@ -978,6 +978,57 @@ class Item_func_time_format: public Item_func_date_format > }; > > > +/* the max length of datetime format models string in Oracle is 144 */ > +#define MAX_DATETIME_FORMAT_MODEL_LEN 144 > + > +class Item_func_tochar :public Item_str_func > +{ > + const MY_LOCALE *locale; > + THD *thd; > + String warning_message; > + bool fixed_length; > + > + /* > + When datetime format models is parsed, use uint16 integers to > + represent the format models and store in fmt_array. > + */ > + uint16 fmt_array[MAX_DATETIME_FORMAT_MODEL_LEN+1]; > + > + bool check_arguments() const override > + { > + return check_argument_types_can_return_text(1, arg_count); > + } > + > +public: > + Item_func_tochar(THD *thd, Item *a, Item *b): > + Item_str_func(thd, a, b), locale(0) > + { > + /* NOTE: max length of warning message is 64 */ > + warning_message.alloc(64); > + warning_message.length(0); > + } As far as I understand, this warning_message was introduced to issue the same error for every row, even if the format string is const_item and is parsed only once, in fix_fields. I don't think it's worth the trouble. Two simpler approaches are: * if the format string is invalid - parse it every time even if const, or * if the const format string is invalid - issue the error only once. so, please, remove warning_message and just use push_warning or my_error where the error is discovered. > + ~Item_func_tochar() { warning_message.free(); } > + String *val_str(String *str) override; > + LEX_CSTRING func_name_cstring() const override > + { > + static LEX_CSTRING name= {STRING_WITH_LEN("to_char") }; > + return name; > + } > + bool fix_length_and_dec() override; > + bool parse_format_string(const String *format, uint *fmt_len); > + > + bool check_vcol_func_processor(void *arg) override > + { > + if (arg_count > 2) > + return false; > + return mark_unsupported_function(func_name(), "()", arg, VCOL_SESSION_FUNC); > + } > + > + Item *get_copy(THD *thd) override > + { return get_item_copy<Item_func_tochar>(thd, this); } > +}; > + > + > class Item_func_from_unixtime :public Item_datetimefunc > { > bool check_arguments() const override > diff --git a/sql/item_timefunc.cc b/sql/item_timefunc.cc > index 04d913b0fca..44d2ec7912d 100644 > --- a/sql/item_timefunc.cc > +++ b/sql/item_timefunc.cc > @@ -1914,6 +1913,805 @@ String *Item_func_date_format::val_str(String *str) > return 0; > } > > +/* > + Oracle has many formatting models, we list all but only part of them > + are implemented, because some models depend on oracle functions > + which mariadb is not supported. > + > + Models for datetime, used by TO_CHAR/TO_DATE. Normal format characters are > + stored as short integer < 256, while format characters are stored as a > + integer > 256 > +*/ > + > +#define FMT_BASE 128 128? or 256? > +#define FMT_AD FMT_BASE+1 > +#define FMT_AD_DOT FMT_BASE+2 > +#define FMT_AM FMT_BASE+3 > +#define FMT_AM_DOT FMT_BASE+4 > +#define FMT_BC FMT_BASE+5 > +#define FMT_BC_DOT FMT_BASE+6 > +#define FMT_CC FMT_BASE+7 > +#define FMT_SCC FMT_BASE+8 > +#define FMT_D FMT_BASE+9 > +#define FMT_DAY FMT_BASE+10 > +#define FMT_DD FMT_BASE+11 > +#define FMT_DDD FMT_BASE+12 > +#define FMT_DL FMT_BASE+13 > +#define FMT_DS FMT_BASE+14 > +#define FMT_DY FMT_BASE+15 > +#define FMT_E FMT_BASE+16 > +#define FMT_EE FMT_BASE+17 > +#define FMT_FF FMT_BASE+18 > +#define FMT_FM FMT_BASE+19 > +#define FMT_FX FMT_BASE+20 > +#define FMT_HH FMT_BASE+21 > +#define FMT_HH12 FMT_BASE+22 > +#define FMT_HH24 FMT_BASE+23 > +#define FMT_IW FMT_BASE+24 > +#define FMT_I FMT_BASE+25 > +#define FMT_IY FMT_BASE+26 > +#define FMT_IYY FMT_BASE+27 > +#define FMT_IYYY FMT_BASE+28 > +#define FMT_J FMT_BASE+29 > +#define FMT_MI FMT_BASE+30 > +#define FMT_MM FMT_BASE+31 > +#define FMT_MON FMT_BASE+32 > +#define FMT_MONTH FMT_BASE+33 > +#define FMT_PM FMT_BASE+34 > +#define FMT_PM_DOT FMT_BASE+35 > +#define FMT_RM FMT_BASE+37 > +#define FMT_RR FMT_BASE+38 > +#define FMT_RRRR FMT_BASE+39 > +#define FMT_SS FMT_BASE+40 > +#define FMT_SSSSSS FMT_BASE+41 > +#define FMT_TS FMT_BASE+42 > +#define FMT_TZD FMT_BASE+43 > +#define FMT_TZH FMT_BASE+44 > +#define FMT_TZM FMT_BASE+45 > +#define FMT_TZR FMT_BASE+46 > +#define FMT_W FMT_BASE+47 > +#define FMT_WW FMT_BASE+48 > +#define FMT_X FMT_BASE+49 > +#define FMT_Y FMT_BASE+50 > +#define FMT_YY FMT_BASE+51 > +#define FMT_YYY FMT_BASE+52 > +#define FMT_YYYY FMT_BASE+53 > +#define FMT_YYYY_COMMA FMT_BASE+54 > +#define FMT_YEAR FMT_BASE+55 > +#define FMT_SYYYY FMT_BASE+56 > +#define FMT_SYEAR FMT_BASE+57 Not enum? Not even safe (with parentheses) #define? enum would be ideal here but at the very least make these defines safe. > + > + > +/** > + Modify the quotation flag and check whether the subsequent process is skipped Could you reword it please? > + > + @param cftm Character or FMT... format descriptor > + @param quotation_flag Points to 'true' if we are inside a quoted string > + > + @return true If we are inside a quoted string or if we found a '"' character > + @return false Otherwise > +*/ > + > +static inline bool check_quotation(uint16 cfmt, bool *quotation_flag) > +{ > + if (cfmt == '"') > + { > + *quotation_flag= !*quotation_flag; > + return true; > + } > + return *quotation_flag; > +} > + > +#define INVALID_CHARACTER(x) (((x) >= 'A' && (x) <= 'Z') ||((x) >= '0' && (x) <= '9') || (x) >= 127 || ((x) < 32)) why not to make this static inline too? side-effect safe, no risk of double evaluation of x. > + > + > +/** > + Special characters are directly output in the result > + > + @return 0 If found not acceptable character > + @return # Number of copied characters > +*/ > + > +static uint parse_special(char cfmt, const char *ptr, const char *end, > + uint16 *array) > +{ > + int offset= 0; > + char tmp1; > + > + /* Non-printable character and Multibyte encoded characters */ > + if (INVALID_CHARACTER(cfmt)) > + return 0; > + > + /* > + * '&' with text is used for variable input, but '&' with other > + * special charaters like '|'. '*' is used as separator > + */ > + if (cfmt == '&' && ptr + 1 < end) > + { > + tmp1= my_toupper(system_charset_info, *(ptr+1)); > + if (tmp1 >= 'A' && tmp1 <= 'Z') > + return 0; > + } > + > + do { > + /* > + Continuously store the special characters in fmt_array until non-special > + characters appear > + */ > + *array++= (uint16) (uchar) *ptr++; > + offset++; > + if (ptr == end) > + break; > + tmp1= my_toupper(system_charset_info, *ptr); > + } while (!INVALID_CHARACTER(tmp1) && tmp1 != '"'); > + return offset; > +} > + > + > +/** > + Parse the format string, convert it to an compact array and calculate the > + length of output string > + > + @param format Format string > + @param fmt_len Function will store max length of formated date string here > + > + @return 0 ok. fmt_len is updated > + @return 1 error. In this case 'warning_string' is set to error message > +*/ > + > +bool Item_func_tochar::parse_format_string(const String *format, uint *fmt_len) > +{ > + const char *ptr, *end; > + uint16 *tmp_fmt= fmt_array; > + uint tmp_len= 0; > + int offset= 0; > + bool quotation_flag= false; > + > + ptr= format->ptr(); > + end= ptr + format->length(); > + > + if (format->length() > MAX_DATETIME_FORMAT_MODEL_LEN) > + { > + warning_message.append(STRING_WITH_LEN("datetime format string is too " > + "long")); > + return 1; > + } > + > + for (; ptr < end; ptr++, tmp_fmt++) > + { > + uint ulen; > + char cfmt, next_char; > + > + cfmt= my_toupper(system_charset_info, *ptr); > + > + /* > + Oracle datetime format support text in double quotation marks like > + 'YYYY"abc"MM"xyz"DD', When this happens, store the text and quotation > + marks, and use the text as a separator in make_date_time_oracle. > + > + NOTE: the quotation mark is not print in return value. for example: > + select TO_CHAR(sysdate, 'YYYY"abc"MM"xyzDD"') will return 2021abc01xyz11 > + */ > + if (check_quotation(cfmt, &quotation_flag)) > + { > + *tmp_fmt= *ptr; > + tmp_len+= 1; > + continue; > + } > + > + switch (cfmt) { > + case 'A': // AD/A.D./AM/A.M. > + if (ptr+1 >= end) > + goto error; > + next_char= my_toupper(system_charset_info, *(ptr+1)); > + if (next_char == 'D') > + { > + *tmp_fmt= FMT_AD; > + ptr+= 1; > + tmp_len+= 2; > + } > + else if (next_char == 'M') > + { > + *tmp_fmt= FMT_AM; > + ptr+= 1; > + tmp_len+= 2; > + } > + else if (next_char == '.' && ptr+3 < end && *(ptr+3) == '.') > + { > + if (my_toupper(system_charset_info, *(ptr+2)) == 'D') > + { > + *tmp_fmt= FMT_AD_DOT; > + ptr+= 3; > + tmp_len+= 4; > + } > + else if (my_toupper(system_charset_info, *(ptr+2)) == 'M') > + { > + *tmp_fmt= FMT_AM_DOT; > + ptr+= 3; > + tmp_len+= 4; > + } > + else > + goto error; > + } > + else > + goto error; > + break; > + case 'B': // BC and B.C > + if (ptr+1 >= end) > + goto error; > + next_char= my_toupper(system_charset_info, *(ptr+1)); > + if (next_char == 'C') > + { > + *tmp_fmt= FMT_BC; > + ptr+= 1; > + tmp_len+= 2; > + } > + else if (next_char == '.' && ptr+3 < end && > + my_toupper(system_charset_info, *(ptr+2)) == 'C' && > + *(ptr+3) == '.') > + { > + *tmp_fmt= FMT_BC_DOT; > + ptr+= 3; > + tmp_len+= 4; > + } > + else > + goto error; > + break; > + case 'P': // PM or P.M. > + next_char= my_toupper(system_charset_info, *(ptr+1)); > + if (next_char == 'M') > + { > + *tmp_fmt= FMT_PM; > + ptr+= 1; > + tmp_len+= 2; > + } > + else if (next_char == '.' && > + my_toupper(system_charset_info, *(ptr+2)) == 'M' && > + my_toupper(system_charset_info, *(ptr+3)) == '.') > + { > + *tmp_fmt= FMT_PM_DOT; > + ptr+= 3; > + tmp_len+= 4; > + } > + else > + goto error; > + break; > + case 'Y': // Y, YY, YYY o YYYYY > + if (ptr + 1 == end || my_toupper(system_charset_info, *(ptr+1)) != 'Y') > + { > + *tmp_fmt= FMT_Y; > + tmp_len+= 1; > + break; > + } > + if (ptr + 2 == end || > + my_toupper(system_charset_info, *(ptr+2)) != 'Y') /* YY */ > + { > + *tmp_fmt= FMT_YY; > + ulen= 2; > + } > + else > + { > + if (ptr + 3 < end && my_toupper(system_charset_info, *(ptr+3)) == 'Y') > + { > + *tmp_fmt= FMT_YYYY; > + ulen= 4; > + } > + else > + { > + *tmp_fmt= FMT_YYY; > + ulen= 3; > + } > + } > + ptr+= ulen-1; > + tmp_len+= ulen; > + break; > + > + case 'R': // RR or RRRR > + if (ptr + 1 == end || my_toupper(system_charset_info, *(ptr+1)) != 'R') > + goto error; > + > + if (ptr + 2 == end || my_toupper(system_charset_info, *(ptr+2)) != 'R') > + { > + *tmp_fmt= FMT_RR; > + ulen= 2; > + } > + else > + { > + if (ptr + 3 >= end || my_toupper(system_charset_info, *(ptr+3)) != 'R') > + goto error; > + *tmp_fmt= FMT_RRRR; > + ulen= 4; > + } > + ptr+= ulen-1; > + tmp_len+= ulen; > + break; > + case 'M': > + { > + char tmp1; > + if (ptr + 1 >= end) > + goto error; > + > + tmp1= my_toupper(system_charset_info, *(ptr+1)); > + if (tmp1 == 'M') > + { > + *tmp_fmt= FMT_MM; > + tmp_len+= 2; > + ptr+= 1; > + } > + else if (tmp1 == 'I') > + { > + *tmp_fmt= FMT_MI; > + tmp_len+= 2; > + ptr+= 1; > + } > + else if (tmp1 == 'O') > + { > + if (ptr + 2 >= end) > + goto error; > + char tmp2= my_toupper(system_charset_info, *(ptr+2)); > + if (tmp2 != 'N') > + goto error; > + > + if (ptr + 4 >= end || > + my_toupper(system_charset_info, *(ptr+3)) != 'T' || > + my_toupper(system_charset_info, *(ptr+4)) != 'H') > + { > + *tmp_fmt= FMT_MON; > + tmp_len+= 3; > + ptr+= 2; > + } > + else > + { > + *tmp_fmt= FMT_MONTH; > + tmp_len+= (locale->max_month_name_length * > + my_charset_utf8mb3_bin.mbmaxlen); > + ptr+= 4; > + } > + } > + else > + goto error; > + } > + break; > + case 'D': // DD, DY, or DAY > + { > + if (ptr + 1 >= end) > + goto error; > + char tmp1= my_toupper(system_charset_info, *(ptr+1)); > + > + if (tmp1 == 'D') > + { > + *tmp_fmt= FMT_DD; > + tmp_len+= 2; > + } > + else if (tmp1 == 'Y') > + { > + *tmp_fmt= FMT_DY; > + tmp_len+= 3; > + } > + else if (tmp1 == 'A') // DAY > + { > + if (ptr + 2 == end || my_toupper(system_charset_info, *(ptr+2)) != 'Y') > + goto error; > + *tmp_fmt= FMT_DAY; > + tmp_len+= locale->max_day_name_length * my_charset_utf8mb3_bin.mbmaxlen; > + ptr+= 1; > + } > + else > + goto error; > + ptr+= 1; > + } > + break; > + case 'H': // HH, HH12 or HH23 > + { > + char tmp1, tmp2, tmp3; > + if (ptr + 1 >= end) > + goto error; > + tmp1= my_toupper(system_charset_info, *(ptr+1)); > + > + if (tmp1 != 'H') > + goto error; > + > + if (ptr+3 >= end) > + { > + *tmp_fmt= FMT_HH; > + ptr+= 1; > + } > + else > + { > + tmp2= *(ptr+2); > + tmp3= *(ptr+3); > + > + if (tmp2 == '1' && tmp3 == '2') > + { > + *tmp_fmt= FMT_HH12; > + ptr+= 3; > + } > + else if (tmp2 == '2' && tmp3 == '4') > + { > + *tmp_fmt= FMT_HH24; > + ptr+= 3; > + } > + else > + { > + *tmp_fmt= FMT_HH; > + ptr+= 1; > + } > + } > + tmp_len+= 2; > + break; > + } > + case 'S': // SS > + if (ptr + 1 == end || my_toupper(system_charset_info, *(ptr+1)) != 'S') > + goto error; > + > + *tmp_fmt= FMT_SS; > + tmp_len+= 2; > + ptr+= 1; > + break; > + case '|': > + /* > + If only one '|' just ignore it, else append others, for example: > + TO_CHAR('2000-11-05', 'YYYY|MM||||DD') --> 200011|||05 > + */ > + if (ptr + 1 == end || *(ptr+1) != '|') > + { > + tmp_fmt--; > + break; > + } > + ptr++; // Skip first '|' > + do > + { > + *tmp_fmt++= *ptr++; > + tmp_len++; > + } while ((ptr < end) && *ptr == '|'); > + ptr--; // Fix ptr for above for loop > + tmp_fmt--; > + break; > + > + default: > + offset= parse_special(cfmt, ptr, end, tmp_fmt); > + if (!offset) > + goto error; > + /* ptr++ is in the for loop, so we must move ptr to offset-1 */ > + ptr+= (offset-1); > + tmp_fmt+= (offset-1); > + tmp_len+= offset; > + break; > + } > + } > + *fmt_len= tmp_len; > + *tmp_fmt= 0; > + return 0; > + > +error: > + warning_message.append(STRING_WITH_LEN("date format not recognized at ")); > + warning_message.append(ptr, MY_MIN(8, end- ptr)); > + return 1; > +} > + > + > +static inline bool append_val(int val, int size, String *str) > +{ > + ulong len= 0; > + char intbuff[15]; > + > + len= (ulong) (int10_to_str(val, intbuff, 10) - intbuff); > + return str->append_with_prefill(intbuff, len, size, '0'); > +} > + > + > +static bool make_date_time_oracle(const uint16 *fmt_array, > + const MYSQL_TIME *l_time, > + const MY_LOCALE *locale, > + String *str) > +{ > + bool quotation_flag= false; > + const uint16 *ptr= fmt_array; > + uint hours_i; > + uint weekday; > + > + str->length(0); > + > + while (*ptr) > + { > + if (check_quotation(*ptr, &quotation_flag)) > + { > + /* don't display '"' in the result, so if it is '"', skip it */ > + if (*ptr != '"') > + { > + DBUG_ASSERT(*ptr <= 255); > + str->append((char) *ptr); > + } > + ptr++; > + continue; > + } > + > + switch (*ptr) { > + > + case FMT_AM: > + case FMT_PM: > + if (l_time->hour > 11) > + str->append("PM", 2); > + else > + str->append("AM", 2); > + break; > + > + case FMT_AM_DOT: > + case FMT_PM_DOT: > + if (l_time->hour > 11) > + str->append(STRING_WITH_LEN("P.M.")); > + else > + str->append(STRING_WITH_LEN("A.M.")); > + break; > + > + case FMT_AD: > + case FMT_BC: > + if (l_time->year > 0) > + str->append(STRING_WITH_LEN("AD")); > + else > + str->append(STRING_WITH_LEN("BC")); > + break; > + > + case FMT_AD_DOT: > + case FMT_BC_DOT: > + if (l_time->year > 0) > + str->append(STRING_WITH_LEN("A.D.")); > + else > + str->append(STRING_WITH_LEN("B.C.")); > + break; > + > + case FMT_Y: > + if (append_val(l_time->year%10, 1, str)) > + goto err_exit; > + break; > + > + case FMT_YY: > + case FMT_RR: > + if (append_val(l_time->year%100, 2, str)) > + goto err_exit; > + break; > + > + case FMT_YYY: > + if (append_val(l_time->year%1000, 3, str)) > + goto err_exit; > + break; > + > + case FMT_YYYY: > + case FMT_RRRR: > + if (append_val(l_time->year, 4, str)) > + goto err_exit; > + break; > + > + case FMT_MM: > + if (append_val(l_time->month, 2, str)) > + goto err_exit; > + break; > + > + case FMT_MON: > + { > + if (l_time->month == 0) > + { > + str->append("00", 2); > + } > + else > + { > + const char *month_name= (locale->ab_month_names-> > + type_names[l_time->month-1]); > + size_t m_len= strlen(month_name); > + str->append(month_name, m_len, system_charset_info); > + } > + } > + break; > + > + case FMT_MONTH: > + { > + if (l_time->month == 0) > + { > + str->append("00", 2); > + } > + else > + { > + const char *month_name= (locale->month_names-> > + type_names[l_time->month-1]); > + size_t month_byte_len= strlen(month_name); > + size_t month_char_len; > + str->append(month_name, month_byte_len, system_charset_info); > + month_char_len= my_numchars_mb(&my_charset_utf8mb3_general_ci, > + month_name, month_name + > + month_byte_len); > + if (str->strfill(' ', locale->max_month_name_length - month_char_len)) > + goto err_exit; > + } > + } > + break; > + > + case FMT_DD: > + if (append_val(l_time->day, 2, str)) > + goto err_exit; > + break; > + > + case FMT_DY: > + { > + if (l_time->day == 0) > + str->append("00", 2); > + else > + { > + weekday= calc_weekday(calc_daynr(l_time->year,l_time->month, > + l_time->day), 0); > + const char *day_name= locale->ab_day_names->type_names[weekday]; > + str->append(day_name, strlen(day_name), system_charset_info); > + } > + } > + break; > + > + case FMT_DAY: > + { > + if (l_time->day == 0) > + str->append("00", 2, system_charset_info); > + else > + { > + const char *day_name; > + size_t day_byte_len, day_char_len; > + weekday=calc_weekday(calc_daynr(l_time->year,l_time->month, > + l_time->day), 0); > + day_name= locale->day_names->type_names[weekday]; > + day_byte_len= strlen(day_name); > + str->append(day_name, day_byte_len, system_charset_info); > + day_char_len= my_numchars_mb(&my_charset_utf8mb3_general_ci, > + day_name, day_name + day_byte_len); > + if (str->strfill(' ', locale->max_day_name_length - day_char_len)) > + goto err_exit; > + } > + } > + break; > + > + case FMT_HH12: > + case FMT_HH: > + hours_i= (l_time->hour%24 + 11)%12+1; > + if (append_val(hours_i, 2, str)) > + goto err_exit; > + break; > + > + case FMT_HH24: > + if (append_val(l_time->hour, 2, str)) > + goto err_exit; > + break; > + > + case FMT_MI: > + if (append_val(l_time->minute, 2, str)) > + goto err_exit; > + break; > + > + case FMT_SS: > + if (append_val(l_time->second, 2, str)) > + goto err_exit; > + break; > + > + default: > + str->append((char) *ptr); > + } > + > + ptr++; > + }; > + return false; > + > +err_exit: > + return true; > +} > + > + > +bool Item_func_tochar::fix_length_and_dec() > +{ > + thd= current_thd; > + CHARSET_INFO *cs= thd->variables.collation_connection; > + Item *arg1= args[1]->this_item(); > + my_repertoire_t repertoire= arg1->collation.repertoire; > + StringBuffer<STRING_BUFFER_USUAL_SIZE> buffer; > + String *str; > + > + locale= thd->variables.lc_time_names; > + if (!thd->variables.lc_time_names->is_ascii) > + repertoire|= MY_REPERTOIRE_EXTENDED; > + collation.set(cs, arg1->collation.derivation, repertoire); > + > + /* first argument must be datetime or string */ > + enum_field_types arg0_mysql_type= args[0]->field_type(); > + > + max_length= 0; > + switch (arg0_mysql_type) { > + case MYSQL_TYPE_TIME: > + case MYSQL_TYPE_DATE: > + case MYSQL_TYPE_DATETIME: > + case MYSQL_TYPE_TIMESTAMP: > + case MYSQL_TYPE_VARCHAR: > + case MYSQL_TYPE_STRING: > + break; > + default: > + { > + my_printf_error(ER_ORACLE_COMPAT_FUNCTION_ERROR, > + ER(ER_ORACLE_COMPAT_FUNCTION_ERROR), > + MYF(0), > + "data type of first argument must be type " > + "date/datetime/time or string"); that's not how MariaDB works, it converts types. In particular, using an integer 20200101 in the date context is perfectly ok. > + return TRUE; > + } > + } > + if (args[1]->basic_const_item() && (str= args[1]->val_str(&buffer))) > + { > + uint ulen; > + fixed_length= 1; > + if (parse_format_string(str, &ulen)) > + { > + my_printf_error(ER_ORACLE_COMPAT_FUNCTION_ERROR, > + ER(ER_ORACLE_COMPAT_FUNCTION_ERROR), > + MYF(0), > + warning_message.c_ptr()); > + return TRUE; > + } > + max_length= (uint32) (ulen * collation.collation->mbmaxlen); > + } > + else > + { > + fixed_length= 0; > + max_length= (uint32) MY_MIN(arg1->max_length * 10 * > + collation.collation->mbmaxlen, > + MAX_BLOB_WIDTH); > + } > + set_maybe_null(); > + return FALSE; > +} > + > + > +String *Item_func_tochar::val_str(String* str) > + { > + StringBuffer<64> format_buffer; > + String *format; > + MYSQL_TIME l_time; > + const MY_LOCALE *lc= locale; > + date_conv_mode_t mode= TIME_CONV_NONE; > + size_t max_result_length= max_length; > + > + if (warning_message.length()) > + goto null_date; > + > + if ((null_value= args[0]->get_date(thd, &l_time, > + Temporal::Options(mode, thd)))) > + return 0; > + > + if (!fixed_length) > + { > + uint ulen; > + if (!(format= args[1]->val_str(&format_buffer)) || !format->length() || > + parse_format_string(format, &ulen)) > + goto null_date; > + max_result_length= ((size_t) ulen) * collation.collation->mbmaxlen; > + } > + > + if (str->alloc(max_result_length)) > + goto null_date; > + > + /* Create the result string */ > + str->set_charset(collation.collation); > + if (!make_date_time_oracle(fmt_array, &l_time, lc, str)) > + return str; > + > +null_date: > + > + if (warning_message.length()) > + { > + push_warning_printf(thd, > + Sql_condition::WARN_LEVEL_WARN, > + ER_ORACLE_COMPAT_FUNCTION_ERROR, > + ER_THD(thd, ER_ORACLE_COMPAT_FUNCTION_ERROR), > + warning_message.c_ptr()); > + if (!fixed_length) > + warning_message.length(0); > + } > + > + null_value= 1; > + return 0; > +} > + > > bool Item_func_from_unixtime::fix_length_and_dec() > { Regards, Sergei

2 1

Re: [Maria-developers] a1440737662: MDEV-20021 sql_mode="oracle" does not support MINUS set operator
by Sergei Golubchik 20 Apr '21

20 Apr '21

Hi, Monty! On Apr 13, Michael Widenius wrote: > revision-id: a1440737662 (mariadb-10.5.2-553-ga1440737662) > parent(s): 04a13e6ab8f > author: Michael Widenius <michael.widenius(a)gmail.com> > committer: Michael Widenius <michael.widenius(a)gmail.com> > timestamp: 2021-03-24 19:51:22 +0200 > message: > > MDEV-20021 sql_mode="oracle" does not support MINUS set operator > > MINUS is mapped to EXCEPT > > diff --git a/sql/sql_yacc.yy b/sql/sql_yacc.yy > index 57ba9df42c0..edd2f353dd0 100644 > --- a/sql/sql_yacc.yy > +++ b/sql/sql_yacc.yy > @@ -16037,6 +16038,7 @@ reserved_keyword_udt_not_param_type: > | MINUTE_MICROSECOND_SYM > | MINUTE_SECOND_SYM > | MIN_SYM > + | MINUS_ORACLE_SYM > | MODIFIES_SYM > | MOD_SYM > | NATURAL this is not good. MINUS should be in reserved_keyword_udt_not_param_type only in oracle mode, and otherwise it should be in keyword_sp_var_and_label (or not a keyword at all, but I don't think it's possible). Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] 04a13e6ab8f: MDEV-24285 support oracle build-in function: sys_guid
by Sergei Golubchik 20 Apr '21

20 Apr '21

Hi, Monty! On Apr 13, Michael Widenius wrote: > revision-id: 04a13e6ab8f (mariadb-10.5.2-552-g04a13e6ab8f) > parent(s): c76ac1e6de8 > author: Michael Widenius <michael.widenius(a)gmail.com> > committer: Michael Widenius <michael.widenius(a)gmail.com> > timestamp: 2021-03-24 19:42:30 +0200 > message: > > MDEV-24285 support oracle build-in function: sys_guid > > SYS_GUID() returns same as UUID(), but without any '-' I'd rather do it as a flag in Item_func_uuid. Less code, no need to copy-paste everything. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] c4de76aeff8: MDEV-17554 Auto-create new partition for system versioned tables with history partitioned by INTERVAL/LIMIT
by Sergei Golubchik 18 Apr '21

18 Apr '21

Hi, Aleksey! Note, it's a review of the `git diff 82e44d60d1e 8175ce8a5f1` so it's not only commit c4de76aeff8 On Apr 04, Aleksey Midenkov wrote: > revision-id: c4de76aeff8 (mariadb-10.5.2-540-gc4de76aeff8) > parent(s): 82e44d60d1e > author: Aleksey Midenkov <midenok(a)gmail.com> > committer: Aleksey Midenkov <midenok(a)gmail.com> > timestamp: 2021-03-31 21:17:55 +0300 > message: > > MDEV-17554 Auto-create new partition for system versioned tables with history partitioned by INTERVAL/LIMIT Overall it's very good! I have few minor questions/comments, see below. But it's almost done, I think. > diff --git a/mysql-test/suite/versioning/r/delete_history.result b/mysql-test/suite/versioning/r/delete_history.result > index cb865a835b3..90c9e4777bb 100644 > --- a/mysql-test/suite/versioning/r/delete_history.result > +++ b/mysql-test/suite/versioning/r/delete_history.result > @@ -154,3 +152,18 @@ select * from t1; > a > 1 > drop table t1; > +# > +# MDEV-17554 Auto-create new partition for system versioned tables with history partitioned by INTERVAL/LIMIT > +# > +# Don't auto-create new partition on DELETE HISTORY: > +create or replace table t (a int) with system versioning > +partition by system_time limit 1000 auto; > +delete history from t; > +show create table t; > +Table Create Table > +t CREATE TABLE `t` ( > + `a` int(11) DEFAULT NULL > +) ENGINE=DEFAULT_ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING > + PARTITION BY SYSTEM_TIME LIMIT 1000 AUTO > +PARTITIONS 2 Hmm, and if DELETE HISTORY would auto-create new partitions, what output would one expect here? I mean, how one can see whether the test result is correct or wrong? > +drop table t; > diff --git a/mysql-test/suite/versioning/t/partition.test b/mysql-test/suite/versioning/t/partition.test > index 445f5844630..57b80ca0b47 100644 > --- a/mysql-test/suite/versioning/t/partition.test > +++ b/mysql-test/suite/versioning/t/partition.test > @@ -1068,11 +1078,412 @@ alter table t1 add partition partitions 2; > --replace_result $default_engine DEFAULT_ENGINE > show create table t1; > alter table t1 add partition partitions 8; > +drop table t1; > + > +--echo # > +--echo # MDEV-17554 Auto-create new partition for system versioned tables with history partitioned by INTERVAL/LIMIT > +--echo # > +create or replace table t1 (x int) with system versioning > +partition by system_time limit 1 auto; > +--replace_result $default_engine DEFAULT_ENGINE > +show create table t1; > + > +--echo # INSERT, INSERT .. SELECT don't increment partitions it's not really "increment", better say "don't auto-create" > +insert into t1 values (1); > + > +create or replace table t2 (y int); > +insert into t2 values (2); > + > +insert into t1 select * from t2; > +insert into t2 select * from t1; why do you need a t2 table in this test? > +--replace_result $default_engine DEFAULT_ENGINE > +show create table t1; > + > +--echo # Too many partitions error > +set timestamp= unix_timestamp('2000-01-01 00:00:00'); > +create or replace table t1 (x int) with system versioning > +partition by system_time interval 1 hour auto; > +set timestamp= unix_timestamp('2001-01-01 00:01:00'); > +--error ER_TOO_MANY_PARTITIONS_ERROR > +update t1 set x= x + 1; > + > +--enable_info > +--echo # Increment from 3 to 5 > +create or replace table t1 (x int) with system versioning > +partition by system_time interval 3600 second > +starts '2000-01-01 00:00:00' auto partitions 3; > + > +insert into t1 values (1); > +--replace_result $default_engine DEFAULT_ENGINE > +show create table t1; > + > +set timestamp= unix_timestamp('2000-01-01 02:00:00'); > +update t1 set x= x + 1; > +--replace_result $default_engine DEFAULT_ENGINE > +show create table t1; > + > +set timestamp= unix_timestamp('2000-01-01 03:00:00'); > +update t1 set x= x + 2; > +--replace_result $default_engine DEFAULT_ENGINE > +show create table t1; > + > +--echo # Increment from 3 to 6, manual names, LOCK TABLES > +set timestamp= unix_timestamp('2000-01-01 00:00:00'); > +create or replace table t1 (x int) with system versioning > +partition by system_time interval 1 hour auto ( > + partition p1 history, > + partition p3 history, > + partition pn current); > + > +insert into t1 values (1); > --replace_result $default_engine DEFAULT_ENGINE > show create table t1; > > +set timestamp= unix_timestamp('2000-01-01 02:00:00'); > +update t1 set x= x + 3; > +--replace_result $default_engine DEFAULT_ENGINE > +show create table t1; > + > +set timestamp= unix_timestamp('2000-01-01 03:00:00'); > +update t1 set x= x + 4; > +--replace_result $default_engine DEFAULT_ENGINE > +show create table t1; > + > +set timestamp= unix_timestamp('2000-01-01 04:00:00'); > +lock tables t1 write; > +update t1 set x= x + 5; > +--replace_result $default_engine DEFAULT_ENGINE > +show create table t1; > +unlock tables; > +set timestamp= default; > + > +--echo # Couple of more LOCK TABLES cases > +create or replace table t1 (x int) with system versioning > +partition by system_time limit 1000 auto; > +lock tables t1 write; > +insert into t1 values (1); > +update t1 set x= x + 1; > +unlock tables; > +--replace_result $default_engine DEFAULT_ENGINE > +show create table t1; what does this test case demonstrate? > + > +--echo # Overflow prevention under LOCK TABLES > +create or replace table t1 (x int) > +with system versioning partition by system_time > +limit 10 auto; > + > +insert into t1 values (1), (2), (3), (4), (5), (6), (7), (8), (9); > +update t1 set x= x + 10; > + > +lock tables t1 write; > +update t1 set x= 1 where x = 11; > +update t1 set x= 2 where x = 12; > +update t1 set x= 3 where x = 13; > +unlock tables; > + > +select count(x) from t1 partition (p0); > +--replace_result $default_engine DEFAULT_ENGINE > +show create table t1; > +drop tables t1; > + > +--echo # Test VIEW, LOCK TABLES > +set timestamp= unix_timestamp('2000-01-01 00:00:00'); > +create or replace table t1 (x int) with system versioning > +partition by system_time interval 1 hour auto partitions 3; > +create or replace view v1 as select * from t1; > + > +insert into v1 values (1); why would a view matter in this test case? > +--replace_result $default_engine DEFAULT_ENGINE > +show create table t1; > + > +set timestamp= unix_timestamp('2000-01-01 01:00:00'); > +lock tables t1 write; > +update t1 set x= x + 3; > +--replace_result $default_engine DEFAULT_ENGINE > +show create table t1; > +unlock tables; > + > +drop view v1; > drop tables t1; > > +--echo # Multiple increments in single command > +set timestamp= unix_timestamp('2000-01-01 00:00:00'); > +create or replace table t1 (x int) with system versioning > +partition by system_time interval 1 hour auto partitions 3; > + > +create or replace table t2 (y int) with system versioning > +partition by system_time interval 1 hour auto; > + > +insert into t1 values (1); > +insert into t2 values (2); > + > +set timestamp= unix_timestamp('2000-01-01 01:00:00'); > +update t1, t2 set x= x + 1, y= y + 1; > +--replace_result $default_engine DEFAULT_ENGINE > +show create table t1; > +--replace_result $default_engine DEFAULT_ENGINE > +show create table t2; > + > +set timestamp= unix_timestamp('2000-01-01 02:00:00'); > +update t1, t2 set x= x + 1, y= y + 1; > +--replace_result $default_engine DEFAULT_ENGINE > +show create table t1; > +--replace_result $default_engine DEFAULT_ENGINE > +show create table t2; > + > +--echo # Here t2 is incremented too, but not updated > +set timestamp= unix_timestamp('2000-01-01 03:00:00'); > +update t1, t2 set t1.x= 0 where t1.x< t2.y; > +--replace_result $default_engine DEFAULT_ENGINE > +show create table t1; > +# Multiupdate_prelocking_strategy::handle_end() is processed after table open. > +# For PS it is possible to skip unneeded auto-creation because the above happens at > +# prepare stage and auto-creation is done at execute stage. > +--replace_result $default_engine DEFAULT_ENGINE 'PARTITIONS 4' 'PARTITIONS ok' 'PARTITIONS 5' 'PARTITIONS ok' eh... I don't think this is really "ok". As far as I remember, Multiupdate_prelocking_strategy knows what tables should be opened for writing and what for reading. Why would a new partition be created for t2? > +show create table t2; > + > +drop tables t1, t2; > + > +--echo # PS, SP, LOCK TABLES > +set timestamp= unix_timestamp('2000-01-01 00:00:00'); > +create or replace table t1 (x int) with system versioning > +partition by system_time interval 1 hour auto; > + > +insert into t1 values (1); > + > +set timestamp= unix_timestamp('2000-01-01 01:00:00'); > +execute immediate 'update t1 set x= x + 5'; > +--replace_result $default_engine DEFAULT_ENGINE > +show create table t1; > + > +prepare s from 'update t1 set x= x + 6'; > +set timestamp= unix_timestamp('2000-01-01 02:00:00'); > +execute s; execute s; > +--replace_result $default_engine DEFAULT_ENGINE > +show create table t1; > + > +set timestamp= unix_timestamp('2000-01-01 03:00:00'); > +lock tables t1 write; > +execute s; execute s; > +--replace_result $default_engine DEFAULT_ENGINE > +show create table t1; > +unlock tables; > +drop prepare s; > + > +create procedure sp() update t1 set x= x + 7; > +set timestamp= unix_timestamp('2000-01-01 04:00:00'); > +call sp; call sp; > +--replace_result $default_engine DEFAULT_ENGINE > +show create table t1; > +set timestamp= unix_timestamp('2000-01-01 05:00:00'); > +lock tables t1 write; > +call sp; call sp; > +--replace_result $default_engine DEFAULT_ENGINE > +show create table t1; > +unlock tables; > +drop procedure sp; > + > +set timestamp= unix_timestamp('2001-01-01 00:00:00'); > +create or replace table t1 (i int) with system versioning > +partition by system_time interval 1 day starts '2000-01-01 00:00:00'; > +insert into t1 values (0); > +set timestamp= unix_timestamp('2001-01-01 00:00:01'); > +prepare s from 'update t1 set i= i + 1'; > +execute s; > +set timestamp= unix_timestamp('2001-01-02 00:00:01'); > +execute s; > +drop prepare s; > + > +if (!$MTR_COMBINATION_HEAP) because of blobs? > +{ > +--echo # Complex table > +set timestamp= unix_timestamp('2000-01-01 00:00:00'); > +create or replace table t1 ( > + x int primary key auto_increment, ... > +--echo # Transaction > +create or replace table t1 (x int) with system versioning engine innodb > +partition by system_time interval 1 hour auto; > +start transaction; > +insert into t1 values (1); > +select * from t1; > + > +--connect con1, localhost, root > +set lock_wait_timeout= 1; > +set innodb_lock_wait_timeout= 1; > +--error ER_LOCK_WAIT_TIMEOUT > +update t1 set x= x + 111; > +select * from t1; what do you test here? (there is no show create table in the test) > + > +# cleanup > +--disconnect con1 > +--connection default > +drop table t1; > + > +set timestamp= unix_timestamp('2000-01-01 00:00:00'); > +create or replace table t1 (x int) with system versioning engine innodb > +partition by system_time interval 1 hour auto; > + > +insert into t1 values (1); > +set timestamp= unix_timestamp('2000-01-01 01:00:00'); > +start transaction; > +update t1 set x= 0; > +--connect con1, localhost, root > +select * from t1; if you add a show create table here, what would it show? > +--connection default > +commit; > +show create table t1; > + > +set timestamp= unix_timestamp('2000-01-01 02:00:00'); > +start transaction; > +update t1 set x= 1; > +--connection con1 > +select * from t1; > +--connection default > +rollback; > +show create table t1; > +--disconnect con1 > +--connection default > +drop table t1; > + > +--echo # > +--echo # MENT-724 Locking timeout caused by auto-creation affects original DML I'd better avoid MENT references here. Let's only mention Jira issues that users can actually see > +--echo # > +set timestamp= unix_timestamp('2000-01-01 00:00:00'); > +create or replace table t1 (x int primary key) with system versioning engine innodb > +partition by system_time interval 1 hour auto; > + > +insert into t1 values (1); > +start transaction; > +insert into t1 values (2); > + > +--connect con1, localhost, root > +set lock_wait_timeout= 1; > +set innodb_lock_wait_timeout= 1; > +set timestamp= unix_timestamp('2000-01-01 01:00:01'); > +update t1 set x= x + 122 where x = 1; isn't it a test that you already have above? with x = x + 111 > +--disconnect con1 > +--connection default > +select * from t1; > + > +# cleanup > +drop table t1; > +set timestamp= default; > + > --echo # > --echo # End of 10.5 tests > --echo # > diff --git a/mysql-test/suite/versioning/r/partition.result b/mysql-test/suite/versioning/r/partition.result > index 2a277b1c2ea..8369b40d98c 100644 > --- a/mysql-test/suite/versioning/r/partition.result > +++ b/mysql-test/suite/versioning/r/partition.result > @@ -1236,27 +1270,752 @@ t1 CREATE TABLE `t1` ( ... > +set timestamp= unix_timestamp('2000-01-01 03:00:00'); > +affected rows: 0 > +lock tables t1 write; > +affected rows: 0 > +execute s; > +affected rows: 1 > +info: Rows matched: 1 Changed: 1 Inserted: 1 Warnings: 0 > +execute s; > +affected rows: 1 > +info: Rows matched: 1 Changed: 1 Inserted: 1 Warnings: 0 > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `x` int(11) DEFAULT NULL > +) ENGINE=DEFAULT_ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING > + PARTITION BY SYSTEM_TIME INTERVAL 1 HOUR STARTS TIMESTAMP'2000-01-01 00:00:00' AUTO > +PARTITIONS 6 why did it add two partitions here? > +affected rows: 1 > +unlock tables; > +affected rows: 0 > +drop prepare s; > +affected rows: 0 > +create procedure sp() update t1 set x= x + 7; > +affected rows: 0 > +set timestamp= unix_timestamp('2000-01-01 04:00:00'); > +affected rows: 0 > +call sp; > +affected rows: 1 > +call sp; > +affected rows: 1 > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `x` int(11) DEFAULT NULL > +) ENGINE=DEFAULT_ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING > + PARTITION BY SYSTEM_TIME INTERVAL 1 HOUR STARTS TIMESTAMP'2000-01-01 00:00:00' AUTO > +PARTITIONS 6 > +affected rows: 1 > +set timestamp= unix_timestamp('2000-01-01 05:00:00'); > +affected rows: 0 > +lock tables t1 write; > +affected rows: 0 > +call sp; > +affected rows: 1 > +call sp; > +affected rows: 1 > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `x` int(11) DEFAULT NULL > +) ENGINE=DEFAULT_ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING > + PARTITION BY SYSTEM_TIME INTERVAL 1 HOUR STARTS TIMESTAMP'2000-01-01 00:00:00' AUTO > +PARTITIONS 8 and two here > +affected rows: 1 > +unlock tables; > +affected rows: 0 > +drop procedure sp; > +affected rows: 0 > +set timestamp= unix_timestamp('2001-01-01 00:00:00'); > +affected rows: 0 > +create or replace table t1 (i int) with system versioning > +partition by system_time interval 1 day starts '2000-01-01 00:00:00'; > +affected rows: 0 > +insert into t1 values (0); > +affected rows: 1 > +set timestamp= unix_timestamp('2001-01-01 00:00:01'); > +affected rows: 0 > +prepare s from 'update t1 set i= i + 1'; > +affected rows: 0 > +info: Statement prepared > +execute s; > +affected rows: 1 > +info: Rows matched: 1 Changed: 1 Inserted: 1 Warnings: 1 > +Warnings: > +Warning 4114 Versioned table `test`.`t1`: last HISTORY partition (`p0`) is out of INTERVAL, need more HISTORY partitions why? > +set timestamp= unix_timestamp('2001-01-02 00:00:01'); > +affected rows: 0 > +execute s; > +affected rows: 1 > +info: Rows matched: 1 Changed: 1 Inserted: 1 Warnings: 1 > +Warnings: > +Warning 4114 Versioned table `test`.`t1`: last HISTORY partition (`p0`) is out of INTERVAL, need more HISTORY partitions > +drop prepare s; > +affected rows: 0 > +# Complex table ... > diff --git a/mysql-test/suite/versioning/t/rpl.test b/mysql-test/suite/versioning/t/rpl.test > index b5be68feece..54258a8bdf1 100644 > --- a/mysql-test/suite/versioning/t/rpl.test > +++ b/mysql-test/suite/versioning/t/rpl.test > @@ -133,4 +133,44 @@ sync_slave_with_master; > connection master; > drop table t1; > > +--echo # > +--echo # MDEV-17554 Auto-create new partition for system versioned tables with history partitioned by INTERVAL/LIMIT > +--echo # > +set timestamp= unix_timestamp('2000-01-01 00:00:00'); > +create or replace table t1 (x int) with system versioning > +partition by system_time interval 1 hour auto; > +insert t1 values (); > +set timestamp= unix_timestamp('2000-01-01 01:00:00'); > +delete from t1; > +--replace_result InnoDB ENGINE MyISAM ENGINE MEMORY ENGINE > +show create table t1; > +--sync_slave_with_master > +--replace_result InnoDB ENGINE MyISAM ENGINE MEMORY ENGINE > +show create table t1; > +--connection master > +drop table t1; > +set timestamp= default; > + > +--echo # > +--echo # MENT-685 DML events for auto-partitioned tables are written into binary log twice same comment about MENT > +--echo # > +create table t1 (a int) with system versioning > +partition by system_time limit 1 auto; > + > +insert into t1 values (1); > +update t1 set a= a + 1; > +update t1 set a= a + 1; > +--replace_result InnoDB ENGINE MyISAM ENGINE MEMORY ENGINE > +show create table t1; > +select * from t1; > + > +--sync_slave_with_master > +--replace_result InnoDB ENGINE MyISAM ENGINE MEMORY ENGINE > +show create table t1; > + > +select * from t1; > +--connection master > +# cleanup > +drop table t1; may be show binlog events? you know, to verify that DML events aren't written into binary log twice > + > --source include/rpl_end.inc > diff --git a/sql/ha_partition.h b/sql/ha_partition.h > index 60a2d7f6762..7ade4419c22 100644 > --- a/sql/ha_partition.h > +++ b/sql/ha_partition.h > @@ -1610,7 +1610,7 @@ class ha_partition :public handler > for (; part_id < part_id_end; ++part_id) > { > handler *file= m_file[part_id]; > - DBUG_ASSERT(bitmap_is_set(&(m_part_info->read_partitions), part_id)); > + bitmap_set_bit(&(m_part_info->read_partitions), part_id); Where was it set before your patch? > file->info(HA_STATUS_VARIABLE | HA_STATUS_NO_LOCK | HA_STATUS_OPEN); > part_recs+= file->stats.records; > } > diff --git a/sql/handler.cc b/sql/handler.cc > index b312635c8ee..6bb6c279193 100644 > --- a/sql/handler.cc > +++ b/sql/handler.cc > @@ -1572,7 +1572,7 @@ int ha_commit_trans(THD *thd, bool all) > DBUG_ASSERT(thd->transaction->stmt.ha_list == NULL || > trans == &thd->transaction->stmt); > > - if (thd->in_sub_stmt) > + if (thd->in_sub_stmt & ~SUB_STMT_AUTO_HIST) > { where is this ha_commit_trans(thd, false) called from? mysql_alter_table that adds a new partition? > DBUG_ASSERT(0); > /* > diff --git a/sql/partition_info.cc b/sql/partition_info.cc > index 871411cf6c4..cf8dd140553 100644 > --- a/sql/partition_info.cc > +++ b/sql/partition_info.cc > @@ -816,10 +816,17 @@ bool partition_info::has_unique_name(partition_element *element) > vers_info->interval Limit by fixed time interval > vers_info->hist_part (out) Working history partition > */ > -void partition_info::vers_set_hist_part(THD *thd) > +uint partition_info::vers_set_hist_part(THD *thd, bool auto_hist) > { > + DBUG_ASSERT(!thd->lex->last_table() || > + !thd->lex->last_table()->vers_conditions.delete_history); is that right? Conceptually you need to test vers_conditions.delete_history for the table that owns this partition_info. Is it always last_table() ? I'd say it'd be more logical to do, like part_field_array[0]->table->pos_in_table_list > + > + uint create_count= 0; > + auto_hist= auto_hist && vers_info->auto_hist; > + > if (vers_info->limit) > { > + DBUG_ASSERT(!vers_info->interval.is_set()); > ha_partition *hp= (ha_partition*)(table->file); > partition_element *next= NULL; > List_iterator<partition_element> it(partitions); > @@ -862,9 +869,183 @@ void partition_info::vers_set_hist_part(THD *thd) > { > vers_info->hist_part= next; > if (next->range_value > thd->query_start()) > - return; > + { > + error= false; > + break; > + } > + } > + if (error) > + { > + if (auto_hist) > + { > + DBUG_ASSERT(thd->query_start() >= vers_info->hist_part->range_value); > + my_time_t diff= thd->query_start() - (my_time_t) vers_info->hist_part->range_value; > + if (diff > 0) > + { > + size_t delta= vers_info->interval.seconds(); > + create_count= (uint) (diff / delta + 1); > + if (diff % delta) > + create_count++; > + } > + else > + create_count= 1; > + } > + else > + { > + my_error(WARN_VERS_PART_FULL, MYF(ME_WARNING|ME_ERROR_LOG), after such an overflow a table becomes essentially corrupted, rows are in a wrong partition. So new history partitions cannot be added anymore without reorganizing the last partition. How is it handled now? (it's just a question, not part of the review, as it's not something you were changing in your patch) > + table->s->db.str, table->s->table_name.str, > + vers_info->hist_part->partition_name, "INTERVAL"); > + } > } > } > + > + /* > + When hist_part is almost full LOCK TABLES my overflow the partition as we s/my/may/ > + can't add new partitions under LOCK TABLES. Reserve one for this case. > + */ > + if (auto_hist && create_count == 0 && > + thd->lex->sql_command == SQLCOM_LOCK_TABLES && > + vers_info->hist_part->id + 1 == vers_info->now_part->id) > + create_count= 1; > + > + return create_count; > +} > + > + > +/** > + @brief Run fast_alter_partition_table() to add new history partitions > + for tables requiring them. > +*/ > +bool vers_add_auto_hist_parts(THD *thd, TABLE_LIST* tl, uint num_parts) > +{ > + bool result= true; > + HA_CREATE_INFO create_info; > + Alter_info alter_info; > + partition_info *save_part_info= thd->work_part_info; > + Query_tables_list save_query_tables; > + Reprepare_observer *save_reprepare_observer= thd->m_reprepare_observer; > + bool save_no_write_to_binlog= thd->lex->no_write_to_binlog; > + thd->m_reprepare_observer= NULL; > + thd->lex->reset_n_backup_query_tables_list(&save_query_tables); > + thd->in_sub_stmt|= SUB_STMT_AUTO_HIST; > + thd->lex->no_write_to_binlog= true; > + TABLE *table= tl->table; > + > + DBUG_ASSERT(!thd->is_error()); > + > + { > + DBUG_ASSERT(table->s->get_table_ref_type() == TABLE_REF_BASE_TABLE); > + DBUG_ASSERT(table->versioned()); > + DBUG_ASSERT(table->part_info); > + DBUG_ASSERT(table->part_info->vers_info); > + alter_info.reset(); > + alter_info.partition_flags= ALTER_PARTITION_ADD|ALTER_PARTITION_AUTO_HIST; > + create_info.init(); > + create_info.alter_info= &alter_info; > + Alter_table_ctx alter_ctx(thd, tl, 1, &table->s->db, &table->s->table_name); > + > + MDL_REQUEST_INIT(&tl->mdl_request, MDL_key::TABLE, tl->db.str, > + tl->table_name.str, MDL_SHARED_NO_WRITE, MDL_TRANSACTION); > + if (thd->mdl_context.acquire_lock(&tl->mdl_request, > + thd->variables.lock_wait_timeout)) > + goto exit; > + table->mdl_ticket= tl->mdl_request.ticket; > + > + create_info.db_type= table->s->db_type(); > + create_info.options|= HA_VERSIONED_TABLE; > + DBUG_ASSERT(create_info.db_type); > + > + create_info.vers_info.set_start(table->s->vers_start_field()->field_name); > + create_info.vers_info.set_end(table->s->vers_end_field()->field_name); > + > + partition_info *part_info= new partition_info(); > + if (unlikely(!part_info)) > + { > + my_error(ER_OUT_OF_RESOURCES, MYF(0)); > + goto exit; > + } > + part_info->use_default_num_partitions= false; > + part_info->use_default_num_subpartitions= false; > + part_info->num_parts= num_parts; > + part_info->num_subparts= table->part_info->num_subparts; > + part_info->subpart_type= table->part_info->subpart_type; > + if (unlikely(part_info->vers_init_info(thd))) > + { > + my_error(ER_OUT_OF_RESOURCES, MYF(0)); > + goto exit; > + } > + > + // NB: set_ok_status() requires DA_EMPTY > + thd->get_stmt_da()->reset_diagnostics_area(); would it not be DA_EMPTY without a reset? this is at the beginning of a statement, I'd expect it normally be DA_EMPTY here. What other DA states can you get here? > + > + thd->work_part_info= part_info; > + if (part_info->set_up_defaults_for_partitioning(thd, table->file, NULL, > + table->part_info->next_part_no(num_parts))) > + { > + push_warning(thd, Sql_condition::WARN_LEVEL_WARN, > + ER_VERS_HIST_PART_FAILED, > + "Auto-increment history partition: " > + "setting up defaults failed"); > + my_error(ER_VERS_HIST_PART_FAILED, MYF(ME_WARNING), > + tl->db.str, tl->table_name.str); > + goto exit; > + } > + bool partition_changed= false; > + bool fast_alter_partition= false; > + if (prep_alter_part_table(thd, table, &alter_info, &create_info, > + &partition_changed, &fast_alter_partition)) > + { > + push_warning(thd, Sql_condition::WARN_LEVEL_WARN, ER_VERS_HIST_PART_FAILED, > + "Auto-increment history partition: " > + "alter partitition prepare failed"); > + my_error(ER_VERS_HIST_PART_FAILED, MYF(ME_WARNING), > + tl->db.str, tl->table_name.str); > + goto exit; > + } > + if (!fast_alter_partition) > + { > + push_warning(thd, Sql_condition::WARN_LEVEL_WARN, ER_VERS_HIST_PART_FAILED, > + "Auto-increment history partition: " > + "fast alter partitition is not possible"); > + my_error(ER_VERS_HIST_PART_FAILED, MYF(ME_WARNING), > + tl->db.str, tl->table_name.str); > + goto exit; > + } > + DBUG_ASSERT(partition_changed); > + if (mysql_prepare_alter_table(thd, table, &create_info, &alter_info, > + &alter_ctx)) > + { > + push_warning(thd, Sql_condition::WARN_LEVEL_WARN, ER_VERS_HIST_PART_FAILED, > + "Auto-increment history partition: " > + "alter prepare failed"); > + my_error(ER_VERS_HIST_PART_FAILED, MYF(ME_WARNING), > + tl->db.str, tl->table_name.str); > + goto exit; > + } > + > + if (fast_alter_partition_table(thd, table, &alter_info, &create_info, > + tl, &table->s->db, &table->s->table_name)) > + { > + push_warning(thd, Sql_condition::WARN_LEVEL_WARN, ER_VERS_HIST_PART_FAILED, > + "Auto-increment history partition: " > + "alter partition table failed"); > + my_error(ER_VERS_HIST_PART_FAILED, MYF(ME_WARNING), > + tl->db.str, tl->table_name.str); > + goto exit; > + } > + } > + > + result= false; > + // NB: we have to return DA_EMPTY for new command may be DBUG_ASSERT(thd->get_stmt_da()->is_ok()); > + thd->get_stmt_da()->reset_diagnostics_area(); > + > +exit: > + thd->work_part_info= save_part_info; > + thd->m_reprepare_observer= save_reprepare_observer; > + thd->lex->restore_backup_query_tables_list(&save_query_tables); > + thd->in_sub_stmt&= ~SUB_STMT_AUTO_HIST; > + thd->lex->no_write_to_binlog= save_no_write_to_binlog; > + return result; > } > > > diff --git a/sql/partition_info.h b/sql/partition_info.h > index 0656238ec07..94093353d60 100644 > --- a/sql/partition_info.h > +++ b/sql/partition_info.h > @@ -72,9 +73,26 @@ struct Vers_part_info : public Sql_alloc > my_time_t start; > INTERVAL step; > enum interval_type type; > - bool is_set() { return type < INTERVAL_LAST; } > + bool is_set() const { return type < INTERVAL_LAST; } > + size_t seconds() const > + { > + if (step.second) > + return step.second; > + if (step.minute) > + return step.minute * 60; > + if (step.hour) > + return step.hour * 3600; > + if (step.day) > + return step.day * 3600 * 24; > + // comparison is used in rough estimates, it doesn't need to be calendar-correct > + if (step.month) > + return step.month * 3600 * 24 * 30; shouldn't it be `* 28` ? you need a most pessimistic estimate to make sure you never miss to create a partition when one is needed. You can sometimes create one when it's not needed yet, but it's less of a problem. > + DBUG_ASSERT(step.year); > + return step.year * 86400 * 30 * 365; that's one `* 30` too many :) > + } > } interval; > ulonglong limit; > + bool auto_hist; > partition_element *now_part; > partition_element *hist_part; > }; > diff --git a/sql/sql_base.cc b/sql/sql_base.cc > index 1a1186aca73..d0e255186da 100644 > --- a/sql/sql_base.cc > +++ b/sql/sql_base.cc > @@ -1625,6 +1625,52 @@ bool is_locked_view(THD *thd, TABLE_LIST *t) > } > > > +bool TABLE::vers_need_hist_part(const THD *thd, const TABLE_LIST *table_list) const > +{ > +#ifdef WITH_PARTITION_STORAGE_ENGINE > + if (part_info && part_info->part_type == VERSIONING_PARTITION && > + !table_list->vers_conditions.delete_history && > + !thd->stmt_arena->is_stmt_prepare() && > + table_list->lock_type >= TL_WRITE_ALLOW_WRITE && > + table_list->mdl_request.type >= MDL_SHARED_WRITE && > + table_list->mdl_request.type < MDL_EXCLUSIVE) > + { > + switch (thd->lex->sql_command) > + { SQLCOM_LOAD with DUP_REPLACE? > + case SQLCOM_INSERT: > + if (thd->lex->duplicates != DUP_UPDATE) > + break; > + /* fall-through: */ > + case SQLCOM_LOCK_TABLES: > + case SQLCOM_DELETE: > + case SQLCOM_UPDATE: > + case SQLCOM_REPLACE: > + case SQLCOM_REPLACE_SELECT: > + case SQLCOM_DELETE_MULTI: > + case SQLCOM_UPDATE_MULTI: > + return true; > + default:; > + break; > + } > + if (thd->rgi_slave && thd->rgi_slave->current_event && thd->lex->sql_command == SQLCOM_END) I wonder, would it be possible, instead of introducing rgi_slave->current_event, just make row events set thd->lex->sql_command appropriately? For example, currently row events increment thd->status_var.com_stat[] each event for its own SQLCOM_xxx, it won't be needed if they'll just set thd->lex->sql_command. > + { > + switch (thd->rgi_slave->current_event->get_type_code()) > + { > + case UPDATE_ROWS_EVENT: > + case UPDATE_ROWS_EVENT_V1: > + case DELETE_ROWS_EVENT: > + case DELETE_ROWS_EVENT_V1: > + return true; > + default:; > + break; > + } > + } > + } > +#endif > + return false; > +} > + > + > /** > Open a base table. > > @@ -1777,6 +1823,11 @@ bool open_table(THD *thd, TABLE_LIST *table_list, Open_table_context *ot_ctx) > DBUG_PRINT("info",("Using locked table")); > #ifdef WITH_PARTITION_STORAGE_ENGINE > part_names_error= set_partitions_as_used(table_list, table); > + if (table->vers_need_hist_part(thd, table_list)) > + { > + /* Rotation is still needed under LOCK TABLES */ a bit confusing comment, you left out stuff that are obvious to you but not to others. I'd suggest something like /* New partitions are not auto-created under LOCK TABLES (TODO: fix it) but rotation can still happen. */ > + table->part_info->vers_set_hist_part(thd, false); > + } > #endif > goto reset; > } > @@ -2032,6 +2083,20 @@ bool open_table(THD *thd, TABLE_LIST *table_list, Open_table_context *ot_ctx) > tc_add_table(thd, table); > } > > + if (!ot_ctx->vers_create_count && what does this condition mean? When is ot_ctx->vers_create_count > 0 ? > + table->vers_need_hist_part(thd, table_list)) > + { > + ot_ctx->vers_create_count= table->part_info->vers_set_hist_part(thd, true); > + if (ot_ctx->vers_create_count) > + { > + MYSQL_UNBIND_TABLE(table->file); > + tc_release_table(table); > + ot_ctx->request_backoff_action(Open_table_context::OT_ADD_HISTORY_PARTITION, > + table_list); > + DBUG_RETURN(TRUE); > + } > + } > + > if (!(flags & MYSQL_OPEN_HAS_MDL_LOCK) && > table->s->table_category < TABLE_CATEGORY_INFORMATION) > { > @@ -3172,10 +3239,18 @@ Open_table_context::recover_from_failed_open() > break; > case OT_DISCOVER: > case OT_REPAIR: > - if ((result= lock_table_names(m_thd, m_thd->lex->create_info, > - m_failed_table, NULL, > - get_timeout(), 0))) > + case OT_ADD_HISTORY_PARTITION: > + result= lock_table_names(m_thd, m_thd->lex->create_info, m_failed_table, > + NULL, get_timeout(), 0); > + if (result) > + { > + if (m_action == OT_ADD_HISTORY_PARTITION) > + { > + m_thd->clear_error(); why? > + result= false; > + } > break; > + } > > tdc_remove_table(m_thd, m_failed_table->db.str, > m_failed_table->table_name.str); > diff --git a/sql/sql_error.h b/sql/sql_error.h > index 318d5076534..92076616adb 100644 > --- a/sql/sql_error.h > +++ b/sql/sql_error.h > @@ -1195,7 +1195,6 @@ class Diagnostics_area: public Sql_state_errno, > > void copy_non_errors_from_wi(THD *thd, const Warning_info *src_wi); > > -private: It doesn't seem you're using get_warning_info() anywhere > Warning_info *get_warning_info() { return m_wi_stack.front(); } > > const Warning_info *get_warning_info() const { return m_wi_stack.front(); } > diff --git a/sql/sql_partition.cc b/sql/sql_partition.cc > index 86ba393fb7b..f4a960afe47 100644 > --- a/sql/sql_partition.cc > +++ b/sql/sql_partition.cc > @@ -7237,7 +7246,8 @@ uint fast_alter_partition_table(THD *thd, TABLE *table, > } > else if ((alter_info->partition_flags & ALTER_PARTITION_ADD) && > (part_info->part_type == RANGE_PARTITION || > - part_info->part_type == LIST_PARTITION)) > + part_info->part_type == LIST_PARTITION || > + alter_info->partition_flags & ALTER_PARTITION_AUTO_HIST)) it'd be good it adding new empty VERSIONING partitions would always go this way, auto or not. but it's a separate task. > { > /* > ADD RANGE/LIST PARTITIONS > @@ -7281,7 +7291,7 @@ uint fast_alter_partition_table(THD *thd, TABLE *table, > write_log_add_change_partition(lpt) || > ERROR_INJECT_CRASH("crash_add_partition_4") || > ERROR_INJECT_ERROR("fail_add_partition_4") || > - mysql_change_partitions(lpt) || > + mysql_change_partitions(lpt, false) || this way you skip trans_commit_stmt/trans_commit_implicit for ALTER TABLE ... ADD RANGE/LIST partition. is it always ok? A safer alternative would've been mysql_change_partitions(lpt, !(alter_info->partition_flags & ALTER_PARTITION_AUTO_HIST)) but it's more complex, so I'd prefer your variant, if we can be sure that it doesn't break anything. > ERROR_INJECT_CRASH("crash_add_partition_5") || > ERROR_INJECT_ERROR("fail_add_partition_5") || > alter_close_table(lpt) || Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

3 12

[Maria-developers] Initial prototype for MDEV-10825 Persist exact view definitions DDL
by Vicențiu Ciorbaru 17 Apr '21

17 Apr '21

Hi Sergei! This email is a followup to the brief discussion on Zulip here: https://mariadb.zulipchat.com/#narrow/stream/118759-general/topic/preserve.… You mentioned we store the view's definition (source) inside the FRM. I've used that information to extend the I_S.views table with a source column. The patch is very small, but I have 2 questions: 1. Is this how the feature should look like? I wonder if we should prepend *create view <view-name>* to the SOURCE column, to make it behave very similar to SHOW CREATE VIEW. Perhaps SOURCE as a column name is not the most well chosen name. 2. I don't know if I should use: table->field[11]->store(tables->source.str, tables->source.length, tables->view_creation_ctx->get_client_cs()); or table->field[11]->store(tables->source.str, tables->source.length, cs); when storing the source data. Here is the patch: https://github.com/MariaDB/server/compare/10.6-mdev-10825 As soon as we agree on the complete specs for the feature, I'll clean up test failures in other tests, etc. Vicențiu

2 1