- developers - lists.mariadb.org

Re: [PATCH] MDEV-31872 Deprecate ENCODE()/DECODE()
by Kristian Nielsen 26 Sep '23

26 Sep '23

Kristian Nielsen <knielsen(a)knielsen-hq.org> writes: > From: Sergei Golubchik <serg(a)mariadb.org> > @@ -2724,6 +2725,10 @@ bool Item_func_encode::seed() > hash_password(rand_nr, key->ptr(), key->length()); > sql_crypt.init(rand_nr); > > + push_warning_printf(thd, Sql_condition::WARN_LEVEL_WARN, ER_WARN_DEPRECATED_SYNTAX, > + ER_THD(thd, ER_WARN_DEPRECATED_SYNTAX_NO_REPLACEMENT), > + func_name_cstring().str); // since 11.3 > + NAK. As I already explained in the MDEV-31872, deprecating these functions is not appropriate and will hurt users, not help them. Don't do this. You need to see this from the user's point of view. She is a DBA, she has an application with columns encoded with ENCODE(). What do you expect her to do, rewrite the application because you or someone does not like these functions? That's very disrespectful. If you want to give a warning for these functions, then you need to create an SQL mode or some other handle for the DBA to disable the warning and keep their application running. And the warning should explain why it's there, and what should be done to avoid it. And you need to carefully justify how burdening the DBA with this task of tracking down the source of the new warning and the way to remove it, is worth the benefit. And deprecation suggests eventual removal, which is even worse, please don't do that. You haven't even updated the documentation https://mariadb.com/kb/en/encode/ to eg. say "encode" or "scramble" instead of encrypt. "base64_encode" also does not do encryption, nor is rot13 cryptographically safe, but that doesn't justify removing them either. Most columns in mariadb databases are not encrypted or encoded at all, how is ENCODE() any less secure that that? - Kristian.

3 10

Re: 025ca90e5df: MDEV-32101 CREATE PACKAGE [BODY] for sql_mode=DEFAULT
by Sergei Golubchik 22 Sep '23

22 Sep '23

Hi, Alexander, On Sep 15, Alexander Barkov wrote: > revision-id: 025ca90e5df (mariadb-11.0.1-231-g025ca90e5df) > parent(s): 8ad1e26b1ba > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2023-09-05 17:33:27 +0400 > message: > > MDEV-32101 CREATE PACKAGE [BODY] for sql_mode=DEFAULT > > This patch adds PACKAGE support with SQL/PSM dialect for sql_mode=DEFAULT: > diff --git a/sql/sql_yacc.yy b/sql/sql_yacc.yy > index 7452f6a083a..22c60bd4277 100644 > --- a/sql/sql_yacc.yy > +++ b/sql/sql_yacc.yy > @@ -18454,6 +18518,22 @@ drop_routine: > if (Lex->stmt_drop_procedure($3, $4)) > MYSQL_YYABORT; > } > + | DROP PACKAGE_MARIADB_SYM opt_if_exists sp_name > + { > + LEX *lex= Lex; > + lex->set_command(SQLCOM_DROP_PACKAGE, $3); > + if (unlikely(lex->sphead)) > + my_yyabort_error((ER_SP_NO_DROP_SP, MYF(0), "PACKAGE")); > + lex->spname= $4; > + } > + | DROP PACKAGE_MARIADB_SYM BODY_MARIADB_SYM opt_if_exists sp_name > + { > + LEX *lex= Lex; > + lex->set_command(SQLCOM_DROP_PACKAGE_BODY, $4); > + if (unlikely(lex->sphead)) > + my_yyabort_error((ER_SP_NO_DROP_SP, MYF(0), "PACKAGE BODY")); > + lex->spname= $5; > + } > ; at least here and in SHOW you can use package: PACKAGE_MARIADB_SYM | PACKAGE_ORACLE_SYM ; and then move drop_routine out of ifdef > > > @@ -18498,6 +18578,248 @@ create_routine: > (Item_result) $8, $10)) > MYSQL_YYABORT; > } > + > + | create_or_replace definer_opt PACKAGE_MARIADB_SYM > + opt_if_not_exists sp_name opt_create_package_chistics_init > + remember_name > + { > + sp_package *pkg; > + if (unlikely(!(pkg= Lex-> > + create_package_start(thd, > + SQLCOM_CREATE_PACKAGE, > + &sp_handler_package_spec, > + $5, $1 | $4)))) > + MYSQL_YYABORT; > + pkg->set_c_chistics(Lex->sp_chistics); > + } > + opt_sqlpsm_package_specification_element_list END > + remember_end_opt > + { > + if (unlikely(Lex->create_package_finalize(thd, $5, NULL, $7, $11))) > + MYSQL_YYABORT; > + } > + | create_or_replace definer_opt PACKAGE_MARIADB_SYM BODY_MARIADB_SYM > + opt_if_not_exists sp_name opt_create_package_chistics_init > + remember_name > + { > + sp_package *pkg; > + if (unlikely(!(pkg= Lex-> > + create_package_start(thd, > + SQLCOM_CREATE_PACKAGE_BODY, > + &sp_handler_package_body, > + $6, $1 | $5)))) > + MYSQL_YYABORT; > + pkg->set_c_chistics(Lex->sp_chistics); > + Lex->sp_block_init(thd); > + } > + sqlpsm_package_implementation_declare_section > + { > + if (unlikely(Lex->sp_block_with_exceptions_finalize_declarations(thd))) > + MYSQL_YYABORT; > + } > + sqlpsm_package_implementation_executable_section > + { > + $10.hndlrs+= $12.hndlrs; > + if (unlikely(Lex->sp_block_finalize(thd, $10))) > + MYSQL_YYABORT; > + } > + remember_end_opt > + { > + if (unlikely(Lex->create_package_finalize(thd, $6, NULL, $8, $14))) > + MYSQL_YYABORT; > + } this can also use `package` rule as above and be moved outside of %ifdef. you'll only need %ifdef MARIADB sp_tail_is: _empty ; opt_sp_name: _empty { $$= NULL; } ; %else sp_tail_is: IS | AS ; opt_sp_name: _empty { $$= NULL; } | sp_name { $$= $1; } ; %endif > + ; > + > + > +opt_sqlpsm_package_specification_element_list: > + _empty > + | sqlpsm_package_specification_element_list > + ; and so on, this can be called simply opt_package_specification_element_list if placed under %ifdef MARIADB > + > +sqlpsm_package_specification_element_list: > + sqlpsm_package_specification_element > + | sqlpsm_package_specification_element_list > + sqlpsm_package_specification_element > + ; ... Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: 14a12f98eb8: MDEV-15935 Clean up string sys_var types.
by Sergei Golubchik 14 Sep '23

14 Sep '23

Hi, Yuchen, ok to push, after making one change (const) below. On Sep 13, Yuchen Pei wrote: > revision-id: 14a12f98eb8 (mariadb-11.0.1-232-g14a12f98eb8) > parent(s): e987b9350cb > author: Yuchen Pei > committer: Yuchen Pei > timestamp: 2023-09-13 11:50:41 +1000 > message: > > MDEV-15935 Clean up string sys_var types. > > Merge sys_var_charptr with sys_var_charptr_base, as well as merge > Sys_var_session_lexstring into Sys_var_lexstring. Also refactored > update methods of sys_var_charptr accordingly. > > Because the class is more generic, session_update() calls > sys_var_charptr::session_update() which does not assume a buffer field > associated with THD, but instead call strdup/free, we get rid of > THD::default_master_connection_buff accordingly You could've mentioned here "this also makes THD smaller by 192 bytes, and there can be many thousands of concurrent THDs" I mean, it's a good result, one of the benefits of your cleanup. > diff --git a/sql/sys_vars.inl b/sql/sys_vars.inl > index 31ebbae01ca..5bac91a0db3 100644 > --- a/sql/sys_vars.inl > +++ b/sql/sys_vars.inl > @@ -492,14 +492,18 @@ class Sys_var_mybool: public Sys_var_typelib > Backing store: char* > > @note > - This class supports only GLOBAL variables, because THD on destruction > - does not destroy individual members of SV, there's no way to free > - allocated string variables for every thread. > + > + Note that the memory management for SESSION_VAR's is manual, the > + value must be strdup'ed in THD::init() and freed in > + plugin_thdvar_cleanup(). TODO: it should be done automatically when > + we'll have more session string variables to justify it. Maybe some > + kind of a loop over all variables, like sys_var_end() in set_var.cc? > */ > -class Sys_var_charptr_base: public sys_var > +class Sys_var_charptr: public sys_var > { > + size_t max_length= 2000; Should be const, right? No need to store max_length in every Sys_var_charptr instance, if you aren't going to modify it per instance. > public: > - Sys_var_charptr_base(const char *name_arg, > + Sys_var_charptr(const char *name_arg, > const char *comment, int flag_args, ptrdiff_t off, size_t size, > CMD_LINE getopt, > const char *def_val, PolyLock *lock=0, Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: 9db04b3dc73: MDEV-15935 Adding global/session system var redirect_url
by Sergei Golubchik 13 Sep '23

13 Sep '23

Hi, Yuchen, Thanks. Ok to push both commits into preview-11.3-preview branch On Sep 13, Yuchen Pei wrote: > revision-id: 9db04b3dc73 (mariadb-11.0.1-233-g9db04b3dc73) > parent(s): 14a12f98eb8 > author: Yuchen Pei > committer: Yuchen Pei > timestamp: 2023-09-13 11:50:42 +1000 > message: > > MDEV-15935 Adding global/session system var redirect_url > > Adding a global/session var `redirect_url' of string type. The initial > value is empty. Can be supplied in mysqld with --redirect-url or set > in --init-connect. A valid redirect_url should be of the format > > {mysql,mariadb}://host[:port] > > where <host> is an arbitrary string not containing colons, and <port> > is a number between 0 and 65535 inclusive. > > The variable will be used by the server to notify clients that they > should connect to another server, specified by the value of the > variable, if not empty. > > The notification is done by the inclusion of the variable in > session_track_system_variable. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: 8ac17273b14: MDEV-15935 Adding global/session system var redirect_url
by Sergei Golubchik 13 Sep '23

13 Sep '23

Hi, Yuchen, Right. This is pretty much the whole feature. Mostly ok, few minor comments below. On Sep 08, Yuchen Pei wrote: > revision-id: 8ac17273b14 (mariadb-11.0.1-104-g8ac17273b14) > parent(s): 365faf7229f > author: Yuchen Pei > committer: Yuchen Pei > timestamp: 2023-09-08 14:46:50 +1000 > message: > > MDEV-15935 Adding global/session system var redirect_url > > Adding a global/session var `redirect_url' of string type. The initial > value is empty. Can be supplied in mysqld with --redirect-url or set > in --init-connect. A valid redirect_url should be of the format > > {mysql,mariadb}://host[:port] > > where <host> is an arbitrary string not containing colons, and <port> > is a number between 0 and 65535 inclusive. > > The variable will be used by the server to notify clients that they > should connect to another server, specified by the value of the > variable, if not empty. > > The notification is done by the inclusion of the variable in > session_track_system_variable. > > diff --git a/mysql-test/suite/sys_vars/r/sysvars_server_embedded.result b/mysql-test/suite/sys_vars/r/sysvars_server_embedded.result > index fa3bf6b4354..85dc924fe5f 100644 > --- a/mysql-test/suite/sys_vars/r/sysvars_server_embedded.result > +++ b/mysql-test/suite/sys_vars/r/sysvars_server_embedded.result > @@ -3232,6 +3232,16 @@ NUMERIC_BLOCK_SIZE 1 > ENUM_VALUE_LIST NULL > READ_ONLY NO > COMMAND_LINE_ARGUMENT REQUIRED > +VARIABLE_NAME REDIRECT_URL > +VARIABLE_SCOPE SESSION > +VARIABLE_TYPE VARCHAR > +VARIABLE_COMMENT URL to redirect to, if not empty see if you can make it a bit more descriptive. if not, ok as is. hmm, and a second thought. it's in the sysvars_server_embedded test. it doesn't make sense there (embedded doesn't accept connections, and doesn't have session tracking, iirc), so, may be hide it under #ifndef EMBEDDED_LIBRARY ? > +NUMERIC_MIN_VALUE NULL > +NUMERIC_MAX_VALUE NULL > +NUMERIC_BLOCK_SIZE NULL > +ENUM_VALUE_LIST NULL > +READ_ONLY NO > +COMMAND_LINE_ARGUMENT REQUIRED > VARIABLE_NAME REQUIRE_SECURE_TRANSPORT > VARIABLE_SCOPE GLOBAL > VARIABLE_TYPE BOOLEAN > diff --git a/sql/sql_class.h b/sql/sql_class.h > index cf2b5b96a22..c86a76f6a08 100644 > --- a/sql/sql_class.h > +++ b/sql/sql_class.h > @@ -878,6 +878,7 @@ typedef struct system_variables > > Time_zone *time_zone; > char *session_track_system_variables; > + char *redirect_url; > > /* Some wsrep variables */ > ulonglong wsrep_trx_fragment_size; > diff --git a/sql/sql_plugin.cc b/sql/sql_plugin.cc > index 4724e2200e6..70aba9cc4d5 100644 > --- a/sql/sql_plugin.cc > +++ b/sql/sql_plugin.cc > @@ -3307,6 +3309,10 @@ void plugin_thdvar_init(THD *thd) > #ifndef EMBEDDED_LIBRARY > thd->session_tracker.sysvars.init(thd); > #endif > + thd->variables.redirect_url= > + my_strdup(PSI_INSTRUMENT_ME, global_system_variables.redirect_url, and here, key_memory_Sys_var_charptr_value > + MYF(MY_WME | MY_THREAD_SPECIFIC)); > + > DBUG_VOID_RETURN; > } > > @@ -3375,6 +3381,8 @@ void plugin_thdvar_cleanup(THD *thd) > #ifndef EMBEDDED_LIBRARY > thd->session_tracker.sysvars.deinit(thd); > #endif > + my_free(thd->variables.redirect_url); > + thd->variables.redirect_url= 0; also, perhaps in cleanup_variables() ? > my_free((char*) thd->variables.default_master_connection.str); > thd->variables.default_master_connection.str= 0; > thd->variables.default_master_connection.length= 0; > diff --git a/sql/sys_vars.cc b/sql/sys_vars.cc > index 064c60ab2b0..a93e68f40d7 100644 > --- a/sql/sys_vars.cc > +++ b/sql/sys_vars.cc > @@ -1343,6 +1343,61 @@ Sys_init_connect( > DEFAULT(""), &PLock_sys_init_connect, NOT_IN_BINLOG, > ON_CHECK(check_init_string)); > > +/** > + Validate a redirect_url string. > + > + A valid string is either empty, or of the format mysql://host[:port], > + where host is an arbitrary string without any colon ':'. > + > + @param str A string to validate > + @param len Length of the string > + @retval false The string is valid > + @retval true The string is invalid > +*/ > +static bool sysvar_validate_redirect_url(sys_var *, THD *, set_var *var) > +{ > + char *str= var->save_result.string_value.str; > + size_t len= var->save_result.string_value.length; > + /* Empty string is valid */ > + if (len == 0) > + return false; > + const char* end= str + len; > + if (!strncmp(str, "mysql://", strlen("mysql://"))) use !strncmp(str, STRING_WITH_LEN("mysql://")) avoids no strlen and no need to write the string twice. same below. > + str+= strlen("mysql://"); ah. Then, I'd suggest instead LEX_CSTRING mysql_prefix={STRING_WITH_LEN("mysql://")}; if (!strncmp(str, mysql_prefix.str, mysql_prefix.length)) str+= mysql_prefix.length; > + else if (!strncmp(str, "mariadb://", strlen("mariadb://"))) > + str+= strlen("mariadb://"); > + else > + return true; > + /* Host name cannot be empty */ > + if (str == end) > + return true; > + /* Find the colon, if any */ > + while (str < end && *str != ':') > + str++; > + /* Found colon */ > + if (str < end) > + { > + /* Should have at least one number after the colon */ > + if (str + 1 == end) > + return true; > + int p= 0; > + while (str < end && isdigit(*++str)) > + p= p * 10 + (*str - '0'); > + /* Should be all numbers after the colon */ > + if (str < end) > + return true; > + /* The port number should be between 0 and 65535 */ > + if (p > 65535) > + return true; check for overflow too. you can either put the p>65535 check inside the loop or at the end check that, like, (str - colon_ptr) < 7. > + } > + return false; > +} > + > +static Sys_var_charptr Sys_redirect_url( > + "redirect_url", "URL to redirect to, if not empty", > + SESSION_VAR(redirect_url), CMD_LINE(REQUIRED_ARG), DEFAULT(""), > + NO_MUTEX_GUARD, NOT_IN_BINLOG, sysvar_validate_redirect_url, 0, 0); By convention, write ON_CHECK(sysvar_validate_redirect_url). And you don't need 0's at the end, so just NO_MUTEX_GUARD, NOT_IN_BINLOG, ON_CHECK(sysvar_validate_redirect_url)); Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 4

Re: [MariaDB commits] [PATCH] MDEV-31872: Add --old_mode=COMPAT_DISCOURAGED
by Sergei Golubchik 12 Sep '23

12 Sep '23

Hi, Kristian, On Sep 12, Kristian Nielsen wrote: > > diff --git a/sql/sys_vars.cc b/sql/sys_vars.cc > index e25d293bbb7..18d08996aa8 100644 > --- a/sql/sys_vars.cc > +++ b/sql/sys_vars.cc > @@ -3863,12 +3863,13 @@ static const char *old_mode_names[]= > "IGNORE_INDEX_ONLY_FOR_JOIN", // deprecated since 11.3 > "COMPAT_5_1_CHECKSUM", // deprecated since 11.3 > "LOCK_ALTER_TABLE_COPY", // deprecated since 11.3 > + "COMPAT_DISCOURAGED", > 0 > }; > > void old_mode_deprecated_warnings(THD *thd, ulonglong v) > { > - v &= ~OLD_MODE_DEFAULT_VALUE; > + v &= ~(OLD_MODE_DEFAULT_VALUE | OLD_MODE_COMPAT_DISCOURAGED); No, this is wrong. old-mode *by definition* is something that is a temporary workaround for something new, old-mode values are by design deprecated from the day one. So, OLD_MODE_COMPAT_DISCOURAGED shouldn't be on the line above, and thus it'll itself be deprecated and will cause a deprecation warning. Which is rather silly, as I wrote in a previous email, and that's why an old-mode is not a good solution for turning deprecated warnings away. > for (uint i=0; old_mode_names[i]; i++) > if ((1ULL<<i) & v) > { > diff --git a/sql/item_strfunc.cc b/sql/item_strfunc.cc > index 4b96271c331..339430a2794 100644 > --- a/sql/item_strfunc.cc > +++ b/sql/item_strfunc.cc > @@ -2720,7 +2720,16 @@ bool Item_func_encode::seed() > hash_password(rand_nr, key->ptr(), key->length()); > sql_crypt.init(rand_nr); > > - warn_deprecated<1103>(current_thd, func_name_cstring().str); > + if (!(current_thd->variables.old_behavior & OLD_MODE_COMPAT_DISCOURAGED)) > + { > + /* > + This function should not be removed despite the deprecation warning. > + That would cause problems for users who can then no longer access > + columns in their database that might have been inserted as > + ENCODE(str,pass_str). > + */ > + warn_deprecated<1103>(current_thd, func_name_cstring().str); > + } No, this is exactly what I wanted to prevent by adding a warn_deprecated<>() wrapper. If something is giving a warning "... is deprecated and will be removed" - it *has to be removed*. We don't want to lie to users to simply scare them. If we promise to do something, we should do it. If we aren't going to remove ENCODE/DECODE functions, we shouldn't deprecate them at all. Which is the result I'm leaning to right now. > > return FALSE; > } Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: 75ea482c4fc: MDEV-30973 GTID in deadlock output
by Sergei Golubchik 11 Sep '23

11 Sep '23

Hi, Aleksey, On Sep 11, Aleksey Midenkov wrote: > revision-id: 75ea482c4fc (mariadb-11.0.1-230-g75ea482c4fc) > parent(s): fe62f70fb4b > author: Aleksey Midenkov > committer: Aleksey Midenkov > timestamp: 2023-09-01 20:30:24 +0300 > message: > > MDEV-30973 GTID in deadlock output > > Example: > > TRANSACTION 40, ACTIVE 0 sec starting index read > mysql tables in use 1, locked 1 > LOCK WAIT 2 lock struct(s), heap size 1152, 1 row lock(s) > MySQL thread id 23, OS thread handle 140201491547840, query id 310, GTID 3-1-1 Updating > delete from tu where a = 9999 and b = 'xxxx' > 2023-04-12 19:53:58 22 [Note] InnoDB: *** (1) WAITING FOR THIS LOCK TO BE GRANTED: > > Deterministic slave_deadlock.test > > Instead of "MDEV-10962 Deadlock with 3 concurrent DELETEs by unique key" > this one uses "MDEV-22698 Deadlock on concurrent acquisition from multiple indexes" > > --- > mysql-test/suite/rpl/t/slave_deadlock.test | 93 ++++++++++++++++++++++++++++++ > sql/sql_show.cc | 10 ++++ please move slave_deadlock.result from the following commit to this one. otherwise ok Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Implementing Binlog GTID Indexes (MDEV-4991)
by Kristian Nielsen 10 Sep '23

10 Sep '23

I am implementing Binlog GTID Indexes, to fix an old performance regression for GTID when slaves connect to the master. I now have a solid design and a working prototype, so I wanted to describe the work to encourage early comments and suggestions. I'm hoping this will make it to the next release (where is the place to actively follow/participate in the release planning?) Ever since I implemented GTID, connecting slaves needed to sequentially scan the leading portion of one binlog file to locate the GTID position to start at. This can be slow as binlog files default to 1GB in size, even worse so when binlog files can now be encrypted. I files MDEV-4991 for this 10 years ago, and it is embarrassing that this has somehow remained unfixed for so long :-/ But now I'm fixing it. The current implementation is in the work-in-progress branch knielsen_mdev4991: https://github.com/MariaDB/server/tree/knielsen_mdev4991 The code is already functional, with the testsuite passing. Still missing is binlog purge, index crash recovery, async path, and general testing and cleanup. The basic idea is for each binlog file master-bin.000001 to write an index file master-bin.000001.idx. The index contains a B+-Tree in which the keys are pairs of (GTID state, binlog offset). A connecting slave's GTID position can be looked up quickly in the tree to find the corresponding binlog offset (and gtid_binlog_state) to start replicating at. Similarly, a non-GTID connecting slave can quickly look up its starting offset (BINLOG_GTID_POS()) to obtain the GTID position corresponding to its starting offset. The index is written out to disk concurrently with the writing of the binlog file. Since the tree is written in sorted order and append-only, the implementation is significantly simplified over a general B+-Tree, and can also be described as a Log-structed Merge Tree with a Btree search structure on top. All GTID state keys (except the first) in the index are delta-compressed, storing only the GTIDs that changed since the last record (which typically will be only one). Additionally the index is sparse, storing only say every 1-in-10 GTIDs in the index (this will be configurable of course). Using a sparse index, the disk space is reduced at the cost only of scanning a few extra events of the binlog file to find the exact position requested. Actual benchmarking of space usage is TBO, but we can expect a typical binlog file of 1 GB containing say 3 million GTIDs to require something like 10 MB of index disk space (1% increase). The file format is page-based (unlike the binlog file), so that it can be written efficiently and easily read using random-access. The index is written in parallel with the binlog file, but no extra fsync()s are needed (except one at the end of the file when the index is closed and synced to disk). The writing of the index will happen asynchroneously from the binlog background thread; this way it will have minimal impact on the performance and scalability of the binlog and the contested LOCK_log. Connecting slaves can read from the currently-being-written ("hot") GTID index by accessing internal memory buffers of pages not yet written to disk. If the server crashes, the existing binlog recovery and binlog checkpoint mechanism will be used to re-create any incomplete indexes as part of the normal recovery scan of binlog files. If somehow a binlog index should be found corrupt or missing (could be eg. old server upgrade or backup script that omits the new .idx files), the code will gracefully fall back to the old (slower) way of sequentially scanning the binlog file to locate the slave starting position. I think this will be a great (if rather late) improvement to one remaining corner of the GTID implementation that currently has sub-optimal performance, and I'm exited to get this completed and reviewed/tested and added to hopefully the next MariaDB release. - Kristian.

1 0

Re: 365faf7229f: MDEV-15935 Clean up string sys_var types.
by Sergei Golubchik 08 Sep '23

08 Sep '23

Hi, Yuchen, On Sep 08, Yuchen Pei wrote: > revision-id: 365faf7229f (mariadb-11.0.1-103-g365faf7229f) > parent(s): 4f39ae75606 > author: Yuchen Pei > committer: Yuchen Pei > timestamp: 2023-09-08 13:12:22 +1000 > message: > > MDEV-15935 Clean up string sys_var types. > > Merge sys_var_charptr with sys_var_charptr_base, as well as merge > Sys_var_session_lexstring into Sys_var_lexstring. Also refactored > update methods of sys_var_charptr accordingly. > > Because the class is more generic, session_update() calls > sys_var_charptr::session_update() which does not assume a buffer field > associated with THD, but instead call strdup/free, we get rid of > THD::default_master_connection_buff accordingly > diff --git a/sql/sql_class.cc b/sql/sql_class.cc > index 18e796655bd..52fa6c27112 100644 > --- a/sql/sql_class.cc > +++ b/sql/sql_class.cc > @@ -1233,10 +1233,12 @@ void THD::init() > avoid temporary tables replication failure. > */ > variables.pseudo_thread_id= thread_id; > - variables.default_master_connection.str= default_master_connection_buff; > - ::strmake(default_master_connection_buff, > - global_system_variables.default_master_connection.str, > - variables.default_master_connection.length); > + variables.default_master_connection.str= > + my_strndup(PSI_INSTRUMENT_ME, may be key_memory_Sys_var_charptr_value instead of PSI_INSTRUMENT_ME? > + global_system_variables.default_master_connection.str, > + global_system_variables.default_master_connection.length, > + MYF(MY_WME | MY_THREAD_SPECIFIC)); > + > mysql_mutex_unlock(&LOCK_global_system_variables); > > user_time.val= start_time= start_time_sec_part= 0; > diff --git a/sql/sql_plugin.cc b/sql/sql_plugin.cc > index d39589ed1ab..4724e2200e6 100644 > --- a/sql/sql_plugin.cc > +++ b/sql/sql_plugin.cc > @@ -3279,6 +3279,9 @@ void plugin_thdvar_init(THD *thd) > #ifndef EMBEDDED_LIBRARY > thd->session_tracker.sysvars.deinit(thd); > #endif > + my_free((char*) thd->variables.default_master_connection.str); > + thd->variables.default_master_connection.str= 0; > + thd->variables.default_master_connection.length= 0; shouldn't this go inside cleanup_variables() ? > > thd->variables= global_system_variables; > > diff --git a/sql/sys_vars.cc b/sql/sys_vars.cc > index 4843450352e..064c60ab2b0 100644 > --- a/sql/sys_vars.cc > +++ b/sql/sys_vars.cc > @@ -1355,12 +1355,11 @@ static bool check_master_connection(sys_var *self, THD *thd, set_var *var) > return false; > } > > -static Sys_var_session_lexstring Sys_default_master_connection( > +static Sys_var_lexstring Sys_default_master_connection( > "default_master_connection", > "Master connection to use for all slave variables and slave commands", > - SESSION_ONLY(default_master_connection), > - NO_CMD_LINE, > - DEFAULT(""), MAX_CONNECTION_NAME, ON_CHECK(check_master_connection)); > + SESSION_ONLY(default_master_connection), NO_CMD_LINE, DEFAULT(""), > + NO_MUTEX_GUARD, NOT_IN_BINLOG, ON_CHECK(check_master_connection)); some length limit would still be nice, I'd say. Doesn't have to be per variable, may be all Sys_var_lexstring could have some common limit, like, arbitrarily, 1K or 10K. > #endif > > static Sys_var_charptr_fscs Sys_init_file( > diff --git a/sql/sys_vars.inl b/sql/sys_vars.inl > index 31ebbae01ca..0f2a07d92fc 100644 > --- a/sql/sys_vars.inl > +++ b/sql/sys_vars.inl > @@ -576,6 +571,21 @@ class Sys_var_charptr_base: public sys_var > new_val= 0; > return new_val; > } > + char *session_update_prepare(set_var *var) do you need this session_update_prepare method? global_update_prepare() and global_update_finish() are used by Sys_var_sesvartrack. Getting rid of them is a task for another day :) > + { > + return update_prepare(var, MYF(MY_WME | MY_THREAD_SPECIFIC)); > + } > + bool session_update(THD *thd, set_var *var) override > + { > + char *new_val= session_update_prepare(var); > + my_free(session_var(thd, char*)); > + session_var(thd, char*)= new_val; > + return (new_val == 0 && var->save_result.string_value.str != 0); > + } > + char *global_update_prepare(set_var *var) > + { > + return update_prepare(var, MYF(MY_WME)); > + } > void global_update_finish(char *new_val) > { > if (flags & ALLOCATED) > @@ -583,50 +593,28 @@ class Sys_var_charptr_base: public sys_var > flags|= ALLOCATED; > global_var(char*)= new_val; > } > - bool global_update(THD *thd, set_var *var) > + bool global_update(THD *thd, set_var *var) override > { > - char *new_val= global_update_prepare(thd, var); > + char *new_val= global_update_prepare(var); > global_update_finish(new_val); > return (new_val == 0 && var->save_result.string_value.str != 0); > } > - void session_save_default(THD *thd, set_var *var)= 0; > - void global_save_default(THD *thd, set_var *var) > + void save_default(set_var *var) > { > char *ptr= (char*)(intptr)option.def_value; > var->save_result.string_value.str= ptr; > var->save_result.string_value.length= ptr ? strlen(ptr) : 0; > } > -}; > - > -class Sys_var_charptr: public Sys_var_charptr_base > -{ > -public: > - Sys_var_charptr(const char *name_arg, > - const char *comment, int flag_args, ptrdiff_t off, size_t size, > - CMD_LINE getopt, > - const char *def_val, PolyLock *lock=0, > - enum binlog_status_enum binlog_status_arg=VARIABLE_NOT_IN_BINLOG, > - on_check_function on_check_func=0, > - on_update_function on_update_func=0, > - const char *substitute=0) : > - Sys_var_charptr_base(name_arg, comment, flag_args, off, size, getopt, > - def_val, lock, binlog_status_arg, > - on_check_func, on_update_func, substitute) > + void session_save_default(THD *, set_var *var) override > { > - SYSVAR_ASSERT(scope() == GLOBAL); > - SYSVAR_ASSERT(size == sizeof(char *)); > + return save_default(var); no, session's default value is global's value. See any other Sys_var_xxx class, e.g. for Sys_var_mybool: void session_save_default(THD *thd, set_var *var) { var->save_result.ulonglong_value= (ulonglong)*(my_bool *)global_value_ptr(thd, 0); } You cannot have a test for that now, but let's have one in the redirect_url commit. > } > - > - bool session_update(THD *thd, set_var *var) > + void global_save_default(THD *, set_var *var) override > { > - DBUG_ASSERT(FALSE); > - return true; > + return save_default(var); > } > - void session_save_default(THD *thd, set_var *var) > - { DBUG_ASSERT(FALSE); } > }; > > - > class Sys_var_charptr_fscs: public Sys_var_charptr > { > using Sys_var_charptr::Sys_var_charptr; > diff --git a/mysql-test/suite/sys_vars/r/mdev_15935.result b/mysql-test/suite/sys_vars/r/mdev_15935.result > new file mode 100644 > index 00000000000..d20c2ad2f19 > --- /dev/null > +++ b/mysql-test/suite/sys_vars/r/mdev_15935.result > @@ -0,0 +1,12 @@ > +# > +# test that the global/session-only enforcement is still in place despite the merge of string sys_var classes > +# > +set session init_connect="whatever"; > +ERROR HY000: Variable 'init_connect' is a GLOBAL variable and should be set with SET GLOBAL > +set session wsrep_auto_increment_control="whatever"; > +ERROR HY000: Variable 'wsrep_auto_increment_control' is a GLOBAL variable and should be set with SET GLOBAL > +set global default_master_connection="whatever"; > +ERROR HY000: Variable 'default_master_connection' is a SESSION variable and can't be used with SET GLOBAL not sure you need that, it's checked in the very base class, you could've have affected that I'd say it's enough if existing tests for Sys_var_session_lexstring and Sys_var_sesvartrack variables succeed and ASAN/MSAN are happy > +# > +# end of test mdev_15935 > +# Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: 04b99a30a3f: MDEV-15935 Adding global/session var redirect_url
by Sergei Golubchik 08 Sep '23

08 Sep '23

Hi, Yuchen, On Sep 01, Yuchen Pei wrote: > revision-id: 04b99a30a3f (mariadb-11.0.1-103-g04b99a30a3f) > parent(s): 4f39ae75606 > author: Yuchen Pei > committer: Yuchen Pei > timestamp: 2023-08-16 11:31:16 +1000 > message: > > MDEV-15935 Adding global/session var redirect_url > > Adding a global/session var `redirect_url' of string type. The initial > value is empty. Can be supplied in mysqld with --redirect-url. A valid > redirect_url should be of the format > > {mysql,mariadb}://host[:port] > > where <host> is an arbitrary string not containing colons, and <port> > is a number between 0 and 65535 inclusive. > > diff --git a/mysql-test/main/redirect.opt b/mysql-test/main/redirect.opt move this to the sys_vars suite, please > --- /dev/null > +++ b/mysql-test/main/redirect.opt > @@ -0,0 +1 @@ > +--redirect_url=mysql://foobar ok, as long as this isn't a complete feature, but just one piece of it. but when client support will be implemented (CONC-599) and the server will actually send this to the connecting clients (MDEV-31609), this test will need to be fixed, as foobar is not something a client can be redirected to. Also, it'd be more interesting to try something like --init-connect='set @@redirect_url="mysql://foobar"' instead of the simple --redirect_url=mysql://foobar because init-connect is much more flexible, it can generate the url on the fly, it can send different users to different locations. So it'd make sense to test init-connect approach. > diff --git a/mysql-test/main/redirect.test b/mysql-test/main/redirect.test > new file mode 100644 > --- /dev/null > +++ b/mysql-test/main/redirect.test > @@ -0,0 +1,56 @@ > +--echo # > +--echo # MDEV-15935 Connection Redirection Mechanism in MariaDB Client/Server Protocol > +--echo # > + > +set @old_redirect_url=@@redirect_url; > +select @@global.redirect_url; > +set global redirect_url=default; > +select @@global.redirect_url; > +--error ER_WRONG_VALUE_FOR_VAR > +set global redirect_url="mariadb.org"; > +--error ER_WRONG_VALUE_FOR_VAR > +set global redirect_url="https://mariadb.org"; before doing all this fun, please remove redirect_url from the session_track_system_variables, or just set it to the empty string. > diff --git a/sql/sql_class.h b/sql/sql_class.h > --- a/sql/sql_class.h > +++ b/sql/sql_class.h > @@ -3585,6 +3587,7 @@ class THD: public THD_count, /* this must be first */ > */ > LEX_CSTRING connection_name; > char default_master_connection_buff[MAX_CONNECTION_NAME+1]; > + char redirect_url_buff[MAX_CONNECTION_NAME+1]; This isn't such a good idea (I'm afraid, I missed it when default_master_connection_buff was added). A couple of years ago Monty have spent significant efforts trying to make THD smaller as it has grown to be too large. Now THD keeps growing again. Let's try not to encourage it, if we can help it. > uint8 password; /* 0, 1 or 2 */ > uint8 failed_com_change_user; > bool slave_thread; > diff --git a/sql/sys_vars.inl b/sql/sys_vars.inl > --- a/sql/sys_vars.inl > +++ b/sql/sys_vars.inl > @@ -637,6 +637,129 @@ class Sys_var_charptr_fscs: public Sys_var_charptr ... > +class Sys_var_redirect_url: public Sys_var_charptr_base please, no. new classes are needed only for variables with very special, unlike anything else, behavior. redirect_url is just a session string variable. See the comment at the top of sys_vars.cc But (!) there are no other session string variables yet. So I'd recomment to fix Sys_var_charptr to support session variables. And then create redirect_url out of it. Practically it might need no changes at all, if you put explicit strmake (or strdup/free) into THD::init like you did anyway. After that, I suspect, you might be able to remove Sys_var_charptr_base, Sys_var_sesvartrack, and Sys_var_session_lexstring, because Sys_var_charptr/Sys_var_lexstring will be able to serve all those use cases. > +{ > + static const size_t max_length = MAX_CONNECTION_NAME; > +public: > + Sys_var_redirect_url(const char *name_arg, > + const char *comment, > + CMD_LINE getopt, > + const char *def_val, PolyLock *lock= 0) : > + Sys_var_charptr_base(name_arg, comment, > + SESSION_VAR(redirect_url), > + getopt, def_val, lock, > + VARIABLE_NOT_IN_BINLOG, 0, 0, 0) > + { > + global_var(LEX_CSTRING).length= strlen(def_val); > + option.var_type|= GET_STR; > + *const_cast<SHOW_TYPE*>(&show_val_type)= SHOW_LEX_STRING; > + } Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1