- developers - lists.mariadb.org

Re: [Maria-developers] 37b6f5cb8ef: decouple commit/rollback code from binlog hton
by Sergei Golubchik 23 Nov '21

23 Nov '21

Hi, Nikita! On Nov 15, Nikita Malyavin wrote: > revision-id: 37b6f5cb8ef (mariadb-10.5.2-484-g37b6f5cb8ef) > parent(s): f3603cbfaeb > author: Nikita Malyavin > committer: Nikita Malyavin > timestamp: 2021-01-29 03:47:52 +1000 > message: > > decouple commit/rollback code from binlog hton let's squash it with "[6/8] store cache managers in a list" Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] ad6e7b87107: introduce cache flipping
by Sergei Golubchik 23 Nov '21

23 Nov '21

Hi, Nikita! Interesting trick. On Nov 13, Nikita Malyavin wrote: > revision-id: ad6e7b87107 (mariadb-10.5.2-477-gad6e7b87107) > parent(s): 85fbd867b4c > author: Nikita Malyavin > committer: Nikita Malyavin > timestamp: 2021-01-27 14:17:39 +1000 > message: > > introduce cache flipping Could you add few lines from the Cache_flip_event_log comment here too please? > diff --git a/sql/log.h b/sql/log.h > index 74c409e1ac7..1e422ae4524 100644 > --- a/sql/log.h > +++ b/sql/log.h > @@ -424,10 +424,9 @@ class Event_log: public MYSQL_LOG > > bool open( > const char *log_name, > - enum_log_type log_type, > const char *new_name, ulong next_file_number, > enum cache_type io_cache_type_arg); > - IO_CACHE *get_log_file() { return &log_file; } > + virtual IO_CACHE *get_log_file() { return &log_file; } > > int write_description_event(enum_binlog_checksum_alg checksum_alg, > bool encrypt, bool dont_set_created); > @@ -435,6 +434,73 @@ class Event_log: public MYSQL_LOG > bool write_event(Log_event *ev, binlog_cache_data *data, IO_CACHE *file); > }; > > +/** > + A single-reader, multiple-writer non-blocking layer for Event_log. > + Provides IO_CACHE for writing and IO_CACHE for reading.\ > + > + Writers use an overrided get_log_file version for their writes, while readers > + should use flip() to initiate reading. > + flip() swaps pointers to allow non-blocking reads. > + > + Writers can block other writers and a reader with a mutex, but a reader only > + swaps two pointers under a lock, so it won't block writers. > + > + TODO should be unnecessary after MDEV-24676 is done > + */ > +class Cache_flip_event_log: public Event_log { > + IO_CACHE alt_buf; > + IO_CACHE *current, *alt; > +public: > + > + Cache_flip_event_log() : Event_log(), alt_buf{}, > + current(&log_file), alt(&alt_buf) {} > + bool open(enum cache_type io_cache_type_arg) > + { > + log_file.dir= mysql_tmpdir; > + alt_buf.dir= log_file.dir; > + bool res= Event_log::open(NULL, NULL, 0, io_cache_type_arg); > + if (res) > + return res; > + > + name= my_strdup(key_memory_MYSQL_LOG_name, "online-alter-binlog", > + MYF(MY_WME)); > + if (!name) > + return false; > + > + res= init_io_cache(&alt_buf, -1, LOG_BIN_IO_SIZE, io_cache_type_arg, 0, 0, > + MYF(MY_WME | MY_NABP | MY_WAIT_IF_FULL)) != 0; > + return res; > + } > + > + /** > + Swaps current and alt_log. Can be called only from the reader thread. > + @return a new IO_CACHE pointer to read from. > + */ > + IO_CACHE *flip() > + { > + IO_CACHE *tmp= current; > + reinit_io_cache(alt, WRITE_CACHE, 0, 0, 0); > + mysql_mutex_lock(get_log_lock()); > + reinit_io_cache(current, READ_CACHE, 0, 0, 0); > + current= alt; > + mysql_mutex_unlock(get_log_lock()); > + alt= tmp; > + > + return alt; > + } > + > + IO_CACHE *get_log_file() override > + { > + return current; may be mysql_mutex_assert_owner(get_log_lock()) ? > + } > + > + void cleanup() > + { > + end_io_cache(&alt_buf); > + Event_log::cleanup(); > + } > +}; > + Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] c80991c79f7: MDEV-25785 Add support for OpenSSL 3.0
by Sergei Golubchik 21 Nov '21

21 Nov '21

Hi, Vladislav! few questions below: On Nov 18, Vladislav Vaintroub wrote: > revision-id: c80991c79f7 (mariadb-10.6.1-213-gc80991c79f7) > parent(s): cee33f1ab7c > author: Vladislav Vaintroub > committer: Vladislav Vaintroub > timestamp: 2021-11-09 02:04:22 +0100 > message: > > MDEV-25785 Add support for OpenSSL 3.0 > > Summary of changes > > - MD_CTX_SIZE is increased > > - EVP_CIPHER_CTX_buf_noconst(ctx) does not work anymore, points > to nobody knows where. The assumption made previously was that > (since the function does not seem to be documented) > was that it points to the last partial source block. > Add own partial block buffer for NOPAD encryption instead > > - SECLEVEL in CipherString in openssl.cnf > had been downgraded to 0, from 1, to make TLSv1.0 and TLSv1.1 possible The definition of SECLEVEL is: Level 0 Everything is permitted. This retains compatibility with previous versions of OpenSSL. Level 1 The security level corresponds to a minimum of 80 bits of security. Any parameters offering below 80 bits of security are excluded. As a result RSA, DSA and DH keys shorter than 1024 bits and ECC keys shorter than 160 bits are prohibited. All export cipher suites are prohibited since they all offer less than 80 bits of security. SSL version 2 is prohibited. Any cipher suite using MD5 for the MAC is also prohibited. Note that signatures using SHA1 and MD5 are also forbidden at this level as they have less than 80 security bits. Only at level 3 does it say "TLS versions below 1.1 are not permitted." I don't see why you had to change from 1 to 0. Do we have "signatures using SHA1 and MD5"? > - ctx_buf buffer now must be aligned to 16 bytes with openssl( > previously with WolfSSL only), ot crashes will happen > > - updated aes-t , to be better debuggable > using function, rather than a huge multiline macro > added test that does "nopad" encryption piece-wise, to test > replacement of EVP_CIPHER_CTX_buf_noconst > > diff --git a/cmake/ssl.cmake b/cmake/ssl.cmake > index 7c2488be8bd..64c93ff9b4f 100644 > --- a/cmake/ssl.cmake > +++ b/cmake/ssl.cmake > @@ -139,9 +139,20 @@ MACRO (MYSQL_CHECK_SSL) > SET(SSL_INTERNAL_INCLUDE_DIRS "") > SET(SSL_DEFINES "-DHAVE_OPENSSL") > > + FOREACH(x INCLUDES LIBRARIES DEFINITIONS) > + SET(SAVE_CMAKE_REQUIRED_${x} ${CMAKE_REQUIRED_${x}}) > + ENDFOREACH() why do you set/restore them? > + # Silence "deprecated in OpenSSL 3.0" > + IF((NOT OPENSSL_VERSION) # 3.0 not determined by older cmake > + OR NOT(OPENSSL_VERSION VERSION_LESS "3.0.0")) > + SET(SSL_DEFINES "${SSL_DEFINES} -DOPENSSL_API_COMPAT=0x10100000L") > + SET(CMAKE_REQUIRED_DEFINITIONS -DOPENSSL_API_COMPAT=0x10100000L) > + ENDIF() This is just postponing the inevitable. They'll drop the old API eventually. As far as I understand the internals were changed in a way that doesn't fit the old API. Is there some safe subset of OpenSSL API that works both in 1.0 and in 3.0 ? It might be more future proof to use only that. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

3 4

Re: [Maria-developers] 4492c869f31: MDEV-26238: Remove inconsistent behaviour of --default-* options in my_print_defaults
by Sergei Golubchik 18 Nov '21

18 Nov '21

Hi, Rucha! Thanks! Looks good now. One comment about tests: On Nov 18, Rucha Deodhar wrote: > revision-id: 4492c869f31 (mariadb-10.6.1-212-g4492c869f31) > parent(s): 3f3ec40c91b > author: Rucha Deodhar > committer: Rucha Deodhar > timestamp: 2021-11-08 16:51:33 +0530 > message: > > MDEV-26238: Remove inconsistent behaviour of --default-* options > in my_print_defaults ... > diff --git a/mysql-test/main/my_print_defaults.test b/mysql-test/main/my_print_defaults.test > index bfd4e563826..b52deaf09ff 100644 > --- a/mysql-test/main/my_print_defaults.test > +++ b/mysql-test/main/my_print_defaults.test > @@ -87,20 +85,74 @@ read_buffer_size=1M ... > +--echo # checking that --defaults* option only works when mentioned at beginning > + > +--echo # Testing --defaults-file at beginning only > +--exec $MYSQL_MY_PRINT_DEFAULTS --defaults-file=$MYSQLTEST_VARDIR/tmp/tmp1.cnf --mysqld > +--remove_file $MYSQLTEST_VARDIR/tmp/tmp3.cnf How does it test that "--defaults* option only works at beginning"? You test that --defaults-file works at the beginning all right. This is redundant, as there are lots of tests in this file that have --defaults-file, but it doesn't hurt. But you don't test the "only" part. You don't test anything that you've changed - I suspect your new test would happily pass without any changes in my_print_defaults.c To test that --defaults-file works *only* at the beginning, you need to verify that it fails not at the beginning. Like error 1 exec $MYSQL_MY_PRINT_DEFAULTS --myslqd --defaults-file=$MYSQLTEST_VARDIR/tmp/tmp1.cnf also with -c error 1 exec $MYSQL_MY_PRINT_DEFAULTS -c $MYSQLTEST_VARDIR/tmp/tmp1.cnf --mysqld This should fail even at the beginning. The point is - you need to test the behavior that you've changed. As your test has introduced failures - you need to test for these failures. But all your tests succeed. > +--echo # Testing --defaults-extra-file works at beginning only > +--exec $MYSQL_MY_PRINT_DEFAULTS --defaults-file=$MYSQLTEST_VARDIR/tmp/tmp1.cnf --defaults-extra-file=$MYSQLTEST_VARDIR/tmp/tmp2.cnf --mysqld Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] MDEV-27036: do not output empty subqueries in traces
by Sergey Petrunia 16 Nov '21

16 Nov '21

Hi Sergei, > commit eb7d6304a2aa168e09d48f6884dadf8c9f868feb > Author: Sergei Krivonos <sergei.krivonos(a)mariadb.com> > Date: Mon Nov 15 01:37:04 2021 +0200 > > do not output empty subqueries in traces This reads as if the trace can have an empty clause describing a subquery. This is very confusing. Actually, the patch removes potential empty "subqueries" arrays. Do you have a testcase for this? I'm trying to find one with grep -rn -A1 '"subqueries"' mysql-test/{main,suite} and I can't. > diff --git a/sql/sql_explain.cc b/sql/sql_explain.cc > index 499b6dbc97a..9a9bf786d62 100644 > --- a/sql/sql_explain.cc > +++ b/sql/sql_explain.cc > @@ -25,6 +25,8 @@ > #include "opt_range.h" > #include "sql_expression_cache.h" > > +#include <algorithm> > + Is this needed? It seems to work without it. > + bool Explain_node::has_json_printable_connection_objects(Explain_query *query) const > + { > + auto n= children.elements(); > + if (n) > + { > + for (; n--;) > + { Why such a convoluted loop inside an if statement? Can you make the code simpler by using for (n=0; n < children.elements(); n++) { if (is_connection_printable_in_json(children.at(i)->connection_type)) return true; } return false; ? > + auto node= query->get_node(children.at(n)); > + /* Derived tables are printed inside Explain_table_access objects */ > + > + if (!is_connection_printable_in_json(node->connection_type)) > + continue; > + > + return true; > + } > + } > + return false; > + } BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://petrunia.net

1 0

Re: [Maria-developers] MDEV-27036: duplicated keys in best_access_path
by Sergey Petrunia 16 Nov '21

16 Nov '21

Hi Sergei, > commit d2b4782200ccfd0850f36d25c75082974e4b86e1 > Author: Sergei Krivonos <sergei.krivonos(a)mariadb.com> > Date: Sun Nov 14 10:37:20 2021 +0200 > > MDEV-27036: duplicated keys in best_access_path > > diff --git a/sql/sql_select.cc b/sql/sql_select.cc > index 3272f8b7f65..29e9c0efe36 100644 > --- a/sql/sql_select.cc > +++ b/sql/sql_select.cc > @@ -7611,16 +7611,14 @@ best_access_path(JOIN *join, > } > else > { > + trace_access_idx.add("used_range_estimates", false); > if (table->quick_keys.is_set(key)) > { > - trace_access_idx.add("used_range_estimates",false) > - .add("cause", > - "not better than ref estimates"); > + cause= "not better than ref estimates"; Issue #1: you make this change here, in the branch marked as ReuseRangeEstimateForRef-2. But there's very similar code right above, in ReuseRangeEstimateForRef-1 branch: /* quick_range couldn't use key! */ records= (double) s->records/rec; trace_access_idx.add("used_range_estimates", false) .add("cause", "not available"); Issue #2: this patch causes some information to be removed from the trace! How about an alternative solution: change the name from "cause" here to "reason"? > } > else > { > - trace_access_idx.add("used_range_estimates", false) > - .add("cause", "not available"); > + cause= "not available"; > } > } > } BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://petrunia.net

1 0

[Maria-developers] MDEV-26929: Please enable optimizer trace run in buildbot JSON
by Sergey Petrunia 09 Nov '21

09 Nov '21

Hi Elena, The patch for https://jira.mariadb.org/browse/MDEV-26929 has been merged from 10.4 into all versions up to 10.8. Could you enable the optimizer trace run in the buildbot please? BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://petrunia.net

2 2

Re: [Maria-developers] 80749146b7d: MDEV-26238: Remove inconsistent behaviour of --default-* options in my_print_defaults
by Sergei Golubchik 07 Nov '21

07 Nov '21

Hi, Rucha! On Nov 05, Rucha Deodhar wrote: > > On Nov 04, Rucha Deodhar wrote: > > > revision-id: 80749146b7d (mariadb-10.6.1-77-g80749146b7d) > > > parent(s): 76149650764 > > > author: Rucha Deodhar > > > committer: Rucha Deodhar > > > timestamp: 2021-09-12 02:11:19 +0530 > > > message: > > > > > > MDEV-26238: Remove inconsistent behaviour of --default-* options > > > in my_print_defaults > > > > > > diff --git a/extra/my_print_defaults.c b/extra/my_print_defaults.c > > > index b7f52382721..607509a309e 100644 > > > --- a/extra/my_print_defaults.c > > > +++ b/extra/my_print_defaults.c > > > @@ -141,50 +124,39 @@ static int get_options(int *argc,char ***argv) > > ... > > > + /* > > > + We already process --defaults* options at the beginning in > > > + get_defaults_options(). So skip --defaults* options and > > > + pass remaining options to handle_options(). > > > + */ > > > + if (is_prefix(org_argv[1], "--no-defaults")) > > > + args_used--; > > > > I'm sorry, I don't understand. Why do you need that? > > I made it to keep the output consistent with previous version. > > If --no-defaults is present in the command line, then it is counted in > args_used when get_defaults_options() is called. Later we skip > "args_used" number of arguments so that only non-default options > mentioned after --default-* options are passed to get_options(). > > The above code decreases this count when --no-defaults is present so > that we don't skip it unlike --defaults-* options and pass it to > get_options(). It is required to pass --no-defaults to get_options() > because if it is not passed then it produces output which looks like > --help ( if no other options are there in addition to --no-defaults). > If it is passed to get_options(), then --no-defaults produces no > output like it does in 10.6. But it's correct, isn't it? If you run my_print_defaults with no groups, it'll print the help text. Try, for example, my_print_defaults --defaults-file=var/my.cnf (presuming you're in mysql-test dir). You'll get the help text, because you didn't specify any groups. Or my_print_defaults --defaults-extra-file=var/my.cnf But my_print_defaults --no-defaults doesn't do it. Doesn't behave like other *defaults* options. We can easily preserve the inconsisent "if --no-defaults, no group is required" behavior with, like if (my_no_defaults) cleanup_and_exit(0); (and add my_no_defaults to my_default.h). But I don't think it's necessary. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 50f43d46299: MDEV-14938 make buildbot to include galera into bintars
by Sergei Golubchik 05 Nov '21

05 Nov '21

Hi, Alexey! On Nov 05, Alexey Bychko wrote: > revision-id: 50f43d46299 (mariadb-10.2.40-106-g50f43d46299) > parent(s): 05c3dced861 > author: Alexey Bychko > committer: Alexey Bychko > timestamp: 2021-10-22 16:30:56 +0700 > message: > > MDEV-14938 make buildbot to include galera into bintars > > this commit adds cmake module to process external directory with built > galera to include galera library and binaries into server build. > it introduces new cmake variable EXT_GALERA_PATH representing relative > or absolute path to compiled galera tree. No compiled galera tree. This is useless. MDEV description says Currently bintars that buildbot produces don't contain libgalera_smm.so. Publishing scripts repackage bintars to add it. We want to produce correct packages in buildbot and release exactly what buildbot builds. It means, buildbot need to produce *exactly* what `prep` script was generating. `prep` script does not build galera. This is a completely separate step, `prep` only packages pre-existing galera files. It's enough to assume that there's ${EXT_GALERA_PATH}/garbd and ${EXT_GALERA_PATH}/libgalera_smm.so. Or if you want it to be really universal, make two different variables, cmake -DBINTAR_GARBD=/path/to/garbd -DBINTAR_LIBGALERA=/path/to/libgalera_smm.so This will work with a compiled galera tree, with installed galera package, with just garbd and libgalera_smm.so in $HOME, with everything. > diff --git a/CMakeLists.txt b/CMakeLists.txt > index 67216e0e443..1107013860c 100644 > --- a/CMakeLists.txt > +++ b/CMakeLists.txt > @@ -153,6 +153,7 @@ INCLUDE(mysql_version) > INCLUDE(cpack_source_ignore_files) > INCLUDE(install_layout) > INCLUDE(wsrep) > +INCLUDE(galera_external) > > # Add macros > INCLUDE(character_sets) > diff --git a/cmake/galera_external.cmake b/cmake/galera_external.cmake > new file mode 100644 > index 00000000000..64ddfc9c498 > --- /dev/null > +++ b/cmake/galera_external.cmake > @@ -0,0 +1,75 @@ > +# EXT_GALERA_PATH > +# a path to built galera git tree > +# or a path to unpacked binary tarball > + > +IF(WIN32 OR NOT EXT_GALERA_PATH) why do you need WIN32 check here? > + RETURN() > +ENDIF() > + > +IF(NOT EXISTS ${EXT_GALERA_PATH}) > + MESSAGE(FATAL_ERROR "Galera dir ${EXT_GALERA_PATH} does not exist!") > +ENDIF() > + > +SET(GALERA_PATH ${CMAKE_SOURCE_DIR}/GALERASYM) > +IF(EXISTS ${GALERA_PATH}) > + FILE({REMOVE ${GALERA_PATH}) > +ENDIF() > +EXECUTE_PROCESS(COMMAND ${CMAKE_COMMAND} -E create_symlink ${EXT_GALERA_PATH} ${GALERA_PATH}) Why do you need this symlink? > + > +FIND_LIBRARY(GALERA_LIB > + NAMES galera_smm galera_enterprise_smm > + PATHS ${GALERA_PATH} > + NO_DEFAULT_PATH > + ) > + > +IF(NOT GALERA_LIB) > + MESSAGE(FATAL_ERROR "Galera library not found in ${GALERA_PATH}") > +ENDIF() > + > +INSTALL(PROGRAMS > + ${GALERA_LIB} > + DESTINATION lib > + RENAME libgalera_smm.so > + COMPONENT Server > + ) > + > +INSTALL(PROGRAMS > + ${GALERA_PATH}/garb/garbd > + DESTINATION bin > + COMPONENT Server > + ) > + > +INSTALL(PROGRAMS > + ${GALERA_PATH}/garb/files/garb-systemd > + DESTINATION bin > + COMPONENT Server > + ) > + > +INSTALL(FILES > + ${GALERA_PATH}/garb/files/garb.cnf > + DESTINATION docs/galera > + COMPONENT Server > + ) MDEV description says Currently bintars that buildbot produces don't contain libgalera_smm.so. Publishing scripts repackage bintars to add it. We want to produce correct packages in buildbot and release exactly what buildbot builds. It means, buildbot need to produce *exactly* what `prep` script was generating. Meaning three galera files. Like: -rwxr-xr-x mariadb-10.2.40-linux-glibc_214-x86_64/bin/garbd -rw-r--r-- mariadb-10.2.40-linux-glibc_214-x86_64/lib/libgalera_smm.so lrwxrwxrwx mariadb-10.2.40-linux-glibc_214-x86_64/lib/galera/libgalera_smm.so -> ../libgalera_smm.so Nothing else. Also, please, verify that the generated directory inside the tarball matches the pattern above. And that the tarbal name does too. > + > +INSTALL(FILES > + ${GALERA_PATH}/AUTHORS > + ${GALERA_PATH}/COPYING > + ${GALERA_PATH}/README > + DESTINATION docs/galera/doc > + COMPONENT Server > + ) > + > +INSTALL(FILES > + ${GALERA_PATH}/asio/LICENSE_1_0.txt > + DESTINATION docs/galera/doc > + RENAME LICENSE.asio > + COMPONENT Server > + ) > + > +INSTALL(FILES > + ${GALERA_PATH}/chromium/LICENSE > + DESTINATION docs/galera/doc > + RENAME LICENSE.chromium > + COMPONENT Server > + ) > + > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] MDEV-23766: Optimizer tracing code is prone to producing invalid JSON
by Sergey Petrunia 05 Nov '21

05 Nov '21

Hi Serg, On Tue, Nov 02, 2021 at 02:19:05AM +0200, Sergei Krivonos wrote: > Applied fixes by your tests patch 638930f85f… <https://github.com/MariaDB/server/commit/638930f85ff40bd39354da81e53e7278a4…> > #1: Please check out the commit adding unit tests: http://lists.askmonty.org/pipermail/commits/2021-November/014773.html It turns out it is not that hard to add them. Any comments/ways to improve this? #2: In the above commit, note that two last unit tests show a scenario that is not currently checked. #3: What is the point of functions Json_writer::on_add_str(), Json_writer::on_start_array, Json_writer::on_start_object() ? I would understand if all the checking code was isolated in these functions. But it isn't. Please either remove the functions or explain the reasoning. #4 See the input below about the commit comment. Please change (feel free to force-push into the branch after that). I think this is all the input, and the patch will be ready for pushing once all of this is addressed. > > Item #2: please improve the commit comment as was requested in my previous > > e-mail: > > > > On Thu, Oct 28, 2021 at 04:58:35PM +0300, Sergey Petrunia wrote: > >> First, input on the commit comment. Please consider it as a request applying > >> to ALL further commits to the MariaDB codebase. > >> > >> > >>> MDEV-23766: implemented requested debug checks > >> > >> Imagine somebody looking at this in a few years. Will they know what checks > >> were requested? They might get a clue by looking at the Jira text, but the > >> idea to make the commit comments concise and self-contained descriptions. > >> > >> Something like this: > >> > >> Line #1: one-line description of the patch: > >> > >>>> MDEV-23766: Make attempts to write invalid JSON cause assertion failures > >> > >> Subsequent lines: > >> > >>>> Make JSON writing API (class Json_writer) check the produced JSON document > >>>> is valid. The following checks are made: > >>>> - JSON objects must contain named members > >>>> - JSON arrays must contain unnamed members. > >>>> - (TODO: add other restrictions we're enforcing). BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://petrunia.net

1 0

Re: [Maria-developers] 80749146b7d: MDEV-26238: Remove inconsistent behaviour of --default-* options in my_print_defaults
by Sergei Golubchik 04 Nov '21

04 Nov '21

Hi, Rucha! On Nov 04, Rucha Deodhar wrote: > revision-id: 80749146b7d (mariadb-10.6.1-77-g80749146b7d) > parent(s): 76149650764 > author: Rucha Deodhar > committer: Rucha Deodhar > timestamp: 2021-09-12 02:11:19 +0530 > message: > > MDEV-26238: Remove inconsistent behaviour of --default-* options > in my_print_defaults > > Analysis: --defaults* option is recognized anywhere in the commandline > instead of only at the beginning because handle_options() recognizes > options in any order. > Fix: use get_defaults_options() which recognizes --defaults* options only at > the beginning. After this is done, we only want to recognize other options > given in any order which can be done using handle_options(). So only skip > --defaults* options and pass rest of them to handle_options(). > Also, removed -e, -g and -c because only my_print_defaults supports them. > > diff --git a/extra/my_print_defaults.c b/extra/my_print_defaults.c > index b7f52382721..607509a309e 100644 > --- a/extra/my_print_defaults.c > +++ b/extra/my_print_defaults.c > @@ -141,50 +124,39 @@ static int get_options(int *argc,char ***argv) ... > int main(int argc, char **argv) > { > - int count= 0, error, no_defaults= 0; > + int count, error, args_used; > char **load_default_groups= 0, *tmp_arguments[6]; > char **argument, **arguments, **org_argv; > int nargs, i= 0; > MY_INIT(argv[0]); > > org_argv= argv; > - if (*argv && !strcmp(*argv, "--no-defaults")) > - { > - argv++; > - ++count; > - no_defaults= 1; > - } > - /* Copy program name and --no-defaults if present*/ > + args_used= get_defaults_options(argv); > + > + /* Copy defaults-xxx arguments & program name */ > + count= args_used; > arguments= tmp_arguments; > - memcpy((char*) arguments, (char*) org_argv, (++count)*sizeof(*org_argv)); > + memcpy((char*) arguments, (char*) org_argv, count*sizeof(*org_argv)); > arguments[count]= 0; I still cannot understand why you're doing it. Why do you need a copy of argv? > > + /* > + We already process --defaults* options at the beginning in > + get_defaults_options(). So skip --defaults* options and > + pass remaining options to handle_options(). > + */ > + if (is_prefix(org_argv[1], "--no-defaults")) > + args_used--; I'm sorry, I don't understand. Why do you need that? all *defaults* options were handled in get_defaults_options(). > + why do you need org_argv? First you do org_argv= argv; then after some code that doesn't change either argv or org_argv, you copy the vallue back: > + argv=org_argv; > + argv+=args_used-1; > + argc-=args_used-1; > + > /* Check out the args */ > if (get_options(&argc,&argv)) > cleanup_and_exit(1); > > - if (!no_defaults) > - { > - if (opt_defaults_file_used) > - arguments[count++]= make_args("--defaults-file=", config_file); > - if (my_defaults_extra_file) > - arguments[count++]= make_args("--defaults-extra-file=", > - my_defaults_extra_file); > - if (my_defaults_group_suffix) > - arguments[count++]= make_args("--defaults-group-suffix=", > - my_defaults_group_suffix); > - arguments[count]= 0; > - } > - > nargs= argc + 1; > if (opt_mysqld) > nargs+= array_elements(mysqld_groups); Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

[Maria-developers] [MDEV-10654] Review request
by Manohar K B 04 Nov '21

04 Nov '21

Hi All, Good day. I have fixed ticket MDEV-10654 and raised PR. Can someone kindly review it? This is the first time I am contributing to MariaDB, so please guide me if I missed anything. Best regards, Manohar.

2 1

Re: [Maria-developers] bb1737b0482: MDEV-12459: The information_schema tables for getting temporary tables info is missing, at least for innodb there is no INNODB_TEMP_TABLE_INFO
by Sergei Golubchik 01 Nov '21

01 Nov '21

Hi, Anel! On Nov 01, Anel Husakovic wrote: > revision-id: bb1737b0482 (mariadb-10.6.1-73-gbb1737b0482) > parent(s): 1bc82aaf0a7 > author: Anel Husakovic > committer: Anel Husakovic > timestamp: 2021-09-06 08:58:46 +0200 > message: > > MDEV-12459: The information_schema tables for getting temporary tables info is missing, at least for innodb there is no INNODB_TEMP_TABLE_INFO > > diff --git a/sql/sql_show.cc b/sql/sql_show.cc > index 6b351726320..5444de8c1ca 100644 > --- a/sql/sql_show.cc > +++ b/sql/sql_show.cc > @@ -69,6 +69,8 @@ > > #include "lex_symbol.h" > #define KEYWORD_SIZE 64 > +#define IS_USER_TEMP_TABLE(A) ((A->tmp_table == TRANSACTIONAL_TMP_TABLE) || \ > + (A->tmp_table == NON_TRANSACTIONAL_TMP_TABLE)) > > extern SYMBOL symbols[]; > extern size_t symbols_length; > @@ -151,6 +153,8 @@ static int show_create_sequence(THD *thd, TABLE_LIST *table_list, > static const LEX_CSTRING *view_algorithm(TABLE_LIST *table); > > bool get_lookup_field_values(THD *, COND *, TABLE_LIST *, LOOKUP_FIELD_VALUES *); > +void process_i_s_table_temporary_tables(THD *thd, TABLE * table, LEX_CSTRING *db_name, > + LEX_CSTRING *table_name, TABLE *tmp_tbl); > > /** > Try to lock a mutex, but give up after a short while to not cause deadlocks > @@ -5161,6 +5166,10 @@ int get_all_tables(THD *thd, TABLE_LIST *tables, COND *cond) > uint table_open_method= tables->table_open_method; > bool can_deadlock; > MEM_ROOT tmp_mem_root; > + Dynamic_array<LEX_CSTRING> system_tables(PSI_INSTRUMENT_MEM); > + All_tmp_tables_list *temp_tables= NULL; > + // Scan for temporary tables > + TMP_TABLE_SHARE *share_temp; > DBUG_ENTER("get_all_tables"); > > bzero(&tmp_mem_root, sizeof(tmp_mem_root)); > @@ -5223,6 +5232,12 @@ int get_all_tables(THD *thd, TABLE_LIST *tables, COND *cond) > init_alloc_root(PSI_INSTRUMENT_ME, &tmp_mem_root, SHOW_ALLOC_BLOCK_SIZE, > SHOW_ALLOC_BLOCK_SIZE, MY_THREAD_SPECIFIC); > > + system_tables.push(INFORMATION_SCHEMA_NAME); > + system_tables.push(PERFORMANCE_SCHEMA_DB_NAME); > + system_tables.push(MYSQL_SCHEMA_NAME); > + system_tables.push(SYS_SCHEMA_NAME); > + system_tables.push(MTR_SCHEMA_NAME); > + > for (size_t i=0; i < db_names.elements(); i++) > { > LEX_CSTRING *db_name= db_names.at(i); > @@ -5234,6 +5249,36 @@ int get_all_tables(THD *thd, TABLE_LIST *tables, COND *cond) > sctx->master_access & (DB_ACLS | SHOW_DB_ACL) || > acl_get(sctx->host, sctx->ip, sctx->priv_user, db_name->str, 0)) > #endif > + > + // Only if there is IS.tables allow temporary tables to be shown > + if (schema_table_idx == SCH_TABLES && !temp_tables) > + { > + for (size_t k=0; k < system_tables.elements(); k++) > + { > + if (db_name != &system_tables.at(k)) > + { > + temp_tables= open_tables_state_backup.temporary_tables; > + if (temp_tables) > + { > + All_tmp_tables_list::Iterator it(*open_tables_state_backup.temporary_tables); > + while ((share_temp= it++)) > + { > + All_share_tables_list::Iterator it2(share_temp->all_tmp_tables); > + while (TABLE *tbl= it2++) > + { > + if (IS_USER_TEMP_TABLE(share_temp)) > + { > + // Now we have the data and we should process_table() manually > + process_i_s_table_temporary_tables(thd, table, db_name, > + &share_temp->table_name, tbl); > + } > + } > + } > + } > + break; > + } > + } > + } hmmmm. What do you do here? I understand, basically, nothing here. Could you explain, line by line? > { > Dynamic_array<LEX_CSTRING*> table_names(PSI_INSTRUMENT_MEM); > int res= make_table_name_list(thd, &table_names, lex, > @@ -5495,6 +5540,8 @@ static int get_schema_tables_record(THD *thd, TABLE_LIST *tables, > table->field[3]->store(STRING_WITH_LEN("SYSTEM VIEW"), cs); > else if (share->table_type == TABLE_TYPE_SEQUENCE) > table->field[3]->store(STRING_WITH_LEN("SEQUENCE"), cs); > + else if (IS_USER_TEMP_TABLE(share)) > + table->field[3]->store(STRING_WITH_LEN("TEMPORARY"), cs); what about INTERNAL_TMP_TABLE and SYSTEM_TMP_TABLE? > else > { > DBUG_ASSERT(share->tmp_table == NO_TMP_TABLE); > @@ -9032,7 +9096,6 @@ ST_FIELD_INFO tables_fields_info[]= > NOT_NULL, "Comment", OPEN_FRM_ONLY), > Column("MAX_INDEX_LENGTH",ULonglong(), NULLABLE, "Max_index_length", > OPEN_FULL_TABLE), > - Column("TEMPORARY", Varchar(1), NULLABLE, "Temporary", OPEN_FRM_ONLY), > CEnd() > }; Let's remove it in a separate commit. After you implement "TEMPORARY" for table type. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] MDEV-23766:Optimizer tracing code is prone to producing invalid JSON
by Sergey Petrunia 31 Oct '21

31 Oct '21

Hi Serg, Ok the patch moving in the right direction but it's not there, yet. Please find the input below. First, input on the commit comment. Please consider it as a request applying to ALL further commits to the MariaDB codebase. > MDEV-23766: implemented requested debug checks Imagine somebody looking at this in a few years. Will they know what checks were requested? They might get a clue by looking at the Jira text, but the idea to make the commit comments concise and self-contained descriptions. Something like this: Line #1: one-line description of the patch: >> MDEV-23766: Make attempts to write invalid JSON cause assertion failures Subsequent lines: >> Make JSON writing API (class Json_writer) check the produced JSON document >> is valid. The following checks are made: >> - JSON objects must contain named members >> - JSON arrays must contain unnamed members. >> - (TODO: add other restrictions we're enforcing). It is not clear how the checker interacts with Single_line_formatting_helper. The code like one the one I'm quoting below shows some non-trivial overlap: > @@ -28,7 +36,16 @@ void Json_writer::append_indent() > > void Json_writer::start_object() > { > +#ifndef NDEBUG > + DBUG_ASSERT(got_name == named_item_expected()); > + named_items_expectation.push_back(true); > + size_t depth= named_items_expectation.size(); > +#endif > fmt_helper.on_start_object(); > +#ifndef NDEBUG > + if (depth != named_items_expectation.size()) > + named_items_expectation.pop_back(); > +#endif > > if (!element_started) > start_element(); This looks very cryptic. Can you elaborate about this? BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://petrunia.net

1 1

Re: [Maria-developers] 85aa3f8cf66: MDEV-26102: dgcov: add support for *.gcda.gcov.json.gz files of gcov 9.1+
by Sergei Golubchik 30 Oct '21

30 Oct '21

Hi, Anel! Looks ok. Except that you have that pointless $gcc_version=~ line that I wrote about in a previous review. Did you verify that this script actually delivers meaningful results? On Oct 30, Anel Husakovic wrote: > revision-id: 85aa3f8cf66 (mariadb-10.2.39-96-g85aa3f8cf66) > parent(s): afe00bb7cce > author: Anel Husakovic > committer: Anel Husakovic > timestamp: 2021-07-29 12:42:44 +0200 > message: > > MDEV-26102: dgcov: add support for *.gcda.gcov.json.gz files of gcov 9.1+ > > Reviewed by: serg(a)mariadb.com > > --- > mysql-test/dgcov.pl | 55 +++++++++++++++++++++++++++++++++++------------------ > 1 file changed, 37 insertions(+), 18 deletions(-) > > diff --git a/mysql-test/dgcov.pl b/mysql-test/dgcov.pl > index 2c00c64d1ff..543099c1b4c 100755 > --- a/mysql-test/dgcov.pl > +++ b/mysql-test/dgcov.pl > @@ -62,12 +62,15 @@ my $res; > + > +my $gcc_version= `gcc -dumpversion`; > +$gcc_version=~ s/(\d).*$/$1/; > > find(\&gcov_one_file, $root); > find(\&write_coverage, $root) if $opt_generate; Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 2c8f52ea53d: MDEV-23328 Server hang due to Galera lock conflict resolution
by Sergei Golubchik 29 Oct '21

29 Oct '21

Hi, Seppo, Jan! this is pretty much the same as in 10.4, so same comments apply, otherwise ok. On Oct 29, sjaakola wrote: > revision-id: 2c8f52ea53d (mariadb-10.5.12-108-g2c8f52ea53d) > parent(s): 910fc72a7f1 > author: sjaakola > committer: Jan Lindström > timestamp: 2021-10-28 12:00:41 +0300 > message: > > MDEV-23328 Server hang due to Galera lock conflict resolution Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 778d88ea2d9: MDEV-23328 Server hang due to Galera lock conflict resolution
by Sergei Golubchik 29 Oct '21

29 Oct '21

Hi, Seppo, Jan! Looks pretty much ok too. A few comments, see below On Oct 29, sjaakola wrote: > revision-id: 778d88ea2d9 (mariadb-10.4.21-76-g778d88ea2d9) > parent(s): e2e4e06f081 > author: sjaakola > committer: Jan Lindström > timestamp: 2021-10-28 11:06:22 +0300 > message: > > MDEV-23328 Server hang due to Galera lock conflict resolution > > diff --git a/sql/log_event.cc b/sql/log_event.cc > index c1c753d69fd..a1562016aa4 100644 > --- a/sql/log_event.cc > +++ b/sql/log_event.cc > @@ -8987,6 +8987,18 @@ bool Xid_log_event::print(FILE* file, PRINT_EVENT_INFO* print_event_info) > > > #if defined(HAVE_REPLICATION) && !defined(MYSQL_CLIENT) > +static bool wsrep_must_replay(THD *thd) > +{ > +#ifdef WITH_WSREP > + mysql_mutex_lock(&thd->LOCK_thd_data); > + bool res= WSREP(thd) && thd->wsrep_trx().state() == wsrep::transaction::s_must_replay; check WSREP(thd) outside of the mutex > + mysql_mutex_unlock(&thd->LOCK_thd_data); > + return res; > +#else > + return false; > +#endif > +} > + > int Xid_log_event::do_apply_event(rpl_group_info *rgi) > { > bool res; > diff --git a/sql/service_wsrep.cc b/sql/service_wsrep.cc > index 6566bb372d8..18067a4d0ec 100644 > --- a/sql/service_wsrep.cc > +++ b/sql/service_wsrep.cc > @@ -29,12 +29,14 @@ extern "C" my_bool wsrep_on(const THD *thd) > > extern "C" void wsrep_thd_LOCK(const THD *thd) > { > + mysql_mutex_lock(&thd->LOCK_thd_kill); > mysql_mutex_lock(&thd->LOCK_thd_data); I presume, you can remove wsrep_thd_kill_LOCK() and wsrep_thd_kill_UNLOCK() that I've added > } > > extern "C" void wsrep_thd_UNLOCK(const THD *thd) > { > mysql_mutex_unlock(&thd->LOCK_thd_data); > + mysql_mutex_unlock(&thd->LOCK_thd_kill); > } > > extern "C" void wsrep_thd_kill_LOCK(const THD *thd) > @@ -295,8 +301,6 @@ extern "C" my_bool wsrep_thd_is_aborting(const MYSQL_THD thd) > default: > return false; > } > - > - return false; this looks like it begs for the compiler warning. please, keep the return where it was. > } Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 002192b92cf: MDEV-23328 Server hang due to Galera lock conflict resolution
by Sergei Golubchik 27 Oct '21

27 Oct '21

Hi, Seppo, Jan! On Oct 27, sjaakola wrote: > revision-id: 002192b92cf (mariadb-10.3.31-60-g002192b92cf) > parent(s): 736da112b65 > author: sjaakola > committer: Jan Lindström > timestamp: 2021-10-27 12:15:48 +0300 > message: > > MDEV-23328 Server hang due to Galera lock conflict resolution This 10.3 version is also ok to push when you have 10.4 and 10.5 versions ready. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] d0e18d694f0: MDEV-23328 Server hang due to Galera lock conflict resolution
by Sergei Golubchik 27 Oct '21

27 Oct '21

Hi, Seppo, Jan! On Oct 27, sjaakola wrote: > revision-id: d0e18d694f0 (mariadb-10.2.40-128-gd0e18d694f0) > parent(s): 408277e8012 > author: sjaakola > committer: Jan Lindström > timestamp: 2021-10-27 12:12:53 +0300 > message: > > MDEV-23328 Server hang due to Galera lock conflict resolution This commit is ok to push, thanks! Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] d1ec0231553: MDEV-26732 Assertion `0' failed in Item::val_native
by Sergei Golubchik 26 Oct '21

26 Oct '21

Hi, Alexander! On Oct 25, Alexander Barkov wrote: > revision-id: d1ec0231553 (mariadb-10.5.4-550-gd1ec0231553) > parent(s): 8b115503563 > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2021-10-07 20:58:18 +0400 > message: > > MDEV-26732 Assertion `0' failed in Item::val_native > > Also fixes MDEV-24619 Wrong result or Assertion `0' in Item::val_native / Type_handler_inet6::Item_val_native_with_conversion would be good to have some explanation here, what was wrong, what was the fix. > diff --git a/plugin/type_inet/mysql-test/type_inet/type_inet6.result b/plugin/type_inet/mysql-test/type_inet/type_inet6.result > index da949481337..ac16f5c06ce 100644 > --- a/plugin/type_inet/mysql-test/type_inet/type_inet6.result > +++ b/plugin/type_inet/mysql-test/type_inet/type_inet6.result > @@ -2159,3 +2159,38 @@ IFNULL(c, '::1') > ::1 > DROP TABLE t2; > DROP TABLE t1; > +# > +# MDEV-26732 Assertion `0' failed in Item::val_native > +# > +SELECT CAST(CONCAT('::', REPEAT('',RAND())) AS INET6) AS f, var_pop('x') FROM dual HAVING f > ''; do you need RAND() here for a test case? can you replace it with a literal? do you need var_pop() here? > +f var_pop('x') > +Warnings: > +Warning 1292 Truncated incorrect DOUBLE value: 'x' > +Warning 1292 Incorrect inet6 value: '' > +SELECT CAST(CONCAT('::', REPEAT('',RAND())) AS INET6) AS f, var_pop(1) FROM dual HAVING f >= '::'; > +f var_pop(1) > +:: 0.0000 > +CREATE TABLE t1(id INET6 NOT NULL PRIMARY KEY, dsc INET6); > +INSERT INTO t1 VALUES ('::1', '1::1'),('::3', '1::3'),('::4', NULL); > +CREATE TABLE t2 SELECT COALESCE(t1.dsc), COUNT(*) FROM t1 GROUP BY t1.id; > +SELECT * FROM t2 ORDER BY 1,2; > +COALESCE(t1.dsc) COUNT(*) > +NULL 1 > +1::1 1 > +1::3 1 > +DROP TABLE t1, t2; > +# > +# MDEV-24619 Wrong result or Assertion `0' in Item::val_native / Type_handler_inet6::Item_val_native_with_conversion > +# > +CREATE TABLE t1 (a INET6); > +INSERT INTO t1 VALUES ('::'),('::'); > +SELECT IF(1, '::', a) AS f FROM t1 GROUP BY 'foo' HAVING f != ''; > +f > +Warnings: > +Warning 1292 Incorrect inet6 value: '' > +SELECT IF(1, '::', a) AS f FROM t1 GROUP BY 'foo' HAVING f != '::'; > +f > +SELECT IF(1, '::', a) AS f FROM t1 GROUP BY 'foo' HAVING f != '::1'; > +f > +:: > +DROP TABLE t1; Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 5

Re: [Maria-developers] 6d0c1f3ae12: MDEV-23328 Server hang due to Galera lock conflict resolution
by Sergei Golubchik 25 Oct '21

25 Oct '21

Hi, Jan! This was good! Just a few questions, see below On Oct 24, Jan Lindström wrote: > revision-id: 6d0c1f3ae12 (mariadb-10.2.40-114-g6d0c1f3ae12) > parent(s): 3153cc799e2 > author: Jan Lindström > committer: Jan Lindström > timestamp: 2021-10-22 07:02:39 +0300 > message: > > MDEV-23328 Server hang due to Galera lock conflict resolution ... > diff --git a/sql/sql_parse.cc b/sql/sql_parse.cc > index 3e1f248b082..5d40106d0ad 100644 > --- a/sql/sql_parse.cc > +++ b/sql/sql_parse.cc > @@ -2159,6 +2159,7 @@ bool dispatch_command(enum enum_server_command command, THD *thd, > } > DBUG_PRINT("quit",("Got shutdown command for level %u", level)); > general_log_print(thd, command, NullS); > + DBUG_EXECUTE_IF("shutdown_unireg_abort", { unireg_abort(1); }); what's the point of this test? > my_eof(thd); > kill_mysql(thd); > error=TRUE; > diff --git a/sql/wsrep_mysqld.cc b/sql/wsrep_mysqld.cc > index e60100e2e90..7466654eead 100644 > --- a/sql/wsrep_mysqld.cc > +++ b/sql/wsrep_mysqld.cc > @@ -835,13 +835,25 @@ void wsrep_thr_init() > DBUG_VOID_RETURN; > } > > +/* This is wrapper for wsrep_break_lock in thr_lock.c */ > +static int wsrep_thr_abort_thd(void *bf_thd_ptr, void *victim_thd_ptr, my_bool signal) When is it called? Under what conditions? > +{ > + THD* victim_thd= (THD *) victim_thd_ptr; > + /* We need to lock THD::LOCK_thd_data to protect victim > + from concurrent usage or disconnect or delete. */ > + mysql_mutex_lock(&victim_thd->LOCK_thd_data); > + int res= wsrep_abort_thd(bf_thd_ptr, victim_thd_ptr, signal); > + return res; > +} > + > diff --git a/storage/innobase/handler/ha_innodb.cc b/storage/innobase/handler/ha_innodb.cc > index 73b3625b6fd..56400264245 100644 > --- a/storage/innobase/handler/ha_innodb.cc > +++ b/storage/innobase/handler/ha_innodb.cc > @@ -19720,46 +19716,113 @@ wsrep_innobase_kill_one_trx( > +/******************************************************************* > +This function is used to kill one transaction in BF. */ > +void > +wsrep_innobase_kill_one_trx( > + MYSQL_THD const bf_thd, > + const trx_t * const bf_trx, > + trx_t *victim_trx, > + my_bool signal) > +{ > + ut_ad(bf_thd); > + ut_ad(victim_trx); > + ut_ad(lock_mutex_own()); > + ut_ad(trx_mutex_own(victim_trx)); > + > + DBUG_ENTER("wsrep_innobase_kill_one_trx"); > + THD *thd= (THD *) victim_trx->mysql_thd; > + > + /* Here we need to lock THD::LOCK_thd_data to protect from > + concurrent usage or disconnect or delete. */ > + DEBUG_SYNC(bf_thd, "wsrep_before_BF_victim_lock"); > + my_sleep(100000); Eh? Forgot to remove after debugging? > + wsrep_thd_LOCK(thd); > + my_sleep(100000); and here > + DEBUG_SYNC(bf_thd, "wsrep_after_BF_victim_lock"); > + Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

3 5

Re: [Maria-developers] 5c4db10d6ff: MDEV-25114: Crash: WSREP: invalid state ROLLED_BACK (FATAL)
by Sergei Golubchik 24 Oct '21

24 Oct '21

Hi, Jan! On Oct 24, Jan LindstrÃ¶m wrote: > revision-id: 5c4db10d6ff (mariadb-10.3.31-58-g5c4db10d6ff) > parent(s): f9b856b0525 > author: Jan Lindström > committer: Jan Lindström > timestamp: 2021-10-22 07:23:14 +0300 > message: > > MDEV-25114: Crash: WSREP: invalid state ROLLED_BACK (FATAL) > > Revert "MDEV-23328 Server hang due to Galera lock conflict resolution" > > This reverts commit 29bbcac0ee841faaa68eeb09c86ff825eabbe6b6. This was incorrectly reverted. 29bbcac0ee841faaa68eeb09c86ff825eabbe6b6 didn't apply to 10.3 verbatim, there were some merge related changes. To revert it properly you need to look at affected files before the corresponding 10.2->10.3 merge. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] dfa33fb309d: MDEV-26664 Store UUIDs in a more efficient manner
by Sergei Golubchik 22 Oct '21

22 Oct '21

Hi, Alexander! On Oct 19, Alexander Barkov wrote: > revision-id: dfa33fb309d (mariadb-10.6.1-84-gdfa33fb309d) > parent(s): 5cfb31854a4 > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2021-10-19 13:27:19 +0400 > message: > > MDEV-26664 Store UUIDs in a more efficient manner > > UUID values > > llllllll-mmmm-Vhhh-vsss-nnnnnnnnnnnn > > are now stored as > > nnnnnnnnnnnn-vsss-Vhhh-mmmm-llllllll > > inside the record: > > - the groups (segments separated by dash) are reordered right-to-left. > - the bytes inside the groups are not reordered (stored as before, > in big-endian format). > > This provides a better sorting order: the earlier UUID was generated, > the higher it appears in the ORDER BY output. > > Also, this change enables a good key prefix compression, > because the constant part is now in the beginning, while > the non-constant part (the timestamp) is in the end. > > diff --git a/plugin/type_uuid/mysql-test/type_uuid/type_uuid.result b/plugin/type_uuid/mysql-test/type_uuid/type_uuid.result > index 6a08ffe8c23..6e2e1cc87ef 100644 > --- a/plugin/type_uuid/mysql-test/type_uuid/type_uuid.result > +++ b/plugin/type_uuid/mysql-test/type_uuid/type_uuid.result > @@ -596,10 +596,10 @@ INSERT INTO t1 VALUES (CAST(CONCAT('2','0000000-0000-0000-0000-000000000003') AS > SELECT * FROM t1 ORDER BY a; > a > 10000000-0000-0000-0000-000000000001 > -10000000-0000-0000-0000-000000000002 > -10000000-0000-0000-0000-000000000003 > 20000000-0000-0000-0000-000000000001 > +10000000-0000-0000-0000-000000000002 > 20000000-0000-0000-0000-000000000002 > +10000000-0000-0000-0000-000000000003 > 20000000-0000-0000-0000-000000000003 please, add a test for `ORDER BY CONCAT(a)` just to test the lexicographical ordering of UUIDs. Or may be `ORDER BY HEX(a)` ? Or `UNHEX` ? Whatever the recommended approach should be. > DROP TABLE t1; > # > diff --git a/plugin/type_uuid/mysql-test/type_uuid/type_uuid_memory.result b/plugin/type_uuid/mysql-test/type_uuid/type_uuid_memory.result > index 42bb74d4b01..eb7a51895b9 100644 > --- a/plugin/type_uuid/mysql-test/type_uuid/type_uuid_memory.result > +++ b/plugin/type_uuid/mysql-test/type_uuid/type_uuid_memory.result > @@ -25,7 +25,7 @@ a > 00000000-0000-0000-0000-0000000000ff > EXPLAIN SELECT * FROM t1 WHERE a='00000000-0000-0000-0000-0000000000ff'; > id select_type table type possible_keys key key_len ref rows Extra > -1 SIMPLE t1 ref a a 17 const 2 Using where > +1 SIMPLE t1 ref a a 17 const 4 Using where why? > diff --git a/plugin/type_uuid/sql_type_uuid.h b/plugin/type_uuid/sql_type_uuid.h > index 9076c874e98..be9fea8ebc9 100644 > --- a/plugin/type_uuid/sql_type_uuid.h > +++ b/plugin/type_uuid/sql_type_uuid.h > @@ -24,8 +24,162 @@ class UUID: public FixedBinTypeStorage<MY_UUID_SIZE, MY_UUID_STRING_LENGTH> ... > + // Compare two in-memory values > + static int cmp(const LEX_CSTRING &a, const LEX_CSTRING &b) > + { this is way more LoC that I would've used. but ok, it's your code and it's a plugin, so as you like > diff --git a/sql/sql_string.h b/sql/sql_string.h > index fe57c8153bb..795f80c3e08 100644 > --- a/sql/sql_string.h > +++ b/sql/sql_string.h > @@ -484,6 +484,11 @@ class Binary_string: public Sql_alloc > if (str.Alloced_length) > Alloced_length= (uint32) (str.Alloced_length - offset); > } > + LEX_CSTRING to_lex_cstring() const > + { > + LEX_CSTRING tmp= {Ptr, str_length}; > + return tmp; > + } > inline LEX_CSTRING *get_value(LEX_CSTRING *res) may be you could remove get_value()? In a separate commit. the name is quite bad and the method makes rather little sense, especially if there's to_lex_cstring() now. > { > res->str= Ptr; > diff --git a/sql/sql_type_fixedbin.h b/sql/sql_type_fixedbin.h > index c6e3d20bcfa..5141cb9fad4 100644 > --- a/sql/sql_type_fixedbin.h > +++ b/sql/sql_type_fixedbin.h > @@ -154,18 +160,13 @@ class FixedBinTypeBundle > FbtImpl::max_char_length()+1)); > return false; > } > - int cmp(const char *str, size_t length) const > - { > - DBUG_ASSERT(length == sizeof(m_buffer)); > - return memcmp(m_buffer, str, length); > - } > int cmp(const Binary_string &other) const what is this one used for? > { > - return cmp(other.ptr(), other.length()); > + return FbtImpl::cmp(FbtImpl::to_lex_cstring(), other.to_lex_cstring()); > } > int cmp(const Fbt &other) const > { > - return memcmp(m_buffer, other.m_buffer, sizeof(m_buffer)); > + return FbtImpl::cmp(FbtImpl::to_lex_cstring(), other.to_lex_cstring()); > } > }; > > diff --git a/sql/sql_type_fixedbin_storage.h b/sql/sql_type_fixedbin_storage.h > index 0e8ac81ab59..6e18335bd4c 100644 > --- a/sql/sql_type_fixedbin_storage.h > +++ b/sql/sql_type_fixedbin_storage.h > @@ -21,6 +21,38 @@ > format and their own (variable size) canonical string representation. > > Examples are INET6 and UUID types. > + > + The MariaDB server uses three binary representations of a data type: > + > + 1. In-memory binary representation (user visible) > + This representation: > + - can be used in INSERT..VALUES (X'AABBCC') > + - can be used in WHERE conditions: WHERE c1=X'AABBCC' > + - is returned by CAST(x AS BINARY(N)) > + - is returned by Field::val_native() and Item::val_native() > + > + 2. In-record binary representation (user invisible) > + This representation: > + - is used in records (is pointed by Field::ptr) > + - must be comparable by memcmp() > + > + 3. Binlog binary (row) representation > + Usually, for string data types the binlog representation > + is based on the in-record representation with trailing byte compression: > + - trailing space compression for text string data types > + - trailing zero compression for binary string data types > + > + We have to have separate in-memory and in-record representations > + because we use HA_KEYTYPE_BINARY for indexing. The engine API > + does not have a way to pass a comparison function as a parameter. > + > + The default implementation below assumes that: > + - the in-memory and in-record representations are equal > + - the binlog representation is compatible with BINARY(N) > + This is OK for simple data types, like INET6. > + > + Data type implementations that need different representations > + can override the default implementation (like e.g. UUID does). > */ thanks, this is very good > > /***********************************************************************/ > @@ -64,5 +137,37 @@ class FixedBinTypeStorage > return true; > } > > + static ulong KEY_pack_flags(uint column_nr) > + { > + /* > + Return zero by default. A particular data type can override > + this method return some flags, e.g. HA_PACK_KEY to enable > + key prefix compression. what if you change it later, will it make old tables corrupted? > + */ > + return 0; > + } Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 3

Re: [Maria-developers] 4b164f176e6: MDEV-25114 Crash: WSREP: invalid state ROLLED_BACK (FATAL)
by Sergei Golubchik 21 Oct '21

21 Oct '21

Hi, Seppo, Jan! Note, this is 10.2 patch below. > commit 4b164f176e6 > Author: Seppo Jaakola <seppo.jaakola(a)galeracluster.com> > Date: Wed Sep 15 09:16:44 2021 +0300 > > MDEV-25114 Crash: WSREP: invalid state ROLLED_BACK (FATAL) I think this should say MDEV-23328 Server hang due to Galera lock conflict resolution it'd be easier to understand the commit comment and code changes if it'll refer to the correct problem it is fixing. The fix for MDEV-25114 was the previous commit, the one that This reverts commit 29bbcac0ee841faaa68eeb09c86ff825eabbe6b6 it should have MDEV-25114 in the commit comment. > > This patch is the plan D variant for fixing potetial mutex locking "potential" > order exercised by BF aborting and KILL command execution. > > In this approach, KILL command is replicated as TOI operation. > This guarantees total isolation for the KILL command execution > in the first node: there is no concurrent replication applying > and no concurrent DDL executing. Therefore there is no risk of > BF aborting to happen in parallel with KILL command execution > either. Potential mutex deadlocks between the different mutex > access paths with KILL command execution and BF aborting cannot > therefore happen. > > TOI replication is used, in this approach, purely as means > to provide isolated KILL command execution in the first node. > KILL command should not (and must not) be applied in secondary > nodes. In this patch, we make this sure by skipping KILL > execution in secondary nodes, in applying phase, where we > bail out if applier thread is trying to execute KILL command. > This is effective, but skipping the applying of KILL command > could happen much earlier as well. I like the idea of using TOI apparatus to provide total isolation for KILL. Actually replicating it to nodes is kind of crude, it'd be more elegant not to replicate at all. But it'll do. > > This patch also fixes mutex locking order and unprotected > THD member accesses on bf aborting case. We try to hold > THD::LOCK_thd_data during bf aborting. Only case where it > is not possible is at wsrep_abort_transaction before > call wsrep_innobase_kill_one_trx where we take InnoDB > mutexes first and then THD::LOCK_thd_data. > > This will also fix possible race condition during > close_connection and while wsrep is disconnecting > connections. > > Added wsrep_bf_kill_debug test case > > Reviewed-by: Jan Lindström <jan.lindstrom(a)mariadb.com> > > diff --git a/sql/sql_class.cc b/sql/sql_class.cc > index 5ada018e540..05ec9a1c369 100644 > --- a/sql/sql_class.cc > +++ b/sql/sql_class.cc > @@ -1804,11 +1804,11 @@ void THD::awake(killed_state state_to_set) > the Vio might be disassociated concurrently. > */ > > -void THD::disconnect() > +void THD::disconnect_mutexed() should be disconnect_no_mutex, cf. awake_no_mutex, set_killed_no_mutex. > { > Vio *vio= NULL; > > - mysql_mutex_lock(&LOCK_thd_data); > + mysql_mutex_assert_owner(&LOCK_thd_data); > > set_killed(KILL_CONNECTION); > > diff --git a/sql/sql_parse.cc b/sql/sql_parse.cc > index 3e1f248b082..2bec9c6b6cd 100644 > --- a/sql/sql_parse.cc > +++ b/sql/sql_parse.cc > @@ -9069,6 +9069,18 @@ static > void sql_kill(THD *thd, longlong id, killed_state state, killed_type type) > { > uint error; > +#ifdef WITH_WSREP > + if (WSREP(thd)) > + { > + WSREP_DEBUG("sql_kill called"); > + if (thd->wsrep_applier) > + { > + WSREP_DEBUG("KILL in applying, bailing out here"); > + return; > + } > + WSREP_TO_ISOLATION_BEGIN(WSREP_MYSQL_DB, NULL, NULL) when can it fail and jump to wsrep_error_label? under what conditions? > + } > +#endif /* WITH_WSREP */ > if (!(error= kill_one_thread(thd, id, state, type))) > { > if (!thd->killed) > diff --git a/sql/wsrep_mysqld.cc b/sql/wsrep_mysqld.cc > index e60100e2e90..00a6dfe2f8a 100644 > --- a/sql/wsrep_mysqld.cc > +++ b/sql/wsrep_mysqld.cc > @@ -835,13 +835,25 @@ void wsrep_thr_init() > DBUG_VOID_RETURN; > } > > +/* This is wrapper for wsrep_break_lock in thr_lock.c */ > +static int wsrep_thr_abort_thd(void *bf_thd_ptr, void *victim_thd_ptr, my_bool signal) > +{ > + THD* victim_thd= (THD *) victim_thd_ptr; > + /* We need to lock THD::LOCK_thd_data to protect victim > + from concurrent usage or disconnect or delete. */ How do you know victim_thd wasn't deleted before you locked LOCK_thd_data below? > + mysql_mutex_lock(&victim_thd->LOCK_thd_data); > + int res= wsrep_abort_thd(bf_thd_ptr, victim_thd_ptr, signal); > + mysql_mutex_unlock(&victim_thd->LOCK_thd_data); > + return res; > +} > > diff --git a/storage/innobase/handler/ha_innodb.cc b/storage/innobase/handler/ha_innodb.cc > index 1f666f19d8a..921b8282e45 100644 > --- a/storage/innobase/handler/ha_innodb.cc > +++ b/storage/innobase/handler/ha_innodb.cc > @@ -19739,27 +19739,65 @@ wsrep_abort_transaction( > { > DBUG_ENTER("wsrep_abort_transaction"); > > + ut_ad(bf_thd); > + ut_ad(victim_thd); > trx_t* victim_trx= thd_to_trx(victim_thd); > - trx_t* bf_trx = (bf_thd) ? thd_to_trx(bf_thd) : NULL; > + trx_t* bf_trx= thd_to_trx(bf_thd); > > > if (victim_trx) { > + wsrep_thd_UNLOCK(victim_thd); what keeps victim_trx from disappearing here? > lock_mutex_enter(); > trx_mutex_enter(victim_trx); > wsrep_innobase_kill_one_trx(bf_thd, bf_trx, victim_trx, signal); > lock_mutex_exit(); > trx_mutex_exit(victim_trx); > wsrep_srv_conc_cancel_wait(victim_trx); > + wsrep_thd_LOCK(victim_thd); > DBUG_VOID_RETURN; > } else { > - wsrep_thd_LOCK(victim_thd); > wsrep_thd_set_conflict_state(victim_thd, MUST_ABORT); > - wsrep_thd_UNLOCK(victim_thd); > wsrep_thd_awake(victim_thd, signal); > } Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 20

Re: [Maria-developers] 688ad5ed67f: MDEV-26830: Wrong ROW_NUMBER in diagnostics upon INSERT IGNORE with CHECK violation
by Sergei Golubchik 18 Oct '21

18 Oct '21

Hi, Rucha! On Oct 18, Rucha Deodhar wrote: > revision-id: 688ad5ed67f (mariadb-10.6.1-138-g688ad5ed67f) > parent(s): c27f04ede5a > author: Rucha Deodhar > committer: Rucha Deodhar > timestamp: 2021-10-18 13:40:56 +0530 > message: > > MDEV-26830: Wrong ROW_NUMBER in diagnostics upon INSERT IGNORE with > CHECK violation > > Analysis: When there is constraint fail we return non-zero value for > view_check_option(). So we continue the loop which doesn't increment the > counter because it increments at the end of the loop. > Fix: Increment m_current_row_for_warning() at the beginning of loop. This > will also fix similar bugs if any, about counter not incrementing > correctly because of continue. > > diff --git a/sql/sql_insert.cc b/sql/sql_insert.cc > index 3dff6722a3d..11289779f7c 100644 > --- a/sql/sql_insert.cc > +++ b/sql/sql_insert.cc > @@ -1007,8 +1007,10 @@ bool mysql_insert(THD *thd, TABLE_LIST *table_list, > goto values_loop_end; > } > > + thd->get_stmt_da()->reset_current_row_for_warning(0); This changes the behavior in array binding case, let's not do it in this commit. Better change the previous reset_current_row_for_warning() to reset to 0, not to 1. The rest is ok. > while ((values= its++)) > { > + thd->get_stmt_da()->inc_current_row_for_warning(); > if (fields.elements || !value_count) > { > /* > @@ -1125,7 +1127,6 @@ bool mysql_insert(THD *thd, TABLE_LIST *table_list, > if (unlikely(error)) > break; > info.accepted_rows++; > - thd->get_stmt_da()->inc_current_row_for_warning(); > } > its.rewind(); > iteration++; > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] a895fce3277: MDEV-26842: MDEV-26842: ROW_NUMBER is not set and differs from the message upon WARN_DATA_TRUNCATED produced by inplace ALTER
by Sergei Golubchik 17 Oct '21

17 Oct '21

Hi, Rucha! On Oct 17, Rucha Deodhar wrote: > revision-id: a895fce3277 (mariadb-10.6.1-138-ga895fce3277) > parent(s): c27f04ede5a > author: Rucha Deodhar > committer: Rucha Deodhar > timestamp: 2021-10-17 17:59:11 +0530 > message: > > MDEV-26842: MDEV-26842: ROW_NUMBER is not set and differs from the message > upon WARN_DATA_TRUNCATED produced by inplace ALTER > > Analysis: When row number is passed as parameter to set_warning() it is only > used for error/warning text but m_current_row_for_warning is not updated. > Hence default value of m_current_row_for_warning is assumed. > Fix: update m_current_row_for_warning when error/warning occurs. > > diff --git a/sql/field.cc b/sql/field.cc > index a5cd595f05b..048798ea509 100644 > --- a/sql/field.cc > +++ b/sql/field.cc > @@ -11086,6 +11086,10 @@ Field::set_warning(Sql_condition::enum_warning_level level, uint code, > will have table == NULL. > */ > THD *thd= get_thd(); > + ulong row_count= current_row ? current_row > + : thd->get_stmt_da()->current_row_for_warning(); > + > + thd->get_stmt_da()->reset_current_row_for_warning(row_count); Isn't it a bit confusing, why do you set current_row_for_warning to its own actual value? May be it'd be clearer to if (current_row) thd->get_stmt_da()->reset_current_row_for_warning(current_row); And also, please, mention either here or in the commit comment that it's for INPLACE alter, where the server cannot know what row has generated the warning, so the value of current_row is supplied by the engine. > if (thd->count_cuted_fields > CHECK_FIELD_EXPRESSION) > { > thd->cuted_fields+= cut_increment; > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 351a0a7d6b7: MDEV-26841: ROW_NUMBER is not set and differs from the message upon ER_WRONG_VALUE_COUNT_ON_ROW for the 1st row
by Sergei Golubchik 17 Oct '21

17 Oct '21

Hi, Rucha! On Oct 17, Rucha Deodhar wrote: > revision-id: 351a0a7d6b7 (mariadb-10.6.1-138-g351a0a7d6b7) > parent(s): c27f04ede5a > author: Rucha Deodhar > committer: Rucha Deodhar > timestamp: 2021-10-17 16:24:47 +0530 > message: > > MDEV-26841: ROW_NUMBER is not set and differs from the message upon > ER_WRONG_VALUE_COUNT_ON_ROW for the 1st row > > diff --git a/sql/sql_insert.cc b/sql/sql_insert.cc > index 3dff6722a3d..9e81c33ff8f 100644 > --- a/sql/sql_insert.cc > +++ b/sql/sql_insert.cc > @@ -229,6 +229,8 @@ static int check_insert_fields(THD *thd, TABLE_LIST *table_list, > } > if (values.elements != table->s->visible_fields) > { > + if (values.elements > table->s->visible_fields) why do you need an if() here? > + thd->get_stmt_da()->inc_current_row_for_warning(); > my_error(ER_WRONG_VALUE_COUNT_ON_ROW, MYF(0), 1L); > DBUG_RETURN(-1); > } Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] fcf2c5fff14: MDEV-26299: Some views force server (and mysqldump) to generate invalid SQL for their definitions
by Sergei Golubchik 16 Oct '21

16 Oct '21

Hi, Oleksandr! On Oct 16, Oleksandr Byelkin wrote: > revision-id: fcf2c5fff14 (mariadb-10.2.40-78-gfcf2c5fff14) > parent(s): d5a15f04f4a > author: Oleksandr Byelkin > committer: Oleksandr Byelkin > timestamp: 2021-10-01 14:46:22 +0200 > message: > > MDEV-26299: Some views force server (and mysqldump) to generate invalid SQL for their definitions > > Do not print illegal table field names for non-top-level SELECT list, > they will not be refered in any case but create problem for parsing > of printed result. > > diff --git a/sql/sql_select.cc b/sql/sql_select.cc > index ff584e936b7..ebee9c85791 100644 > --- a/sql/sql_select.cc > +++ b/sql/sql_select.cc > @@ -25797,6 +25797,11 @@ void st_select_lex::print(THD *thd, String *str, enum_query_type query_type) > > //Item List > bool first= 1; > + /* > + outer_select() can not be used here because it is for name resolution > + and will return NULL at any end of name resolution chain (view/derived) > + */ > + bool top_level= (get_master()->get_master() == 0); > List_iterator_fast<Item> it(item_list); > Item *item; > while ((item= it++)) > @@ -25806,7 +25811,8 @@ void st_select_lex::print(THD *thd, String *str, enum_query_type query_type) > else > str->append(','); > > - if (is_subquery_function() && item->is_autogenerated_name) > + if ((is_subquery_function() && item->is_autogenerated_name) || > + !item->name) Where can item->name be NULL? I thought the either specified by the user or autogenerated, but never NULL. > { > /* > Do not print auto-generated aliases in subqueries. It has no purpose > @@ -25815,7 +25821,26 @@ void st_select_lex::print(THD *thd, String *str, enum_query_type query_type) > item->print(str, query_type); > } > else > - item->print_item_w_name(str, query_type); > + { > + /* > + Despite presence of item->name_length it is incorrect in many cases > + till we start using LEX_STRING for the name. > + */ > + size_t len= strlen(item->name); > + > + /* > + Do not print illegal names (if it is not top level SELECT). > + Top level view checked (and correct name are assigned), > + other cases of top level SELECT are not important, because > + it is not "table field". > + */ > + if (top_level || > + !item->is_autogenerated_name || > + !check_table_name(item->name, len, TRUE)) why check_table_name() if it isn't a table? it's an incorrect check to use for columns. > + item->print_item_w_name(str, query_type); > + else > + item->print(str, query_type); > + } > } > > /* > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 2

Re: [Maria-developers] baaaa971662: MDEV-22883: Assertion `!parse_error || lex.sphead == 0' failed in Table_triggers_list::check_n_load | SIGSEGV in strxmov
by Sergei Golubchik 15 Oct '21

15 Oct '21

Hi, Oleksandr! On Oct 15, Oleksandr Byelkin wrote: > revision-id: baaaa971662 (mariadb-10.2.40-61-gbaaaa971662) > parent(s): dce490e9d4d > author: Oleksandr Byelkin > committer: Oleksandr Byelkin > timestamp: 2021-09-19 19:23:49 +0200 > message: > > MDEV-22883: Assertion `!parse_error || lex.sphead == 0' failed in Table_triggers_list::check_n_load | SIGSEGV in strxmov > > Abort execution of SHOW in case of fatal error. Does this patch fixes all test cases from the bug report? There are quite a few. > diff --git a/sql/sql_show.cc b/sql/sql_show.cc > index 721bb053343..105b2719882 100644 > --- a/sql/sql_show.cc > +++ b/sql/sql_show.cc > @@ -4849,7 +4849,8 @@ static int fill_schema_table_from_frm(THD *thd, TABLE *table, > */ > DBUG_ASSERT(thd->open_tables == NULL); > thd->mdl_context.rollback_to_savepoint(open_tables_state_backup->mdl_system_tables_svp); > - thd->clear_error(); > + if (!thd->is_fatal_error) > + thd->clear_error(); Hmm. THD::clear_error() starts with a comment: /** Clear the current error, if any. We do not clear is_fatal_error or is_fatal_sub_stmt_error since we assume this is never called if the fatal error is set. Unfortunately, it doesn't have an assert to go with this, otherwise all these bugs would've probably failed identically and early. Please add DBUG_ASSERT(!is_fatal_error); DBUG_ASSERT(!is_fatal_sub_stmt_error); or something like that to THD::clear_error() > return res; > } > > @@ -5064,7 +5065,7 @@ int get_all_tables(THD *thd, TABLE_LIST *tables, COND *cond) > continue; > } > > - if (thd->killed == ABORT_QUERY) > + if (thd->killed == ABORT_QUERY || thd->is_fatal_error) > { > error= 0; > goto err; that's kind of weird. Why does it continue executing if the statement is killed? Perhaps it'd better be if (thd->killed || thd->is_fatal_error) { if (thd->killed == ABORT_QUERY) error= 0; goto err; Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 2f812e800c0: MDEV-22447: SIGSEGV in Item::val_uint on EXPLAIN
by Sergei Golubchik 15 Oct '21

15 Oct '21

Hi, Oleksandr! On Oct 15, Oleksandr Byelkin wrote: > revision-id: 2f812e800c0 (mariadb-10.2.40-57-g2f812e800c0) > parent(s): 696de6d06c0 > author: Oleksandr Byelkin > committer: Oleksandr Byelkin > timestamp: 2021-09-16 14:12:48 +0200 > message: > > MDEV-22447: SIGSEGV in Item::val_uint on EXPLAIN > > take into account possible NULL > > diff --git a/sql/sql_select.cc b/sql/sql_select.cc > index ff584e936b7..d1b738e94a0 100644 > --- a/sql/sql_select.cc > +++ b/sql/sql_select.cc > @@ -389,7 +389,7 @@ bool handle_select(THD *thd, LEX *lex, select_result *result, > ER_QUERY_EXCEEDED_ROWS_EXAMINED_LIMIT, > ER_THD(thd, ER_QUERY_EXCEEDED_ROWS_EXAMINED_LIMIT), > thd->accessed_rows_and_keys, > - thd->lex->limit_rows_examined->val_uint()); > + thd->lex->get_limit_rows_examined()); > thd->abort_on_warning= saved_abort_on_warning; > thd->reset_killed(); > } How comes, it gets into ER_QUERY_EXCEEDED_ROWS_EXAMINED_LIMIT if limit_rows_examined is NULL? Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] cd4cac2b95a: MDEV-17964: Assertion `status == 0' failed in add_role_user_mapping_action
by Sergei Golubchik 15 Oct '21

15 Oct '21

Hi, Vicențiu! The patch is ok, please push. But given the time it took us to understand the issue, I'm afraid that simply stating that "HASH implementation shuffles elements around" is not enough. Better to explain how elements are shuffled around. For example: in this case the hash has 131 element, DROP ROLE removes the element [119]. Its [119]->next was element [129], so [129] is moved to [119]. Now we need to compact the hash, removing the last element [130]. It gets one bit off its hash value and becomes element [2]. The existing element [2] is moved to [129], and old [130] is moved to [2]. We cannot simply move [130] to [129] and make [2]->next=130, it won't work if [2] is itself in the collision list and doesn't belong in [2]. On Oct 15, Vicențiu Ciorbaru wrote: > revision-id: cd4cac2b95a (mariadb-10.2.40-91-gcd4cac2b95a) > parent(s): 2bb8d7c2f36 > author: Vicențiu Ciorbaru > committer: Vicențiu Ciorbaru > timestamp: 2021-10-13 13:28:50 +0300 > message: > > MDEV-17964: Assertion `status == 0' failed in add_role_user_mapping_action > > This happens upon CREATE USER and DROP ROLE. > > The underlying problem is that our HASH implementation shuffles > elements around when performing an update or delete. This means that > when doing a scan through the HASH table by index, in search of > elements to delete or update one must restart the scan to make sure > nothing is missed if at least one delete / update happened. > > The handle_grant_struct code assumed that it is safe to continue by > only reexamining the currently modified / deleted element index, but > that is not true. > > Missing to delete an element in the hash triggered the assertion in > the test case. DROP ROLE would not clear all necessary role->role or > role->user mappings. > > To fix the problem we ensure that the scan is restarted, only if an > element was deleted / updated, similar to how bubble-sort keeps > sorting until it finds no more elements to swap. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 97458a7767b: MDEV-26832: ROW_NUMBER in SIGNAL/RESIGNAL causes a syntax error
by Sergei Golubchik 15 Oct '21

15 Oct '21

Hi, Rucha! Looks quite good. A couple of small comments, see below. After you fix them, no need to ask for a new review, just push into bb-10.7-row_number On Oct 15, Rucha Deodhar wrote: > revision-id: 97458a7767b (mariadb-10.6.1-131-g97458a7767b) > parent(s): a6cf8b34a83 > author: Rucha Deodhar > committer: Rucha Deodhar > timestamp: 2021-10-15 01:54:51 +0530 > message: > > MDEV-26832: ROW_NUMBER in SIGNAL/RESIGNAL causes a syntax error > > Analysis: Parser was missing ROW_NUMBER as syntax for SIGNAL and RESIGNAL. > Fix: Fix parser and fix how m_row_number is copied like other attributes > to avoid ROW_NUMBER from assuming default value. > > diff --git a/mysql-test/main/get_diagnostics.test b/mysql-test/main/get_diagnostics.test > index 83ea1dee343..186d1bfc438 100644 > --- a/mysql-test/main/get_diagnostics.test > +++ b/mysql-test/main/get_diagnostics.test > @@ -1552,3 +1552,53 @@ GET DIAGNOSTICS CONDITION 2 @rnum = ROW_NUMBER, @msg = MESSAGE_TEXT, @err = MYSQ ... > + > +CREATE PROCEDURE resignal_syntax() > +BEGIN > + DECLARE CONTINUE HANDLER > + FOR 1146 > + BEGIN > + RESIGNAL SET > + MESSAGE_TEXT = '`temptab` does not exist', ROW_NUMBER= 105; > + END; > + SELECT `c` FROM `temptab`; > +END| > + > +DELIMITER ;| > + > +--error ER_NO_SUCH_TABLE > +CALL resignal_syntax(); You test that the above isn't a syntax error. But you also need to verify that row_number was, indeed, set. Like, do GET DIAGNOSTICS and print the ROW_NUMBER. > + > +DROP PROCEDURE resignal_syntax; > diff --git a/sql/sql_error.cc b/sql/sql_error.cc > index a2178831697..84065ddce15 100644 > --- a/sql/sql_error.cc > +++ b/sql/sql_error.cc > @@ -190,6 +190,11 @@ static void copy_string(MEM_ROOT *mem_root, String* dst, const String* src) > dst->length(0); > } > > +void copy_value(ulong *dst, const ulong *src) > +{ > + memcpy(dst, src, sizeof(*src)); > +} > + > void > Sql_condition::copy_opt_attributes(const Sql_condition *cond) > { > @@ -204,6 +209,7 @@ Sql_condition::copy_opt_attributes(const Sql_condition *cond) > copy_string(m_mem_root, & m_table_name, & cond->m_table_name); > copy_string(m_mem_root, & m_column_name, & cond->m_column_name); > copy_string(m_mem_root, & m_cursor_name, & cond->m_cursor_name); > + copy_value(& m_row_number, & cond->m_row_number); That makes no sense. Other values were strings and had to be strdup-ed, but that's a number, you can simply do m_row_number= cond->m_row_number; > } Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 6ec64caec5d: MDEV-26650: Failed ALTER USER/GRANT statement removes the password from the cache
by Sergei Golubchik 15 Oct '21

15 Oct '21

Hi, Oleksandr! All good, ok to push. Just add a comment right after memcpy, that /* if acl_user->nauth was >= nauth, then we've just updated the actual acl_user, not the copy. We rely on the fact that acl_update_user() cannot fail anywhere below. */ On Oct 15, Oleksandr Byelkin wrote: > revision-id: 6ec64caec5d (mariadb-10.4.21-64-g6ec64caec5d) > parent(s): a736a3174a4 > author: Oleksandr Byelkin > committer: Oleksandr Byelkin > timestamp: 2021-10-14 16:19:09 +0200 > message: > > MDEV-26650: Failed ALTER USER/GRANT statement removes the password from the cache > > Starting from 10.4 AUTH is not part of ACL_USER so changes have to be done > over a copy, and bring in the cache only in case of success. > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] MDEV-26742 Assertion `field->type_handler() == this' failed...
by Sergei Golubchik 12 Oct '21

12 Oct '21

Hi, Alexander! See comments below > diff --git a/plugin/type_inet/mysql-test/type_inet/type_inet6_innodb.test b/plugin/type_inet/mysql-test/type_inet/type_inet6_innodb.test > index dd6049abbf3..3f3a6fdac2c 100644 > --- a/plugin/type_inet/mysql-test/type_inet/type_inet6_innodb.test > +++ b/plugin/type_inet/mysql-test/type_inet/type_inet6_innodb.test > @@ -12,6 +12,30 @@ > SET default_storage_engine=InnoDB; > --source type_inet6_engines.inc > > +--echo # > +--echo # MDEV-26742 Assertion `field->type_handler() == this' failed in FixedBinTypeBundle<NATIVE_LEN, MAX_CHAR_LEN>::Type_handler_fbt::stored_field_cmp_to_item > +--echo # > + > +CREATE TABLE t1 (pk inet6, c text) engine=myisam; > +INSERT INTO t1 VALUES ('::',1); > +CREATE TABLE t2 (d text, KEY (d)) engine=innodb ; > +INSERT INTO t2 VALUES (2); > +--error ER_TRUNCATED_WRONG_VALUE > +UPDATE t2 JOIN t1 ON ( t1.pk > t2.d) SET t1.c = 1; > +SET sql_mode=''; > +UPDATE t2 JOIN t1 ON ( t1.pk > t2.d) SET t1.c = 1; > +SET sql_mode=DEFAULT; > +SELECT * FROM t1; > +SELECT * FROM t2; > +DROP TABLE t1, t2; > + > +CREATE TABLE t1 (pk inet6, c text) engine=myisam; > +INSERT INTO t1 VALUES ('::',1); > +CREATE TABLE t2 ( d text, KEY (d)) engine=innodb ; I don't get it, why do you recreate tables with exactly the same definition and exactly the same content? > +INSERT INTO t2 VALUES (2); > +SELECT * FROM t2 JOIN t1 ON ( t1.pk > t2.d); > +DROP TABLE t1, t2; > + > > --echo # > --echo # End of 10.5 tests > diff --git a/plugin/type_inet/mysql-test/type_inet/type_inet6_myisam.test b/plugin/type_inet/mysql-test/type_inet/type_inet6_myisam.test > index c5183f01cf0..1ccd654181c 100644 > --- a/plugin/type_inet/mysql-test/type_inet/type_inet6_myisam.test > +++ b/plugin/type_inet/mysql-test/type_inet/type_inet6_myisam.test > @@ -10,6 +10,21 @@ > SET default_storage_engine=MyISAM; > --source type_inet6_engines.inc > > +--echo # > +--echo # MDEV-26742 Assertion `field->type_handler() == this' failed in FixedBinTypeBundle<NATIVE_LEN, MAX_CHAR_LEN>::Type_handler_fbt::stored_field_cmp_to_item > +--echo # > + > +CREATE TABLE t1 (c varchar(64), key(c)) engine=myisam; > +INSERT INTO t1 VALUES ('::1'),('::2'); > +SELECT * FROM t1 WHERE c>CAST('::1' AS INET6); > +DROP TABLE t1; > + > +CREATE TABLE t1 (c varchar(64), key(c)) engine=myisam; > +INSERT INTO t1 VALUES ('::1'),('0::1'); also don't see much point in recreating > +SELECT * FROM t1 WHERE c=CAST('::1' AS INET6); > +EXPLAIN SELECT * FROM t1 WHERE c=CAST('::1' AS INET6); > +DROP TABLE t1; > + > > --echo # > --echo # End of 10.5 tests > diff --git a/plugin/type_uuid/mysql-test/type_uuid/type_uuid_innodb.test b/plugin/type_uuid/mysql-test/type_uuid/type_uuid_innodb.test > index 062e25d366b..27e3214af15 100644 > --- a/plugin/type_uuid/mysql-test/type_uuid/type_uuid_innodb.test > +++ b/plugin/type_uuid/mysql-test/type_uuid/type_uuid_innodb.test > @@ -12,5 +12,27 @@ SET default_storage_engine=InnoDB; > --source type_uuid_engines.inc > > --echo # > ---echo # End of 10.5 tests > +--echo # MDEV-26742 Assertion `field->type_handler() == this' failed in FixedBinTypeBundle<NATIVE_LEN, MAX_CHAR_LEN>::Type_handler_fbt::stored_field_cmp_to_item > +--echo # > + > +CREATE TABLE t1 ( pk uuid, c text) engine=myisam; > +INSERT INTO t1 VALUES ('00000000-0000-0000-0000-000000000000',1); > +CREATE TABLE t2 ( d text, KEY (d)) engine=innodb ; > +INSERT INTO t2 VALUES (2); > +--error ER_TRUNCATED_WRONG_VALUE > +UPDATE t2 JOIN t1 ON ( t1.pk > t2.d) SET t1.c = 1; > +SET sql_mode=''; > +UPDATE t2 JOIN t1 ON ( t1.pk > t2.d) SET t1.c = 1; > +SET sql_mode=DEFAULT; > +DROP TABLE t1, t2; > + > +CREATE TABLE t1 (pk uuid, c text) engine=myisam; > +INSERT INTO t1 VALUES ('00000000-0000-0000-0000-000000000000',1); > +CREATE TABLE t2 ( d text, KEY (d)) engine=innodb ; > +INSERT INTO t2 VALUES (2); same here > +SELECT * FROM t2 JOIN t1 ON ( t1.pk > t2.d); > +DROP TABLE t1, t2; > + > +--echo # > +--echo # End of 10.7 tests > --echo # > diff --git a/sql/field.cc b/sql/field.cc > index 46a3a1deea3..fc3456e2ccb 100644 > --- a/sql/field.cc > +++ b/sql/field.cc > @@ -7368,7 +7368,9 @@ bool > Field_longstr::cmp_to_string_with_same_collation(const Item_bool_func *cond, > const Item *item) const > { > + Type_handler_hybrid_field_type cmp(type_handler_for_comparison()); Can this be anything else but type_handler_long_blob ? if not, then I'd suggest Type_handler_hybrid_field_type cmp(type_handler_long_blob); DBUG_ASSERT(type_handler_for_comparison() == &type_handler_long_blob); > + return !cmp.aggregate_for_comparison(item->type_handler_for_comparison()) && > + cmp.type_handler() == type_handler_for_comparison() && > - return item->cmp_type() == STRING_RESULT && > charset() == cond->compare_collation(); This, basically, means, that cmp_type() is now meaningless, one cannot use it anymore? There's no INET6_RESULT and UUID_RESULT for comparison, so what's the point of cmp_type() now? One more question. What type do we have where you need to aggregate here and simple item->type_handler_for_comparison() != &type_handler_long_blob won't do? > } > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 4

[Maria-developers] How can I separate networking test from mysql_test
by Lukas Javorsky 11 Oct '21

11 Oct '21

Hello there, I've open this topic on the maria-discuss mailing list, but, I didn't get any response. Maybe this mailing list could be a better fit for this request. > Hi guys, > I've found few particular tests from the test suite that is failing due to the OpenSSL-3 version and I want to separate them and give them to the OpenSSL team in Red Hat. > > The idea behind this is to give them a set of commands/config options (reproducer) for each test and they will try to find out what's the exact problem there and help us develop the patch. > > The tests that I'm interested in are (there are more of them, I just need to see how to reproduce few of them and I'll do the same for the other ones as well): > encryption.corrupted_during_recovery > encryption.innodb-checksum-algorithm > encryption.innodb-discard-import-change > encryption.innodb_page_encryption_key_change > binlog_encryption.rpl_loaddata_local > binlog_encryption.rpl_parallel_free_deferred_event > > IMPORTANT: These tests are failing only with the 'cbc' cipher in use, the 'ctr' is passing. > > Could you please help me with how to make a reproducer for each test, so I can create them and give them to the OpenSSL team? > > Thank you for any help you can provide > Lukas -- S pozdravom/ Best regards Lukáš Javorský Associate Software Engineer, Core service - Databases Red Hat <https://www.redhat.com> Purkyňova 115 (TPB-C) 612 00 Brno - Královo Pole ljavorsk(a)redhat.com <https://www.redhat.com>

1 0

Re: [Maria-developers] dcbdec60893: MDEV-26650: Failed ALTER USER/GRANT statement removes the password from the cache
by Sergei Golubchik 07 Oct '21

07 Oct '21

Hi, Oleksandr! Ok to push! A couple of typos below On Oct 07, Oleksandr Byelkin wrote: > revision-id: dcbdec60893 (mariadb-10.4.21-56-gdcbdec60893) > parent(s): a75813d4673 > author: Oleksandr Byelkin > committer: Oleksandr Byelkin > timestamp: 2021-10-07 11:39:02 +0200 > message: > > MDEV-26650: Failed ALTER USER/GRANT statement removes the password from the cache > > Starting from 10.4 AUTH is not part of ACL_USER so have to be safely copied. > Patch made in line with preriouse code ideas. "previous" > diff --git a/sql/sql_acl.cc b/sql/sql_acl.cc > index 0e2c044e3a1..cef51038540 100644 > --- a/sql/sql_acl.cc > +++ b/sql/sql_acl.cc > @@ -188,6 +188,9 @@ class ACL_USER_PARAM > class ACL_USER :public ACL_USER_BASE, > public ACL_USER_PARAM > { > +protected: > + // default shellow copy constructor (protect from non-authorized usage) "shallow" > + ACL_USER(const ACL_USER &)= default; > public: Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

[Maria-developers] INSTEAD OF triggers
by Alexandre hadjinlian guerra 05 Oct '21

05 Oct '21

Hello Im looking forward to implement this feature in MariaDB, and will look after trigger definition code/modules in MariaDB, but any wisdom that could be shared to speed up my efforts would be greatly appreciated. The idea is to provide the same functionality as describe here https://aws.amazon.com/pt/blogs/database/implementing-table-partitioning-in… Thanks

3 2

Re: [Maria-developers] 4156071ebe2: MDEV-26606: ERROR_INDEX property value isn't passed from inside a stored procedure
by Sergei Golubchik 28 Sep '21

28 Sep '21

Hi, Rucha! On Sep 28, Rucha Deodhar wrote: > revision-id: 4156071ebe2 (mariadb-10.6.1-91-g4156071ebe2) > parent(s): 8dd4794c4e1 > author: Rucha Deodhar > committer: Rucha Deodhar > timestamp: 2021-09-26 19:58:33 +0530 > message: > > MDEV-26606: ERROR_INDEX property value isn't passed from inside a stored > procedure > > Analysis: m_current_row_for_warning is reset to 1 during cleanup phase of > stored procedure. When we perform a copy because some statement of procedure > created warning, this reset value os passed to push _warning(). > Hence the output is always 1. > Fix: Add a parameter in relevant functions to pass correct value of > error index and don't use m_current_row_for_warning directly. first thought about test cases, ERROR_INDEX should show the same row number as in a warning, so let's check whether warning is correct or not too. In this case we have CREATE OR REPLACE TABLE t1 (pk TINYINT PRIMARY KEY); CREATE OR REPLACE PROCEDURE sp(a INT) INSERT INTO t1 VALUES (2),(a); INSERT INTO t1 VALUES(1); --error ER_WARN_DATA_OUT_OF_RANGE CALL sp(100000); prints ERROR 22003: Out of range value for column 'a' at row 2 which is fine. Would be good to have it in the test. Otherwise looks good. One question below. > diff --git a/sql/sql_class.cc b/sql/sql_class.cc > index 31f5faa72ed..4d7093763ae 100644 > --- a/sql/sql_class.cc > +++ b/sql/sql_class.cc > @@ -1149,7 +1149,8 @@ Sql_condition* THD::raise_condition(uint sql_errno, > if (likely(!(is_fatal_error && (sql_errno == EE_OUTOFMEMORY || > sql_errno == ER_OUTOFMEMORY)))) > { > - cond= da->push_warning(this, sql_errno, sqlstate, level, ucid, msg); > + cond= da->push_warning(this, sql_errno, sqlstate, level, ucid, msg, > + da->current_row_for_warning()); > } > DBUG_RETURN(cond); > } > diff --git a/sql/sql_error.cc b/sql/sql_error.cc > index 7b6e6a83d2f..ca9da3dbf89 100644 > --- a/sql/sql_error.cc > +++ b/sql/sql_error.cc > @@ -663,7 +663,8 @@ void Warning_info::reserve_space(THD *thd, uint count) > > Sql_condition *Warning_info::push_warning(THD *thd, > const Sql_condition_identity *value, > - const char *msg) > + const char *msg, > + ulong current_error_index) > { > Sql_condition *cond= NULL; > > @@ -673,7 +674,7 @@ Sql_condition *Warning_info::push_warning(THD *thd, > m_warn_list.elements() < thd->variables.max_error_count) > { > cond= new (& m_warn_root) Sql_condition(& m_warn_root, *value, msg, > - m_current_row_for_warning); > + current_error_index); > if (cond) > m_warn_list.push_back(cond); > } > @@ -689,7 +690,8 @@ Sql_condition *Warning_info::push_warning(THD *thd, > const Sql_condition *sql_condition) > { > Sql_condition *new_condition= push_warning(thd, sql_condition, > - sql_condition->get_message_text()); > + sql_condition->get_message_text(), > + sql_condition->m_error_index); > > if (new_condition) > new_condition->copy_opt_attributes(sql_condition); > diff --git a/sql/sql_error.h b/sql/sql_error.h > index b6ed7f5061a..a3e00e8419d 100644 > --- a/sql/sql_error.h > +++ b/sql/sql_error.h > @@ -747,7 +747,8 @@ class Warning_info > */ > Sql_condition *push_warning(THD *thd, > const Sql_condition_identity *identity, > - const char* msg); > + const char* msg, > + ulong current_error_index); > > /** > Add a new SQL-condition to the current list and increment the respective > @@ -1178,10 +1179,12 @@ class Diagnostics_area: public Sql_state_errno, > const char* sqlstate, > Sql_condition::enum_warning_level level, > const Sql_user_condition_identity &ucid, > - const char* msg) > + const char* msg, > + ulong current_error_index) > { > Sql_condition_identity tmp(sql_errno_arg, sqlstate, level, ucid); > - return get_warning_info()->push_warning(thd, &tmp, msg); > + return get_warning_info()->push_warning(thd, &tmp, msg, > + current_error_index); > } > > Sql_condition *push_warning(THD *thd, > @@ -1191,7 +1194,7 @@ class Diagnostics_area: public Sql_state_errno, > const char* msg) > { > return push_warning(thd, sqlerrno, sqlstate, level, > - Sql_user_condition_identity(), msg); > + Sql_user_condition_identity(), msg, 0); what is this push_warning used for? does it never need a proper current_error_index? > } > void mark_sql_conditions_for_removal() > { get_warning_info()->mark_sql_conditions_for_removal(); } > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] INSTEAD OF triggers
by Daniel Black 27 Sep '21

27 Sep '21

Opps, meant to send to list too. On Sat, Sep 25, 2021 at 5:28 PM Daniel Black <daniel(a)mariadb.org> wrote: > > Hi Alexandre, > > Given the aws blog is written around that Oracle SE doesn't have > partitions and EE does, does it still make sense with MariaDB (that > includes partitions as standard - > https://mariadb.com/kb/en/partitioning-tables/)? > > Take a look at the way partitions are implemented, they may offer what > you plan to write in a slightly different way. > > If you are sure you want to embark on this journey, there is a > feature request for this issue > https://jira.mariadb.org/browse/MDEV-12151 > > Recommend starting by describing exactly what you plan to implement in > this issue. Reference existing implementations like > https://www.postgresql.org/docs/current/sql-createtrigger.html make > notes of differences and what you plan to cover (and not cover) > > I assume you've come across the generic > https://mariadb.org/get-involved/getting-started-for-developers/ . > Start with a fork from the 10.7 branch > > First bit is sql/sql_yacc.yy trg_action_time that is the parsing syntax. > > If you `git grep TRG_ACTION_AFTER` the code you should contain the > places that will need a TRG_ACTION_INSTEAD_OF handling. > > Also look at the set of tests in $ find mysql-test -name \*trigger\* > > Any problems, find us on > https://mariadb.zulipchat.com/#narrow/stream/118759-general for some > more realtime help. > > On Sat, Sep 25, 2021 at 1:44 AM Alexandre hadjinlian guerra > <alexhguerra(a)gmail.com> wrote: > > > > Hello > > > > Im looking forward to implement this feature in MariaDB, and will look after trigger definition code/modules in MariaDB, but any wisdom that could be shared to speed up my efforts would be greatly appreciated. > > > > The idea is to provide the same functionality as describe here > > https://aws.amazon.com/pt/blogs/database/implementing-table-partitioning-in… > > > > Thanks > > > > _______________________________________________ > > Mailing list: https://launchpad.net/~maria-developers > > Post to : maria-developers(a)lists.launchpad.net > > Unsubscribe : https://launchpad.net/~maria-developers > > More help : https://help.launchpad.net/ListHelp

1 0

Re: [Maria-developers] b248988c8b6: MDEV-19275 Provide SQL service to plugins.
by Sergei Golubchik 14 Sep '21

14 Sep '21

Hi, Alexey! see questions below On Sep 07, Alexey Botchkov wrote: > revision-id: b248988c8b6 (mariadb-10.6.1-74-gb248988c8b6) > parent(s): 42ad4fa3464 > author: Alexey Botchkov > committer: Alexey Botchkov > timestamp: 2021-09-07 14:08:13 +0400 > message: > > MDEV-19275 Provide SQL service to plugins. > > SQL service added. > It provides the limited set of client library functions > to be used by plugin. > > diff --git a/include/mysql/service_sql.h b/include/mysql/service_sql.h > new file mode 100644 > index 00000000000..5050793f655 > --- /dev/null > +++ b/include/mysql/service_sql.h > @@ -0,0 +1,99 @@ > +/* Copyright (C) 2021 MariaDB Corporation > + > + This program is free software; you can redistribute it and/or modify > + it under the terms of the GNU General Public License as published by > + the Free Software Foundation; version 2 of the License. > + > + This program is distributed in the hope that it will be useful, > + but WITHOUT ANY WARRANTY; without even the implied warranty of > + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > + GNU General Public License for more details. > + > + You should have received a copy of the GNU General Public License > + along with this program; if not, write to the Free Software > + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02111-1301 USA */ > + > +#if defined(MYSQL_SERVER) && !defined MYSQL_SERVICE_SQL > +#define MYSQL_SERVICE_SQL > + > +#include <mysql.h> > + > +/** > + @file > + SQL service > + > + Interface for plugins to execute SQL queries on the local server. > + > + Functions of the service are the 'server-limited' client library: > + mysql_init > + mysql_real_connect_local > + mysql_real_connect > + mysql_errno > + mysql_error > + mysql_real_query > + mysql_affected_rows > + mysql_num_rows > + mysql_store_result > + mysql_free_result > + mysql_fetch_row > + mysql_close > +*/ > + > + > +#ifdef __cplusplus > +extern "C" { > +#endif > + > +extern struct sql_service_st { > + MYSQL * STDCALL (*mysql_init)(MYSQL *mysql); > + MYSQL *(*mysql_real_connect_local)(MYSQL *mysql, > + const char *host, const char *user, const char *db, > + unsigned long clientflag); > + MYSQL * STDCALL (*mysql_real_connect)(MYSQL *mysql, const char *host, > + const char *user, const char *passwd, const char *db, unsigned int port, > + const char *unix_socket, unsigned long clientflag); > + unsigned int STDCALL (*mysql_errno)(MYSQL *mysql); > + const char * STDCALL (*mysql_error)(MYSQL *mysql); > + int STDCALL (*mysql_real_query)(MYSQL *mysql, const char *q, > + unsigned long length); > + my_ulonglong STDCALL (*mysql_affected_rows)(MYSQL *mysql); > + my_ulonglong STDCALL (*mysql_num_rows)(MYSQL_RES *res); > + MYSQL_RES * STDCALL (*mysql_store_result)(MYSQL *mysql); > + void STDCALL (*mysql_free_result)(MYSQL_RES *result); > + MYSQL_ROW STDCALL (*mysql_fetch_row)(MYSQL_RES *result); > + void STDCALL (*mysql_close)(MYSQL *sock); > +} *sql_service; > + > +#ifdef MYSQL_DYNAMIC_PLUGIN > + > +#define mysql_init sql_service->mysql_init > +#define mysql_real_connect_local sql_service->mysql_real_connect_local > +#define mysql_real_connect sql_service->mysql_real_connect > +#define mysql_errno(M) sql_service->mysql_errno(M) > +#define mysql_error(M) sql_service->mysql_error(M) > +#define mysql_real_query sql_service->mysql_real_query > +#define mysql_affected_rows sql_service->mysql_affected_rows > +#define mysql_num_rows sql_service->mysql_num_rows > +#define mysql_store_result sql_service->mysql_store_result > +#define mysql_free_result sql_service->mysql_free_result > +#define mysql_fetch_row sql_service->mysql_fetch_row > +#define mysql_close sql_service->mysql_close > + > +#else > + > +MYSQL *mysql_real_connect_local(MYSQL *mysql, > + const char *host, const char *user, const char *db, > + unsigned long clientflag); normally the service file is the authoritative source of the service documentation, everything there is to know about the service should be there. In this case it's mostly the normal client C API, so no need to document it. But mysql_real_connect_local is new, so please, add a comment here, explaining what it is. For example, I'd think it connects to the current server, internally. But then, why does it need host and user? The comment should cover it. > + > +/* The rest of the function declarations mest be taken from the mysql.h */ > + > +#endif /*MYSQL_DYNAMIC_PLUGIN*/ > + > + > +#ifdef __cplusplus > +} > +#endif > + > +#endif /*MYSQL_SERVICE_SQL */ > + > + > diff --git a/plugin/test_sql_service/CMakeLists.txt b/plugin/test_sql_service/CMakeLists.txt > index aa9ecfe685e..7c61a1c3c7a 100644 > --- a/plugin/test_sql_service/CMakeLists.txt > +++ b/plugin/test_sql_service/CMakeLists.txt > @@ -15,4 +15,5 @@ > > SET(SOURCES test_sql_service.c) > > -MYSQL_ADD_PLUGIN(test_sql_service ${SOURCES} MODULE_ONLY RECOMPILE_FOR_EMBEDDED) > +ADD_DEFINITIONS(-DMYSQL_SERVER) why? > +MYSQL_ADD_PLUGIN(test_sql_service ${SOURCES} MODULE_ONLY) > diff --git a/plugin/test_sql_service/test_sql_service.c b/plugin/test_sql_service/test_sql_service.c > index 062f10fce58..e1155a98c40 100644 > --- a/plugin/test_sql_service/test_sql_service.c > +++ b/plugin/test_sql_service/test_sql_service.c > @@ -14,71 +14,113 @@ > Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1335 USA */ > > > -#define PLUGIN_VERSION 0x100 > -#define PLUGIN_STR_VERSION "1.0.0" > - > -#define _my_thread_var loc_thread_var > +#define PLUGIN_VERSION 0x20000 > +#define PLUGIN_STR_VERSION "2.0" > > #include <my_config.h> > -#include <assert.h> > #include <my_global.h> > #include <my_base.h> > -#include <typelib.h> > -//#include <mysql_com.h> /* for enum enum_server_command */ > -#include <mysql/plugin.h> > #include <mysql/plugin_audit.h> > -//#include <string.h> > - > +#include <mysql.h> shouldn't be needed > > -LEX_STRING * thd_query_string (MYSQL_THD thd); > -unsigned long long thd_query_id(const MYSQL_THD thd); > -size_t thd_query_safe(MYSQL_THD thd, char *buf, size_t buflen); > -const char *thd_user_name(MYSQL_THD thd); > -const char *thd_client_host(MYSQL_THD thd); > -const char *thd_client_ip(MYSQL_THD thd); > -LEX_CSTRING *thd_current_db(MYSQL_THD thd); > -int thd_current_status(MYSQL_THD thd); > -enum enum_server_command thd_current_command(MYSQL_THD thd); > - > -int maria_compare_hostname(const char *wild_host, long wild_ip, long ip_mask, > - const char *host, const char *ip); > -void maria_update_hostname(const char **wild_host, long *wild_ip, long *ip_mask, > - const char *host); > > /* Status variables for SHOW STATUS */ > static long test_passed= 0; > +static char *sql_text_local, *sql_text_global; > +static char qwe_res[1024]= ""; > + > static struct st_mysql_show_var test_sql_status[]= > { > {"test_sql_service_passed", (char *)&test_passed, SHOW_LONG}, > + {"test_sql_query_result", qwe_res, SHOW_CHAR}, > {0,0,0} > }; > > static my_bool do_test= TRUE; > -static void run_test(MYSQL_THD thd, struct st_mysql_sys_var *var, > - void *var_ptr, const void *save); > -static MYSQL_SYSVAR_BOOL(run_test, do_test, PLUGIN_VAR_OPCMDARG, > - "Perform the test now.", NULL, run_test, FALSE); > +static int run_test(MYSQL_THD thd, struct st_mysql_sys_var *var, void *save, > + struct st_mysql_value *value); > +static int run_sql_local(MYSQL_THD thd, struct st_mysql_sys_var *var, void *save, > + struct st_mysql_value *value); > +static int run_sql_global(MYSQL_THD thd, struct st_mysql_sys_var *var, void *save, > + struct st_mysql_value *value); > + > +static void noop_update(MYSQL_THD thd, struct st_mysql_sys_var *var, > + void *var_ptr, const void *save); > + > +static MYSQL_SYSVAR_BOOL(run_test, do_test, > + PLUGIN_VAR_OPCMDARG, > + "Perform the test now.", > + run_test, NULL, FALSE); > + > +static MYSQL_SYSVAR_STR(execute_sql_local, sql_text_local, > + PLUGIN_VAR_OPCMDARG, > + "Create the new local connection, execute SQL statement with it.", > + run_sql_local, noop_update, FALSE); > + > +static MYSQL_SYSVAR_STR(execute_sql_global, sql_text_global, > + PLUGIN_VAR_OPCMDARG, > + "Execute SQL statement using the global connection.", > + run_sql_global, noop_update, FALSE); Why did you test local and global connections? What difference in behavior can one expect? > + > static struct st_mysql_sys_var* test_sql_vars[]= > { > MYSQL_SYSVAR(run_test), > + MYSQL_SYSVAR(execute_sql_local), > + MYSQL_SYSVAR(execute_sql_global), > NULL > }; > > diff --git a/sql/mysqld.cc b/sql/mysqld.cc > index ef6f40778fe..8a941a68e53 100644 > --- a/sql/mysqld.cc > +++ b/sql/mysqld.cc > @@ -5064,6 +5075,9 @@ static int init_server_components() > > tc_log= 0; // ha_initialize_handlerton() needs that > > + if (ddl_log_initialize()) > + unireg_abort(1); why? > + > if (plugin_init(&remaining_argc, remaining_argv, > (opt_noacl ? PLUGIN_INIT_SKIP_PLUGIN_TABLE : 0) | > (opt_abort ? PLUGIN_INIT_SKIP_INITIALIZATION : 0))) > diff --git a/sql/sql_class.h b/sql/sql_class.h > index e569fcd32d6..964626be3d4 100644 > --- a/sql/sql_class.h > +++ b/sql/sql_class.h > diff --git a/sql/sql_prepare.cc b/sql/sql_prepare.cc > index 09ad632dd98..4725855c130 100644 > --- a/sql/sql_prepare.cc > +++ b/sql/sql_prepare.cc > @@ -4857,6 +4870,7 @@ Prepared_statement::execute_server_runnable(Server_runnable *server_runnable) > if (!(lex= new (mem_root) st_lex_local)) > return TRUE; > > + thd->set_time(); why? May be it should be executed using the timestamp of the top-level statement? > thd->set_n_backup_statement(this, &stmt_backup); > thd->set_n_backup_active_arena(this, &stmt_backup); > thd->stmt_arena= this; > @@ -6153,23 +6163,193 @@ static my_bool loc_read_query_result(MYSQL *mysql) > + > extern "C" MYSQL *mysql_real_connect_local(MYSQL *mysql, > - const char *host, const char *user, const char *passwd, const char *db) > + const char *host, const char *user, const char *db, you aren't using `host` or `db` for anything. And I don't understand what you're using `user` for. > + unsigned long clientflag) > { > - //char name_buff[USERNAME_LENGTH]; > - > + THD *thd_orig= current_thd; > + THD *new_thd; > + Protocol_local *p; > DBUG_ENTER("mysql_real_connect_local"); > > /* Test whether we're already connected */ > diff --git a/sql/thread_pool_info.cc b/sql/thread_pool_info.cc > index 90ac6871784..e3ffd160a11 100644 > --- a/sql/thread_pool_info.cc > +++ b/sql/thread_pool_info.cc > @@ -14,9 +14,9 @@ along with this program; if not, write to the Free Software > Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02111 - 1301 USA*/ > > #include <mysql_version.h> > -#include <mysql/plugin.h> > > #include <my_global.h> > +#include <mysql/plugin.h> shouldn't be needed, mysql/plugin.h should not need my_global.h > #include <sql_class.h> > #include <sql_i_s.h> > #include <mysql/plugin.h> > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 6

Re: [Maria-developers] 48261114b9d: MDEV-26572 Improve simple multibyte collation performance on the ASCII range
by Sergei Golubchik 13 Sep '21

13 Sep '21

Hi, Alexander! please add more benchmark results to the MDEV, all the variants that you've tried and speedups (or slowdowns) for each. otherwise, basically, no comments, see below. the only real concern is to know exactly when it helps and when it doesn't. On Sep 11, Alexander Barkov wrote: > revision-id: 48261114b9d (mariadb-10.6.1-77-g48261114b9d) > parent(s): 76149650764 > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2021-09-09 15:20:57 +0400 > message: > > MDEV-26572 Improve simple multibyte collation performance on the ASCII range > > diff --git a/strings/ctype-ascii.h b/strings/ctype-ascii.h > new file mode 100644 > index 00000000000..66c9cfb3c6d > --- /dev/null > +++ b/strings/ctype-ascii.h > @@ -0,0 +1,189 @@ > +#ifndef CTYPE_ASCII_H_INCLUDED > +#define CTYPE_ASCII_H_INCLUDED The convention is CTYPE_ASCII_INCLUDED (every file has .h, so it's omitted from the include guard) > + > +#include "myisampack.h" > + > +/* > + Magic expression. It uses the fact that for any byte value X in > + the range 0..31 (0x00..0x1F) the expression (X+31)*5 returns > + the 7th bit (0x80) set only for the following six (out of 32) values: > + 0x00, 0x1B, 0x1C, 0x1D, 0x1E, 0x1F. > + These values correspond to offsets of non-letter characters > + in the ASCII table: > + > + The following macro sets the bit 0x20 for the following characters: > + ---------------- -------------------------------- > + Magic bit 10000000000000000000000000011111 > + ASCII 0x00..0x1F ................................ Control > + ASCII 0x20..0x3F ................................ Punctuation, digits > + ASCII 0x40..0x5F @ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_ > + ASCII 0x60..0x7F `abcdefghijklmnopqrstuvwxyz{|}~. > + ---------------- -------------------------------- > + We shift the magic bit 0x80 right twice to make it 0x20. > + So on the ranges [40..5F] and [60..7F] the expression > + has the bit 0x20 set for all non-letter characters. > + Note, other bits contain garbage. > + > + Requirements: > + All bytes must be in the range [00..7F], > + to avoid overflow and carry to the next byte. > +*/ > +#define MY_ASCII_20_IS_SET_IF_NOT_LETTER_MAGIC(i) \ > + (((((i)+0x1F1F1F1F1F1F1F1FULL) & 0x1F1F1F1F1F1F1F1F) * 5) >> 2) > + > + > +/* > + The following macro returns the bit 0x20 set to: > + - 1 for input bytes in the ranges [60..7F] or [E0..FF] > + - 0 otherwise > + Bytes in the ranges [40..7F] and [C0..FF] have the bit 0x40 set. > + Bytes in the ranges [60..7F] smd [E0..FF] have the bit 0x20 set. s/smd/and/ > + Hex BinHi BinLo > + ---- -1-- ---- > + 0x[4C]X .10. .... > + 0x[5D]X .10. .... > + 0x[6E]X .11. .... > + 0x[7F]X .11. .... > +*/ > +#define MY_ASCII_20_IS_SET_IF_RANGE_60_7F_OR_E0_FF(i) (((i) >> 1) & ((i))) > + > + > +/* > + The following macro evaluates to exactly 0x20 for all > + lower case ASCII letters [a-z], and to 0x00 otherwise: > + > + Value Range Character range Subrange > + -------- -------- -------------------------------- ------- > + 00000000 0x00..0x3F Control, punctuation, digits > + 00100000 0x40..0x5F @ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_ letters A-Z > + 00000000 0x40..0x5F @ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_ non-letters > + 00100000 0x60..0x7F `abcdefghijklmnopqrstuvwxyz{|}~. letters a-z > + 00000000 0x60..0x7F `abcdefghijklmnopqrstuvwxyz{|}~. non-letters > + > + Requirements: > + All bytes must be in the range [00..7F]. > + See the comments in MY_ASCII_20_IS_SET_IF_NOT_LETTER_MAGIC(). > +*/ > + > +#define MY_ASCII_20_IF_IS_LOWER_LETTER(i) \ > + (MY_ASCII_20_IS_SET_IF_RANGE_60_7F_OR_E0_FF(i) & \ > + ~MY_ASCII_20_IS_SET_IF_NOT_LETTER_MAGIC(i) & \ > + 0x2020202020202020) > + > +/* > + Convert lower case ASCII letters to upper case by unsetting > + the bit 0x20 with help of the magic expression. > + > + Requirements: > + All bytes must be in the range [00..7F]. > + See the comments in MY_ASCII_20_IS_SET_IF_NOT_LETTER_MAGIC() > +*/ > +#define MY_ASCII_TOUPPER_MAGIC(i) \ > + (i ^ MY_ASCII_20_IF_IS_LOWER_LETTER(i)) > + > + > +/* > + Convert a string (consisting of 8 bytes stored in uint64) > + to upper case algorithmically. > + > + Requirements: > + All bytes must be in the range [00..0x7F]. > + See the comments in MY_ASCII_20_IS_SET_IF_NOT_LETTER_MAGIC(). > + The result on 8bit data is unpredictable!!! > + The caller should make sure not to pass 8bit data. > +*/ > +static inline ulonglong my_ascii_to_upper_magic_uint64(ulonglong i) > +{ > + return MY_ASCII_TOUPPER_MAGIC(i); > +} > + > + > +/* > + Check if: > + - both strings "a" and "b" have at least 4 bytes, and > + - both strings have only 7bit data. > +*/ > +static inline int > +my_strcoll_ascii_4bytes_found(const uchar *a, const uchar *ae, > + const uchar *b, const uchar *be) > +{ > + return a + 4 <= ae && b + 4 <= be && > + (uint4korr(b) & 0x80808080) == 0 && > + (uint4korr(a) & 0x80808080) == 0; > +} > + > + > +/* > + Compare the leading four 7bit ASCII bytes in two strings case insensitively > + by converting letters [a-z] to upper case [A-Z]. > + > + Requirements: > + - The input strings must have at least four bytes, and > + - The leading four bytes in both strings must be 7bit ASCII. > + The caller must make sure to provide only strings that meet > + these requirements. The result on 8-bit data is unpredictable > + as 8-bit bytes may cause overflow in my_ascii_to_upper_magic_uint64(). > + See comments above. > +*/ > +static inline int > +my_strcoll_ascii_toupper_4bytes(const uchar *a, const uchar *b) > +{ > + ulonglong abn= (((ulonglong) mi_uint4korr(a)) << 32) | mi_uint4korr(b); > + abn= my_ascii_to_upper_magic_uint64(abn); > + if ((uint) (abn >> 32) == (uint32) abn) s/uint/uint32/ ? > + return 0; > + return ((uint32) (abn >> 32)) < ((uint32) abn) ? -1 : + 1; > +} > + > + > +/* > + Compare the leading eight 7bit ASCII bytes in two strings case insensitively > + by converting letters [a-z] to upper case [A-Z]. > + > + Requirements: > + - The input strings must have at least eight bytes, and > + - The leading eight bytes in both strings must be 7bit ASCII. > + See comments in my_strcoll_ascii_toupper_4bytes(). > +*/ > +static inline int > +my_strcoll_ascii_toupper_8bytes(const uchar *a, const uchar *b) > +{ > + /* > + TODO: > + Try to get advantage of SIMD instructions by massive comparison > + (16 bytes at a time) of characters against (x>='a' && x<='z') using: > + - either explicit intrinsics I thought your SIMD version was slower, wasn't it? > + - or a loop that can get vectorized automatically by some compilers. > + */ > + ulonglong an= mi_uint8korr(a); > + ulonglong bn= mi_uint8korr(b); > + an= my_ascii_to_upper_magic_uint64(an); > + bn= my_ascii_to_upper_magic_uint64(bn); > + return an == bn ? 0 : an < bn ? -1 : +1; > +} > + > + > +/* > + Compare the leading four 7bit ASCII bytes in two strings in binary style. > +*/ > +static inline int > +my_strcoll_mb7_bin_4bytes(const uchar *a, const uchar *b) > +{ > + uint32 an= mi_uint4korr(a); > + uint32 bn= mi_uint4korr(b); > + return an == bn ? 0 : an < bn ? -1 : +1; > +} > + > + > +/* > + Compare the leading four 7bit ASCII bytes in two strings in binary style. > +*/ > +static inline int > +my_strcoll_mb7_bin_8bytes(const uchar *a, const uchar *b) > +{ > + ulonglong an= mi_uint8korr(a); > + ulonglong bn= mi_uint8korr(b); > + return an == bn ? 0 : an < bn ? -1 : +1; > +} > + > +#endif /* CTYPE_ASCII_H_INCLUDED */ > diff --git a/strings/strcoll.ic b/strings/strcoll.ic > index 6aca0d0c460..72e0131e1c5 100644 > --- a/strings/strcoll.ic > +++ b/strings/strcoll.ic > @@ -39,6 +41,42 @@ > #endif > > > +/* > + For binary collations: > + - on 32bit platforms perform only 4 byte optimization > + - on 64bit platforms perform both 4 byte and 8 byte optimization > +*/ > +#if defined(STRCOLL_MB7_BIN) > +#define MY_STRCOLL_MB7_4BYTES(a,b) my_strcoll_mb7_bin_4bytes((a),(b)) > +#if defined(__x86_64__) as Wlad has already said, what about not gcc-on-intel 64bit platforms? > +#define STRCOLL_MB7_8BYTES > +#define MY_STRCOLL_MB7_8BYTES(a,b) my_strcoll_mb7_bin_8bytes((a),(b)) > +#endif > +#endif > + > + > +/* > + For case insensitive collations with trivial mapping from [a-z] to [A-Z] > + perform optimization only on 64 bit platforms. > + There is no sense to perform my_ascii_to_upper_magic_uint64() based > + optimization on 32bit platforms. The idea of this optimization > + is that it handles 8bytes at a time, using a 64bit CPU register. > + Enabling this optimization on 32bit platform may only slow things down. > +*/ > +#if defined(STRCOLL_MB7_TOUPPER) > +#if defined(__x86_64__) > +#define MY_STRCOLL_MB7_4BYTES(a,b) my_strcoll_ascii_toupper_4bytes((a),(b)) > +#define MY_STRCOLL_MB7_8BYTES(a,b) my_strcoll_ascii_toupper_8bytes((a),(b)) > +#endif /* Architecture test */ > +#endif /* STRCOLL_MB7_TOUPPER */ > + > + > +/* > + A helper macro to shift two pointers forward, to the given amount. > +*/ > +#define MY_STRING_SHIFT_PTR_PTR(a,b,len) do { a+= len; b+= len; } while(0) > + > + > /* > Weight of an illegal byte, must follow these rules: > 1. Must be greater than weight of any normal character in the collation. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] a00410a2ee4: MDEV-26238: Remove inconsistent behaviour of --default-* options in my_print_defaults
by Sergei Golubchik 11 Sep '21

11 Sep '21

Hi, Rucha! On Sep 11, Rucha Deodhar wrote: > revision-id: a00410a2ee4 (mariadb-10.6.1-77-ga00410a2ee4) > parent(s): 76149650764 > author: Rucha Deodhar > committer: Rucha Deodhar > timestamp: 2021-09-11 16:30:38 +0530 > message: > > MDEV-26238: Remove inconsistent behaviour of --default-* options > in my_print_defaults > > Analysis: --defaults* option is recognized anywhere in the commandline > instead of only at the beginning because handle_options() recognizes > options in any order. > Fix: use get_defaults_options() which recognizes --defaults* options only at > the beginning. After this is done, we only want to recognize other options > given in any order which can be done using handle_options(). So (only) skip > --defaults* options, combine it with program name to pass it to handle_options() > (because we skip program name in handle_options() too). Also, removed -e, -g and > -c because only my_print_defaults supports them. > > diff --git a/extra/my_print_defaults.c b/extra/my_print_defaults.c > index b7f52382721..0253614b410 100644 > --- a/extra/my_print_defaults.c > ++ b/extra/my_print_defaults.c > @@ -141,50 +124,49 @@ static int get_options(int *argc,char ***argv) > return 0; > } > > int main(int argc, char **argv) > { > int count, error, args_used; > char **load_default_groups= 0, *tmp_arguments[6]; > char **argument, **arguments, **org_argv, **new_argv, **pos; > int nargs, i= 0, index=1; > MY_INIT(argv[0]); > > org_argv= argv; > args_used= get_defaults_options(argv); > > /* Copy defaults-xxx arguments & program name */ > count= args_used; > arguments= tmp_arguments; > memcpy((char*) arguments, (char*) org_argv, count*sizeof(*org_argv)); > arguments[count]= 0; you're still overdoing it. you can do just argv+= args_used-1; argc+= args_used-1; get_options(&argc,&argv); no need to allocate or copy anything. Isn't it simpler? > /* > We already process --defaults* options at the beginning in > get_defaults_options(). So skip only --defaults* options and > pass program name and remaining options to handle_options(). > */ > > new_argv= (char**)malloc(argc*sizeof(*org_argv)); > memcpy((char*)new_argv, (char*)org_argv, 1*(sizeof(*new_argv))); > org_argv++; > > /* We want to skip --defaults* option only */ > if (!is_prefix(*org_argv, "--no-defaults")) > org_argv+= args_used-1; > > for (pos= org_argv; *pos; pos++) > new_argv[index++]= *(pos); > > new_argv[index]= 0; > argc= index; > > memcpy((char*)argv, (char*)new_argv, index*(sizeof(*new_argv))); > > /* Check out the args. */ > if (get_options(&argc,&argv)) > cleanup_and_exit(1); > > nargs= argc + 1; > if (opt_mysqld) > nargs+= array_elements(mysqld_groups); Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] eb121bbe93e: Vanilla cleanups and refactorings
by Sergei Golubchik 11 Sep '21

11 Sep '21

Hi, Aleksey! On Sep 10, Aleksey Midenkov wrote: > revision-id: eb121bbe93e (mariadb-10.6.1-78-geb121bbe93e) > parent(s): d90f54e2206 > author: Aleksey Midenkov > committer: Aleksey Midenkov > timestamp: 2021-09-10 17:42:10 +0300 > message: > > diff --git a/dbug/tests.c b/dbug/tests.c > index 3e77bf82236..100388891eb 100644 > --- a/dbug/tests.c > +++ b/dbug/tests.c > @@ -13,6 +13,17 @@ char *push1=0; > #include <my_pthread.h> > #include <string.h> > > + > +#ifndef DBUG_OFF > +#define DBUG_EVALUATE(keyword,a1,a2) \ > + (_db_keyword_(0,(keyword), 0) ? (a1) : (a2)) > +#define DBUG_EVALUATE_IF(keyword,a1,a2) \ > + (_db_keyword_(0,(keyword), 1) ? (a1) : (a2)) > +#else > +#define DBUG_EVALUATE(keyword,a1,a2) (a2) > +#define DBUG_EVALUATE_IF(keyword,a1,a2) (a2) > +#endif > + could you fix the test properly, please? > const char *func3() > { > DBUG_ENTER("func3"); Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 3

Re: [Maria-developers] 0e09bc41cba: MDEV-9245 password "reuse prevention" validation plugin
by Sergei Golubchik 11 Sep '21

11 Sep '21

Hi, Oleksandr! Looks very good. See comments/questions below. The main thing - please add some tests for SQL errors in the plugin. On Sep 11, Oleksandr Byelkin wrote: > revision-id: 0e09bc41cba (mariadb-10.6.1-78-g0e09bc41cba) > parent(s): 67cd5da8c34 > author: Oleksandr Byelkin > committer: Oleksandr Byelkin > timestamp: 2021-09-10 13:47:07 +0200 > message: > > MDEV-9245 password "reuse prevention" validation plugin > > diff --git a/mysql-test/suite/plugins/t/password_reuse_check.test b/mysql-test/suite/plugins/t/password_reuse_check.test > --- /dev/null > +++ b/mysql-test/suite/plugins/t/password_reuse_check.test > @@ -0,0 +1,58 @@ > +--source include/not_embedded.inc > + > +if (!$PASSWORD_REUSE_CHECK_SO) { > + skip No PASSWORD_REUSE_CHECK plugin; > +} > + > +install soname "password_reuse_check"; > + > +set @save_password_reuse_check_interval= @@password_reuse_check_interval; this is kind of pointless, if you uninstall the plugin anyway. it's not harmless, just useless. > + > +set global password_reuse_check_interval= 0; ... > diff --git a/plugin/password_reuse_check/CMakeLists.txt b/plugin/password_reuse_check/CMakeLists.txt > --- /dev/null > +++ b/plugin/password_reuse_check/CMakeLists.txt > @@ -0,0 +1,2 @@ > + > +MYSQL_ADD_PLUGIN(password_reuse_check password_reuse_check.c) did you try to compile it statically? > diff --git a/plugin/password_reuse_check/password_reuse_check.c b/plugin/password_reuse_check/password_reuse_check.c > --- /dev/null > +++ b/plugin/password_reuse_check/password_reuse_check.c ... > +static void bin_to_hex512(char *to, const unsigned char *str) > +{ > + const unsigned char *str_end= str + (512/8); > + for (; str != str_end; ++str) > + { > + *to++= digits[((unsigned char) *str) >> 4]; > + *to++= digits[((unsigned char) *str) & 0x0F]; > + } > +} > + > + > +/** > + Send SQL error as ER_UNKNOWN_ERROR for information > + > + @param mysql Connection handler > +*/ > + > +static void report_sql_error(MYSQL *mysql) > +{ > + my_printf_error(ER_UNKNOWN_ERROR, "password_reuse_check:[%d] %s", ME_WARNING, > + mysql_errno(mysql), mysql_error(mysql)); > +} No tests for that? You can, for example, drop the table, and create a non-insertable view with the same name. CREATE VIEW mysql.password_reuse_check_history AS SELECT 1; That should make the plugin to fail, right? > + > + > +/** > + Create the history of passwords table for this plugin. > + > + @param mysql Connection handler > + > + @retval 1 - Error > + @retval 0 - OK > +*/ Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 818407474d6: MDEV-22166 CONVERT PARTITION: move out partition into a table
by Sergei Golubchik 10 Sep '21

10 Sep '21

Hi, Aleksey! I presume that changes to the logging code is what Monty asked. So I'll only comment on a couple of minor issues: On Sep 10, Aleksey Midenkov wrote: > revision-id: 818407474d6 (mariadb-10.6.1-80-g818407474d6) > parent(s): bb4b95975f7 > author: Aleksey Midenkov > committer: Aleksey Midenkov > timestamp: 2021-09-10 18:47:31 +0300 > message: > > MDEV-22166 CONVERT PARTITION: move out partition into a table > > diff --git a/sql/sql_partition.cc b/sql/sql_partition.cc > index db371d23ba7..cbdb40472d8 100644 > --- a/sql/sql_partition.cc > +++ b/sql/sql_partition.cc > @@ -5371,8 +5367,12 @@ that are reorganised. > tab_part_info->is_auto_partitioned= FALSE; > } > } > - else if (alter_info->partition_flags & ALTER_PARTITION_DROP) > + else if ((alter_info->partition_flags & ALTER_PARTITION_DROP) | > + (alter_info->partition_flags & ALTER_PARTITION_CONVERT_OUT)) > { > + const char * const cmd= > + (alter_info->partition_flags & ALTER_PARTITION_CONVERT_OUT) ? > + "EXTRACT" : "DROP"; "CONVERT" > /* > Drop a partition from a range partition and list partitioning is > always safe and can be made more or less immediate. It is necessary > diff --git a/sql/sql_table.h b/sql/sql_table.h > index aacb6c99f15..a580c7d348d 100644 > --- a/sql/sql_table.h > +++ b/sql/sql_table.h > @@ -20,6 +20,17 @@ > #include <my_sys.h> // pthread_mutex_t > #include "m_string.h" // LEX_CUSTRING > > +inline bool suicide() > +{ > + DBUG_SUICIDE(); > + return false; > +} as I wrote earlier, this is wrong. DBUG_SUICIDE() must always be an expression. > + > +#define ERROR_INJECT_CRASH(code) \ > + (DBUG_IF(code) && suicide()) > +#define ERROR_INJECT_ERROR(code) \ > + (DBUG_IF(code) && (my_error(ER_UNKNOWN_ERROR, MYF(0)), 1)) > + > class Alter_info; > class Alter_table_ctx; > class Column_definition; Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 2

Re: [Maria-developers] ff0ece70855: MDEV-26531: mtr shows some hunks failed
by Sergei Golubchik 10 Sep '21

10 Sep '21

Hi, Anel! On Sep 10, Anel Husakovic wrote: > revision-id: ff0ece70855 (mariadb-10.6.1-73-gff0ece70855) > parent(s): 1bc82aaf0a7 > author: Anel Husakovic > committer: Anel Husakovic > timestamp: 2021-09-03 21:15:58 +0200 > message: > > MDEV-26531: mtr shows some hunks failed > > Reviewed by: wlad(a)mariadb.com > > --- > mysql-test/mariadb-test-run.pl | 10 +++++++--- > 1 file changed, 7 insertions(+), 3 deletions(-) > > diff --git a/mysql-test/mariadb-test-run.pl b/mysql-test/mariadb-test-run.pl > index 6dcd25a9c86..7e0282b89e4 100755 > --- a/mysql-test/mariadb-test-run.pl > +++ b/mysql-test/mariadb-test-run.pl > @@ -3278,11 +3278,15 @@ sub do_before_run_mysqltest($) > # don't rebuild a file if it's up to date > unless (-e $dest and -M $dest < -M $resfile > and -M $dest < -M $base_result) { > - run_system(@cmd); > + if(run_system(@cmd) != 0){ > + die("system(@cmd) : $?"); I'd rather see it done more perl-ish, like run_system(@cmd) or die "system(@cmd): $?" > + } > } > } else { > $cmd[-3] = $dest = $opt_tmpdir . '/' . basename($dest); > - run_system(@cmd); > + if(run_system(@cmd) != 0){ > + die("system(@cmd) : $?"); > + } > } > $tinfo->{result_file} = $dest; > } > @@ -4887,7 +4891,7 @@ sub report_failure_and_restart ($) { > > sub run_system(@) { > mtr_verbose("Running '$_[0]'"); > - my $ret= system(@_) >> 8; > + my $ret= system(@_); this seems wrong. `perldoc -f system` says The return value is the exit status of the program as returned by the "wait" call. To get the actual exit value, shift right by eight (see below). > return $ret; > } Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 4645a1e1950: MDEV-26238: Remove inconsistent behaviour of --default-* options in my_print_defaults
by Sergei Golubchik 10 Sep '21

10 Sep '21

Hi, Rucha! On Sep 08, Rucha Deodhar wrote: > Hi, Sergei ! > Thanks for the review. > > > > diff --git a/extra/my_print_defaults.c b/extra/my_print_defaults.c > > > index b7f52382721..e2ec0b00923 100644 > > > --- a/extra/my_print_defaults.c > > > +++ b/extra/my_print_defaults.c > > > @@ -48,20 +48,6 @@ static struct my_option my_long_options[] = > > > > you forgot the "config-file", which is here, just about "debug" > > I am sorry, I don't understand. Oops. Sorry. I looked at my_print_defaults.c in the checked out branch and I saw: {"config-file", 'c', "Deprecated, please use --defaults-file instead. " "Name of config file to read; if no extension is given, default " "extension (e.g., .ini or .cnf) will be added", (char**) &config_file, (char**) &config_file, 0, GET_STR, REQUIRED_ARG, 0, 0, 0, 0, 0, 0}, #ifdef DBUG_OFF {"debug", '#', "This is a non-debug version. Catch this and exit", 0,0, 0, GET_DISABLED, OPT_ARG, 0, 0, 0, 0, 0, 0}, #else so I wrote that "config-file" should also be removed. But now I see that it's present in 10.2, but not in 10.4, it was already removed. So, ignore that my comment, you didn't forget config-file, it was removed in 10.4, I looked at the wrong branch. > > > {"debug", '#', "Output debug log", (char**) &default_dbug_option, > > > (char**) &default_dbug_option, 0, GET_STR, OPT_ARG, 0, 0, 0, 0, 0, 0}, > > > #endif > > > - {"defaults-file", 'c', > > > - "Read this file only, do not read global or per-user config " > > > - "files; should be the first option", > > > - (char**) &config_file, (char*) &config_file, 0, GET_STR, REQUIRED_ARG, > > > @@ -141,49 +124,44 @@ static int get_options(int *argc,char ***argv) > > > return 0; > > > } > > > > > > -static char *make_args(const char *s1, const char *s2) > > > -{ > > > - char *s= malloc(strlen(s1) + strlen(s2) + 1); > > > - strmov(strmov(s, s1), s2); > > > - return s; > > > -} > > > - > > > int main(int argc, char **argv) > > > { > > > - int count= 0, error, no_defaults= 0; > > > + int count, error, args_used; > > > char **load_default_groups= 0, *tmp_arguments[6]; > > > - char **argument, **arguments, **org_argv; > > > - int nargs, i= 0; > > > + char **argument, **arguments, **org_argv, **new_argv, **pos; > > > + int nargs, i= 0, index=0; > > > MY_INIT(argv[0]); > > > > > > org_argv= argv; > > > - if (*argv && !strcmp(*argv, "--no-defaults")) > > > - { > > > - argv++; > > > - ++count; > > > - no_defaults= 1; > > > - } > > > - /* Copy program name and --no-defaults if present*/ > > > + args_used= get_defaults_options(argv); > > > + > > > + /* Copy defaults-xxx arguments & program name */ > > > + count= args_used; > > > arguments= tmp_arguments; > > > - memcpy((char*) arguments, (char*) org_argv, (++count)*sizeof(*org_argv)); > > > + memcpy((char*) arguments, (char*) org_argv, count*sizeof(*org_argv)); > > > arguments[count]= 0; > > > > > > - /* Check out the args */ > > > - if (get_options(&argc,&argv)) > > > - cleanup_and_exit(1); > > > - > > > - if (!no_defaults) > > > + /* > > > + We already process --defaults* options at the beginning in > > > + get_defaults_options(). So, now we we make a new commandline which > > > + has all options except --defaults* options and pass it to handle_options() > > > + instead of original commandline. > > > + */ > > > + new_argv= (char**)malloc(argc*sizeof(*org_argv)); > > > + for (pos= org_argv; *pos; pos++) > > > { > > > + if (!is_prefix(*pos, "--defaults-extra-file=") && > > > + !is_prefix(*pos, "--defaults-file=") && > > > + !is_prefix(*pos, "--defaults-group-suffix=")) > > > + new_argv[index++]= *(pos); > > > > Why are you doing it? > > After get_defaults_options, you know how many arguments were default, > > just skip args_used, and invoke handle_options on the rest. > > I did it to filter all the --defaults* options, anywhere on the command > line. > > get_defaults_options() counts the program name in addition to --defaults* > options, so args_used will at least be 1. So skipping argv using agrs_used > will at least skip the program name even before it is passed to > handle_options(). But argv is also skipped in handle_options() to skip the > program name. > So one extra argument will get skipped too. correct, you need to skip args_used-1, good point. > Say --defaults* options are somewhere at the end, they will not get counted > in get_defaults_options(). So --defaults* will be passed to > handle_options() without getting skipped. And handle_options() will give an > error even if the options are valid because > it will not find --defaults* options in my_long_options[]. This is what we want, right? That's why you're doing it. Try `mariadb` client or `mariadb-dump` or `mariadbd` server, anything, and put --defaults* option not at the beginning. It'll be "unknown option". The idea is to make my_print_defaults *not special*, to behave like any other executable. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 6768e3a2830: MDEV-22166 MIGRATE PARTITION: move out partition into a table
by Sergei Golubchik 10 Sep '21

10 Sep '21

Hi, Aleksey! It's great that you made it atomic. Frankly, I didn't think of it, MDEV defines the feature as being exactly equivalent to the 3-command sequence, so non-atomic. But, of course, making it atomic makes perfect sense. See other comments below. On Aug 23, Aleksey Midenkov wrote: > commit 6768e3a2830 > Author: Aleksey Midenkov <midenok(a)gmail.com> > Date: Thu Jul 29 22:54:12 2021 +0300 > > diff --git a/sql/lex.h b/sql/lex.h > index cbf9d9d51b2..55eefc39a7e 100644 > --- a/sql/lex.h > +++ b/sql/lex.h > @@ -240,6 +240,7 @@ SYMBOL symbols[] = { > { "EXPLAIN", SYM(DESCRIBE)}, > { "EXTENDED", SYM(EXTENDED_SYM)}, > { "EXTENT_SIZE", SYM(EXTENT_SIZE_SYM)}, > + { "EXTRACT", SYM(EXTRACT_SYM)}, So, you did EXTRACT after all, not MIGRATE? ADD/EXTRACT aren't opposites in SQL. It would be rather consistent to have ALTER ... ADD PARTITION (...) FROM table_name and ALTER ... DROP PARTITION ... TO table_name except that DROP...TO looks kind of strange, and ADD...FROM does not convey the meaning that the table disappears. The second issue also applies to IMPORT/EXPORT. Another option, also kind of strange, but also consistent and makes it clear that the table/partition disappears: ALTER ... MOVE TABLE table_name TO PARTITION (...) ALTER ... MOVE PARTITION ... TO TABLE table_name or the same with CHANGE instead of MOVE to reuse an existing keyword. I'm not saying anything of the above is perfect, and better variants are very much welcome. But pretty much anything is better than ADD/EXTRACT. > { "FALSE", SYM(FALSE_SYM)}, > { "FAST", SYM(FAST_SYM)}, > { "FAULTS", SYM(FAULTS_SYM)}, > diff --git a/sql/sql_yacc.yy b/sql/sql_yacc.yy > index dc2eb4425f0..a41ea7d3e4a 100644 > --- a/sql/sql_yacc.yy > +++ b/sql/sql_yacc.yy > @@ -7605,6 +7605,25 @@ alter_commands: > if (Lex->stmt_alter_table_exchange_partition($6)) > MYSQL_YYABORT; > } > + | EXTRACT_SYM PARTITION_SYM alt_part_name_item > + AS TABLE_SYM table_ident have_partitioning > + { > + LEX *lex= Lex; > + lex->first_select_lex()->db= $6->db; > + if (lex->first_select_lex()->db.str == NULL && > + lex->copy_db_to(&lex->first_select_lex()->db)) > + MYSQL_YYABORT; > + if (unlikely(check_table_name($6->table.str,$6->table.length, > + FALSE)) || > + ($6->db.str && unlikely(check_db_name((LEX_STRING*) &$6->db)))) > + my_yyabort_error((ER_WRONG_TABLE_NAME, MYF(0), $6->table.str)); good point, LEX::stmt_alter_table_exchange_partition() doesn't seem to do it > + lex->name= $6->table; > + > + lex->m_sql_cmd= new (thd->mem_root) Sql_cmd_alter_table(); > + if (unlikely(lex->m_sql_cmd == NULL)) > + MYSQL_YYABORT; > + lex->alter_info.partition_flags|= ALTER_PARTITION_EXTRACT; why wouldn't you combine the common code from here and EXCHANGE PARTITION? less code duplication and as a bonus EXCHANGE will get proper table/db name checks. > + } > ; > > remove_partitioning: > diff --git a/sql/sql_alter.h b/sql/sql_alter.h > index a0f89c28d2a..6aa65d232ca 100644 > --- a/sql/sql_alter.h > +++ b/sql/sql_alter.h > @@ -345,6 +348,8 @@ class Alter_table_ctx > /** Indicates that we are altering temporary table */ > bool tmp_table; > > + DDL_LOG_STATE *ddl_log_state; What do you need this for? > + > private: > char new_filename[FN_REFLEN + 1]; > char new_alias_buff[NAME_LEN + 1]; > diff --git a/sql/sql_table.cc b/sql/sql_table.cc > index 779d125af70..033d77a827d 100644 > --- a/sql/sql_table.cc > +++ b/sql/sql_table.cc > @@ -10016,7 +10146,7 @@ bool mysql_alter_table(THD *thd, const LEX_CSTRING *new_db, > } > > // In-place execution of ALTER TABLE for partitioning. > - DBUG_RETURN(fast_alter_partition_table(thd, table, alter_info, > + DBUG_RETURN(fast_alter_partition_table(thd, table, alter_info, &alter_ctx, > create_info, table_list, > &alter_ctx.db, > &alter_ctx.table_name)); you no longer need to pass alter_ctx.db and alter_ctx.table_name, if you pass the whole alter_ctx down. > diff --git a/include/my_dbug.h b/include/my_dbug.h > index e25bfcf28a7..cf969ce5750 100644 > --- a/include/my_dbug.h > +++ b/include/my_dbug.h > @@ -104,6 +104,8 @@ extern int (*dbug_sanity)(void); > (_db_keyword_(0,(keyword), 0) ? (a1) : (a2)) > #define DBUG_EVALUATE_IF(keyword,a1,a2) \ > (_db_keyword_(0,(keyword), 1) ? (a1) : (a2)) > +#define DBUG_TRUE_IF(keyword) DBUG_EVALUATE_IF(keyword, 1, 0) > +#define DBUG_FALSE_IF(keyword) DBUG_EVALUATE_IF(keyword, 0, 1) hmm. DBUG_FALSE_IF is not used anywhere, DBUG_TRUE_IF is used once as if (DBUG_TRUE_IF("error_extract_partition_00") || unlikely(error= file->ha_rename_table(from_name, to_name))) and this is typically written as if (DBUG_EVALUATE_IF("error_extract_partition_00", 1, error= file->ha_rename_table(from_name, to_name))) so I see no need for either DBUG_TRUE_IF or DBUG_FALSE_IF, please, remove both > #define DBUG_PRINT(keyword,arglist) \ > do if (_db_pargs_(__LINE__,keyword)) _db_doprnt_ arglist; while(0) > > diff --git a/sql/sql_partition.cc b/sql/sql_partition.cc > index 85d880cbbb4..4172a61812e 100644 > --- a/sql/sql_partition.cc > +++ b/sql/sql_partition.cc > @@ -5432,7 +5432,7 @@ uint prep_alter_part_table(THD *thd, TABLE *table, Alter_info *alter_info, > } while (++part_count < tab_part_info->num_parts); > if (num_parts_found != num_parts_dropped) > { > - my_error(ER_DROP_PARTITION_NON_EXISTENT, MYF(0), "DROP"); > + my_error(ER_DROP_PARTITION_NON_EXISTENT, MYF(0), cmd); Not part of your patch, I know, but the error message here is "Error in list of partitions to %-.64s" shouldn't it say "Partition does not exist" or something? > goto err; > } > if (table->file->is_fk_defined_on_table_or_index(MAX_KEY)) > @@ -7268,6 +7420,48 @@ uint fast_alter_partition_table(THD *thd, TABLE *table, > if (alter_partition_lock_handling(lpt)) > goto err; > } > + else if (alter_info->partition_flags & ALTER_PARTITION_EXTRACT) > + { > + if (mysql_write_frm(lpt, WFRM_WRITE_EXTRACTED) || > + write_log_drop_shadow_frm(lpt) || > + ERROR_INJECT_CRASH("crash_extract_partition_1") || > + ERROR_INJECT_ERROR("fail_extract_partition_1") || > + mysql_write_frm(lpt, WFRM_WRITE_SHADOW) || > + ERROR_INJECT_CRASH("crash_extract_partition_2") || > + ERROR_INJECT_ERROR("fail_extract_partition_2") || > + wait_while_table_is_used(thd, table, HA_EXTRA_NOT_USED) || > + ERROR_INJECT_CRASH("crash_extract_partition_3") || > + ERROR_INJECT_ERROR("fail_extract_partition_3") || > + write_log_extract_partition(lpt) || > + ERROR_INJECT_CRASH("crash_extract_partition_4") || > + ERROR_INJECT_ERROR("fail_extract_partition_4") || > + alter_close_table(lpt) || > + ERROR_INJECT_CRASH("crash_extract_partition_5") || > + ERROR_INJECT_ERROR("fail_extract_partition_5") || > + alter_partition_extract(lpt) || > + ERROR_INJECT_CRASH("crash_extract_partition_7") || > + ERROR_INJECT_ERROR("fail_extract_partition_7") || > + (frm_install= TRUE, FALSE) || > + mysql_write_frm(lpt, WFRM_INSTALL_SHADOW|WFRM_BACKUP_ORIGINAL) || > + log_partition_alter_to_ddl_log(lpt) || > + (frm_install= FALSE, FALSE) || > + ERROR_INJECT_CRASH("crash_extract_partition_8") || > + ERROR_INJECT_ERROR("fail_extract_partition_8") || > + (ddl_log_complete(lpt->part_info), false) || as far as I understand this ^^^ should be done *after* binlogging. I suspect that if you crash here, blnlog won't get this ALTER TABLE. > + ((!thd->lex->no_write_to_binlog) && > + (write_bin_log(thd, FALSE, > + thd->query(), thd->query_length()), FALSE)) || if you crash here, what will roll forward on recovery and drop the backup? The log is already closed here, isn't it? I don't understand why you need a backup. Why not to install shadow after the binlog was written? > + mysql_write_frm(lpt, WFRM_DROP_BACKUP) || > + ERROR_INJECT_CRASH("crash_extract_partition_9") || > + ERROR_INJECT_ERROR("fail_extract_partition_9")) > + { > + (void) ddl_log_revert(thd, lpt->part_info); > + handle_alter_part_error(lpt, true, true, frm_install); > + goto err; > + } > + if (alter_partition_lock_handling(lpt)) > + goto err; > + } > else if ((alter_info->partition_flags & ALTER_PARTITION_ADD) && > (part_info->part_type == RANGE_PARTITION || > part_info->part_type == LIST_PARTITION)) > diff --git a/mysql-test/include/lcase_names.inc b/mysql-test/include/lcase_names.inc > new file mode 100644 > index 00000000000..e69de29bb2d better put here some comment, like in other dummy *.inc files see e.g. platform.inc or protocol.inc > diff --git a/mysql-test/suite/atomic/alter_partition,aria.rdiff b/mysql-test/suite/atomic/alter_partition,aria.rdiff > new file mode 100644 > index 00000000000..20c0510c5ed > --- /dev/null > +++ b/mysql-test/suite/atomic/alter_partition,aria.rdiff > @@ -0,0 +1,219 @@ > +--- alter_partition.result > ++++ alter_partition,aria.reject > +@@ -1,4 +1,4 @@ > +-set @@default_storage_engine=MyISAM; > ++set @@default_storage_engine=Aria; > + # Crash recovery > + create or replace procedure prepare_table() > + begin > +@@ -13,12 +13,12 @@ > + end $ > + # QUERY: ALTER TABLE t1 EXTRACT PARTITION p1 AS TABLE tp1 > + # CRASH POINT: ddl_log_create_before_create_frm > +-t1#P#p0.MYD > +-t1#P#p0.MYI > +-t1#P#p1.MYD may be better `replace_result MAI MYI MAD MYD` ? if that's the only difference. To avoid big rdiff files. > +-t1#P#p1.MYI > +-t1#P#pn.MYD > +-t1#P#pn.MYI > ++t1#P#p0.MAD > ++t1#P#p0.MAI Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 20

Re: [Maria-developers] ca4ef7185da: MDEV-9245 password "reuse prevention" validation plugin
by Sergei Golubchik 09 Sep '21

09 Sep '21

Hi, Oleksandr! you forgot to check in the result file On Sep 07, Oleksandr Byelkin wrote: > revision-id: ca4ef7185da (mariadb-10.6.1-79-gca4ef7185da) > parent(s): 271afbc88e4 > author: Oleksandr Byelkin > committer: Oleksandr Byelkin > timestamp: 2021-09-07 14:35:05 +0200 > message: > > MDEV-9245 password "reuse prevention" validation plugin > > diff --git a/plugin/reuse_password_check/CMakeLists.txt b/plugin/reuse_password_check/CMakeLists.txt > --- /dev/null > +++ b/plugin/reuse_password_check/CMakeLists.txt > @@ -0,0 +1,4 @@ > + > +ADD_DEFINITIONS(-DMYSQL_SERVER) as I wrote to HF, this should not be needed. > + > +MYSQL_ADD_PLUGIN(reuse_password_check reuse_password_check.c MODULE_ONLY) Unless you have a reason why this must be "MODULE_ONLY", please remove MODULE_ONLY and try to compile and test with -DPLUGIN_REUSE_PASSWORD_CHECK=STATIC also, I'd suggest "password_reuse_check" not "reuse_password_check". > diff --git a/plugin/reuse_password_check/reuse_password_check.c b/plugin/reuse_password_check/reuse_password_check.c > --- /dev/null > +++ b/plugin/reuse_password_check/reuse_password_check.c > @@ -0,0 +1,236 @@ > +/* Copyright (c) 2021, Oleksandr Byelkin and MariaDB > + > + This program is free software; you can redistribute it and/or modify > + it under the terms of the GNU General Public License as published by > + the Free Software Foundation; version 2 of the License. > + > + This program is distributed in the hope that it will be useful, > + but WITHOUT ANY WARRANTY; without even the implied warranty of > + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > + GNU General Public License for more details. > + > + You should have received a copy of the GNU General Public License > + along with this program; if not, write to the Free Software > + Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1335 USA */ > + > +#include <my_config.h> > +#include <my_global.h> > +#include <my_base.h> > +#include <mysql/plugin_password_validation.h> > +#include <mysql.h> only plugin_password_validation.h should be needed here. > + > + > +#include <mysqld_error.h> this might be needed all right > +#include <mysql/service_sha2.h> but this isn't needed either > + > +#define HISTORY_DB_NAME "reuse_password_check_history" > + > +#define SQL_BUFF_LEN 2048 > + > +// 0 - unlimited, otherwise number of days to check > +static unsigned interval= 0; > + > +// helping string for bin_to_hex512 > +static char digits[]= "0123456789ABCDEF"; > + > + > +/** > + Convert string of 512 bits (64 bytes) to hex representation > + > + @param to pointer to the result puffer > + (should be at least 64*2 bytes) > + @param str pointer to 512 bits (64 bytes string) > +*/ > + > +static void bin_to_hex512(char *to, const unsigned char *str) > +{ > + const unsigned char *str_end= str + (512/8); > + for (; str != str_end; ++str) > + { > + *to++= digits[((uchar) *str) >> 4]; > + *to++= digits[((uchar) *str) & 0x0F]; > + } > +} > + > + > +/** > + Send SQL error as ER_UNKNOWN_ERROR for information > + > + @param mysql Connection handler > +*/ > + > +static void report_sql_error(MYSQL *mysql) > +{ > + my_printf_error(ER_UNKNOWN_ERROR, "[%d] %s", ME_WARNING, > + mysql_errno(mysql), mysql_error(mysql)); may be include plugin name here? > +} > + > + > +/** > + Create the history of passwords table for this plugin. > + > + @param mysql Connection handler > + > + @retval 1 - Error > + @retval 0 - OK > +*/ > + > +static int create_table(MYSQL *mysql) > +{ > + if (mysql_real_query(mysql, > + // 512/8 = 64 > + STRING_WITH_LEN("CREATE TABLE mysql." HISTORY_DB_NAME > + " ( hash binary(64)," > + " time timestamp," make it "default current_timestamp", just to be explicit. even though it's implicitly default current_timestamp already. > + " primary key (hash), index tm (time) )" > + " ENGINE=Aria"))) > + { > + report_sql_error(mysql); > + return 1; > + } > + return 0; > +} > + > + > +/** > + Run this query and create table if needed > + > + @param mysql Connection handler > + @param query The query to run > + @param len length of the query text > + > + @retval 1 - Error > + @retval 0 - OK > +*/ > + > +static int run_query_with_table_creation(MYSQL *mysql, const char *query, > + size_t len) > +{ > + if (unlikely(mysql_real_query(mysql, query, len))) > + { > + unsigned int rc= mysql_errno(mysql); > + if (rc != ER_NO_SUCH_TABLE) > + { > + // suppress this error in case of try to add the same password twice > + if (rc != ER_DUP_ENTRY) > + report_sql_error(mysql); > + return 1; > + } > + if (create_table(mysql)) > + return 1; > + if (unlikely(mysql_real_query(mysql, query, len))) > + { > + report_sql_error(mysql); > + return 1; > + } > + } > + return 0; > +} > + > + > +/** > + Password validator > + > + @param username User name (part of whole login name) > + @param password Password to validate > + @param hostname Host name (part of whole login name) > + > + @retval 1 - Password is not OK or an error happened > + @retval 0 - Password is OK > +*/ > + > +static int validate(const MYSQL_CONST_LEX_STRING *username, > + const MYSQL_CONST_LEX_STRING *password, > + const MYSQL_CONST_LEX_STRING *hostname) > +{ > + MYSQL *mysql= NULL; > + size_t key_len= username->length + password->length + hostname->length; > + size_t buff_len= (key_len > SQL_BUFF_LEN ? key_len : SQL_BUFF_LEN); > + size_t len; > + char *buff= malloc(buff_len); > + unsigned char hash[512/8]; > + char escaped_hash[512/8*2 + 1]; > + if (!buff) > + return 1; > + > + mysql= mysql_init(NULL); > + if (!mysql) > + { > + free(buff); > + return 1; > + } > + > + memcpy(buff, hostname->str, hostname->length); > + memcpy(buff + hostname->length, username->str, username->length); > + memcpy(buff + hostname->length + username->length, password->str, > + password->length); > + buff[key_len]= 0; > + bzero(hash, sizeof(hash)); > + my_sha512(hash, buff, key_len); > + if (mysql_real_connect_local(mysql, NULL, NULL, NULL, 0) == NULL) > + goto sql_error; > + > + if (interval) > + { > + // trim the table > + len= snprintf(buff, buff_len, > + "DELETE FROM mysql." HISTORY_DB_NAME > + " WHERE time < DATE_SUB(NOW(), interval %d day)", > + interval); > + if (unlikely(run_query_with_table_creation(mysql, buff, len))) > + goto sql_error; > + } > + > + bin_to_hex512(escaped_hash, hash); > + escaped_hash[512/8*2]= '\0'; > + len= snprintf(buff, buff_len, > + "INSERT INTO mysql." HISTORY_DB_NAME "(hash) " > + "values (x'%s')", > + escaped_hash); > + if (unlikely(run_query_with_table_creation(mysql, buff, len))) > + goto sql_error; > + > + free(buff); > + mysql_close(mysql); > + return 0; // OK > + > +sql_error: > + free(buff); > + if (mysql) > + mysql_close(mysql); > + return 1; // Error > +} > + > +static MYSQL_SYSVAR_UINT(interval, interval, PLUGIN_VAR_RQCMDARG, > + "How old (in days) passwords to check (0 means unlimited)", NULL, NULL, better "Password history retention period in days (0 means unlimited)" > > + 0, 0, 365*100, 1); > + > + > +static struct st_mysql_sys_var* sysvars[]= { > + MYSQL_SYSVAR(interval), > + NULL > +}; > + > +static struct st_mariadb_password_validation info= > +{ > + MariaDB_PASSWORD_VALIDATION_INTERFACE_VERSION, > + validate > +}; > + > +maria_declare_plugin(simple_password_check) ^^^ must be the same name as in CMakeLists.txt, otherwise static linking won't work. > +{ > + MariaDB_PASSWORD_VALIDATION_PLUGIN, > + &info, > + "reuse_password_check", > + "Oleksandr Byelkin", > + "Reusage of password check", Better "Password reuse check" or, as it's a description, "Prevent password reuse" > + PLUGIN_LICENSE_GPL, > + NULL, > + NULL, > + 0x0100, > + NULL, > + sysvars, > + "1.0", > + MariaDB_PLUGIN_MATURITY_ALPHA > +} > +maria_declare_plugin_end; > > diff --git a/mysql-test/suite/plugins/t/reuse_password_check.test b/mysql-test/suite/plugins/t/reuse_password_check.test > --- /dev/null > +++ b/mysql-test/suite/plugins/t/reuse_password_check.test > @@ -0,0 +1,58 @@ > +--source include/not_embedded.inc > + > +if (!$REUSE_PASSWORD_CHECK_SO) { > + skip No REUSE_PASSWORD_CHECK plugin; > +} > + > +install soname "reuse_password_check"; > + > +set @save_reuse_password_check_interval= @@reuse_password_check_interval; > + > +set global reuse_password_check_interval= 0; > + > +--echo # Default value (sould be unlimited i.e. 0) > +SHOW GLOBAL VARIABLES like "reuse_password_check%"; > + > +--echo # insert user > +grant select on *.* to user_name@localhost identified by 'test_pwd'; what about sql mode that doesn't allow to create users with grant? I don't see where you've disabled that. > + > +#--error ER_NOT_VALID_PASSWORD > +#grant select on *.* to user_name@localhost identified by 'test_pwd'; > +#show warnings; why is that commented out? > + > +--error ER_CANNOT_USER > +alter user user_name@localhost identified by 'test_pwd'; > +show warnings; > + > +# Plugin does not work for it > +#--error ER_NOT_VALID_PASSWORD > +#SET PASSWORD FOR user_name@localhost = PASSWORD('test_pwd'); > + > +--echo # check exparation > + > +set global reuse_password_check_interval= 10; > + > +--error ER_CANNOT_USER > +alter user user_name@localhost identified by 'test_pwd'; > +show warnings; > +select hex(hash) from mysql.reuse_password_check_history; > + > +--echo # emulate old password > +update mysql.reuse_password_check_history set time= date_sub(now(), interval > +11 day); > + > +alter user user_name@localhost identified by 'test_pwd'; > +show warnings; > + > +drop user user_name@localhost; > + > +show create table mysql.reuse_password_check_history; > +select count(*) from mysql.reuse_password_check_history; > + > +drop table mysql.reuse_password_check_history; > + > +set @save_reuse_password_check_interval= @@reuse_password_check_interval; > + > +set global reuse_password_check_interval= @save_reuse_password_check_interval; > + > +uninstall plugin reuse_password_check; Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] 9e1c15355a9: resolve the possible deadlock issue through intern_plugin_lock
by Sergei Golubchik 08 Sep '21

08 Sep '21

Hi, Nikita! Looks good so far but I noticed too late that it's just one patch on top of others, not a complete thing. See few comments below. Shall I review everything? Is it feature complete except that it skips inactive threads? On Sep 07, Nikita Malyavin wrote: > revision-id: 9e1c15355a9 (mariadb-10.6.3-44-g9e1c15355a9) > parent(s): d7c36600144 > author: Nikita Malyavin > committer: Nikita Malyavin > timestamp: 2021-09-07 18:37:51 +0300 > message: > > resolve the possible deadlock issue through intern_plugin_lock > > diff --git a/sql/sql_plugin.cc b/sql/sql_plugin.cc > index be043bf229a..ecd885ed296 100644 > --- a/sql/sql_plugin.cc > +++ b/sql/sql_plugin.cc > @@ -1053,6 +1053,15 @@ plugin_ref plugin_lock(THD *thd, plugin_ref ptr) > } > this needs a function comment. the function is quite specialized and very different from normal plugin lock functions. it requires a THD (normally it's optional), it does not return a locked plugin like the others (because, exactly, it's stored in the THD). because it's not what people usually expect from a plugin_lock* function, the comment should describe all these differences. > +void plugin_lock_by_var_if_different(THD *thd, sys_var *var, sys_var *var_prev) > +{ > + sys_var_pluginvar *pv= var->cast_pluginvar(); > + sys_var_pluginvar *pv_prev= var_prev ? var_prev->cast_pluginvar() : NULL; > + if (pv && (!pv_prev || pv->plugin != pv_prev->plugin)) > + intern_plugin_lock(thd->lex, plugin_int_to_ref(pv->plugin)); > +} > + > + > /* > Notes on lifetime: > > diff --git a/storage/perfschema/pfs_variable.cc b/storage/perfschema/pfs_variable.cc > index 984b68e699b..85656532c72 100644 > --- a/storage/perfschema/pfs_variable.cc > +++ b/storage/perfschema/pfs_variable.cc > @@ -100,6 +102,15 @@ bool PFS_system_variable_cache::init_show_var_array(enum_var_type scope, bool st > > mysql_prlock_unlock(&LOCK_system_variables_hash); > > + for (uint i= 0; i < m_show_var_array.elements(); i++) why separately, not in the loop above? > + { > + sys_var *var= (sys_var *)m_show_var_array.at(i).value; > + if (!var) continue; can var be NULL? > + sys_var *prev= i == 0 ? NULL : (sys_var *)m_show_var_array.at(i-1).value; hmm, normally this is done with sys_var *prev= NULL; before the loop and prev= var; in the loop, after plugin_lock_by_var_if_different(). > + > + plugin_lock_by_var_if_different(m_current_thd, var, prev); > + } > + > /* Increase cache size if necessary. */ > m_cache.reserve(m_show_var_array.elements()); > > @@ -316,7 +327,27 @@ class PFS_system_variable_cache_apc: public Apc_target::Apc_call > > void call_in_target_thread() override > { > - (m_pfs->*m_func)(m_param); > + call(m_pfs, m_func, m_param); > + } > +public: > + static void call(PFS_system_variable_cache *pfs, Request func, uint param) > + { > + THD *safe_thd= pfs->safe_thd(); > + > + if (pfs->query_scope() == OPT_SESSION) what's the point doing apc call if the query_scope() is not OPT_SESSION? It seems like here you need a DBUG_ASSERT, and the if() should be much earlier up the stack. > + { > + mysql_mutex_lock(&LOCK_global_system_variables); > + if (!safe_thd->variables.dynamic_variables_ptr || > + global_system_variables.dynamic_variables_head > > + safe_thd->variables.dynamic_variables_head) > + { > + mysql_prlock_rdlock(&LOCK_system_variables_hash); > + sync_dynamic_session_variables(safe_thd, false); > + mysql_prlock_unlock(&LOCK_system_variables_hash); > + } > + mysql_mutex_unlock(&LOCK_global_system_variables); > + } > + (pfs->*func)(param); > } > }; Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 2

Re: [Maria-developers] 4645a1e1950: MDEV-26238: Remove inconsistent behaviour of --default-* options in my_print_defaults
by Sergei Golubchik 07 Sep '21

07 Sep '21

Hi, Rucha! On Sep 07, Rucha Deodhar wrote: > revision-id: 4645a1e1950 (mariadb-10.6.1-72-g4645a1e1950) > parent(s): 21ce69123c1 > author: Rucha Deodhar > committer: Rucha Deodhar > timestamp: 2021-09-02 22:10:37 +0530 > message: > > MDEV-26238: Remove inconsistent behaviour of --default-* options > in my_print_defaults > > Analysis: --defaults* option is recognized anywhere in the commandline > instead of only at the beginning because handle_options() recognizes > options in any order. > Fix: use get_defaults_options() which recognizes --defaults* options only at > the beginning. After this is done, we only want to recognize other options > given in any order which can be done using handle_options(). But > handle_options() returns error and exits if an option is not found. > We don't want to exit and give error because --defaults* options are still > valid. So make a new commandline which doesn't have --defaults* > option and pass it to handle_options(). Also, removed -e, -g and -c because > only my_print_defaults supports them. > > diff --git a/extra/my_print_defaults.c b/extra/my_print_defaults.c > index b7f52382721..e2ec0b00923 100644 > --- a/extra/my_print_defaults.c > +++ b/extra/my_print_defaults.c > @@ -48,20 +48,6 @@ static struct my_option my_long_options[] = you forgot the "config-file", which is here, just about "debug" > {"debug", '#', "Output debug log", (char**) &default_dbug_option, > (char**) &default_dbug_option, 0, GET_STR, OPT_ARG, 0, 0, 0, 0, 0, 0}, > #endif > - {"defaults-file", 'c', > - "Read this file only, do not read global or per-user config " > - "files; should be the first option", > - (char**) &config_file, (char*) &config_file, 0, GET_STR, REQUIRED_ARG, > - 0, 0, 0, 0, 0, 0}, > - {"defaults-extra-file", 'e', > - "Read this file after the global config file and before the config " > - "file in the users home directory; should be the first option", > - (void *)&my_defaults_extra_file, (void *)&my_defaults_extra_file, 0, > - GET_STR, REQUIRED_ARG, 0, 0, 0, 0, 0, 0}, > - {"defaults-group-suffix", 'g', > - "In addition to the given groups, read also groups with this suffix", > - (char**) &my_defaults_group_suffix, (char**) &my_defaults_group_suffix, > - 0, GET_STR, REQUIRED_ARG, 0, 0, 0, 0, 0, 0}, > {"mysqld", 0, "Read the same set of groups that the mysqld binary does.", > &opt_mysqld, &opt_mysqld, 0, GET_BOOL, NO_ARG, 0, 0, 0, 0, 0, 0}, > {"no-defaults", 'n', "Return an empty string (useful for scripts).", > @@ -109,9 +95,6 @@ get_one_option(const struct my_option *opt __attribute__((unused)), > const char *filename __attribute__((unused))) > { > switch (opt->id) { > - case 'c': > - opt_defaults_file_used= 1; > - break; > case 'n': > cleanup_and_exit(0); > case 'I': > @@ -141,49 +124,44 @@ static int get_options(int *argc,char ***argv) > return 0; > } > > -static char *make_args(const char *s1, const char *s2) > -{ > - char *s= malloc(strlen(s1) + strlen(s2) + 1); > - strmov(strmov(s, s1), s2); > - return s; > -} > - > int main(int argc, char **argv) > { > - int count= 0, error, no_defaults= 0; > + int count, error, args_used; > char **load_default_groups= 0, *tmp_arguments[6]; > - char **argument, **arguments, **org_argv; > - int nargs, i= 0; > + char **argument, **arguments, **org_argv, **new_argv, **pos; > + int nargs, i= 0, index=0; > MY_INIT(argv[0]); > > org_argv= argv; > - if (*argv && !strcmp(*argv, "--no-defaults")) > - { > - argv++; > - ++count; > - no_defaults= 1; > - } > - /* Copy program name and --no-defaults if present*/ > + args_used= get_defaults_options(argv); > + > + /* Copy defaults-xxx arguments & program name */ > + count= args_used; > arguments= tmp_arguments; > - memcpy((char*) arguments, (char*) org_argv, (++count)*sizeof(*org_argv)); > + memcpy((char*) arguments, (char*) org_argv, count*sizeof(*org_argv)); > arguments[count]= 0; > > - /* Check out the args */ > - if (get_options(&argc,&argv)) > - cleanup_and_exit(1); > - > - if (!no_defaults) > + /* > + We already process --defaults* options at the beginning in > + get_defaults_options(). So, now we we make a new commandline which > + has all options except --defaults* options and pass it to handle_options() > + instead of original commandline. > + */ > + new_argv= (char**)malloc(argc*sizeof(*org_argv)); > + for (pos= org_argv; *pos; pos++) > { > + if (!is_prefix(*pos, "--defaults-extra-file=") && > + !is_prefix(*pos, "--defaults-file=") && > + !is_prefix(*pos, "--defaults-group-suffix=")) > + new_argv[index++]= *(pos); Why are you doing it? After get_defaults_options, you know how many arguments were default, just skip args_used, and invoke handle_options on the rest. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 271afbc88e4: Pre requiste MDEV-9245: add host to validate password interface
by Sergei Golubchik 07 Sep '21

07 Sep '21

Hi, Oleksandr! this is ok! On Sep 07, Oleksandr Byelkin wrote: > revision-id: 271afbc88e4 (mariadb-10.6.1-78-g271afbc88e4) > parent(s): 47207ea291a > author: Oleksandr Byelkin > committer: Oleksandr Byelkin > timestamp: 2021-09-07 11:52:28 +0200 > message: > > Pre requiste MDEV-9245: add host to validate password interface > > Hostname added to the interface. > > Cracklib plugin fixed according to the new interface. > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] b48dfdedd59: MDEV-26307 multi-source-replication support mysql syntax(for channel)
by Sergei Golubchik 02 Sep '21

02 Sep '21

Hi, Michael! On Sep 02, Michael Widenius wrote: > revision-id: b48dfdedd59 (mariadb-10.6.1-64-gb48dfdedd59) > parent(s): 5f0b6b017a8 > author: Michael Widenius > committer: Michael Widenius > timestamp: 2021-08-24 11:22:29 +0300 > message: > > MDEV-26307 multi-source-replication support mysql syntax(for channel) > > diff --git a/sql/sql_yacc.yy b/sql/sql_yacc.yy > index 0c1c68cd644..704812e5dd8 100644 > --- a/sql/sql_yacc.yy > +++ b/sql/sql_yacc.yy > @@ -2307,6 +2309,34 @@ connection_name: > } > ; > > +optional_for_channel: > + /* empty */ > + { > + /*do nothing */ > + } > + | for_channel > + > + ; > + > +for_channel: > + FOR_SYM CHANNEL_SYM TEXT_STRING_sys > + { > + if (Lex->mi.connection_name.str != NULL) > + { > + my_yyabort_error((ER_WRONG_ARGUMENTS, MYF(0), "CONNECTION_NAME AND FOR CHANNEL CAN NOT BE SPECIFIED AT THE SAME TIME)")); No, please, no LONG ENGLISH MESSAGES IN UPPERCASE WITH STRAY PARENTHESYS) embedded as a part of the error message. In your tests it looks like ERROR HY000: Incorrect arguments to CONNECTION_NAME AND FOR CHANNEL CAN NOT BE SPECIFIED AT THE SAME TIME) or, for example, ERROR HY000: Felaktiga argument till CONNECTION_NAME AND FOR CHANNEL CAN NOT BE SPECIFIED AT THE SAME TIME) The correct approach is to have the parser ensure the syntax is correct. Meaning, it should be START_SYM SLAVE optional_connection_name slave_thread_opts | START_SYM SLAVE slave_thread_opts optional_for_channel the incorrect but still kind of acceptable approach is to fake this and check in the code, like you did, but use my_parse_error, not ER_WRONG_ARGUMENTS > + } > + else > + { > + Lex->mi.connection_name= $3; > +#ifdef HAVE_REPLICATION > + if (unlikely(check_master_connection_name(&$3))) > + my_yyabort_error((ER_WRONG_ARGUMENTS, MYF(0), "MASTER_CONNECTION_NAME")); > +#endif > + } > + > + } > + ; > + > /* create a table */ > > create: > @@ -8030,7 +8060,7 @@ opt_to: > ; > > slave: > - START_SYM SLAVE optional_connection_name slave_thread_opts > + START_SYM SLAVE optional_connection_name slave_thread_opts optional_for_channel > { > LEX *lex=Lex; > lex->sql_command = SQLCOM_SLAVE_START; Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] dc2ace70f1b: Syntax with MIGRATE keyword
by Sergei Golubchik 01 Sep '21

01 Sep '21

Hi, Aleksey! On Sep 01, Aleksey Midenkov wrote: > revision-id: dc2ace70f1b (mariadb-10.6.1-68-gdc2ace70f1b) > parent(s): 19fbfab084f > author: Aleksey Midenkov > committer: Aleksey Midenkov > timestamp: 2021-08-30 22:37:08 +0300 > message: > > Syntax with MIGRATE keyword > > ALTER TABLE tbl_name > [alter_option [, alter_option] ...] | > [partition_options] > > partition_option: { > ... > | MIGRATE [OUT] PARTITION partition_name TO [TABLE] tbl_name > } > > Examples: > > ALTER TABLE t1 MIGRATE PARTITION p1 TO tp1; > ALTER TABLE t1 MIGRATE PARTITION p2 TO TABLE tp2; > ALTER TABLE t1 MIGRATE OUT PARTITION p3 TO TABLE tp3; > > diff --git a/mysql-test/suite/parts/r/alter_table.result b/mysql-test/suite/parts/r/alter_table.result > index bc1bf661d4c..7a2d45c51c1 100644 > --- a/mysql-test/suite/parts/r/alter_table.result > +++ b/mysql-test/suite/parts/r/alter_table.result > @@ -258,9 +258,46 @@ show grants for current_user; > Grants for alan@% > GRANT USAGE ON *.* TO `alan`@`%` > GRANT INSERT, CREATE, DROP, ALTER ON `test`.* TO `alan`@`%` > -alter table t1 extract partition p1 as table tp1; > +alter table t1 migrate partition p1 to table tp1; > disconnect alan; > connection default; > drop database EXISTENT; > drop user alan; > drop tables t1, tp1, tp2, tp4; > +# Cunning syntax what's so cunning here? > +create or replace table t1 (x int) > +partition by range(x) ( > diff --git a/sql/handler.h b/sql/handler.h > index c455625aafe..7ac9e929ecc 100644 > --- a/sql/handler.h > +++ b/sql/handler.h > @@ -824,7 +824,7 @@ typedef bool Log_func(THD*, TABLE*, bool, const uchar*, const uchar*); > #define ALTER_PARTITION_TRUNCATE (1ULL << 11) > // Set for REORGANIZE PARTITION > #define ALTER_PARTITION_TABLE_REORG (1ULL << 12) > -#define ALTER_PARTITION_EXTRACT (1ULL << 13) > +#define ALTER_PARTITION_MIGRATE_OUT (1ULL << 13) > > /* > This is master database for most of system tables. However there > diff --git a/sql/sql_yacc.yy b/sql/sql_yacc.yy > index 49dd244f2ff..e1be6ea9557 100644 > --- a/sql/sql_yacc.yy > +++ b/sql/sql_yacc.yy > @@ -7492,6 +7492,16 @@ ident_or_empty: > | ident > ; > > +move_out_partition: > + MIGRATE_SYM PARTITION_SYM > + | MIGRATE_SYM OUT_SYM PARTITION_SYM Where does this OUT_SYM came out from? It doesn't add any values, it doesn't make the syntax more natural. Why did you add it? > + ; > + > +to_table: > + TO_SYM TABLE_SYM > + | TO_SYM Let's keep the TABLE_SYM. Saying MIGRATE PARTITION p1 TO p2 is very vague, the syntax should explicitly say that it migrates a PARTITION to a TABLE. The statement mentions many different types of objects, if should be clear about what object type every identifier refers to. > + ; > + > alter_commands: > /* empty */ > | DISCARD TABLESPACE > @@ -7619,15 +7629,15 @@ alter_commands: > MYSQL_YYABORT; > Lex->alter_info.partition_flags|= ALTER_PARTITION_EXCHANGE; > } > - | EXTRACT_SYM PARTITION_SYM alt_part_name_item > - AS TABLE_SYM table_ident have_partitioning > + | move_out_partition alt_part_name_item > + to_table table_ident have_partitioning please, let it be just MIGRATE_SYM PARTITION_SYM alt_part_name_item TO_SYM TABLE_SYM table_ident have_partitioning > { > - if (Lex->stmt_alter_table($6)) > + if (Lex->stmt_alter_table($4)) > MYSQL_YYABORT; > Lex->m_sql_cmd= new (thd->mem_root) Sql_cmd_alter_table(); > if (unlikely(Lex->m_sql_cmd == NULL)) > MYSQL_YYABORT; > - Lex->alter_info.partition_flags|= ALTER_PARTITION_EXTRACT; > + Lex->alter_info.partition_flags|= ALTER_PARTITION_MIGRATE_OUT; > } > ; Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 5

Re: [Maria-developers] 19fbfab084f: Compilation fix
by Sergei Golubchik 01 Sep '21

01 Sep '21

Hi, Aleksey! On Sep 01, Aleksey Midenkov wrote: > revision-id: 19fbfab084f (mariadb-10.6.1-67-g19fbfab084f) > parent(s): 8009680fbc3 > author: Aleksey Midenkov > committer: Aleksey Midenkov > timestamp: 2021-08-30 22:02:40 +0300 > message: > > Compilation fix > > diff --git a/sql/sql_table.h b/sql/sql_table.h > index 83e88c82f0e..8cb9ebe0445 100644 > --- a/sql/sql_table.h > +++ b/sql/sql_table.h > @@ -20,8 +20,14 @@ > #include <my_sys.h> // pthread_mutex_t > #include "m_string.h" // LEX_CUSTRING > > +inline bool suicide() > +{ > + DBUG_SUICIDE(); > + return false; > +} > + > #define ERROR_INJECT_CRASH(code) \ > - (DBUG_IF(code) && (DBUG_SUICIDE(), 0)) > + (DBUG_IF(code) && suicide()) What did that change? A compiler didn't like (void,0) ? Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 2

Re: [Maria-developers] 87953156c76: Vanilla cleanup: DBUG_EVALUATE_IF() macro replaced by shorter form DBUG_IF()
by Sergei Golubchik 01 Sep '21

01 Sep '21

Hi, Aleksey! On Sep 01, Aleksey Midenkov wrote: > revision-id: 87953156c76 (mariadb-10.6.1-61-g87953156c76) > parent(s): b67fe0c009d > author: Aleksey Midenkov > committer: Aleksey Midenkov > timestamp: 2021-08-30 21:24:35 +0300 > message: > > Vanilla cleanup: DBUG_EVALUATE_IF() macro replaced by shorter form DBUG_IF() Thanks, I agree, this looks much better > diff --git a/dbug/user.r b/dbug/user.r > index 8d8a9ce6910..3740a60851c 100644 > --- a/dbug/user.r > +++ b/dbug/user.r > @@ -785,18 +785,20 @@ EX:\fC > +.B 0 > +Like DBUG_EXECUTE_IF this could be used to conditionally execute > "dangerous" actions. > .SP 1 > EX:\fC > .br > if (prepare_transaction () || > .br > - DBUG_EVALUATE ("crashme", (DBUG_ABORT(), 0), 0) || > + (DBUG_IF("crashme") && (DBUG_ABORT(), 0)) || oh, a bug in the manual! good catch. > .br > commit_transaction () )\fR > .SP 1 > diff --git a/include/my_dbug.h b/include/my_dbug.h > index e25bfcf28a7..2afcb010e7c 100644 > --- a/include/my_dbug.h > +++ b/include/my_dbug.h > @@ -102,8 +102,7 @@ extern int (*dbug_sanity)(void); > do {if (_db_keyword_(0, (keyword), 1)) { a1 }} while(0) > #define DBUG_EVALUATE(keyword,a1,a2) \ > (_db_keyword_(0,(keyword), 0) ? (a1) : (a2)) you've kept DBUG_EVALUATE, although it was only added for consistency together with DBUG_EVALUATE_IF. And DBUG_EVALUATE is not used anywhere, I've just grepped. Remove it too? > -#define DBUG_EVALUATE_IF(keyword,a1,a2) \ > - (_db_keyword_(0,(keyword), 1) ? (a1) : (a2)) > +#define DBUG_IF(keyword) _db_keyword_(0, (keyword), 1) > #define DBUG_PRINT(keyword,arglist) \ > do if (_db_pargs_(__LINE__,keyword)) _db_doprnt_ arglist; while(0) > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] f14cfd22115: Vanilla refactoring: binlog query in DDL_LOG_RENAME_ACTION
by Aleksey Midenkov 24 Aug '21

24 Aug '21

Hi Sergei! Really, this is not pure refactoring. I've been planning to remove it if it will not be useful for MDEV-22165 (that task should be based on this branch). Meanwhile, can you please review other commits? On Mon, Aug 23, 2021 at 7:35 PM Sergei Golubchik <serg(a)mariadb.com> wrote: > > Hi, Aleksey! > > On Aug 23, Aleksey Midenkov wrote: > > revision-id: f14cfd22115 (mariadb-10.6.1-63-gf14cfd22115) > > parent(s): 03225474d60 > > author: Aleksey Midenkov > > committer: Aleksey Midenkov > > timestamp: 2021-08-17 20:01:48 +0300 > > message: > > > > Vanilla refactoring: binlog query in DDL_LOG_RENAME_ACTION > > > > This turned out to be not needed for MDEV-22166. But it still might be > > useful for MDEV-22165 or later. > > There are three changes here. > Adding 'locked' to ddl_log_store_query, creating > st_ddl_recovery::binlog_query(), adding 'case RENAME_PHASE_BINLOG' > > st_ddl_recovery::binlog_query() is ok, if you want to keep it. > But the other two aren't a refactoring, it's creating new functionality > which isn't used in this patch, dead code basically. And your commit > comment seems to imply that this new functionality isn't needed > in the following feature patch either. > > While your other refactoring patches are indeed pure refactoring, this > one isn't, so, please, remove it from the patchset if it isn't used > there. You can keep st_ddl_recovery::binlog_query() if you'd like - it > is indeed pure refactoring. > > Regards, > Sergei > VP of MariaDB Server Engineering > and security(a)mariadb.org -- All the best, Aleksey Midenkov @midenok

2 1

Re: [Maria-developers] bc55e3c438f: MDEV-4742 - provide function to sort string in "natural" order
by Sergei Golubchik 20 Aug '21

20 Aug '21

Hi, Vladislav! On Aug 20, Vladislav Vaintroub wrote: > commit bc55e3c438f > Author: Vladislav Vaintroub <wlad(a)mariadb.com> > Date: Thu Jul 29 13:28:11 2021 +0200 > > MDEV-4742 - provide function to sort string in "natural" order > > The numbers should be compared as numbers, while the rest should be compared > as string. > > Introduce natural_sort_key() function that transforms original string > so that the lexicographic order of such keys is suitable for > natural sort. This comment doesn't mention the second optional argument. Neither does Jira. > diff --git a/mysql-test/main/natural_sort_key.test b/mysql-test/main/natural_sort_key.test > new file mode 100644 > index 00000000000..bee0d8e76bd > --- /dev/null > +++ b/mysql-test/main/natural_sort_key.test > @@ -0,0 +1,64 @@ > +--source include/have_innodb.inc > +--source include/have_sequence.inc > + > +SET NAMES utf8mb4; > +SELECT NATURAL_SORT_KEY(NULL); > + > +# Sort without tables > +SELECT '' c WHERE 0 UNION VALUES('a10'),('a9'),('a1000'), ('a0'),('b'),('b0') ORDER BY NATURAL_SORT_KEY(c); > + > +#Test that max packet overflow produces NULL plus warning > +SELECT NATURAL_SORT_KEY(repeat('a1',@@max_allowed_packet/2-1)); > + > +#Test with virtual(index only) key > +CREATE TABLE t1( > + c VARCHAR(30) CHARACTER SET latin1 COLLATE latin1_bin, > + k VARCHAR(45) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci AS (NATURAL_SORT_KEY(CONVERT(c USING utf8mb4))) INVISIBLE, > + KEY(k,c)) ENGINE=InnoDB; > +INSERT INTO t1 values > + ('A1'),('a1'),('A100'),('a100'),('A2'),('ä2'),('a2'),('A99'), > + ('äb'),('B1'),('B100'),('B9'),('C'),('100'); > +EXPLAIN SELECT c FROM t1 ORDER BY k,c; > +-- echo #Natural sort order. > +# We sort by 2 colums, for stable sort,as we do not currenly have a case and accent insensitive Unicode collation. > +SELECT c FROM t1 ORDER BY k,c; > +-- echo #Unnatural but unicode aware) sort order > +SELECT c FROM t1 ORDER BY CONVERT(c USING utf8mb4) COLLATE utf8mb4_unicode_ci,c; > +# CREATE TABLE AS SELECT, to see that length of the column is correct. > +CREATE TABLE t2 AS SELECT c, NATURAL_SORT_KEY(c) FROM t1 WHERE 0; you could've simply used --enable_metadata > +SHOW CREATE TABLE t2; > +DROP TABLE t1,t2; > + > +#Show encoding of numbers, including fractions, and leading whitespace. > +SELECT RPAD(val,21,' ') value , RPAD(NATURAL_SORT_KEY(val),26,' ') sortkey , LENGTH(NATURAL_SORT_KEY(val)) - LENGTH(val) encoding_overhead > +FROM > +( > +SELECT 0 val > +UNION VALUES ('0.0'),('0.1'), ('0,15'),('1.001'),('1.002'),('1.010'),('1.02'),('1.1'),('1.3'),('1'),('01'),('0001') > +UNION SELECT CONCAT('1',repeat('0',seq)) FROM seq_1_to_20 > +) AS numbers ORDER BY sortkey; > + > +--echo # Disable fractions handling by passing NULL as second parameter to NATURAL_SORT_KEY > +SELECT val value > +FROM > +( > +SELECT 0 val > +UNION VALUES ('0.1'),('1.001'),('1.002'),('1.010'),('1.02'),('1.1'),('1.3'),('1'), ('0,1'),('1,001'),('1,002'),('1,010'),('1,02'),('1,1'),('1,3'),('1')) > +AS numbers ORDER BY NATURAL_SORT_KEY(val, NULL); > + > +--echo # Use ',' as decimal separator for NATURAL_SORT_KEY > +SELECT val value, NATURAL_SORT_KEY(val,',') sortkey > +FROM > +( > +SELECT 0 val > +UNION VALUES ('0,1'),('1,001'),('1,002'),('1,010'),('1,02'),('1,1'),('1,3'),('1')) > +AS numbers ORDER BY sortkey; > + > +--echo # Use '.' as decimal separator for NATURAL_SORT_KEY > +SELECT val value,NATURAL_SORT_KEY(val,'.') sortkey > +FROM > +( > +SELECT 0 val > +UNION VALUES ('0.1'),('1.001'),('1.002'),('1.010'),('1.02'),('1.1'),('1.3'),('1')) > +AS numbers ORDER BY sortkey; > +SET NAMES DEFAULT; > diff --git a/sql/item_strfunc.cc b/sql/item_strfunc.cc > index 7e8ff667e75..a2efd414750 100644 > --- a/sql/item_strfunc.cc > +++ b/sql/item_strfunc.cc > @@ -5293,6 +5293,232 @@ String *Item_temptable_rowid::val_str(String *str) > return &str_value; > } > > +/** > + Here is how we generate a key suitable for lexicographic comparison > + from a numeric string (representing positive number). > + > + We prepend variable length prefix, that only encodes string length > + The longer the string, the lexicographically larger is the prefix. > + > + Rules for generating prefix is following. > + > + 1. Small length (1 to 9) > + Prefix is single char '0' + length -1 > + This gives the range '0' - '8' > + 2. Medium length(10 to 18) > + Prefix is 2 chars - concat('9', '0'+length-10)' > + This gives range '90'-'98' > + 3. Large length(19) > + Prefix is "990" > + 4. Huge length (20 to 2^32-1) > + Prefix stars with '99', then log10 of the length is added as char > + then the number itself is added > + The range is '99120' - '9994294967295' please also explain here how you handle leading zeros > +*/ > + > +#define NATSORT_BUFSIZE 16 > +static size_t natsort_num2str(size_t n, char *s) > +{ > + if (n < 10) > + { > + s[0]= '0' + (char)n -1; > + return 1; > + } > + > + if (n < 19) > + { > + s[0]= '9'; > + s[1]= '0'+ (char)n- 10; > + return 2; > + } > + > + if (n == 19) > + { > + s[0]= '9'; > + s[1]= '9'; > + s[2]= '0'; > + return 3; > + } > + > + /* Here, n > 19*/ > + s[0]= '9'; > + s[1]= '9'; > + size_t log10n= 0; > + for (size_t tmp= n/10; tmp; tmp /= 10) > + log10n++; > + DBUG_ASSERT(log10n && log10n < 10); first, I don't see where the caller ensures that log10n < 10 (in other words, that n < 9999999999) second, please, never write assert(a && b), always do two separate asserts, so that when it fires, it'd be clear what condition was false > + s[2]='0' + (char)log10n; > + return ll2str(n,s+3,10,0) - s; > +} > + > +/** > + Convert a string to natural sort key. > + @param[in] in - input string > + @param[out] s - output string > + > + We assume that memory is preallocated for the output > + string, so that appends do not fail. > + > + The lead zeros count, if positive, is stored after > + the numeric string. > + > + Fractional parts are stored and compared as strings. > + Even if they have numbers in them, they should just > + be compared as strings. > +*/ > +static void to_natsort_key(const String *in, String *s, my_wc_t decimal_sep) > +{ > + CHARSET_INFO *cs= in->charset(); > + uchar *pos= (uchar *) in->ptr(); > + uchar *end= pos + in->length(); > + size_t num_len= 0; > + size_t leading_zeros= 0; > + uchar *num_start= nullptr; > + bool in_fraction= false; > + size_t fraction_len=0; > + > + for (;;) > + { > + my_wc_t c= 0; > + int charlen= cs->mb_wc(&c, pos, end); > + bool is_digit= charlen >= 1 && (c >= '0' && c <= '9'); what do you mean `>= 1` ? how can it be `> 1` and still satisfy the secon part of the condition? > + > + if (!is_digit && (num_start || leading_zeros)) > + { > + /* > + Handle end of digits run. > + > + Write prefix, natsort_num2str(length) is without leading zeros > + Write numeric value, without leading zeros > + If there were leading zeros, write variable length suffix > + */ > + char buf[NATSORT_BUFSIZE]; > + if (!num_len) > + { > + // number was zeros only > + leading_zeros--; > + num_len= 1; > + } > + > + s->append(buf, natsort_num2str(num_len, buf)); > + if (!num_start) > + s->append('0'); > + else > + s->append((const char *) num_start, pos - num_start, cs); > + > + if (leading_zeros) > + s->append(buf, natsort_num2str(leading_zeros, buf)); hmm. How would it sort "1;", "001;", "1a" ? > + > + /* Reset state.*/ > + num_len= 0; > + num_start= nullptr; > + leading_zeros= 0; > + in_fraction= c == decimal_sep; I don't really like this idea. First, see https://natsort.readthedocs.io/en/master/index.html there are many ways one can potentially want to customize natsort. (of course, none of them have to be in this MDEV) One of those customizations, indeed, is treating floating point numbers as floating point numbers. But comma vs dot is locale thing, and a one string argument it's not how you should tell the function what decimal separator the current locale uses. So. Simple approach - remove the second argument, no floating point support. Complex approach - have a second argument that enables floating point support, but that can be used in the future to specify more customizations, not that is hard-wired to specify decimal separator duplicating and bypassing the locale. > + fraction_len= 0; > + } > + > + if (charlen < 1) > + break; > + > + if (is_digit && !in_fraction) > + { > + /* Do not count leading zeros in num_len, and num_pos */ > + if (c != '0' || num_len) > + { > + num_len++; > + if (!num_start) > + num_start= pos; > + } > + else > + leading_zeros++; > + } > + else > + { > + /* Append a non-digit, or fractional part of the number (handling the same as string) */ > + s->append((const char *) pos, charlen); > + if (in_fraction) > + { > + if (!is_digit && fraction_len) > + in_fraction= false; > + fraction_len++; > + } > + } > + pos+= charlen; > + } > +} > + > + > +String *Item_func_natural_sort_key::val_str(String *s) > +{ > + if (args[0]->is_null()) > + { > + null_value= true; > + return nullptr; > + } > + > + String *in= args[0]->val_str(); > + CHARSET_INFO *cs= in->charset(); > + ulong max_allowed_packet= current_thd->variables.max_allowed_packet; > + size_t reserve_length= (size_t) in->length() + in->length() / 2 + 2; > + if (s->alloc((uint32)reserve_length)) > + goto error_exit; > + s->length(0); > + s->set_charset(cs); > + > + to_natsort_key(in, s, decimal_separator()); > + > + DBUG_ASSERT(s->length() < reserve_length); > + > + if (s->length() > max_allowed_packet) better check this inside to_natsort_key(). max_allowed_packet is supposed to prevent the string from growing too large, instead of just letting it grow and erroring out afterwards. > + { > + push_warning_printf(current_thd, Sql_condition::WARN_LEVEL_WARN, > + ER_WARN_ALLOWED_PACKET_OVERFLOWED, > + ER(ER_WARN_ALLOWED_PACKET_OVERFLOWED), func_name(), > + max_allowed_packet); > + goto error_exit; > + } > + return s; > + > +error_exit: > + null_value=true; > + return nullptr; > +} > + > + > +my_wc_t Item_func_natural_sort_key::decimal_separator() > +{ > + if (m_decimal_separator != DECIMAL_SEP_UNDEFINED) > + return m_decimal_separator; > + > + DBUG_ASSERT(arg_count > 1); > + String *s= args[1]->val_str(); > + if (s && s->length()) > + { > + my_wc_t c; > + if (s->charset()->mb_wc(&c, (const uchar *) s->ptr(), > + (const uchar *) s->end())) > + return c; > + } > + return DECIMAL_SEP_NONE; > +} > + > +bool Item_func_natural_sort_key::fix_length_and_dec(void) > +{ > + if (agg_arg_charsets_for_string_result(collation, args, 1)) > + return true; > + DBUG_ASSERT(collation.collation != NULL); > + uint32 max_clen= args[0]->max_char_length(); > + fix_char_length(max_clen + max_clen/2 + ((max_clen <= 3)?1:0)); > + set_maybe_null(); better, only set maybe_null if the first argument maybe_null or the first argument maybe_too_long > + > + if (arg_count> 1 && args[1]->can_eval_in_optimize()) > + { > + /* Initialize decimal separator */ > + m_decimal_separator= decimal_separator(); > + } > + return false; > +} > + > #ifdef WITH_WSREP > > #include "wsrep_mysqld.h" Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] a19495728de: MDEV-16355 Add option for mysqldump to read data as of specific timestamp from system-versioned tables
by Sergei Golubchik 15 Aug '21

15 Aug '21

Hi, Aleksey! I'm reviewing it separately from other commits in the branch, because it looks to be functionally independent from, say, MDEV-16029 mysqldump: dump and restore historical data and doesn't have to wait for the latter to be pushed first. I think I was able to guess what the patch would look like, if you'd apply it to vanilla 10.7 without MDEV-16029. But just to be sure, please, let me look it over again after the cherry-pick. Otherwise no comments code-wise. A couple of comments related to naming, see below. On Aug 15, Aleksey Midenkov wrote: > revision-id: a19495728de (versioning-1.0.5-45-ga19495728de) > parent(s): be4605ae086 > author: Aleksey Midenkov > committer: Aleksey Midenkov > timestamp: 2018-07-19 14:47:35 +0300 > message: > > MDEV-16355 Add option for mysqldump to read data as of specific timestamp from system-versioned tables > > mysqldump changes: > > * --asof-timestamp option specifies historical point; "asof" isn't a word. I'd suggest --as-of-timestamp or simply --as-of > * with --dump-history it dumps all prior history, without it dumps only single point in time; > * query forging protection for --asof-timestamp parameter. > > diff --git a/client/mysqldump.c b/client/mysqldump.c > index eee12555271..f426e8c58ef 100644 > --- a/client/mysqldump.c > +++ b/client/mysqldump.c > @@ -1099,6 +1102,13 @@ static int get_options(int *argc, char ***argv) > my_progname_short); > return(EX_USAGE); > } > + if (opt_asof_timestamp && strchr(opt_asof_timestamp, '\'')) > + { > + fprintf(stderr, > + "%s: --asof-timestamp wrong argument!\n", > + my_progname_short); May be, use the wording that is closer to the error one would get from any other invalid character, like, if one would do --asof-timestamp='!@#$%' That is, instead of "wrong argument", something like fprintf(stderr, "%s: Incorrect DATETIME value: '%s'\n", opt_asof_timestamp, my_progname_short); Because MariaDB [test]> select timestamp'2010!@#$%'; ERROR 1525 (HY000): Incorrect DATETIME value: '2010!@#$%' > + return(EX_USAGE); > + } > if (strcmp(default_charset, charset_info->csname) && > !(charset_info= get_charset_by_csname(default_charset, > MY_CS_PRIMARY, MYF(MY_WME)))) Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] cdb6db8ce38: MDEV-25444 mysql --binary-mode is not able to replay some mysqlbinlog outputs
by Sergei Golubchik 15 Aug '21

15 Aug '21

Hi, Sachin! On Jul 12, Sachin Kumar wrote: > revision-id: cdb6db8ce38 (mariadb-10.3.26-155-gcdb6db8ce38) > parent(s): 72753d2b65f > author: Sachin Kumar <sachin.setiya(a)mariadb.com> > committer: Sachin Kumar <sachin.setiya(a)mariadb.com> > timestamp: 2021-05-19 15:46:57 +0100 > message: > > MDEV-25444 mysql --binary-mode is not able to replay some mysqlbinlog outputs > > Problem:- Some binary data is inserted into the table using Jconnector. When > binlog dump of the data is applied using mysql cleint it gives syntax error. > > Reason:- > After investigating it turns out to be a issue of mysql client not able to properly > handle \\\0 <0 in binary>. In all binary files where mysql client fails to insert > these 2 bytes are commom (0x5c00) > > Solution:- > I have changed mysql.cc to include for the possibility that binary string can > have \\\0 in it. > > --- > client/mysql.cc | 8 ++++++-- > mysql-test/main/binary_zero_insert.result | 6 ++++++ > mysql-test/main/binary_zero_insert.test | 15 +++++++++++++++ > mysql-test/std_data/binary_zero_insert.bin | Bin 0 -> 87 bytes > 4 files changed, 27 insertions(+), 2 deletions(-) > > diff --git a/client/mysql.cc b/client/mysql.cc > index 2a7c4eaf3e5..8530c105820 100644 > --- a/client/mysql.cc > +++ b/client/mysql.cc > @@ -2319,8 +2319,12 @@ static bool add_line(String &buffer, char *line, size_t line_length, > { > // Found possbile one character command like \c > > + inchar = (uchar) *++pos; > + // In Binary mode , when in_string is not null \0 should not be treated as > + // end statement. This can happen when we are in middle of binary data which > + // can contain \0 and its quoted with ' '. > + if (!real_binary_mode && !*in_string && !inchar) Two comments here. 1. if you check the history where this "real_binary_mode" came from, you'd arrive at commit 2db4392bf4cb and the test file mysql_binary_mode.test So, I'd say it make sense to add your test there too. And in the same style, wthout a separate binary file. in my experimening I did it as simple as --- a/mysql-test/main/mysql_binary_mode.test +++ b/mysql-test/main/mysql_binary_mode.test @@ -16,14 +16,14 @@ let $table_name_right= `SELECT 0x410D0A42`; let $table_name_wrong= `SELECT 0x410A42`; # 0x410042 => 'A\0B' -let $char0= `SELECT 0x410042`; +let $char0= `SELECT 0x4100425C0043`; (there already was a test for a \0 byte, but not for an escaped \\\0) but if you'd like you can add a separate test with a header and everything. > - if (!(inchar = (uchar) *++pos)) > - break; // readline adds one '\' 2. What does the old code mean? Why returning if inchar==0 ? What does the comment mean? It's the code from the last millenium, does it even apply now? For example, there can be no \0 bytes in the string, because of if (!real_binary_mode && strlen(line) != line_length) ... in other words, shouldn't we just remove the if? I tried, at least the main suite didn't break. > + break; // readline adds one '\' > if (*in_string || inchar == 'N') // \N is short for NULL > { // Don't allow commands in string > *out++='\\'; Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 2

[Maria-developers] Pushing down aggregation functions
by Berrocal, Eduardo 04 Aug '21

04 Aug '21

Hello!, Continuing in my series of emails about engine condition pushdown... I wonder if it is possible to push down aggregation functions such as SUM(), MIN(), MAX() or AVG() to the storage engine. Currently, and as far as I can see (I may be wrong here), it is not possible. However, this is something that can be beneficial if the storage engine can perform some of these basic computations itself, like suming over a whole column and returning just the result. The storage engine can even be smart enough to use some hardware advantages like accelerators or computational storage. If you guys think this is indeed possible without drastic code changes, it would be amazing to be pointed at where this is. At first, I would only push down functions if they operate over just one column of the same table. Things can be made more complicated in the future, of course. Thanks a lot!, Eduardo.

2 1

Re: [Maria-developers] 203aa38f9b6: MDEV-26102: gcov 9.1 doesn't support intermediate format any more, but json instead - refactoring needed
by Sergei Golubchik 29 Jul '21

29 Jul '21

Hi, Anel! On Jul 28, Anel Husakovic wrote: > revision-id: 203aa38f9b6 (mariadb-10.2.39-96-g203aa38f9b6) > parent(s): afe00bb7cce > author: Anel Husakovic > committer: Anel Husakovic > timestamp: 2021-07-27 16:09:41 +0200 > message: > > MDEV-26102: gcov 9.1 doesn't support intermediate format any more, but json instead - refactoring needed I've renamed the MDEV to have a shorter title, use the new one in your next commit > > Reviewed by: serg(a)mariadb.com that's not right, I haven't reviewed it yet :) > diff --git a/mysql-test/dgcov.pl b/mysql-test/dgcov.pl > index 2c00c64d1ff..e2aadec33e0 100755 > --- a/mysql-test/dgcov.pl > +++ b/mysql-test/dgcov.pl > @@ -25,6 +25,8 @@ use warnings; > use Getopt::Long; > use File::Find; > use Cwd qw/realpath/; > +use IO::Uncompress::Gunzip qw(gunzip $GunzipError); > +use JSON::PP; move this down, dcgov should not try to load these modules for for gcc < 9 > my $opt_verbose=0; > my $opt_generate; > @@ -32,6 +34,7 @@ my $opt_help; > my $opt_purge; > my $opt_only_gcov; > my $opt_skip_gcov; > +my $fname; and this > > my %cov; > my $file_no=0; > @@ -62,12 +65,15 @@ my $res; > my $cmd; > if ($opt_purge) > { > - $cmd= "find . -name '*.da' -o -name '*.gcda' -o -name '*.gcov' -o ". > + $cmd= "find . -name '*.da' -o -name '*.gcda*' -o -name '*.gcov' -o ". > "-name '*.dgcov' | grep -v 'README\.gcov' | xargs rm -f ''"; > logv "Running: $cmd"; > system($cmd)==0 or die "system($cmd): $? $!"; > exit 0; > } > + > +my $gcc_version= `gcc -dumpversion`; > +$gcc_version=~ s/(\d).*$/$1/; you don't need this second line, I suspect > > find(\&gcov_one_file, $root); > find(\&write_coverage, $root) if $opt_generate; > @@ -162,6 +168,7 @@ sub gcov_one_file { > } > > # now, read the generated file > + if ($gcc_version <9){ > for my $gcov_file (<$_*.gcov>) { > open FH, '<', "$gcov_file" or die "open(<$gcov_file): $!"; > my $fname; > @@ -183,6 +189,18 @@ sub gcov_one_file { > $cov{$fname}->{$1}+=$2; > } > close(FH); > + } > + } > + else { put both `use` lines here > + my $gcov_file_json; > + gunzip "$_.gcov.json.gz" => \$gcov_file_json or die "gunzip($_.gcov.json.gz): $GunzipError"; > + my $obj= decode_json $gcov_file_json; > + for my $file (@{$obj->{files}}) { > + $fname= $file->{file}; and write `my $fname` here > + for my $line (@{$file->{lines}}){ > + $cov{$fname}->{$line->{line_number}}= $line->{count}; shouldn't it be += $line->{count} ? The point is to accumulate all counters, see how it's done for non-json intermediate format. > + } > + } > } > } > > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 3

Re: [Maria-developers] ce289840131: MDEV-24511 null field is created with CREATE..SELECT
by Nikita Malyavin 28 Jul '21

28 Jul '21

Hello, Sergei! Looks fine for me. So if you both prefer it more, let's use this variant. Regards, Nikita On Wed, Jul 28, 2021 at 12:08 PM Sergei Golubchik <serg(a)mariadb.org> wrote: > Hi, Nikita! > > On Jul 27, Nikita Malyavin wrote: > > revision-id: ce289840131 (mariadb-10.3.30-31-gce289840131) > > parent(s): b50ea900636 > > author: Nikita Malyavin > > committer: Nikita Malyavin > > timestamp: 2021-07-22 22:38:31 +0300 > > message: > > > > MDEV-24511 null field is created with CREATE..SELECT > > > > The regression has been initially appeared by MDEV-12588 patch > (441349aa). > > > > The reason CREATE... SELECT NULL does not create null field is that > > it calls handle_select(), where the field is created directly from Item, > > by calling type_handler->type_handler_for_tmp_table(), instead of plain > > type_handler. > > > > In UNION case, we'll already have Item_field's containing UNION'ed tmp > > table fields to the moment of select_create::prepare() call. > > > > There, item's type_handler is used directly in > > st_select_lex_unit::join_union_item_types(), instead of > > type_handler_for_tmp_table() or type_handler_for_union(). > > > > Using those two involve too many changes in CREATE...SELECT...UNION > > behavior, which may be reasonably undesired to be pushed in 10.3 > > > > However, patching Field_null::type_handler() directly seems to be > accurate > > enough. > > > > diff --git a/mysql-test/main/union.result b/mysql-test/main/union.result > > index a892f6c9e40..b19427948a6 100644 > > --- a/mysql-test/main/union.result > > +++ b/mysql-test/main/union.result > > @@ -1609,7 +1609,7 @@ NULL binary(0) YES NULL > > CREATE TABLE t5 SELECT NULL UNION SELECT NULL; > > DESC t5; > > Field Type Null Key Default Extra > > -NULL null YES NULL > > +NULL binary(0) YES NULL > > CREATE TABLE t6 > > SELECT * FROM (SELECT * FROM (SELECT NULL)a) b UNION SELECT a FROM t1; > > DESC t6; > > diff --git a/sql/field.h b/sql/field.h > > index a3385736c0a..2af5346bea9 100644 > > --- a/sql/field.h > > +++ b/sql/field.h > > @@ -2526,7 +2526,7 @@ class Field_null :public Field_str { > > :Field_str(ptr_arg, len_arg, null, 1, > > unireg_check_arg, field_name_arg, collation) > > {} > > - const Type_handler *type_handler() const { return &type_handler_null; > } > > + const Type_handler *type_handler() const { return > &type_handler_string; } > > Information_schema_character_attributes > > information_schema_character_attributes() const > > { > > This looks suspiciously risky to me. The problem was specifically in > UNION, but you change the very basic idea of what type the Field_null > is. > > I cannot clearly see what else might this possibly affect, field's type > is used everywhere. > > It would be safer and more logical to fix specifically field creation > for UNION. > > I've discussed it with Bar, who implemented this whole Type_handler > hierarchy, he suggested to use Type_handler::union_element_finalize() > for that. I've attached the patch that does that. > > union_element_finalize() is for 10.4, so I also backported it to 10.3, > it's very straightforward though. > > What do you think? > > Regards, > Sergei > VP of MariaDB Server Engineering > and security(a)mariadb.org > -- Nikita Malyavin, Software Engineer MariaDB Corporation

1 0

Re: [Maria-developers] ce289840131: MDEV-24511 null field is created with CREATE..SELECT
by Sergei Golubchik 28 Jul '21

28 Jul '21

Hi, Nikita! On Jul 27, Nikita Malyavin wrote: > revision-id: ce289840131 (mariadb-10.3.30-31-gce289840131) > parent(s): b50ea900636 > author: Nikita Malyavin > committer: Nikita Malyavin > timestamp: 2021-07-22 22:38:31 +0300 > message: > > MDEV-24511 null field is created with CREATE..SELECT > > The regression has been initially appeared by MDEV-12588 patch (441349aa). > > The reason CREATE... SELECT NULL does not create null field is that > it calls handle_select(), where the field is created directly from Item, > by calling type_handler->type_handler_for_tmp_table(), instead of plain > type_handler. > > In UNION case, we'll already have Item_field's containing UNION'ed tmp > table fields to the moment of select_create::prepare() call. > > There, item's type_handler is used directly in > st_select_lex_unit::join_union_item_types(), instead of > type_handler_for_tmp_table() or type_handler_for_union(). > > Using those two involve too many changes in CREATE...SELECT...UNION > behavior, which may be reasonably undesired to be pushed in 10.3 > > However, patching Field_null::type_handler() directly seems to be accurate > enough. > > diff --git a/mysql-test/main/union.result b/mysql-test/main/union.result > index a892f6c9e40..b19427948a6 100644 > --- a/mysql-test/main/union.result > +++ b/mysql-test/main/union.result > @@ -1609,7 +1609,7 @@ NULL binary(0) YES NULL > CREATE TABLE t5 SELECT NULL UNION SELECT NULL; > DESC t5; > Field Type Null Key Default Extra > -NULL null YES NULL > +NULL binary(0) YES NULL > CREATE TABLE t6 > SELECT * FROM (SELECT * FROM (SELECT NULL)a) b UNION SELECT a FROM t1; > DESC t6; > diff --git a/sql/field.h b/sql/field.h > index a3385736c0a..2af5346bea9 100644 > --- a/sql/field.h > +++ b/sql/field.h > @@ -2526,7 +2526,7 @@ class Field_null :public Field_str { > :Field_str(ptr_arg, len_arg, null, 1, > unireg_check_arg, field_name_arg, collation) > {} > - const Type_handler *type_handler() const { return &type_handler_null; } > + const Type_handler *type_handler() const { return &type_handler_string; } > Information_schema_character_attributes > information_schema_character_attributes() const > { This looks suspiciously risky to me. The problem was specifically in UNION, but you change the very basic idea of what type the Field_null is. I cannot clearly see what else might this possibly affect, field's type is used everywhere. It would be safer and more logical to fix specifically field creation for UNION. I've discussed it with Bar, who implemented this whole Type_handler hierarchy, he suggested to use Type_handler::union_element_finalize() for that. I've attached the patch that does that. union_element_finalize() is for 10.4, so I also backported it to 10.3, it's very straightforward though. What do you think? Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] c34533186ab: MDEV-18734 ASAN heap-use-after-free upon sorting by blob column from partitioned table
by Sergei Golubchik 26 Jul '21

26 Jul '21

Hi, Aleksey! As before, despite the email subject, it's git diff dfa2d0bc13..e533b5fb307 Don't be confused :) On Jul 21, Aleksey Midenkov wrote: > diff --git a/sql/ha_partition.cc b/sql/ha_partition.cc > index c53732a2b51..70ed14448ea 100644 > --- a/sql/ha_partition.cc > +++ b/sql/ha_partition.cc > @@ -5130,7 +5135,18 @@ bool ha_partition::init_record_priority_queue() > i= bitmap_get_next_set(&m_part_info->read_partitions, i)) > { > DBUG_PRINT("info", ("init rec-buf for part %u", i)); > - int2store(ptr, i); > + if (table->s->blob_fields) > + { > + for (uint j= 0; j < table->s->blob_fields; ++j, ++objs) > + { > + blob_storage[j]= new (objs) Ordered_blob_storage; > + if (!blob_storage[j]) > + DBUG_RETURN(true); is this possible? > + } > + *((Ordered_blob_storage ***) ptr)= blob_storage; > + blob_storage+= table->s->blob_fields; > + } > + int2store(ptr + sizeof(String **), i); > ptr+= m_priority_queue_rec_len; > } > m_start_key.key= (const uchar*)ptr; > @@ -6291,6 +6333,43 @@ int ha_partition::handle_ordered_index_scan_key_not_found() > } > > > +void ha_partition::swap_blobs(uchar * rec_buf, Ordered_blob_storage ** storage, bool restore) > +{ > + uint *ptr, *end; > + uint blob_n= 0; > + table->move_fields(table->field, rec_buf, table->record[0]); > + for (ptr= table->s->blob_field, end= ptr + table->s->blob_fields; > + ptr != end; ++ptr, ++blob_n) > + { > + DBUG_ASSERT(*ptr < table->s->fields); > + Field_blob *blob= (Field_blob*) table->field[*ptr]; > + DBUG_ASSERT(blob->flags & BLOB_FLAG); > + DBUG_ASSERT(blob->field_index == *ptr); > + if (!bitmap_is_set(table->read_set, *ptr) || blob->is_null()) > + continue; > + > + Ordered_blob_storage &s= *storage[blob_n]; > + > + if (restore) > + { > + if (!s.blob.is_empty()) > + blob->swap(s.blob, s.set_read_value); don't you need here something like else blob->reset(); to make blob empty? > + } > + else > + { > + bool set_read_value; > + String *cached= blob->cached(set_read_value); > + if (cached) > + { > + cached->swap(s.blob); > + s.set_read_value= set_read_value; > + } > + } > + } > + table->move_fields(table->field, table->record[0], rec_buf); On swap_blobs() you always move all fields from rec_buf to table->record[0]? Why? This looks very suspicious. > +} > + > + > /* > Common routine to handle index_next with ordered results > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 3

Re: [Maria-developers] c4835af7e8c: MDEV-16026: Forbid global system_versioning_asof in non-default time zone
by Sergei Golubchik 26 Jul '21

26 Jul '21

Hi, Nikita! I suggest to squash commits from c4835af7e8c to 80e3007a5a1 into one. It'll be simpler, no intermediate juggling with MYSQL_TIME anymore. This squashed commit is ok to push (make sure to mention both MDEVs in the commit comment, but not necessarily in the first line) On Jul 26, Nikita Malyavin wrote: > revision-id: c4835af7e8c (mariadb-10.3.30-30-gc4835af7e8c) > parent(s): ec3289b6de4 > author: Nikita Malyavin > committer: Nikita Malyavin > timestamp: 2021-07-21 00:21:31 +0300 > message: > > MDEV-16026: Forbid global system_versioning_asof in non-default time zone > > * store `system_versioning_asof` in unix time; > * both session and global vars are processed in session timezone; > * setting `default` does not copy global varibale abymore. Instead, it sets > system_time to SYSTEM_TIME_UNSPECIFIED, which means that no 'AS OF' time > is applied and `now()` can be assumed > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] a4a5c2d5963: MDEV-25908: -e does not work for my_print_defaults
by Sergei Golubchik 26 Jul '21

26 Jul '21

Hi, Rucha! On Jul 26, Rucha Deodhar wrote: > revision-id: a4a5c2d5963 (mariadb-10.5.11-52-ga4a5c2d5963) > parent(s): 675b72d0da1 > author: Rucha Deodhar > committer: Rucha Deodhar > timestamp: 2021-07-23 19:51:57 +0530 > message: > > MDEV-25908: -e does not work for my_print_defaults > > Analysis: get_defaults_options() rewrites the value for my_defaults_file, > my_defaults_extra_file and my_defaults_group_suffix to 0. So the config file > can't be read. > Fix: Let handle_options() handle --defaults* option and create a new command > line to pass it to load_defaults(). > > diff --git a/libmariadb b/libmariadb > index 802ce584a26..180c543704d 160000 > --- a/libmariadb > +++ b/libmariadb > @@ -1 +1 @@ > -Subproject commit 802ce584a26fdc0ba67fcf35e277bf3c7440956a > +Subproject commit 180c543704d627a50a52aaf60e24ca14e0ec4686 ^^^^^ This is wrong, please, remove it from your commit Otherwise ok to push Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

[Maria-developers] Accessing read records during join for condition pushdown
by Berrocal, Eduardo 23 Jul '21

23 Jul '21

Hello list (first email here, hopefully sending it to the right address), I have been trying to solve this problem for a while now, and I am stuck. I wonder if what I am trying to do is even possible. I am implementing a new storage engine which handles engine condition pushdown. Now, when MariaDB is working on a JOIN using more than one table (for simplicity, let's assume 2), how can I access the read records for table 1 when working on a condition pushdown for table 2? When joining like this: table1.field1 = table2.field1, I can see 2 Item objects of type FIELD in the condition. However, I can't find where the read records for table1 are so I can effectively filter records in table2 by those values. Thanks a lot for your help, [A close up of a sign Description automatically generated] Eduardo Berrocal García de Carellán Senior Software Engineer Intel Corporation | intel.com<http://intel.com/>

3 9

Re: [Maria-developers] 412e696fd2b: MDEV-25395 server recovery hits replication event checksum error
by Sergei Golubchik 23 Jul '21

23 Jul '21

Hi, Andrei! fdle changes - ok, I've seen them in 21117. cur_log/etc - clear. What are the changes around prev_event_pos for? On Jul 23, Andrei Elkin wrote: > revision-id: 412e696fd2b (mariadb-10.3.26-136-g412e696fd2b) > parent(s): 2fa526e26e8 > author: Andrei Elkin > committer: Andrei Elkin > timestamp: 2021-04-15 17:11:07 +0300 > message: > > MDEV-25395 server recovery hits replication event checksum error > > When @@global.binlog_checksum is changing from none to > crc32 and only the first of two Binlog_checkpoint_log_event gets written to > the crc32 rotated binlog file before the server crashes, the recovery faces a checksum > verification error. > > The reason was that recovery algorithm did not realize > potentially varying binlog checksum at scanning of > a sequence of binlog files. > > That is corrected now. > The master-checksum-verify server recovery aborts now and stops > the server due to the checksum error. > A simulation test confirms the fixes. > > Induced changes: > > rpl_cant_read_event_incident got affected because the test > adopts an external not properly closed binlog file that contains > events which can't pass checksum verification. > A work-around is done to accept that binlog and evade recovery. > > binlog_recover_checksum_error needed refinement as > the master-checksum-verify configured server is made to stop > due to the checksum error. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 1c7d4f9c06f: MDEV-25282 Auto-shutdown on idle when socket-activated
by Sergei Golubchik 23 Jul '21

23 Jul '21

Hi, Daniel! The approach is fine, see some comments below On Jul 23, Daniel Black wrote: > revision-id: 1c7d4f9c06f (mariadb-10.6.1-78-g1c7d4f9c06f) > parent(s): 71964c76fe0 > author: Daniel Black > committer: Daniel Black > timestamp: 2021-06-16 17:15:59 +1000 > message: > > MDEV-25282 Auto-shutdown on idle when socket-activated > > Adds max_idle_execution system variable with a UINT_MAX > default value that corresponds to the time in seconds > under which the mariadbd executable will run in an idle state > with no connections. > > Under systemd socket activiation this variable will get a 10 minute > default value. This will enable a service to be activated on > connection and fall back to a shutdown state after 10 minutes of > no queries. The systemd socket activation can restart the service > on the next connection transparently. > > A global variable of server_last_activity is updated on > accepted connections (where max_idle_execution < UINT_MAX) > and when the connection count (standard or extra) is down > to <= 1 to keep the number of updates on a single variable > low (an not create another performance MDEV-21212 problem). > > When the main accept loop times out on the max_idle_execution > seconds, and then the server_last_activity is checked along > with if current connection count (standard + extra) is 0 > (in case a recently started connection hasn't finished > a query). > > To make this neater, in main accept loop moved code to > handle_new_socket_connection that encompases accepting a connection > and the timeout mechanism has been separated too. > > Changed when looping though possible connections, loop until > the end of the connection list and hereby assume two connection can > occur on the same poll/select call and both will be accepted. > > diff --git a/mysql-test/main/max_idle_execution.test b/mysql-test/main/max_idle_execution.test > new file mode 100644 > --- /dev/null > +++ b/mysql-test/main/max_idle_execution.test > @@ -0,0 +1,38 @@ > +--source include/not_embedded.inc > + > +--let $_server_id= `SELECT @@server_id` > +--let $_expect_file_name= $MYSQLTEST_VARDIR/tmp/mysqld.$_server_id.expect > +--exec echo "wait" > $_expect_file_name there's no reason to exec echo, as you can write directly --write_file $_expect_file_name wait EOF > +disable_reconnect; > +disconnect default; > + > +--echo 'allow server to time out in 10 seconds' > + > +--sleep 10 better wait_until_disconnected.inc here and everywhere below > + > +--echo 'should have timed out' > +# mtr weirdness > +--replace_regex /[\/0-9]+.*// > +--error 0,ER_SERVER_SHUTDOWN,ER_CONNECTION_KILLED,2002,2006,201 > +--connect fail_me,localhost,root you won't need all that ^^^ > + > +--exec echo "restart" > $_expect_file_name > +--sleep 5 > +--connect con0,localhost,root,,test > + > +--source include/wait_until_connected_again.inc > +SELECT VARIABLE_VALUE AS THREAD_CONNECTED FROM INFORMATION_SCHEMA.GLOBAL_STATUS WHERE VARIABLE_NAME='THREADS_CONNECTED'; > +SELECT 'we are back'; > +SELECT VARIABLE_VALUE < 10 AS UPTIME_LESS_THAN_10_SECONDS FROM INFORMATION_SCHEMA.GLOBAL_STATUS WHERE VARIABLE_NAME='UPTIME'; > + > +--sleep 13 > +SELECT 'still here because the connection is open, but disconnecting now'; > +disconnect con0; > +--sleep 5 > + > +--connect con2,127.0.0.1,root,,test,$MASTER_EXTRA_PORT, > +SELECT "extra connection just created: still here, only 5 seconds since last query"; > + > +--sleep 13 > +connect default,localhost,root,,test,,; > +SELECT 'active extra connection kept the server up'; This will be a fairly slow test. 46 seconds of waits. May be you can cut away some? E.g. second test, whether a connection keeps the server up, can reasonably be considered a subset of the third test, whether an extra connection keeps the server up. So you can remove the second test. > diff --git a/mysql-test/main/mysqld--help.result b/mysql-test/main/mysqld--help.result > --- a/mysql-test/main/mysqld--help.result > +++ b/mysql-test/main/mysqld--help.result > @@ -576,6 +576,10 @@ The following specify which files/extra groups are read (specified before remain > --max-error-count=# Max number of errors/warnings to store for a statement > --max-heap-table-size=# > Don't allow creation of heap tables bigger than this > + --max-idle-execution=# > + If no new connections or running queries within this time > + (in seconds) shutdown the server (Automatically > + configured unless set explicitly) > --max-join-size=# Joins that are probably going to read more than > max_join_size records return an error > --max-length-for-sort-data=# > diff --git a/sql/handle_connections_win.cc b/sql/handle_connections_win.cc > --- a/sql/handle_connections_win.cc > +++ b/sql/handle_connections_win.cc > @@ -593,7 +593,7 @@ void network_init_win() > } > } > > -void handle_connections_win() > +void handle_connections_win(Atomic_counter<uint> *connection_count) why do you pass it as an argument? first, you always invoke handle_connections_win with the same value of the argument, second other global variables like server_last_activity are used directly. > { > int n_waits; > > @@ -631,12 +631,26 @@ void handle_connections_win() > { > DBUG_ASSERT(wait_events.size() <= MAXIMUM_WAIT_OBJECTS); > DWORD idx = WaitForMultipleObjects((DWORD)wait_events.size(), > - wait_events.data(), FALSE, INFINITE); > + wait_events.data(), FALSE, > + max_idle_execution < UINT_MAX ? > + max_idle_execution * 1000 : INFINITE); > - DBUG_ASSERT((int)idx >= 0 && (int)idx < (int)wait_events.size()); > + DBUG_ASSERT(idx == WAIT_TIMEOUT || > + ((int)idx >= 0 && (int)idx < (int)wait_events.size())); did you test it on Windows? > > if (idx == SHUTDOWN_IDX) > break; > > + if (idx == WAIT_TIMEOUT) > + { > + if (*connection_count == 0 && > + microsecond_interval_timer() > (server_last_activity + max_idle_execution * 1000000)) > + { > + sql_print_information("max_idle_execution time reached starting shutdown"); > + break; > + } > + continue; > + } > + > all_listeners[idx - LISTENER_START_IDX]->completion_callback(); > } > > diff --git a/sql/mysqld.cc b/sql/mysqld.cc > --- a/sql/mysqld.cc > +++ b/sql/mysqld.cc > @@ -474,7 +474,8 @@ ulong malloc_calls; > ulong specialflag=0; > ulong binlog_cache_use= 0, binlog_cache_disk_use= 0; > ulong binlog_stmt_cache_use= 0, binlog_stmt_cache_disk_use= 0; > -ulong max_connections, max_connect_errors; > +ulong max_connections, max_connect_errors, max_idle_execution; > +ulonglong server_last_activity; > uint max_password_errors; > ulong extra_max_connections; > uint max_digest_length= 0; > @@ -4093,6 +4094,13 @@ static int init_common_variables() > SYSVAR_AUTOSIZE(back_log, MY_MIN(900, (50 + max_connections / 5))); > } > > + /* > + max_idle_execution, defaults to 10mins under systemd socket activation, > + otherwise 136 years or so. > + */ > + if (IS_SYSVAR_AUTOSIZE(&max_idle_execution)) > + SYSVAR_AUTOSIZE(max_idle_execution, sd_listen_fds(0) ? 6000 : UINT_MAX); is that really 10 minutes? > + > unireg_init(opt_specialflag); /* Set up extern variabels */ > if (!(my_default_lc_messages= > my_locale_by_name(lc_messages))) > @@ -6064,17 +6072,60 @@ static void set_non_blocking_if_supported(MYSQL_SOCKET sock) > } > > > +static void handle_socket_timeout() > +{ please add a comment, explaining why a race condition is impossible here, that is, why no other thread can increment connection_count or extra_connection_count just after you saw them being zeros. > + if (connection_count == 0 && extra_connection_count == 0 && > + microsecond_interval_timer() > (server_last_activity + max_idle_execution * 1000000)) * 1000000ULL ? > + { > + sql_print_information("max_idle_execution time reached starting shutdown"); > + abort_loop= 1; > + } > +} > + > + > +static void handle_new_socket_connection(MYSQL_SOCKET sock) > +{ > + struct sockaddr_storage cAddr; > + uint error_count= 0; > + > + for (uint retry= 0; retry < MAX_ACCEPT_RETRY; retry++) > + { > + size_socket length= sizeof(struct sockaddr_storage); > + MYSQL_SOCKET new_sock; > + > + new_sock= mysql_socket_accept(key_socket_client_connection, sock, > + (struct sockaddr *)(&cAddr), > + &length); > + if (mysql_socket_getfd(new_sock) != INVALID_SOCKET) > + handle_accepted_socket(new_sock, sock); > + else if (socket_errno != SOCKET_EINTR && socket_errno != SOCKET_EAGAIN) > + { > + /* > + accept(2) failed on the listening port. > + There is not much details to report about the client, > + increment the server global status variable. > + */ > + statistic_increment(connection_errors_accept, &LOCK_status); > + if ((error_count++ & 255) == 0) // This can happen often > + sql_perror("Error in accept"); > + if (socket_errno == SOCKET_ENFILE || socket_errno == SOCKET_EMFILE) > + sleep(1); // Give other threads some time > + break; > + } > + } > +} > + > + > void handle_connections_sockets() > { > - MYSQL_SOCKET sock= mysql_socket_invalid(); > + MYSQL_SOCKET sock; > - uint error_count=0; > - struct sockaddr_storage cAddr; > int retval; > #ifdef HAVE_POLL > // for ip_sock, unix_sock and extra_ip_sock > Dynamic_array<struct pollfd> fds(PSI_INSTRUMENT_MEM); > #else > fd_set readFDs,clientFDs; > + struct timespec timeout; > #endif > > DBUG_ENTER("handle_connections_sockets"); > @@ -6099,6 +6150,7 @@ void handle_connections_sockets() > } > #endif > > + server_last_activity= microsecond_interval_timer(); > sd_notify(0, "READY=1\n" > "STATUS=Taking your SQL requests now...\n"); > > @@ -6106,10 +6158,12 @@ void handle_connections_sockets() > while (!abort_loop) > { > #ifdef HAVE_POLL > - retval= poll(fds.get_pos(0), fds.size(), -1); > + /* poll timeout in milliseconds */ > + retval= poll(fds.get_pos(0), fds.size(), max_idle_execution * 1000); max value of max_idle_execution is UINT_MAX. The third argument of poll() is int. Although it seems to "work", that is, you get a negative number which poll() interprets as infinite, I seriously suspect it's a UB. > #else > + timeout= { max_idle_execution, 0}; > readFDs=clientFDs; > - retval= select(FD_SETSIZE, &readFDs, NULL, NULL, NULL); > + retval= select(FD_SETSIZE, &readFDs, NULL, NULL, &timeout); > #endif > > if (retval < 0) > @@ -6132,50 +6186,27 @@ void handle_connections_sockets() > break; > > /* Is this a new connection request ? */ > -#ifdef HAVE_POLL > - for (size_t i= 0; i < fds.size(); ++i) > + sock= mysql_socket_invalid(); > + for (size_t i= 0; i < listen_sockets.size(); i++) > { > +#ifdef HAVE_POLL > if (fds.at(i).revents & POLLIN) > - { > - sock= listen_sockets.at(i); > - break; > - } > - } > #else // HAVE_POLL > - for (size_t i=0; i < listen_sockets.size(); i++) > - { > if (FD_ISSET(mysql_socket_getfd(listen_sockets.at(i)), &readFDs)) > +#endif // HAVE_POLL > { > sock= listen_sockets.at(i); > - break; > + handle_new_socket_connection(sock); > } > } > -#endif // HAVE_POLL > - > - for (uint retry=0; retry < MAX_ACCEPT_RETRY; retry++) > + /* timeout */ > + if (mysql_socket_getfd(sock) == INVALID_SOCKET) > { > - size_socket length= sizeof(struct sockaddr_storage); > - MYSQL_SOCKET new_sock; > - > - new_sock= mysql_socket_accept(key_socket_client_connection, sock, > - (struct sockaddr *)(&cAddr), > - &length); > - if (mysql_socket_getfd(new_sock) != INVALID_SOCKET) > - handle_accepted_socket(new_sock, sock); > - else if (socket_errno != SOCKET_EINTR && socket_errno != SOCKET_EAGAIN) > - { > - /* > - accept(2) failed on the listening port. > - There is not much details to report about the client, > - increment the server global status variable. > - */ > - statistic_increment(connection_errors_accept, &LOCK_status); > - if ((error_count++ & 255) == 0) // This can happen often > - sql_perror("Error in accept"); > - if (socket_errno == SOCKET_ENFILE || socket_errno == SOCKET_EMFILE) > - sleep(1); // Give other threads some time > - break; > - } > + handle_socket_timeout(); > + } > + else if (max_idle_execution < UINT_MAX) > + { > + server_last_activity= microsecond_interval_timer(); > } > } > sd_notify(0, "STOPPING=1\n" > diff --git a/sql/sql_class.h b/sql/sql_class.h > --- a/sql/sql_class.h > +++ b/sql/sql_class.h > @@ -3949,6 +3950,9 @@ class THD: public THD_count, /* this must be first */ > set_time_for_next_stage(); > if (utime_after_query >= utime_after_lock + variables.long_query_time) > server_status|= SERVER_QUERY_WAS_SLOW; > + /* If we're tracking idle execution, and we're down to the last connection */ > + if (max_idle_execution < UINT_MAX && *scheduler->connection_count <= 1) > + server_last_activity= utime_after_query; 1. you update it concurrenty. Must be atomic, right? 2. do you really want to shutdown the server if there's an idle connection? I'd think it'd be less surprising to shutdown only when there are no connections at all. If there is a runaway idle connection, it'll die after interactive_timeout or wait_timeout. And then you shutdown the server after max_idle_execution. So instead of shutting down the server when there are connections, I'd suggest to autoset interactive_timeout and wait_timeout to be in line with max_idle_execution. Like, 10 mins, or, may be if (IS_SYSVAR_AUTOSIZE(net_wait_timeout) && net_wait_timeout > max_idle_execution) SYSVAR_AUTOSIZE(net_wait_timeout, max_idle_execution); > } > inline ulonglong found_rows(void) > { > diff --git a/sql/sys_vars.cc b/sql/sys_vars.cc > --- a/sql/sys_vars.cc > +++ b/sql/sys_vars.cc > @@ -1724,6 +1724,14 @@ Sys_max_connect_errors( > VALID_RANGE(1, UINT_MAX), DEFAULT(MAX_CONNECT_ERRORS), > BLOCK_SIZE(1)); > > +static Sys_var_ulong Sys_max_idle_execution( ulong is non portable. why not make the variable uint, if max value is UINT_MAX anyway? > + "max_idle_execution", > + "If no new connections or running queries within this time (in seconds) " > + "shutdown the server", may be something like "Default to 10 minutes in systemd socket activation setup" ? I know, it means asking for troubles :( > + AUTO_SET GLOBAL_VAR(max_idle_execution), CMD_LINE(REQUIRED_ARG), > + VALID_RANGE(10, UINT_MAX), DEFAULT(UINT_MAX), > + BLOCK_SIZE(1)); > + > static Sys_var_on_access_global<Sys_var_uint, > PRIV_SET_SYSTEM_GLOBAL_VAR_MAX_PASSWORD_ERRORS> > Sys_max_password_errors( > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] ae1f4e4d727: MDEV-23836: Assertion `! is_set() || m_can_overwrite_status' in Diagnostics_area::set_error_status (interrupted ALTER TABLE under LOCK)
by Sergei Golubchik 23 Jul '21

23 Jul '21

Hi, Rucha! On Jul 23, Rucha Deodhar wrote: > On Fri, Jul 23, 2021 at 1:55 PM Sergei Golubchik <serg(a)mariadb.org> wrote: > > > > Consider this - thd->killed can be changed literally any time. We cannot > > test for it after every second line, can we? Clearly some places are > > better for this test than others. > > > There was no specific reason except it seemed a better choice for this > particular test case. > But yes, it can also arrive before warning or later. Now I think it would > be better check it before exiting handle_if_exists_option() > and return appropriate value. Yes, you're right, it would be better. but KILL can come after handle_if_exists_options() too :) Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

[Maria-developers] MariaDB Connector/J 3.x Status and Plans
by Ondrej Dubaj 23 Jul '21

23 Jul '21

Hello, I would like to ask about the current status of MariaDB Connector/J 3.x. I maintain the package in Fedora and RHEL. Currently, we are facing a decision about which major version to pack to RHEL 9, which is currently being developed. We would like to ship RHEL 9 with the latest stable version of the Java connector, and avoid rebasing to any next major release during the RHEL 9 life, if possible. However the 3.x version is only available in ALPHA quality and I can't see any plan for it to reach further stages in the Jira roadmap for the next half a year. https://jira.mariadb.org/secure/Dashboard.jspa I believe the best solution for us would be you releasing the GA, or at least BETA version ASAP, or at least in the near future. We still have time to update the package from the ALPHA version to GA version before the RHEL 9 development freeze, but the decision whether to take this road has to be made ASAP. After RHEL 9 GA, it will be very complicated for us to deal with rebasing to a new, incompatible version of mariadb-connector-j, where this rebase can break existing user applications and brings a lot of problems with it. Currently mariadb-java-client-2.7.2 is present in the repositories, but as already said, we want to aim for the 3.x version. You, of course, might have very different plans for the connector and different idea about how ready it is to become BETA or GA. We would like to hear your plans with the Java connector 3.x version and to discuss our possible paths to take. Thank you for your time in advance. Regards, Ondrej Dubaj

2 2

Re: [Maria-developers] ae1f4e4d727: MDEV-23836: Assertion `! is_set() || m_can_overwrite_status' in Diagnostics_area::set_error_status (interrupted ALTER TABLE under LOCK)
by Sergei Golubchik 23 Jul '21

23 Jul '21

Hi, Rucha! On Jul 23, Rucha Deodhar wrote: > > > diff --git a/sql/sql_table.cc b/sql/sql_table.cc > > > index f58ce8f997d..0b722984fca 100644 > > > --- a/sql/sql_table.cc > > > +++ b/sql/sql_table.cc > > > @@ -6489,6 +6489,12 @@ handle_if_exists_options(THD *thd, TABLE *table, Alter_info *alter_info, > > > ER_BAD_FIELD_ERROR, > > > ER_THD(thd, ER_BAD_FIELD_ERROR), > > > sql_field->change.str, table->s->table_name.str); > > > + /* > > > + if thread is killed (for example because local memory used execeeds > > > + maximum session memory used), return TRUE and do rollback (done later). > > > + */ > > > + if (thd->killed) > > > + DBUG_RETURN(true); > > > it.remove(); > > > > Looks like a completely arbitrary place to put a check in. > > What if the kill signal arrives later? > > We want to stop execution of the query as soon as we get kill. In this case > it happens when we are giving out a warning. > So I put the check here to return error right after giving warning. You can get KILL anytime. Was there any reason to put this check here and not few lines earlier or later? What if KILL would arrive before the warning? or after it.remove() ? Consider this - thd->killed can be changed literally any time. We cannot test for it after every second line, can we? Clearly some places are better for this test than others. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] ae1f4e4d727: MDEV-23836: Assertion `! is_set() || m_can_overwrite_status' in Diagnostics_area::set_error_status (interrupted ALTER TABLE under LOCK)
by Sergei Golubchik 22 Jul '21

22 Jul '21

Hi, Rucha! On Jul 22, Rucha Deodhar wrote: > revision-id: ae1f4e4d727 (mariadb-10.5.11-1-gae1f4e4d727) > parent(s): dc82effa5df > author: Rucha Deodhar > committer: Rucha Deodhar > timestamp: 2021-06-21 20:24:43 +0530 > message: > > MDEV-23836: Assertion `! is_set() || m_can_overwrite_status' in > Diagnostics_area::set_error_status (interrupted ALTER TABLE under LOCK) > > Analysis: KILL_QUERY is not ignored when local memory used exceeds maximum > session memory. Hence the query proceeds, OK is sent and we end up > reopening tables that are marked for reopen. During this, kill status is > eventually checked and assertion failure happens during trying to send error > message because OK has already been sent. > Fix: If local memory used exceeds maximum session memory and query is killed > before it has completed execution, return TRUE. > diff --git a/sql/sql_table.cc b/sql/sql_table.cc > index f58ce8f997d..0b722984fca 100644 > --- a/sql/sql_table.cc > +++ b/sql/sql_table.cc > @@ -6489,6 +6489,12 @@ handle_if_exists_options(THD *thd, TABLE *table, Alter_info *alter_info, > ER_BAD_FIELD_ERROR, > ER_THD(thd, ER_BAD_FIELD_ERROR), > sql_field->change.str, table->s->table_name.str); > + /* > + if thread is killed (for example because local memory used execeeds > + maximum session memory used), return TRUE and do rollback (done later). > + */ > + if (thd->killed) > + DBUG_RETURN(true); > it.remove(); Looks like a completely arbitrary place to put a check in. What if the kill signal arrives later? Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] f5196787294: MDEV-23178: Qualified asterisk not supported in INSERT .. RETURNING
by Sergei Golubchik 22 Jul '21

22 Jul '21

Hi, Rucha! ok to push! thanks On Jul 22, Rucha Deodhar wrote: > revision-id: f5196787294 (mariadb-10.5.11-28-gf5196787294) > parent(s): b1e57179242 > author: Rucha Deodhar > committer: Rucha Deodhar > timestamp: 2021-07-22 18:54:36 +0530 > message: > > MDEV-23178: Qualified asterisk not supported in INSERT .. RETURNING > > Analysis: When we have INSERT/REPLACE returning with qualified asterisk in the > RETURNING clause, '*' is not resolved properly because of wrong context. > context->table_list is NULL or has incorrect table because context->table_list > has tables from the FROM clause. For INSERT/REPLACE...SELECT...RETURNING, > context->table_list has table we are inserting from. While in other > INSERT/REPLACE syntax, context->table_list is NULL because there is no FROM > clause. > Fix: If filling fields instead of '*' for qualified asterisk in RETURNING, > use first_name_resolution_table for correct resolution of item. > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] d91c4eea668: MDEV-23178: Qualified asterisk not supported in INSERT .. RETURNING
by Sergei Golubchik 22 Jul '21

22 Jul '21

Hi, Rucha! On Jul 22, Rucha Deodhar wrote: > revision-id: d91c4eea668 (mariadb-10.5.11-28-gd91c4eea668) > parent(s): b1e57179242 > author: Rucha Deodhar > committer: Rucha Deodhar > timestamp: 2021-07-17 18:47:15 +0530 > message: > > MDEV-23178: Qualified asterisk not supported in INSERT .. RETURNING > > Analysis: When we have INSERT/REPLACE returning with qualified asterisk in the > RETURNING clause, '*' is not resolved properly because of wrong context. > context->table_list is NULL or has incorrect table because context->table_list > has tables from the FROM clause. For INSERT/REPLACE...SELECT...RETURNING, > context->table_list has table we are inserting from. While in other > INSERT/REPLACE syntax, context->table_list is NULL because there is no FROM > clause. > Fix: If filling fields instead of '*' for qualified asterisk in RETURNING, > use first_name_resolution_table for correct resolution of item. > @@ -271,6 +289,8 @@ INSERT INTO t2(id2, val2) SELECT * FROM t1 WHERE id1=2 RETURNING(SELECT > --error ER_SUBQUERY_NO_1_ROW > INSERT INTO t2(id2,val2) SELECT * FROM t1 WHERE id1=2 RETURNING (SELECT > id2 FROM t2); > +#--error ER_BAD_TABLE_ERROR > +#INSERT INTO t2(id2,val2) SELECT t1.* FROM t1 WHERE id1=2 RETURNING t1.*; Why is this test commented out? (same for REPLACE) > @@ -8031,10 +8034,20 @@ insert_fields(THD *thd, Name_resolution_context *context, const char *db_name, > If table names are qualified, then loop over all tables used in the query, > else treat natural joins as leaves and do not iterate over their underlying > tables. > + If filling fields instead of '*' for qualified asterisk in RETURNING clause, > + use first_name_resolution_table for correct resolution of item because > + context->table_list is either NULL or has incorrect table because > + context->table_list has tables from the FROM clause. > + In case of INSERT/REPLACE...SELECT..RETURNING context->table_list will have > + table we are inserting and NULL for other cases because there is no > + FROM clause. > */ > - TABLE_LIST *first= context->first_name_resolution_table; > - TABLE_LIST *TABLE_LIST::* next= &TABLE_LIST::next_name_resolution_table; > - if (table_name) > + if (returning_field || !table_name) > + { > + first= context->first_name_resolution_table; > + next= &TABLE_LIST::next_name_resolution_table; These are the values already set before, so assignments are redundant. Either remove initializers, like TABLE_LIST *first; TABLE_LIST *TABLE_LIST::* next; if (returning_field || !table_name) or reverse your condition: TABLE_LIST *first= context->first_name_resolution_table; TABLE_LIST *TABLE_LIST::* next= &TABLE_LIST::next_name_resolution_table; if (table_name && !returning_field) > + } > + else > { > first= context->table_list; > next= &TABLE_LIST::next_local; Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 5e16a49f5d9: MDEV-26013 distinct not work properly in some cases for spider tables
by Sergei Golubchik 22 Jul '21

22 Jul '21

Hi, Nayuta! On Jul 21, Nayuta Yanagisawa wrote: > revision-id: 5e16a49f5d9 (mariadb-10.4.20-32-g5e16a49f5d9) > parent(s): 78735dcaf75 > author: Nayuta Yanagisawa > committer: Nayuta Yanagisawa > timestamp: 2021-07-21 09:58:34 +0000 > message: > > MDEV-26013 distinct not work properly in some cases for spider tables > > diff --git a/sql/sql_select.cc b/sql/sql_select.cc > index d8116b20cbe..cfd04d93662 100644 > --- a/sql/sql_select.cc > +++ b/sql/sql_select.cc > @@ -3274,9 +3274,17 @@ bool JOIN::make_aggr_tables_info() > > if (ht && ht->create_group_by) > { > - /* Check if the storage engine can intercept the query */ > - Query query= {&all_fields, select_distinct, tables_list, conds, > - group_list, order ? order : group_list, having}; > + /* > + Check if the storage engine can intercept the query. > + > + JOIN::optimize_stage2() might convert DISTINCT into GROUP BY and then > + optimize away GROUP BY. In such a case, we need to notify a storage engine > + supporting a group by handler of the existence of the original DISTINCT. > + Thus, we set select_distinct || (no_order && group_optimized_away) to > + Query::distinct. > + */ > + Query query= {&all_fields, select_distinct || (no_order && group_optimized_away), > + tables_list, conds, group_list, order ? order : group_list, having}; If DISTINCT was coverted to a GROUP BY, why would the engine need to know whether there was DISTINCT or not originally? There is no DISTINCT on the execution plan now, that should be sufficient, shouldn't it? Why does the query fail with GROUP BY? > group_by_handler *gbh= ht->create_group_by(thd, &query); > > if (gbh) Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 4

Re: [Maria-developers] 2f9bbf392072: MDEV-18734 optimization by taking ownership via String::swap()
by Sergei Golubchik 21 Jul '21

21 Jul '21

Hi, Aleksey! Don't be confused, the commit header and the comment come from your last commit, but the diff below includes all three commits that mention MDEV-18734. On Nov 02, Aleksey Midenkov wrote: > revision-id: 2f9bbf392072 (mariadb-10.2.31-543-g2f9bbf392072) > parent(s): 1f4960e3f2ac > author: Aleksey Midenkov <midenok(a)gmail.com> > committer: Aleksey Midenkov <midenok(a)gmail.com> > timestamp: 2020-11-02 14:26:04 +0300 > message: > > MDEV-18734 optimization by taking ownership via String::swap() > > ha_partition stores records in array of m_ordered_rec_buffer and uses > it for prio queue in ordered index scan. When the records are restored > from the array the blob buffers may be already freed or > rewritten. This can happen when blob buffers are cached in > Field_blob::value or Field_blob::read_value. > > Previous solution worked via copying blob buffer into > mem_root-allocated buffer. That was not optimal as it requires more > memory and processing time. > > This patch avoids memory copy by taking temporary ownership of cached > blob buffers via String::swap(). It also fixes problem when > value.str_length is wrong after Field_blob::store(). > diff --git a/mysql-test/suite/federated/federated_partition.test b/mysql-test/suite/federated/federated_partition.test > index ef1e27ec5054..33ee025442f4 100644 > --- a/mysql-test/suite/federated/federated_partition.test > +++ b/mysql-test/suite/federated/federated_partition.test > @@ -50,4 +50,30 @@ drop table federated.t1_2; > > --echo End of 5.1 tests > > +--echo # > +--echo # MDEV-18734 ASAN heap-use-after-free upon sorting by blob column from partitioned table > +--echo # > +connection slave; > +use federated; > +create table t1_1 (x int, b text, key(x)); > +create table t1_2 (x int, b text, key(x)); > +connection master; > +--replace_result $SLAVE_MYPORT SLAVE_PORT > +eval create table t1 (x int, b text, key(x)) engine=federated > + partition by range columns (x) ( > + partition p1 values less than (40) connection='mysql://root@127.0.0.1:$SLAVE_MYPORT/federated/t1_1', > + partition pn values less than (maxvalue) connection='mysql://root@127.0.0.1:$SLAVE_MYPORT/federated/t1_2' > +); > +insert t1 values (1, 1), (2, 2), (3, 3), (4, 4), (5, 5), (6, 6), (7, 7), (8, 8); > +insert t1 select x + 8, x + 8 from t1; > +insert t1 select x + 16, x + 16 from t1; > +insert t1 select x + 49, repeat(x + 49, 100) from t1; > +--echo # This produces wrong result before MDEV-17573 > +select * from t1; What do you mean, simple `select * from t1` produces incorrect result? > +flush tables; > +select x, b from t1 where x > 30 and x < 60 order by b; Is this ASAN-only test? > +drop table t1; > +connection slave; > +drop table t1_1, t1_2; > + > source include/federated_cleanup.inc; > diff --git a/mysql-test/suite/vcol/t/partition.test b/mysql-test/suite/vcol/t/partition.test > index 889724fb1c5c..f10ee0491ccc 100644 > --- a/mysql-test/suite/vcol/t/partition.test > +++ b/mysql-test/suite/vcol/t/partition.test > @@ -30,3 +30,28 @@ subpartition by hash(v) subpartitions 3 ( > insert t1 set i= 0; > set statement sql_mode= '' for update t1 set i= 1, v= 2; > drop table t1; > + > +--echo # > +--echo # MDEV-18734 ASAN heap-use-after-free in my_strnxfrm_simple_internal upon update on versioned partitioned table > +--echo # > +--echo # Cover queue_fix() in ha_partition::handle_ordered_index_scan() > +create table t1 ( > + x int auto_increment primary key, > + b text, v mediumtext as (b) virtual, > + index (v(10)) > +) partition by range columns (x) ( > + partition p1 values less than (3), > + partition p2 values less than (6), > + partition pn values less than (maxvalue)); > +insert into t1 (b) values ('q'), ('a'), ('b'); > +update t1 set b= 'bar' where v > 'a'; > +drop table t1; This test didn't fail for me in an ASAN build on the vanilla 10.5 without your fix. That is, I'm not sure it actually tests the fix. > + > +--echo # Cover return_top_record() in ha_partition::handle_ordered_index_scan() > +create table t1 (x int primary key, b tinytext, v text as (b) virtual) > +partition by range columns (x) ( > + partition p1 values less than (4), > + partition pn values less than (maxvalue)); > +insert into t1 (x, b) values (1, ''), (2, ''), (3, 'a'), (4, 'b'); > +update t1 set b= 'bar' where x > 0 order by v limit 2; This fails without a fix all right. Still, I wonder whether it's possible to create a test that'll fail in a normal optimized build w/o ASAN > +drop table t1; > diff --git a/sql/field.cc b/sql/field.cc > index bdaaecc20269..0fd40c979d2c 100644 > --- a/sql/field.cc > +++ b/sql/field.cc > @@ -8310,6 +8310,7 @@ int Field_blob::store(const char *from,uint length,CHARSET_INFO *cs) > copy_length= copier.well_formed_copy(field_charset, > (char*) value.ptr(), new_length, > cs, from, length); > + value.length(copy_length); good! Could this be a distinct bug with its own test case? > Field_blob::store_length(copy_length); > bmove(ptr+packlength,(uchar*) &tmp,sizeof(char*)); > > diff --git a/sql/ha_partition.cc b/sql/ha_partition.cc > index 9d82df0235c7..7c658082d397 100644 > --- a/sql/ha_partition.cc > +++ b/sql/ha_partition.cc > @@ -5125,14 +5128,14 @@ bool ha_partition::init_record_priority_queue() > uint alloc_len; > uint used_parts= bitmap_bits_set(&m_part_info->read_partitions); > /* Allocate record buffer for each used partition. */ > - m_priority_queue_rec_len= m_rec_length + PARTITION_BYTES_IN_POS; > + m_priority_queue_rec_len= m_rec_length + ORDERED_REC_OFFSET; > if (!m_using_extended_keys) > m_priority_queue_rec_len += m_file[0]->ref_length; > alloc_len= used_parts * m_priority_queue_rec_len; > /* Allocate a key for temporary use when setting up the scan. */ > alloc_len+= table_share->max_key_length; > > - if (!(m_ordered_rec_buffer= (uchar*)my_malloc(alloc_len, MYF(MY_WME)))) > + if (!(m_ordered_rec_buffer= (uchar*) alloc_root(&m_ordered_root, alloc_len))) I don't see why you need a memroot here. memroot is needed when you allocate later at difeerent points in time some initially unpredictable amount of memory that has the same lifetime and needs to be freed all at once. But here it seems that you only allocate once. You could just make m_priority_queue_rec_len = m_rec_length + PARTITION_BYTES_IN_POS + table->s->blob_fields * sizeof(Ordered_blob_storage) (plus necessary alignment). > DBUG_RETURN(true); > > /* > @@ -6178,7 +6203,11 @@ int ha_partition::handle_ordered_index_scan(uchar *buf, bool reverse_order) > */ > error= file->read_range_first(m_start_key.key? &m_start_key: NULL, > end_range, eq_range, TRUE); > - memcpy(rec_buf_ptr, table->record[0], m_rec_length); > + if (!error) > + { > + memcpy(rec_buf_ptr, table->record[0], m_rec_length); > + } did you have a bug because of that? something didn't work? > + > reverse_order= FALSE; > break; > } > @@ -6310,6 +6349,43 @@ int ha_partition::handle_ordered_index_scan_key_not_found() > } > > > +void ha_partition::swap_blobs(uchar * rec_buf, Ordered_blob_storage ** storage, bool restore) > +{ > + uint *ptr, *end; > + uint blob_n= 0; > + table->move_fields(table->field, rec_buf, table->record[0]); > + for (ptr= table->s->blob_field, end= ptr + table->s->blob_fields; > + ptr != end; ++ptr, ++blob_n) > + { > + DBUG_ASSERT(*ptr < table->s->fields); > + Field_blob *blob= (Field_blob*) table->field[*ptr]; > + DBUG_ASSERT(blob->flags & BLOB_FLAG); > + DBUG_ASSERT(blob->field_index == *ptr); > + if (!bitmap_is_set(table->read_set, *ptr) || blob->is_null()) > + continue; > + > + Ordered_blob_storage &s= *storage[blob_n]; > + > + if (restore) > + { > + if (!s.blob.is_empty()) > + blob->swap(s.blob, s.set_read_value); looks somewhat risky, why blob.length == 0 means that you don't need to restore the value? > + } > + else > + { > + bool set_read_value; > + String *cached= blob->cached(set_read_value); > + if (cached) > + { > + cached->swap(s.blob); > + s.set_read_value= set_read_value; is it indeed possible here for a value to be either in `value` or in `read_value` ? When happens what? > + } > + } > + } > + table->move_fields(table->field, table->record[0], rec_buf); > +} > + > + > /* > Common routine to handle index_next with ordered results > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 11

Re: [Maria-developers] 5d19829df5f: MDEV-25908: -e does not work for my_print_defaults
by Sergei Golubchik 21 Jul '21

21 Jul '21

Hi, Rucha! Here the test is ok, buildbot is green. But you've only removed -e. What about -c, -n, -g ? On Jul 21, Rucha Deodhar wrote: > revision-id: 5d19829df5f (mariadb-10.6.1-39-g5d19829df5f) > parent(s): 2c4d1fb5440 > author: Rucha Deodhar > committer: Rucha Deodhar > timestamp: 2021-07-15 13:29:25 +0530 > message: > > MDEV-25908: -e does not work for my_print_defaults > > Analysis: -e is very inconsistent with how --default* options work in any other > executable. > Fix: Removed the support for -e starting from 10.7 to avoid inconsistency in the > long term. > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 61dcf8d940c: MDEV-25908: -e does not work for my_print_defaults
by Sergei Golubchik 21 Jul '21

21 Jul '21

Hi, Rucha! On Jul 21, Rucha Deodhar wrote: > revision-id: 61dcf8d940c (mariadb-10.5.11-35-g61dcf8d940c) > parent(s): f88d130e710 > author: Rucha Deodhar > committer: Rucha Deodhar > timestamp: 2021-07-15 16:07:18 +0530 > message: > > MDEV-25908: -e does not work for my_print_defaults > > Analysis: get_defaults_options() rewrites the value for my_defaults_file, > my_defaults_extra_file and my_defaults_group_suffix to 0. So the config file > can't be read. > Fix: Let handle_options() handle --defaults* option and create a new command > line to pass it to load_defaults(). The code change looks ok. But I thought we've seen from your previous push that this test doesn't work. Haven't we? See how it failed all over in buildbot: https://buildbot.askmonty.org/buildbot/grid?category=main&branch=bb-10.5-MD… Please, generally, first check that your commit passes buildbot, then ask for a review. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] cf6d83e7d68: MDEV-24760 SELECT..CASE statement syntax error at Spider Engine table
by Sergei Golubchik 21 Jul '21

21 Jul '21

Hi, Nayuta! Great, ok to push! On Jul 21, Nayuta Yanagisawa wrote: > revision-id: cf6d83e7d68 (mariadb-10.3.30-28-gcf6d83e7d68) > parent(s): efa311ab8ea > author: Nayuta Yanagisawa > committer: Nayuta Yanagisawa > timestamp: 2021-07-19 15:16:25 +0000 > message: > > MDEV-24760 SELECT..CASE statement syntax error at Spider Engine table > > The root cause of the bug is in `spider_db_mbase_util::open_item_func()`. > The function handles an instance of the `Item_func` class based on its > `Item_func::Functype`. > The `Functype` of `CASE WHEN ... THEN` is `CASE_SEARCHED_FUNC`. > However, the Spider SE doesn't recognize this `Functype` because > `CASE_SEARCHED_FUNC` is newly added by 4de0d92. This results in the wrong > handling of `CASE WHEN ... THEN`. > > The above also applies to `CASE_SIMPLE_FUNC`. > > --- /dev/null > +++ b/storage/spider/mysql-test/spider/bugfix/r/mdev_24760.result ... > +) ENGINE=Spider DEFAULT CHARSET=utf8 COMMENT='table "tbl_a"' > +PARTITION BY HASH(id) ( > +PARTITION pt1 COMMENT='srv "s_2_1"' > +); > +SELECT id, CASE WHEN name='Alice' THEN "A" WHEN name='Bob' THEN "B" END FROM tbl_a; > +id CASE WHEN name='Alice' THEN "A" WHEN name='Bob' THEN "B" END > +1 A > +2 B > +SELECT id, CASE name WHEN 'Alice' THEN "A" WHEN 'Bob' THEN "B" END FROM tbl_a; > +id CASE name WHEN 'Alice' THEN "A" WHEN 'Bob' THEN "B" END > +1 A > +2 B a _very_ slight preference - could you use different names for local (spider) and remote tables? That is, by all means, please, follow the conventions that are used in existing spider tests. But if it's possible to use different tables names for remote and local without violating spider test conventions, I think it'd make tests easier to read. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 38a888da0f1: MDEV-17554 Auto-create new partition for system versioned tables with history partitioned by INTERVAL/LIMIT
by Sergei Golubchik 20 Jul '21

20 Jul '21

Hi, Aleksey! Like last time, it's a review of the diff bc04ded2353 22158b3c05a, so not only commit 38a888da0f1. The main thing below - I didn't understand a couple of changes and asked for clarifications. On May 30, Aleksey Midenkov wrote: > revision-id: 38a888da0f1 (mariadb-10.5.2-653-g38a888da0f1) > parent(s): bc04ded2353 > author: Aleksey Midenkov <midenok(a)gmail.com> > committer: Aleksey Midenkov <midenok(a)gmail.com> > timestamp: 2021-04-22 23:35:06 +0300 > message: > > MDEV-17554 Auto-create new partition for system versioned tables with history partitioned by INTERVAL/LIMIT > diff --git a/mysql-test/suite/versioning/r/delete_history.result b/mysql-test/suite/versioning/r/delete_history.result > --- a/mysql-test/suite/versioning/r/delete_history.result > +++ b/mysql-test/suite/versioning/r/delete_history.result > @@ -154,3 +152,21 @@ select * from t1; > a > 1 > drop table t1; > +# > +# MDEV-17554 Auto-create new partition for system versioned tables with history partitioned by INTERVAL/LIMIT > +# > +# Don't auto-create new partition on DELETE HISTORY: > +set timestamp= unix_timestamp('2000-01-01 00:00:00'); > +create or replace table t (a int) with system versioning > +partition by system_time interval 1 hour auto; > +set timestamp= unix_timestamp('2000-01-01 10:00:00'); perhaps, make it only 02:00:00 to avoid any chance that a partition won't be created because of max-auto-create limit. it's possible we put the limit at 10, but 2 is very unlikely. or does max-auto-create limit always result in a warning? > +delete history from t; > +set timestamp= default; > +show create table t; > +Table Create Table > +t CREATE TABLE `t` ( > + `a` int(11) DEFAULT NULL > +) ENGINE=DEFAULT_ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING > + PARTITION BY SYSTEM_TIME INTERVAL 1 HOUR STARTS TIMESTAMP'2000-01-01 00:00:00' AUTO > +PARTITIONS 2 > +drop table t; > diff --git a/mysql-test/suite/versioning/r/partition.result b/mysql-test/suite/versioning/r/partition.result > --- a/mysql-test/suite/versioning/r/partition.result > +++ b/mysql-test/suite/versioning/r/partition.result > @@ -1236,27 +1270,826 @@ t1 CREATE TABLE `t1` ( > PARTITION `p5` HISTORY ENGINE = DEFAULT_ENGINE, > PARTITION `pn` CURRENT ENGINE = DEFAULT_ENGINE) > alter table t1 add partition partitions 8; > +drop table t1; > +# > +# End of 10.5 tests > +# > +# > +# MDEV-17554 Auto-create new partition for system versioned tables with history partitioned by INTERVAL/LIMIT > +# > +create or replace table t1 (x int) with system versioning > +partition by system_time limit 1 auto; > show create table t1; > Table Create Table > t1 CREATE TABLE `t1` ( > `x` int(11) DEFAULT NULL > ) ENGINE=DEFAULT_ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING > - PARTITION BY SYSTEM_TIME LIMIT 1 > -(PARTITION `p2` HISTORY ENGINE = DEFAULT_ENGINE, > - PARTITION `p8` HISTORY ENGINE = DEFAULT_ENGINE, > + PARTITION BY SYSTEM_TIME LIMIT 1 AUTO > +PARTITIONS 2 > +insert into t1 values (1); > +create or replace table t2 (y int); > +insert into t2 values (2); > +insert into t1 select * from t2; > +insert into t2 select * from t1; > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `x` int(11) DEFAULT NULL > +) ENGINE=DEFAULT_ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING > + PARTITION BY SYSTEM_TIME LIMIT 1 AUTO > +PARTITIONS 2 > +# Too many partitions error > +set timestamp= unix_timestamp('2000-01-01 00:00:00'); > +create or replace table t1 (x int) with system versioning > +partition by system_time interval 1 hour auto; > +set timestamp= unix_timestamp('2001-01-01 00:01:00'); > +update t1 set x= x + 1; > +ERROR HY000: Too many partitions (including subpartitions) were defined > +# Partition overflow error and manual fix seems to be a wrong comment? no manual fix here. > +set timestamp= unix_timestamp('2000-01-01 00:00:00'); > +create or replace table t1 (x int) with system versioning > +partition by system_time interval 1 hour; > +insert into t1 values (333); > +set timestamp= unix_timestamp('2000-01-01 01:00:00'); > +update t1 set x= x + 1; > +Warnings: > +Warning 4114 Versioned table `test`.`t1`: last HISTORY partition (`p0`) is out of INTERVAL, need more HISTORY partitions > +drop table t1; > +# Partition overflow error and manual fix > +set timestamp= unix_timestamp('2000-01-01 00:00:00'); > +create or replace table t1 (x int) with system versioning > +partition by system_time interval 1 hour; > +insert into t1 values (440); > +set timestamp= unix_timestamp('2000-01-01 00:10:00'); > +update t1 set x= x + 1; > +# Check how pruning boundaries work > +explain partitions select * from t1 for system_time as of '2000-01-01 00:59:58'; > +id select_type table partitions type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 p0,pn # NULL NULL NULL NULL # # > +explain partitions select * from t1 for system_time as of '2000-01-01 00:59:59'; > +id select_type table partitions type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 pn # NULL NULL NULL NULL # # Why the first select looks in both partitions and the second one - only in pn? > +explain partitions select * from t1 for system_time as of '2000-01-01 01:00:00'; > +id select_type table partitions type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 pn # NULL NULL NULL NULL # # > +select * from t1 for system_time as of '2000-01-01 00:09:59'; > +x > +440 > +set timestamp= unix_timestamp('2000-01-01 02:00:00'); > +update t1 set x= x + 1; > +Warnings: > +Warning 4114 Versioned table `test`.`t1`: last HISTORY partition (`p0`) is out of INTERVAL, need more HISTORY partitions > +select * from t1 for system_time as of '2000-01-01 01:00:00'; > +x > +select * from t1 partition (p0) order by x; > +x > +440 > +441 > +alter table t1 add partition partitions 3; > +select * from t1 for system_time as of '2000-01-01 01:00:00'; > +x > +441 > +select * from t1 partition (p0) order by x; > +x > +440 would be good to have EXPLAINs after the overflow and after the manual fix. to show how ALTER changes what partition will be pruned > +drop table t1; > +create or replace table t1 (x int) with system versioning > +partition by system_time interval 3600 second > +starts '2000-01-01 00:00:00' auto partitions 3; > +affected rows: 0 > +insert into t1 values (1); > +affected rows: 1 > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `x` int(11) DEFAULT NULL > +) ENGINE=DEFAULT_ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING > + PARTITION BY SYSTEM_TIME INTERVAL 3600 SECOND STARTS TIMESTAMP'2000-01-01 00:00:00' AUTO > +PARTITIONS 3 > ... > diff --git a/mysql-test/suite/versioning/r/rpl_mix.result b/mysql-test/suite/versioning/r/rpl_mix.result > --- a/mysql-test/suite/versioning/r/rpl_mix.result > +++ b/mysql-test/suite/versioning/r/rpl_mix.result > @@ -8,4 +8,59 @@ DELETE HISTORY FROM t1; > connection slave; > connection master; > drop table t1; > +# > +# MDEV-25347 DML events for auto-partitioned tables are written into binary log twice > +# > +flush binary logs; > +create table t1 (a int) with system versioning > +partition by system_time limit 1 auto; > +insert into t1 values (1); > +update t1 set a= a + 1; > +update t1 set a= a + 1; better do a+2 the second time. otherwise it's a bit confusing, the bug says "written twice" and you indeed have update a=a+1 written twice :) with clearly different statements the test will prove that every event is written once, and not one is skipped and the other is duplicated > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` int(11) DEFAULT NULL > +) ENGINE=ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING > + PARTITION BY SYSTEM_TIME LIMIT 1 AUTO > +PARTITIONS 3 > +select * from t1; > +a > +3 > +include/show_binlog_events.inc > +Log_name Pos Event_type Server_id End_log_pos Info > +master-bin.000002 # Binlog_checkpoint # # master-bin.000002 > +master-bin.000002 # Gtid # # GTID #-#-# > +master-bin.000002 # Query # # use `test`; create table t1 (a int) with system versioning > +partition by system_time limit 1 auto > +master-bin.000002 # Gtid # # BEGIN GTID #-#-# > +master-bin.000002 # Annotate_rows # # insert into t1 values (1) > +master-bin.000002 # Table_map # # table_id: # (test.t1) > +master-bin.000002 # Write_rows_v1 # # table_id: # flags: STMT_END_F > +master-bin.000002 # Query # # COMMIT > +master-bin.000002 # Gtid # # BEGIN GTID #-#-# > +master-bin.000002 # Annotate_rows # # update t1 set a= a + 1 > +master-bin.000002 # Table_map # # table_id: # (test.t1) > +master-bin.000002 # Update_rows_v1 # # table_id: # > +master-bin.000002 # Write_rows_v1 # # table_id: # flags: STMT_END_F > +master-bin.000002 # Query # # COMMIT > +master-bin.000002 # Gtid # # BEGIN GTID #-#-# > +master-bin.000002 # Annotate_rows # # update t1 set a= a + 1 > +master-bin.000002 # Table_map # # table_id: # (test.t1) > +master-bin.000002 # Update_rows_v1 # # table_id: # > +master-bin.000002 # Write_rows_v1 # # table_id: # flags: STMT_END_F > +master-bin.000002 # Query # # COMMIT > +connection slave; > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` int(11) DEFAULT NULL > +) ENGINE=ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING > + PARTITION BY SYSTEM_TIME LIMIT 1 AUTO > +PARTITIONS 3 > +select * from t1; > +a > +3 > +connection master; > +drop table t1; > include/rpl_end.inc > diff --git a/mysql-test/suite/versioning/r/rpl_row.result b/mysql-test/suite/versioning/r/rpl_row.result > --- a/mysql-test/suite/versioning/r/rpl_row.result > +++ b/mysql-test/suite/versioning/r/rpl_row.result > @@ -11,4 +11,59 @@ connection slave; > connection master; > drop table t1; > set binlog_row_image= @old_row_image; > +# > +# MDEV-25347 DML events for auto-partitioned tables are written into binary log twice > +# > +flush binary logs; > +create table t1 (a int) with system versioning > +partition by system_time limit 1 auto; > +insert into t1 values (1); > +update t1 set a= a + 1; > +update t1 set a= a + 1; same. as a general rule it's helpful to use slightly different statement, to be able to map them to binlog events uniquely. a+2, or 1+a, or whatever. > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` int(11) DEFAULT NULL > +) ENGINE=ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING > + PARTITION BY SYSTEM_TIME LIMIT 1 AUTO > +PARTITIONS 3 > +select * from t1; > +a > +3 > +include/show_binlog_events.inc > +Log_name Pos Event_type Server_id End_log_pos Info > +master-bin.000002 # Binlog_checkpoint # # master-bin.000002 > +master-bin.000002 # Gtid # # GTID #-#-# > +master-bin.000002 # Query # # use `test`; create table t1 (a int) with system versioning > +partition by system_time limit 1 auto > +master-bin.000002 # Gtid # # BEGIN GTID #-#-# > +master-bin.000002 # Annotate_rows # # insert into t1 values (1) > +master-bin.000002 # Table_map # # table_id: # (test.t1) > +master-bin.000002 # Write_rows_v1 # # table_id: # flags: STMT_END_F > +master-bin.000002 # Query # # COMMIT > +master-bin.000002 # Gtid # # BEGIN GTID #-#-# > +master-bin.000002 # Annotate_rows # # update t1 set a= a + 1 > +master-bin.000002 # Table_map # # table_id: # (test.t1) > +master-bin.000002 # Update_rows_v1 # # table_id: # > +master-bin.000002 # Write_rows_v1 # # table_id: # flags: STMT_END_F > +master-bin.000002 # Query # # COMMIT > +master-bin.000002 # Gtid # # BEGIN GTID #-#-# > +master-bin.000002 # Annotate_rows # # update t1 set a= a + 1 > +master-bin.000002 # Table_map # # table_id: # (test.t1) > +master-bin.000002 # Update_rows_v1 # # table_id: # > +master-bin.000002 # Write_rows_v1 # # table_id: # flags: STMT_END_F > +master-bin.000002 # Query # # COMMIT > +connection slave; > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` int(11) DEFAULT NULL > +) ENGINE=ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING > + PARTITION BY SYSTEM_TIME LIMIT 1 AUTO > +PARTITIONS 3 > +select * from t1; > +a > +3 > +connection master; > +drop table t1; > include/rpl_end.inc > diff --git a/mysql-test/suite/versioning/t/partition.test b/mysql-test/suite/versioning/t/partition.test > --- a/mysql-test/suite/versioning/t/partition.test > +++ b/mysql-test/suite/versioning/t/partition.test > @@ -1068,13 +1078,456 @@ alter table t1 add partition partitions 2; > --replace_result $default_engine DEFAULT_ENGINE > show create table t1; > alter table t1 add partition partitions 8; > +drop table t1; > + > +--echo # > +--echo # End of 10.5 tests > +--echo # > + > +--echo # > +--echo # MDEV-17554 Auto-create new partition for system versioned tables with history partitioned by INTERVAL/LIMIT > +--echo # > +create or replace table t1 (x int) with system versioning > +partition by system_time limit 1 auto; > +--replace_result $default_engine DEFAULT_ENGINE > +show create table t1; > + in the previous version of this patch you had clarifying comments here, like --echo # INSERT, INSERT .. SELECT don't auto-create partitions or --echo # Number of partitions goes from 3 to 5 I kind of miss them now, they were helpful > +insert into t1 values (1); > + > +create or replace table t2 (y int); > +insert into t2 values (2); > + > +insert into t1 select * from t2; > +insert into t2 select * from t1; > +--replace_result $default_engine DEFAULT_ENGINE > +show create table t1; > + > ... > diff --git a/sql/ha_partition.h b/sql/ha_partition.h > --- a/sql/ha_partition.h > +++ b/sql/ha_partition.h > @@ -1607,7 +1607,7 @@ class ha_partition :public handler > for (; part_id < part_id_end; ++part_id) > { > handler *file= m_file[part_id]; > - DBUG_ASSERT(bitmap_is_set(&(m_part_info->read_partitions), part_id)); > + bitmap_set_bit(&(m_part_info->read_partitions), part_id); would be good to have a comment before the method that it's only used in vers_set_hist_part(), if (vers_info->limit). because if it's a generic method to return the number of rows in a partition (incl. subpartitions) then it's totally not clear why it modifies read_partitions bitmap. or may be you'd want to rename the method to make it look more specific for LIMIT partitions in vers_set_hist_part() ? > file->info(HA_STATUS_VARIABLE | HA_STATUS_NO_LOCK | HA_STATUS_OPEN); > part_recs+= file->stats.records; > } > diff --git a/sql/handler.cc b/sql/handler.cc > --- a/sql/handler.cc > +++ b/sql/handler.cc > @@ -1574,6 +1574,8 @@ int ha_commit_trans(THD *thd, bool all) > DBUG_ASSERT(thd->transaction->stmt.ha_list == NULL || > trans == &thd->transaction->stmt); > > + DBUG_ASSERT(!thd->in_sub_stmt); > if (thd->in_sub_stmt) > { > DBUG_ASSERT(0); This is a bit redundant, don't you think? One DBUG_ASSERT would be enough. (personally, I'd prefer the one you added, not DBUG_ASSERT(0)) > diff --git a/sql/lock.cc b/sql/lock.cc > --- a/sql/lock.cc > +++ b/sql/lock.cc > @@ -662,16 +662,28 @@ MYSQL_LOCK *mysql_lock_merge(MYSQL_LOCK *a,MYSQL_LOCK *b) > DBUG_PRINT("enter", ("a->lock_count: %u b->lock_count: %u", > a->lock_count, b->lock_count)); > > - if (!(sql_lock= (MYSQL_LOCK*) > - my_malloc(key_memory_MYSQL_LOCK, sizeof(*sql_lock) + > - sizeof(THR_LOCK_DATA*)*((a->lock_count+b->lock_count)*2) + > - sizeof(TABLE*)*(a->table_count+b->table_count),MYF(MY_WME)))) > - DBUG_RETURN(0); // Fatal error > + const size_t lock_size= sizeof(*sql_lock) + > + sizeof(THR_LOCK_DATA *) * ((a->lock_count + b->lock_count) * 2) + > + sizeof(TABLE *) * (a->table_count + b->table_count); > + if (thd) > + { > + sql_lock= (MYSQL_LOCK *) thd->alloc(lock_size); > + if (!sql_lock) > + DBUG_RETURN(0); > + sql_lock->flags= GET_LOCK_ON_THD; why? the commit comment for MDEV-23639 doesn't mention it. generally, I think it's a good idea and, likely, should be used more than just in recover_from_failed_open(). But why right now and in this commit? Does this commit (22158b3c) rely on locks being on THD memroot? How so? > + } > + else > + { > + sql_lock= (MYSQL_LOCK *) > + my_malloc(key_memory_MYSQL_LOCK, lock_size, MYF(MY_WME)); > + if (!sql_lock) > + DBUG_RETURN(0); > + sql_lock->flags= 0; > + } > sql_lock->lock_count=a->lock_count+b->lock_count; > sql_lock->table_count=a->table_count+b->table_count; > sql_lock->locks=(THR_LOCK_DATA**) (sql_lock+1); > sql_lock->table=(TABLE**) (sql_lock->locks+sql_lock->lock_count*2); > - sql_lock->flags= 0; > memcpy(sql_lock->locks,a->locks,a->lock_count*sizeof(*a->locks)); > memcpy(sql_lock->locks+a->lock_count,b->locks, > b->lock_count*sizeof(*b->locks)); > @@ -705,8 +717,10 @@ MYSQL_LOCK *mysql_lock_merge(MYSQL_LOCK *a,MYSQL_LOCK *b) > a->lock_count, b->lock_count); > > /* Delete old, not needed locks */ > - my_free(a); > - my_free(b); > + if (!(a->flags & GET_LOCK_ON_THD)) > + my_free(a); > + if (!(b->flags & GET_LOCK_ON_THD)) > + my_free(b); this even looks like a bug, no check that old locks weren't on THD memroot. (probably impossible in this particular case, but still rather fragile) thanks! > DBUG_RETURN(sql_lock); > } > > diff --git a/sql/share/errmsg-utf8.txt b/sql/share/errmsg-utf8.txt > --- a/sql/share/errmsg-utf8.txt > +++ b/sql/share/errmsg-utf8.txt > @@ -7985,3 +7985,5 @@ ER_JSON_TABLE_MULTIPLE_MATCHES > eng "Can't store multiple matches of the path in the column '%s' of JSON_TABLE '%s'." > ER_WITH_TIES_NEEDS_ORDER > eng "FETCH ... WITH TIES requires ORDER BY clause to be present" > +ER_VERS_HIST_PART_FAILED > + eng "Versioned table %`s.%`s: adding HISTORY partition failed" Can you come up with a test for this error? > diff --git a/sql/sql_base.cc b/sql/sql_base.cc > --- a/sql/sql_base.cc > +++ b/sql/sql_base.cc > @@ -931,7 +934,7 @@ void close_thread_table(THD *thd, TABLE **table_ptr) > DBUG_PRINT("tcache", ("table: '%s'.'%s' %p", table->s->db.str, > table->s->table_name.str, table)); > DBUG_ASSERT(!table->file->keyread_enabled()); > - DBUG_ASSERT(!table->file || table->file->inited == handler::NONE); > + DBUG_ASSERT(table->file->inited == handler::NONE); indeed > > /* > The metadata lock must be released after giving back > @@ -1625,6 +1625,125 @@ bool is_locked_view(THD *thd, TABLE_LIST *t) > } > > > +#ifdef WITH_PARTITION_STORAGE_ENGINE > +/** > + Switch part_info->hist_part and request partition creation if needed. > + > + @retval true Error or partition creation was requested. > + @retval false No error > +*/ > +bool TABLE::vers_switch_partition(THD *thd, TABLE_LIST *table_list, > + Open_table_context *ot_ctx) > +{ > + if (!part_info || part_info->part_type != VERSIONING_PARTITION || > + table_list->vers_conditions.delete_history || > + thd->stmt_arena->is_stmt_prepare() || > + table_list->lock_type < TL_WRITE_ALLOW_WRITE || > + table_list->mdl_request.type < MDL_SHARED_WRITE || > + table_list->mdl_request.type == MDL_EXCLUSIVE) > + { > + return false; > + } > + > + switch (thd->lex->sql_command) > + { > + case SQLCOM_INSERT: > + if (thd->lex->duplicates != DUP_UPDATE) > + return false; > + break; > + case SQLCOM_LOAD: > + if (thd->lex->duplicates != DUP_REPLACE) > + return false; > + break; > + case SQLCOM_LOCK_TABLES: > + case SQLCOM_DELETE: > + case SQLCOM_UPDATE: > + case SQLCOM_REPLACE: > + case SQLCOM_REPLACE_SELECT: > + case SQLCOM_DELETE_MULTI: > + case SQLCOM_UPDATE_MULTI: > + break; > + default: > + /* > + TODO: make row events set thd->lex->sql_command appropriately. > + > + Sergei Golubchik: f.ex. currently row events increment > + thd->status_var.com_stat[] each event for its own SQLCOM_xxx, it won't be > + needed if they'll just set thd->lex->sql_command. > + */ > + if (thd->rgi_slave && thd->rgi_slave->current_event && > + thd->lex->sql_command == SQLCOM_END) > + { > + switch (thd->rgi_slave->current_event->get_type_code()) > + { > + case UPDATE_ROWS_EVENT: > + case UPDATE_ROWS_EVENT_V1: > + case DELETE_ROWS_EVENT: > + case DELETE_ROWS_EVENT_V1: > + break; > + default:; > + return false; > + } > + } > + break; > + } > + > + TABLE *table= this; > + > + /* > + NOTE: The semantics of vers_set_hist_part() is double: even when we twofold > + don't need auto-create, we need to update part_info->hist_part. > + */ > + uint *create_count= table_list->vers_skip_create ? > + NULL : &ot_ctx->vers_create_count; > + table_list->vers_skip_create= true; > + if (table->part_info->vers_set_hist_part(thd, create_count)) > + { > + MYSQL_UNBIND_TABLE(table->file); > + tc_release_table(table); > + return true; > + } > + if (ot_ctx->vers_create_count) > + { > + Open_table_context::enum_open_table_action action; > + TABLE_LIST *table_arg; > + mysql_mutex_lock(&table->s->LOCK_share); > + if (!table->s->vers_skip_auto_create) > + { > + table->s->vers_skip_auto_create= true; > + action= Open_table_context::OT_ADD_HISTORY_PARTITION; > + table_arg= table_list; > + } > + else > + { > + /* > + NOTE: this may repeat multiple times until creating thread acquires > + MDL_EXCLUSIVE. Since auto-creation is rare operation this is acceptable. > + We could suspend this thread on cond-var but we must first exit > + MDL_SHARED_WRITE first and we cannot store cond-var into TABLE_SHARE > + because it is already released and there is no guarantee that it will > + be same instance if we acquire it again. > + */ > + table_list->vers_skip_create= false; > + ot_ctx->vers_create_count= 0; > + action= Open_table_context::OT_REOPEN_TABLES; > + table_arg= NULL; > + } I'm afraid I don't understand. All this business with vers_skip_create and vers_skip_auto_create, it wasn't in the previous version of the patch. So, I believe it was a fix for one of the MDEV bugs reported and fixed meanwhile. What MDEV was it? Could you explain what was the issue and what was the fix, please? > + mysql_mutex_unlock(&table->s->LOCK_share); > + if (!thd->locked_tables_mode) > + { > + MYSQL_UNBIND_TABLE(table->file); > + tc_release_table(table); > + } > + ot_ctx->request_backoff_action(action, table_arg); > + return true; > + } > + > + return false; > +} > +#endif /* WITH_PARTITION_STORAGE_ENGINE */ > + > + > /** > Open a base table. > > @@ -2572,11 +2699,13 @@ void Locked_tables_list::mark_table_for_reopen(THD *thd, TABLE *table) > table_list; table_list= table_list->next_global) > { > if (table_list->table->s == share) > + { > table_list->table->internal_set_needs_reopen(true); > + some_table_marked_for_reopen= 1; > + } > } > /* This is needed in the case where lock tables where not used */ > table->internal_set_needs_reopen(true); why ^^^ this doesn't count as "some_table_marked_for_reopen" ? > - some_table_marked_for_reopen= 1; > } > > > @@ -3172,13 +3304,49 @@ Open_table_context::recover_from_failed_open() > break; > case OT_DISCOVER: > case OT_REPAIR: > - if ((result= lock_table_names(m_thd, m_thd->lex->create_info, > - m_failed_table, NULL, > - get_timeout(), 0))) > + case OT_ADD_HISTORY_PARTITION: > + if (!m_thd->locked_tables_mode) > + result= lock_table_names(m_thd, m_thd->lex->create_info, m_failed_table, > + NULL, get_timeout(), 0); > + else > + { > + DBUG_ASSERT(!result); > + DBUG_ASSERT(m_action == OT_ADD_HISTORY_PARTITION); It seems you've solved recover_from_failed_open under LOCK TABLES, so it could be possible to make OT_DISCOVER and OT_REPAIR to work there too. Is that right? It's just a question, I don't imply it should be part of this MDEV. > + } > + /* > + We are now under MDL_EXCLUSIVE mode. Other threads have no table share > + acquired: they are blocked either at open_table_get_mdl_lock() in > + open_table() or at lock_table_names() here. > + */ > + if (result) > + { > + if (m_action == OT_ADD_HISTORY_PARTITION) > + { > + TABLE_SHARE *share= tdc_acquire_share(m_thd, m_failed_table, > + GTS_TABLE, NULL); > + if (share) > + { > + share->vers_skip_auto_create= false; > + tdc_release_share(share); > + } > + if (m_thd->get_stmt_da()->sql_errno() == ER_LOCK_WAIT_TIMEOUT) > + { > + // MDEV-23642 Locking timeout caused by auto-creation affects original DML > + m_thd->clear_error(); > + vers_create_count= 0; > + result= false; > + } > + } > break; > + } > > - tdc_remove_table(m_thd, m_failed_table->db.str, > - m_failed_table->table_name.str); > + /* > + We don't need to remove share under OT_ADD_HISTORY_PARTITION. > + Moreover fast_alter_partition_table() works with TABLE instance. > + */ > + if (m_action != OT_ADD_HISTORY_PARTITION) > + tdc_remove_table(m_thd, m_failed_table->db.str, > + m_failed_table->table_name.str); > > switch (m_action) > { > @@ -3206,6 +3374,70 @@ Open_table_context::recover_from_failed_open() > case OT_REPAIR: > result= auto_repair_table(m_thd, m_failed_table); > break; > + case OT_ADD_HISTORY_PARTITION: > + { > + result= false; > + TABLE *table= open_ltable(m_thd, m_failed_table, TL_WRITE, > + MYSQL_OPEN_HAS_MDL_LOCK | MYSQL_OPEN_IGNORE_LOGGING_FORMAT); > + if (table == NULL) > + { > + m_thd->clear_error(); > + break; > + } > + > + DBUG_ASSERT(vers_create_count); > + result= vers_create_partitions(m_thd, m_failed_table, vers_create_count); > + vers_create_count= 0; > + if (!m_thd->transaction->stmt.is_empty()) > + trans_commit_stmt(m_thd); > + DBUG_ASSERT(!result || > + !m_thd->locked_tables_mode || > + m_thd->lock->lock_count); > + if (result) > + break; > + if (!m_thd->locked_tables_mode) > + { > + /* > + alter_partition_lock_handling() does mysql_lock_remove() but > + does not clear thd->lock completely. > + */ > + DBUG_ASSERT(m_thd->lock->lock_count == 0); > + if (!(m_thd->lock->flags & GET_LOCK_ON_THD)) > + my_free(m_thd->lock); > + m_thd->lock= NULL; > + } > + else if (m_thd->locked_tables_mode == LTM_PRELOCKED) > + { > + MYSQL_LOCK *lock; > + MYSQL_LOCK *merged_lock; > + > + /* > + In LTM_LOCK_TABLES table was reopened via locked_tables_list, > + but not in prelocked environment where we have to reopen > + the table manually. > + */ > + Open_table_context ot_ctx(m_thd, MYSQL_OPEN_REOPEN); > + if (open_table(m_thd, m_failed_table, &ot_ctx)) > + { > + result= true; > + break; > + } > + TABLE *table= m_failed_table->table; > + table->reginfo.lock_type= m_thd->update_lock_default; > + m_thd->in_lock_tables= 1; > + lock= mysql_lock_tables(m_thd, &table, 1, > + MYSQL_OPEN_REOPEN | MYSQL_LOCK_USE_MALLOC); > + m_thd->in_lock_tables= 0; > + if (lock == NULL || > + !(merged_lock= mysql_lock_merge(m_thd->lock, lock, m_thd))) > + { > + result= true; > + break; > + } > + m_thd->lock= merged_lock; What MDEV does it fix? I wonder why other cases in recover_from_failed_open doesn't do that > + } > + break; > + } > case OT_BACKOFF_AND_RETRY: > case OT_REOPEN_TABLES: > case OT_NO_ACTION: > @@ -4370,6 +4611,8 @@ bool open_tables(THD *thd, const DDL_options_st &options, > { > if (ot_ctx.can_recover_from_failed_open()) > { > + // FIXME: is this really used? > + DBUG_ASSERT(0); I take it, you weren't able to get this assert to fire? > close_tables_for_reopen(thd, start, > ot_ctx.start_of_statement_svp()); > if (ot_ctx.recover_from_failed_open()) Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 13

[Maria-developers] Progress of improve IO_CACHE (MDEV-24676)
by Vladisav Kakurin 20 Jul '21

20 Jul '21

To this moment I have implemented medium-grained granularity, and also move IO_CACHE to C++ entitled RingBuffer. In future, it will be fine-grained with slots-pattern. https://github.com/MagHErmit/server/commit/7c066a97403115f6d7702049b741d466…

1 0

Re: [Maria-developers] [Commits] 6f1628b: MDEV-25565 Crash on 2-nd execution of SP/PS for query calculating window functions
by Vicențiu Ciorbaru 20 Jul '21

20 Jul '21

Hi Igor! The patch looks good, however I suggest the following changes to make it smaller and easier to read & maintain. 1. You do not need to have special if statements for save_partition_list and save_order_list when you do the cleanup. 2. You do not need to have special code that saves the partition list and the order list *only* when it is overwritten. You can instead always set save_partition_list to be equal to partition_list in the Window_spec's constructor. Same for save_order_list and order_list. Then, during cleanup, you can always execute order_list= save_order_list; partition_list= save_partition_list; The advantage of this change is that it removes all the references to save_xxxx_list in sql_window.cc. You then only have a simple cleanup function sql_union.cc. 3. I would prefer to name the variables "initial_order_list" and "initial_partition_list". The patch then becomes: diff --git a/sql/sql_union.cc b/sql/sql_union.cc index e5648e6989b..54a33bb7863 100644 --- a/sql/sql_union.cc +++ b/sql/sql_union.cc @@ -30,6 +30,7 @@ #include "filesort.h" // filesort_free_buffers #include "sql_view.h" #include "sql_cte.h" +#include "item_windowfunc.h" bool mysql_union(THD *thd, LEX *lex, select_result *result, SELECT_LEX_UNIT *unit, ulong setup_tables_done_option) @@ -1551,6 +1552,21 @@ static void cleanup_order(ORDER *order) } +static void cleanup_window_funcs(List<Item_window_func> &win_funcs) +{ + List_iterator_fast<Item_window_func> it(win_funcs); + Item_window_func *win_func; + while ((win_func= it++)) + { + Window_spec *win_spec= win_func->window_spec; + if (!win_spec) + continue; + win_spec->partition_list= win_spec->initial_partition_list; + win_spec->order_list= win_spec->initial_order_list; + } +} + + bool st_select_lex::cleanup() { bool error= FALSE; @@ -1559,6 +1575,8 @@ bool st_select_lex::cleanup() cleanup_order(order_list.first); cleanup_order(group_list.first); + cleanup_window_funcs(window_funcs); + if (join) { List_iterator<TABLE_LIST> ti(leaf_tables); diff --git a/sql/sql_window.h b/sql/sql_window.h index e0c1563e5bb..cf84e26c822 100644 --- a/sql/sql_window.h +++ b/sql/sql_window.h @@ -99,8 +99,10 @@ class Window_spec : public Sql_alloc LEX_STRING *window_ref; SQL_I_List<ORDER> *partition_list; + SQL_I_List<ORDER> *initial_partition_list; SQL_I_List<ORDER> *order_list; + SQL_I_List<ORDER> *initial_order_list; Window_frame *window_frame; @@ -111,7 +113,8 @@ class Window_spec : public Sql_alloc SQL_I_List<ORDER> *ord_list, Window_frame *win_frame) : window_names_are_checked(false), window_ref(win_ref), - partition_list(part_list), order_list(ord_list), + partition_list(part_list), initial_partition_list(part_list), + order_list(ord_list), initial_order_list(ord_list), window_frame(win_frame), referenced_win_spec(NULL) {} virtual char *name() { return NULL; } On Sat, 10 Jul 2021 at 04:57, IgorBabaev <igor(a)mariadb.com> wrote: > revision-id: 6f1628b917d365ecfc9c4c9951011613f4212592 > (mariadb-10.2.31-1048-g6f1628b) > parent(s): fb0b28932ce82903f2fcfb690a71bff52355507f > author: Igor Babaev > committer: Igor Babaev > timestamp: 2021-07-09 18:56:34 -0700 > message: > > MDEV-25565 Crash on 2-nd execution of SP/PS for query calculating window > functions > from view > > A crash of the server happened when executing a stored procedure whose the > only query calculated window functions over a mergeable view specified > as a select from non-mergeable view. The crash could be reproduced if > the window specifications of the window functions were identical and both > contained PARTITION lists and ORDER BY lists. A crash also happened on > the second execution of the prepared statement created for such query. > If to use derived tables or CTE instead of views the problem still > manifests itself crashing the server. > > When optimizing the window specifications of a window function the > server can substitute the partition lists and the order lists for > the corresponding lists from another window specification in the case > when the lists are identical. This substitution is not permanent and should > be rolled back before the second execution. It was not done and this > ultimately led to a crash when resolving the column names at the second > execution of SP/PS. > > --- > mysql-test/r/win.result | 287 > ++++++++++++++++++++++++++++++++++++++++++++++++ > mysql-test/t/win.test | 147 +++++++++++++++++++++++++ > sql/sql_union.cc | 26 +++++ > sql/sql_window.cc | 12 ++ > sql/sql_window.h | 5 +- > 5 files changed, 476 insertions(+), 1 deletion(-) > > diff --git a/mysql-test/r/win.result b/mysql-test/r/win.result > index 8a31dcc..bc017ea 100644 > --- a/mysql-test/r/win.result > +++ b/mysql-test/r/win.result > @@ -3911,5 +3911,292 @@ sum(i) over () IN ( SELECT 1 FROM t1 a) > 0 > DROP TABLE t1; > # > +# MDEV-25565: 2-nd call of SP with SELECT from view / derived table / CTE > +# returning the result of calculation of 2 window > +# functions that use the same window specification > +# > +create table t1 (a int); > +insert into t1 values (3), (7), (1), (7), (1), (1), (3), (1), (5); > +create view v2 as select a from t1 group by a; > +create view v1 as select * from v2; > +create procedure sp1() select v1.a, > +sum(v1.a) over (partition by v1.a order by v1.a) as k, > +avg(v1.a) over (partition by v1.a order by v1.a) as m > +from v1; > +call sp1(); > +a k m > +1 1 1.0000 > +3 3 3.0000 > +5 5 5.0000 > +7 7 7.0000 > +call sp1(); > +a k m > +1 1 1.0000 > +3 3 3.0000 > +5 5 5.0000 > +7 7 7.0000 > +prepare stmt from "select v1.a, > +sum(v1.a) over (partition by v1.a order by v1.a) as k, > +avg(v1.a) over (partition by v1.a order by v1.a) as m > +from v1"; > +execute stmt; > +a k m > +1 1 1.0000 > +3 3 3.0000 > +5 5 5.0000 > +7 7 7.0000 > +execute stmt; > +a k m > +1 1 1.0000 > +3 3 3.0000 > +5 5 5.0000 > +7 7 7.0000 > +deallocate prepare stmt; > +create procedure sp2() select * from > +( select dt1.a, > +sum(dt1.a) over (partition by dt1.a order by dt1.a) as k, > +avg(dt1.a) over (partition by dt1.a order by dt1.a) as m > +from (select * from v2) as dt1 > +) as dt; > +call sp2(); > +a k m > +1 1 1.0000 > +3 3 3.0000 > +5 5 5.0000 > +7 7 7.0000 > +call sp2(); > +a k m > +1 1 1.0000 > +3 3 3.0000 > +5 5 5.0000 > +7 7 7.0000 > +prepare stmt from "select * from > +( select dt1.a, > +sum(dt1.a) over (partition by dt1.a order by dt1.a) as k, > +avg(dt1.a) over (partition by dt1.a order by dt1.a) as m > +from (select * from v2) as dt1 > +) as dt"; > +execute stmt; > +a k m > +1 1 1.0000 > +3 3 3.0000 > +5 5 5.0000 > +7 7 7.0000 > +execute stmt; > +a k m > +1 1 1.0000 > +3 3 3.0000 > +5 5 5.0000 > +7 7 7.0000 > +deallocate prepare stmt; > +create procedure sp3() select * from > +( select dt1.a, > +sum(dt1.a) over (partition by dt1.a order by dt1.a) as k, > +avg(dt1.a) over (partition by dt1.a order by dt1.a) as m > +from ( select * from (select * from t1 group by a) as dt2 ) as dt1 > +) as dt; > +call sp3(); > +a k m > +1 1 1.0000 > +3 3 3.0000 > +5 5 5.0000 > +7 7 7.0000 > +call sp3(); > +a k m > +1 1 1.0000 > +3 3 3.0000 > +5 5 5.0000 > +7 7 7.0000 > +prepare stmt from "select * from > +( select dt1.a, > +sum(dt1.a) over (partition by dt1.a order by dt1.a) as k, > +avg(dt1.a) over (partition by dt1.a order by dt1.a) as m > +from ( select * from (select * from t1 group by a) as dt2 ) as dt1 > +) as dt"; > +execute stmt; > +a k m > +1 1 1.0000 > +3 3 3.0000 > +5 5 5.0000 > +7 7 7.0000 > +execute stmt; > +a k m > +1 1 1.0000 > +3 3 3.0000 > +5 5 5.0000 > +7 7 7.0000 > +deallocate prepare stmt; > +create procedure sp4() with cte1 as (select * from (select * from t1 > group by a) as dt2), > +cte as > +( select cte1.a, > +sum(cte1.a) over (partition by cte1.a order by cte1.a) as k, > +avg(cte1.a) over (partition by cte1.a order by cte1.a) as m > +from cte1 ) > +select * from cte; > +call sp4(); > +a k m > +1 1 1.0000 > +3 3 3.0000 > +5 5 5.0000 > +7 7 7.0000 > +call sp4(); > +a k m > +1 1 1.0000 > +3 3 3.0000 > +5 5 5.0000 > +7 7 7.0000 > +prepare stmt from "with cte1 as (select * from (select * from t1 group by > a) as dt2), > +cte as > +( select cte1.a, > +sum(cte1.a) over (partition by cte1.a order by cte1.a) as k, > +avg(cte1.a) over (partition by cte1.a order by cte1.a) as m > +from cte1 ) > +select * from cte"; > +execute stmt; > +a k m > +1 1 1.0000 > +3 3 3.0000 > +5 5 5.0000 > +7 7 7.0000 > +execute stmt; > +a k m > +1 1 1.0000 > +3 3 3.0000 > +5 5 5.0000 > +7 7 7.0000 > +deallocate prepare stmt; > +create procedure sp5() with cte1 as (select * from v2), > +cte as > +( select cte1.a, > +sum(cte1.a) over (partition by cte1.a order by cte1.a) as k, > +avg(cte1.a) over (partition by cte1.a order by cte1.a) as m > +from cte1 ) > +select * from cte; > +call sp5(); > +a k m > +1 1 1.0000 > +3 3 3.0000 > +5 5 5.0000 > +7 7 7.0000 > +call sp5(); > +a k m > +1 1 1.0000 > +3 3 3.0000 > +5 5 5.0000 > +7 7 7.0000 > +prepare stmt from "with cte1 as (select * from v2), > +cte as > +( select cte1.a, > +sum(cte1.a) over (partition by cte1.a order by cte1.a) as k, > +avg(cte1.a) over (partition by cte1.a order by cte1.a) as m > +from cte1 ) > +select * from cte"; > +execute stmt; > +a k m > +1 1 1.0000 > +3 3 3.0000 > +5 5 5.0000 > +7 7 7.0000 > +execute stmt; > +a k m > +1 1 1.0000 > +3 3 3.0000 > +5 5 5.0000 > +7 7 7.0000 > +deallocate prepare stmt; > +create procedure sp6() with > +cte1 as (with cte2 as (select * from t1 group by a) select * from cte2), > +cte as > +( select cte1.a, > +sum(cte1.a) over (partition by cte1.a order by cte1.a) as k, > +avg(cte1.a) over (partition by cte1.a order by cte1.a) as m > +from cte1 ) > +select * from cte; > +call sp6(); > +a k m > +1 1 1.0000 > +3 3 3.0000 > +5 5 5.0000 > +7 7 7.0000 > +call sp6(); > +a k m > +1 1 1.0000 > +3 3 3.0000 > +5 5 5.0000 > +7 7 7.0000 > +prepare stmt from "with > +cte1 as (with cte2 as (select * from t1 group by a) select * from cte2), > +cte as > +( select cte1.a, > +sum(cte1.a) over (partition by cte1.a order by cte1.a) as k, > +avg(cte1.a) over (partition by cte1.a order by cte1.a) as m > +from cte1 ) > +select * from cte"; > +execute stmt; > +a k m > +1 1 1.0000 > +3 3 3.0000 > +5 5 5.0000 > +7 7 7.0000 > +execute stmt; > +a k m > +1 1 1.0000 > +3 3 3.0000 > +5 5 5.0000 > +7 7 7.0000 > +deallocate prepare stmt; > +create procedure sp7() with > +cte2 as (select * from v1), > +cte1 as (select * from cte2), > +cte as > +( select cte1.a, > +sum(cte1.a) over (partition by cte1.a order by cte1.a) as k, > +avg(cte1.a) over (partition by cte1.a order by cte1.a) as m > +from cte1 ) > +select * from cte; > +call sp7(); > +a k m > +1 1 1.0000 > +3 3 3.0000 > +5 5 5.0000 > +7 7 7.0000 > +call sp7(); > +a k m > +1 1 1.0000 > +3 3 3.0000 > +5 5 5.0000 > +7 7 7.0000 > +prepare stmt from "with > +cte2 as (select * from v1), > +cte1 as (select * from cte2), > +cte as > +( select cte1.a, > +sum(cte1.a) over (partition by cte1.a order by cte1.a) as k, > +avg(cte1.a) over (partition by cte1.a order by cte1.a) as m > +from cte1 ) > +select * from cte"; > +execute stmt; > +a k m > +1 1 1.0000 > +3 3 3.0000 > +5 5 5.0000 > +7 7 7.0000 > +execute stmt; > +a k m > +1 1 1.0000 > +3 3 3.0000 > +5 5 5.0000 > +7 7 7.0000 > +deallocate prepare stmt; > +drop procedure sp1; > +drop procedure sp2; > +drop procedure sp3; > +drop procedure sp4; > +drop procedure sp5; > +drop procedure sp6; > +drop procedure sp7; > +drop view v1,v2; > +drop table t1; > +# > # End of 10.2 tests > # > diff --git a/mysql-test/t/win.test b/mysql-test/t/win.test > index c07a81f..72e789d 100644 > --- a/mysql-test/t/win.test > +++ b/mysql-test/t/win.test > @@ -2557,5 +2557,152 @@ SELECT sum(i) over () IN ( SELECT 1 FROM t1 a) > FROM t1; > DROP TABLE t1; > > --echo # > +--echo # MDEV-25565: 2-nd call of SP with SELECT from view / derived > table / CTE > +--echo # returning the result of calculation of 2 window > +--echo # functions that use the same window specification > +--echo # > + > +create table t1 (a int); > +insert into t1 values (3), (7), (1), (7), (1), (1), (3), (1), (5); > + > +create view v2 as select a from t1 group by a; > +create view v1 as select * from v2; > + > +let $q1= > +select v1.a, > + sum(v1.a) over (partition by v1.a order by v1.a) as k, > + avg(v1.a) over (partition by v1.a order by v1.a) as m > +from v1; > + > +eval create procedure sp1() $q1; > +call sp1(); > +call sp1(); > + > +eval prepare stmt from "$q1"; > +execute stmt; > +execute stmt; > +deallocate prepare stmt; > + > +let $q2= > +select * from > + ( select dt1.a, > + sum(dt1.a) over (partition by dt1.a order by dt1.a) as k, > + avg(dt1.a) over (partition by dt1.a order by dt1.a) as m > + from (select * from v2) as dt1 > + ) as dt; > + > +eval create procedure sp2() $q2; > +call sp2(); > +call sp2(); > + > +eval prepare stmt from "$q2"; > +execute stmt; > +execute stmt; > +deallocate prepare stmt; > + > +let $q3= > +select * from > + ( select dt1.a, > + sum(dt1.a) over (partition by dt1.a order by dt1.a) as k, > + avg(dt1.a) over (partition by dt1.a order by dt1.a) as m > + from ( select * from (select * from t1 group by a) as dt2 ) as dt1 > + ) as dt; > + > +eval create procedure sp3() $q3; > +call sp3(); > +call sp3(); > + > +eval prepare stmt from "$q3"; > +execute stmt; > +execute stmt; > +deallocate prepare stmt; > + > +let $q4= > +with cte1 as (select * from (select * from t1 group by a) as dt2), > + cte as > + ( select cte1.a, > + sum(cte1.a) over (partition by cte1.a order by cte1.a) as > k, > + avg(cte1.a) over (partition by cte1.a order by cte1.a) as m > + from cte1 ) > +select * from cte; > + > +eval create procedure sp4() $q4; > +call sp4(); > +call sp4(); > + > +eval prepare stmt from "$q4"; > +execute stmt; > +execute stmt; > +deallocate prepare stmt; > + > +let $q5= > +with cte1 as (select * from v2), > + cte as > + ( select cte1.a, > + sum(cte1.a) over (partition by cte1.a order by cte1.a) as > k, > + avg(cte1.a) over (partition by cte1.a order by cte1.a) as m > + from cte1 ) > +select * from cte; > + > +eval create procedure sp5() $q5; > +call sp5(); > +call sp5(); > + > +eval prepare stmt from "$q5"; > +execute stmt; > +execute stmt; > +deallocate prepare stmt; > + > +let $q6= > +with > +cte1 as (with cte2 as (select * from t1 group by a) select * from cte2), > + cte as > + ( select cte1.a, > + sum(cte1.a) over (partition by cte1.a order by cte1.a) as > k, > + avg(cte1.a) over (partition by cte1.a order by cte1.a) as m > + from cte1 ) > +select * from cte; > + > +eval create procedure sp6() $q6; > +call sp6(); > +call sp6(); > + > +eval prepare stmt from "$q6"; > +execute stmt; > +execute stmt; > +deallocate prepare stmt; > + > +let $q7= > +with > + cte2 as (select * from v1), > + cte1 as (select * from cte2), > + cte as > + ( select cte1.a, > + sum(cte1.a) over (partition by cte1.a order by cte1.a) as > k, > + avg(cte1.a) over (partition by cte1.a order by cte1.a) as m > + from cte1 ) > +select * from cte; > + > +eval create procedure sp7() $q7; > +call sp7(); > +call sp7(); > + > +eval prepare stmt from "$q7"; > +execute stmt; > +execute stmt; > +deallocate prepare stmt; > + > + > +drop procedure sp1; > +drop procedure sp2; > +drop procedure sp3; > +drop procedure sp4; > +drop procedure sp5; > +drop procedure sp6; > +drop procedure sp7; > +drop view v1,v2; > +drop table t1; > + > +--echo # > --echo # End of 10.2 tests > --echo # > diff --git a/sql/sql_union.cc b/sql/sql_union.cc > index 7baedfb..f3c90b8 100644 > --- a/sql/sql_union.cc > +++ b/sql/sql_union.cc > @@ -30,6 +30,7 @@ > #include "filesort.h" // filesort_free_buffers > #include "sql_view.h" > #include "sql_cte.h" > +#include "item_windowfunc.h" > > bool mysql_union(THD *thd, LEX *lex, select_result *result, > SELECT_LEX_UNIT *unit, ulong setup_tables_done_option) > @@ -1550,6 +1551,29 @@ static void cleanup_order(ORDER *order) > } > > > +static void cleanup_window_funcs(List<Item_window_func> &win_funcs) > +{ > + List_iterator_fast<Item_window_func> it(win_funcs); > + Item_window_func *win_func; > + while ((win_func= it++)) > + { > + Window_spec *win_spec= win_func->window_spec; > + if (!win_spec) > + continue; > + if (win_spec->save_partition_list) > + { > + win_spec->partition_list= win_spec->save_partition_list; > + win_spec->save_partition_list= NULL; > + } > + if (win_spec->save_order_list) > + { > + win_spec->order_list= win_spec->save_order_list; > + win_spec->save_order_list= NULL; > + } > + } > +} > + > + > bool st_select_lex::cleanup() > { > bool error= FALSE; > @@ -1558,6 +1582,8 @@ bool st_select_lex::cleanup() > cleanup_order(order_list.first); > cleanup_order(group_list.first); > > + cleanup_window_funcs(window_funcs); > + > if (join) > { > List_iterator<TABLE_LIST> ti(leaf_tables); > diff --git a/sql/sql_window.cc b/sql/sql_window.cc > index 612c6e6..3ef751b 100644 > --- a/sql/sql_window.cc > +++ b/sql/sql_window.cc > @@ -479,9 +479,15 @@ int > compare_window_funcs_by_window_specs(Item_window_func *win_func1, > Let's use only one of the lists. > */ > if (!win_spec1->name() && win_spec2->name()) > + { > + win_spec1->save_partition_list= win_spec1->partition_list; > win_spec1->partition_list= win_spec2->partition_list; > + } > else > + { > + win_spec2->save_partition_list= win_spec2->partition_list; > win_spec2->partition_list= win_spec1->partition_list; > + } > > cmp= compare_order_lists(win_spec1->order_list, > win_spec2->order_list); > @@ -494,9 +500,15 @@ int > compare_window_funcs_by_window_specs(Item_window_func *win_func1, > Let's use only one of the lists. > */ > if (!win_spec1->name() && win_spec2->name()) > + { > + win_spec1->save_order_list= win_spec2->order_list; > win_spec1->order_list= win_spec2->order_list; > + } > else > + { > + win_spec1->save_order_list= win_spec2->order_list; > win_spec2->order_list= win_spec1->order_list; > + } > > cmp= compare_window_frames(win_spec1->window_frame, > win_spec2->window_frame); > diff --git a/sql/sql_window.h b/sql/sql_window.h > index e0c1563..417d0bc 100644 > --- a/sql/sql_window.h > +++ b/sql/sql_window.h > @@ -99,8 +99,10 @@ class Window_spec : public Sql_alloc > LEX_STRING *window_ref; > > SQL_I_List<ORDER> *partition_list; > + SQL_I_List<ORDER> *save_partition_list; > > SQL_I_List<ORDER> *order_list; > + SQL_I_List<ORDER> *save_order_list; > > Window_frame *window_frame; > > @@ -111,7 +113,8 @@ class Window_spec : public Sql_alloc > SQL_I_List<ORDER> *ord_list, > Window_frame *win_frame) > : window_names_are_checked(false), window_ref(win_ref), > - partition_list(part_list), order_list(ord_list), > + partition_list(part_list), save_partition_list(NULL), > + order_list(ord_list), save_order_list(NULL), > window_frame(win_frame), referenced_win_spec(NULL) {} > > virtual char *name() { return NULL; } > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits

1 0

Re: [Maria-developers] 19aebbd1b89: MDEV-16481: set global system_versioning_asof=sf() crashes in specific case
by Sergei Golubchik 19 Jul '21

19 Jul '21

Hi, Nikita! On Jul 12, Nikita Malyavin wrote: > revision-id: 19aebbd1b89 (mariadb-10.3.26-129-g19aebbd1b89) > parent(s): 10c163e4a18 > author: Nikita Malyavin <nikita.malyavin(a)mariadb.com> > committer: Nikita Malyavin <nikita.malyavin(a)mariadb.com> > timestamp: 2021-04-06 21:32:43 +0300 > message: > > MDEV-16481: set global system_versioning_asof=sf() crashes in specific case > > * sys_vars.h: add `MYSQL_TIME` field to `set_var::save_result` > * sys_vars.ic: get rid of calling `var->value->get_date()` from `Sys_var_vers_asof::update()` > * versioning.sysvars: add test; remove double warning > > diff --git a/sql/set_var.h b/sql/set_var.h > index 12e025e4696..6351fb5b666 100644 > --- a/sql/set_var.h > +++ b/sql/set_var.h > @@ -296,6 +296,7 @@ class set_var :public set_var_base > plugin_ref *plugins; ///< for Sys_var_pluginlist > Time_zone *time_zone; ///< for Sys_var_tz > LEX_STRING string_value; ///< for Sys_var_charptr and others > + MYSQL_TIME time; ///< for Sys_var_vers_asof old sizeof(save_result) is 16 bytes, sizeof(MYSQL_TIME) is 40. It's just an observation, set_var isn't a structure that's constantly in memory in large quantities. So, not a problem at all. > const void *ptr; ///< for Sys_var_struct > } save_result; > LEX_CSTRING base; /**< for structured variables, like keycache_name.variable_name */ > diff --git a/sql/sys_vars.ic b/sql/sys_vars.ic > index 8d094d3a1d1..eff6a59e65b 100644 > --- a/sql/sys_vars.ic > +++ b/sql/sys_vars.ic > @@ -2650,50 +2650,47 @@ class Sys_var_vers_asof: public Sys_var_enum > virtual bool do_check(THD *thd, set_var *var) > { > if (!Sys_var_enum::do_check(thd, var)) > + { > + var->save_result.time.time_type= MYSQL_TIMESTAMP_NONE; > return false; > - MYSQL_TIME ltime; > - bool res= var->value->get_date(&ltime, TIME_NO_ZERO_IN_DATE|TIME_NO_ZERO_DATE); > - if (!res) > + } > + bool res= var->value->get_date(&var->save_result.time, > + TIME_NO_ZERO_IN_DATE | TIME_NO_ZERO_DATE); > + if (res) > { > - var->save_result.ulonglong_value= SYSTEM_TIME_AS_OF; > + var->save_result.time.time_type= MYSQL_TIMESTAMP_ERROR; > } > return res; > } > > private: > - bool update(set_var *var, vers_asof_timestamp_t &out, system_variables& pool) > + bool update(THD *thd, set_var *var, vers_asof_timestamp_t *out) > { > - bool res= false; > - uint error; > - MYSQL_TIME ltime; > - out.type= static_cast<enum_var_type>(var->save_result.ulonglong_value); > - if (out.type == SYSTEM_TIME_AS_OF) > + MYSQL_TIME &ltime= var->save_result.time; > + uint error= 0; > + > + if (var->value && ltime.time_type >= 0) // any valid value is ok > { > - if (var->value) > - { > - res= var->value->get_date(&ltime, TIME_NO_ZERO_IN_DATE > - | TIME_NO_ZERO_DATE); > - out.unix_time= pool.time_zone->TIME_to_gmt_sec(&ltime, &error); > - out.second_part= ltime.second_part; > - res|= (error != 0); > - } > - else // set DEFAULT from global var > - { > - out.type= SYSTEM_TIME_UNSPECIFIED; > - res= false; > - } > + out->type = SYSTEM_TIME_AS_OF; > + out->unix_time = thd->variables.time_zone->TIME_to_gmt_sec(&ltime, > + &error); why are you doing it here? You could've converted in ::check(), and as a bonus you wouldn't need MYSQL_TIME in the save_result. And what would you do if TIME_to_gmt_sec would fail? ::update() isn't expected to fail, ::check() should guarantee that as much as possible. > + out->second_part= ltime.second_part; > } > - return res; > + else // DEFAULT is set > + { > + out->type= SYSTEM_TIME_UNSPECIFIED; > + } > + return (error != 0); > } > > public: > virtual bool global_update(THD *thd, set_var *var) > { > - return update(var, global_var(vers_asof_timestamp_t), thd->variables); > + return update(thd, var, &global_var(vers_asof_timestamp_t)); > } > virtual bool session_update(THD *thd, set_var *var) > { > - return update(var, session_var(thd, vers_asof_timestamp_t), thd->variables); > + return update(thd, var, &session_var(thd, vers_asof_timestamp_t)); > } > > private: > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 3

Re: [Maria-developers] a11756e4fba: MDEV-23178: Qualified asterisk not supported in INSERT .. RETURNING
by Sergei Golubchik 16 Jul '21

16 Jul '21

Hi, Rucha! On Jul 15, Rucha Deodhar wrote: > revision-id: a11756e4fba (mariadb-10.5.11-27-ga11756e4fba) > parent(s): f0f47cbca18 > author: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > committer: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > timestamp: 2021-07-05 20:14:19 +0530 > message: > > MDEV-23178: Qualified asterisk not supported in INSERT .. RETURNING > > Analysis: When we have INSERT/REPLACE returning with qualified asterisk in the > RETURNING clause, '*' is not resolved properly because of wrong context. > context->table_list is NULL or has incorrect table because context->table_list > has tables from the FROM clause. > Fix: If filling fields instead of '*' for qualified asterisk in RETURNING, > use first_name_resolution_table for correct resolution of item. please, add a couple of tests with a qualified asterisk, where it's qualified _not_ by the table you're inserting into. Like INSERT INTO t1(id1,val1) VALUES(14,'m') RETURNING t2.* not very interesting case, an obvious error ^^^ INSERT INTO t2 SELECT t1.* FROM t1 WHERE id1=2 RETURNING t1.*; but what will happen here? ^^^ > diff --git a/sql/sql_base.cc b/sql/sql_base.cc > index 1476d708d75..60d14c6dbea 100644 > --- a/sql/sql_base.cc > +++ b/sql/sql_base.cc > @@ -8032,12 +8032,13 @@ insert_fields(THD *thd, Name_resolution_context *context, const char *db_name, > else treat natural joins as leaves and do not iterate over their underlying > tables. > */ > - for (TABLE_LIST *tables= (table_name ? context->table_list : > - context->first_name_resolution_table); > + TABLE_LIST *tables= returning_field || !table_name ? context->first_name_resolution_table : > + context->table_list; > + for (; > tables; > - tables= (table_name ? tables->next_local : > - tables->next_name_resolution_table) > + tables= returning_field || !table_name ? tables->next_name_resolution_table : > + tables->next_local > ) > { > Field *field; > TABLE *table= tables->table; 1. please also update the comment to mention RETURNING. 2. Could you apply the following patch as a separate commit before your fix: ============================ --- a/sql/sql_base.cc +++ b/sql/sql_base.cc @@ -8098,12 +8098,14 @@ insert_fields(THD *thd, Name_resolution_context *context, const char *db_name, else treat natural joins as leaves and do not iterate over their underlying tables. */ - for (TABLE_LIST *tables= (table_name ? context->table_list : - context->first_name_resolution_table); - tables; - tables= (table_name ? tables->next_local : - tables->next_name_resolution_table) - ) + TABLE_LIST *first= context->first_name_resolution_table; + TABLE_LIST *TABLE_LIST::* next= &TABLE_LIST::next_name_resolution_table; + if (table_name) + { + first= context->table_list; + next= &TABLE_LIST::next_local; + } + for (TABLE_LIST *tables= first; tables; tables= tables->*next) { Field *field; TABLE *table= tables->table; ============================ it'll remove the condition from inside the loop, will put it in one place only. And will make your change simpler and clearer. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] 70252733d41: MDEV-25908: -e does not work for my_print_defaults
by Sergei Golubchik 15 Jul '21

15 Jul '21

Hi, Rucha! On Jul 15, Rucha Deodhar wrote: > revision-id: 70252733d41 (mariadb-10.5.11-35-g70252733d41) > parent(s): f88d130e710 > author: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > committer: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > timestamp: 2021-07-14 22:55:21 +0530 > message: > > MDEV-25908: -e does not work for my_print_defaults > > Analysis: get_defaults_options() rewrites the value for my_defaults_file, > my_defaults_extra_file and my_defaults_group_suffix to 0. So the config file > can't be read. > Fix: Let handle_options() handle --defaults* option and create a new command > line to pass it to load_defaults(). > > --- > extra/my_print_defaults.c | 37 +++++++++++++++++----- > mysql-test/main/my_print_defaults.result | 35 ++++++++++++++++++++- > mysql-test/main/my_print_defaults.test | 54 +++++++++++++++++++++++++++++++- > mysql-test/main/mysqldump.result | 1 + > 4 files changed, 117 insertions(+), 10 deletions(-) > > diff --git a/extra/my_print_defaults.c b/extra/my_print_defaults.c > index 1dec667949f..409ef273b44 100644 > --- a/extra/my_print_defaults.c > +++ b/extra/my_print_defaults.c > @@ -141,28 +141,49 @@ static int get_options(int *argc,char ***argv) > return 0; > } > > +static char *make_args(const char *s1, const char *s2) > +{ > + char *s= malloc(strlen(s1) + strlen(s2) + 1); > + strmov(strmov(s, s1), s2); > + return s; > +} > > int main(int argc, char **argv) > { > - int count, error, args_used; > - char **load_default_groups= 0, *tmp_arguments[6]; > + int count= 0, error, no_defaults= 0; > + char **load_default_groups= 0; > char **argument, **arguments, **org_argv; > int nargs, i= 0; > MY_INIT(argv[0]); > > org_argv= argv; > - args_used= get_defaults_options(argv); > - > - /* Copy defaults-xxx arguments & program name */ > - count=args_used; > - arguments= tmp_arguments; > - memcpy((char*) arguments, (char*) org_argv, count * sizeof(*org_argv)); > + if (*argv && !strcmp(*argv, "--no-defaults")) > + { > + argv++; > + ++count; > + no_defaults= 1; > + } > + /* Copy program name and --no-defaults if present*/ > + memcpy((char*) arguments, (char*) org_argv, (++count)*sizeof(*org_argv)); isn't this wrong? `arguments` is uninitialized at this point. > arguments[count]= 0; > > /* Check out the args */ > if (get_options(&argc,&argv)) > cleanup_and_exit(1); > > +--echo # Testing -e > +--exec $MYSQL_MY_PRINT_DEFAULTS -e $MYSQLTEST_VARDIR/tmp/tmp1.cnf --mysqld mysql.server > +--echo #Testing --defaults-extra-file > +--exec $MYSQL_MY_PRINT_DEFAULTS --defaults-extra-file=$MYSQLTEST_VARDIR/tmp/tmp1.cnf --mysqld mysql.server > + > +--echo # Testing -c option > +--exec $MYSQL_MY_PRINT_DEFAULTS -c $MYSQLTEST_VARDIR/tmp/tmp2.cnf --mysqld mysql.server > +--echo # Testing --defaults-file > +--exec $MYSQL_MY_PRINT_DEFAULTS --defaults-file=$MYSQLTEST_VARDIR/tmp/tmp2.cnf --mysqld mysql.server > + > +--echo # Testing -g option > +--exec $MYSQL_MY_PRINT_DEFAULTS -c $MYSQLTEST_VARDIR/tmp/tmp3.cnf --mysqld mysql.server -g .1 > +--echo # Testing --defaults-group-suffix > +--exec $MYSQL_MY_PRINT_DEFAULTS -c $MYSQLTEST_VARDIR/tmp/tmp3.cnf --mysqld mysql.server --defaults-group-suffix=.1 you don't seem have a --no-defaults test Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] da70f190279: MDEV-25985 Spider handle ">=" as ">" in some cases
by Sergei Golubchik 13 Jul '21

13 Jul '21

Hi, Nayuta! The code change is good, no comments about it at all. But why did you create a new suite spider/bugfix and all those files in it? I know that some later spider version has it. Still, I'd expect it'd be much simpler and a smaller change to drop the test file into the exising spider suite. On Jul 12, Nayuta Yanagisawa wrote: > revision-id: da70f190279 (mariadb-10.2.39-44-gda70f190279) > parent(s): 99f700a820e > author: Nayuta Yanagisawa <nayuta.yanagisawa(a)hey.com> > committer: Nayuta Yanagisawa <nayuta.yanagisawa(a)hey.com> > timestamp: 2021-07-08 14:12:45 +0000 > message: > > MDEV-25985 Spider handle ">=" as ">" in some cases > > The function spider_db_append_key_where_internal() converts > HA_READ_AFTER_KEY to '>'. The conversion seems to be correct for > single-column indexes because HA_READ_AFTER_KEY means "read the > key after the provided value." > > However, how about multi-column indexes? Assume that there is a > multi-column index on c1 and c2 and we search with the condition > 'c1 >= 100 AND c2 > 200'. The key_range.flag corresponds to the > search condition could be HA_READ_AFTER_KEY. In such a case, > we could not simply convert HA_READ_AFTER_KEY to '>'. > > The correct conversion is to convert HA_READ_AFTER_KEY to '>' > only for the last column in key_part_map and to convert > HA_READ_AFTER_KEY to '>=' for the other column. > > The similar discussion also applies to the conversion from > key_range.flag to a sign of inequality. > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] a6e96ce3c96: MDEV-21916: COM_STMT_BULK_EXECUTE with RETURNING insert wrong values
by Sergei Golubchik 12 Jul '21

12 Jul '21

Hi, Oleksandr! ok to push! On Jul 12, Oleksandr Byelkin wrote: > revision-id: a6e96ce3c96 (mariadb-10.5.4-666-ga6e96ce3c96) > parent(s): f4425d3a3db > author: Oleksandr Byelkin <sanja(a)mariadb.com> > committer: Oleksandr Byelkin <sanja(a)mariadb.com> > timestamp: 2021-06-09 10:43:24 +0200 > message: > > MDEV-21916: COM_STMT_BULK_EXECUTE with RETURNING insert wrong values > > The problem is that array binding uses net buffer to read parameters for each > execution while each execiting with RETURNING write in the same buffer. > > Solution is to allocate new net buffer to avoid changing buffer we are reading > from. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] b1881e07797: MDEV-20131 Assertion `!pk->has_virtual ()' failed in instant_alter_column_possible
by Sergei Golubchik 12 Jul '21

12 Jul '21

Hi, Sachin! On Jul 12, Sachin Kumar wrote: > revision-id: b1881e07797 (mariadb-10.4.20-9-gb1881e07797) > parent(s): 8a2b4d531dc > author: Sachin Kumar <sachin.setiya(a)mariadb.com> > committer: Sachin Kumar <sachin.setiya(a)mariadb.com> > timestamp: 2021-06-11 14:26:19 +0100 > message: > > MDEV-20131 Assertion `!pk->has_virtual ()' failed in instant_alter_column_possible > > Problem:- Primary key on long unique columns is not allowed but when > `CREATE TABLE t2 (a TEXT, PRIMARY KEY(a(1871))) ENGINE=InnoDB;` > is executed with innodb_page_size=8k or 4k, primary key on DB_ROW_HASH_1 column > is created. Reason of this is in mysql_prepare_create_table we check against > max_key_part_length() to see whether key_part length is more then engine supported > key_part length , and if it is true error is thrown for primary key and long > unique index is created for unique key. But in this case max_key_part_length() > returns 3072 which is more the max_key_length() , which later in code triggers > creating of long unique index. > > Solution:- max_supported_key_part_length() should return according to innodb > page size. > > diff --git a/storage/innobase/handler/ha_innodb.cc b/storage/innobase/handler/ha_innodb.cc > index ce2e4b6990f..e5e864bffc5 100644 > --- a/storage/innobase/handler/ha_innodb.cc > +++ b/storage/innobase/handler/ha_innodb.cc > @@ -6537,14 +6537,21 @@ ha_innobase::clone( > DBUG_RETURN(new_handler); > } > > - > -uint > -ha_innobase::max_supported_key_part_length() const > -/*==============================================*/ > -{ > - /* A table format specific index column length check will be performed > - at ha_innobase::add_index() and row_create_index_for_mysql() */ > - return(REC_VERSION_56_MAX_INDEX_COL_LEN); > +uint ha_innobase::max_supported_key_part_length() const > +{ > + /* A table format specific index column length check will be performed > + at ha_innobase::add_index() and row_create_index_for_mysql() */ > + /* FIXME: rewrite this as well as ha_innobase::max_supported_key_length() > + using an API that considers the PRIMARY KEY as well as secondary index > + metadata and the ROW_FORMAT and KEY_BLOCK_SIZE */ > + switch (srv_page_size) { > + case 4096: > + return 1173; > + case 8192: > + return 1536; > + default: > + return REC_VERSION_56_MAX_INDEX_COL_LEN; > + } dunno, why would you duplicate the whole switch logic? can you just - return(REC_VERSION_56_MAX_INDEX_COL_LEN); - return MY_MIN(max_supported_key_length(), REC_VERSION_56_MAX_INDEX_COL_LEN); > } > > /******************************************************************//** > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 85186734823: MDEV-24248: my_print_defaults is not taking all the values when using -e
by Sergei Golubchik 12 Jul '21

12 Jul '21

Hi, Rucha! On Jul 12, Rucha Deodhar wrote: > revision-id: 85186734823 (mariadb-10.5.4-665-g85186734823) > parent(s): bf5c050fd28 > author: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > committer: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > timestamp: 2021-06-14 10:27:13 +0530 > message: > > MDEV-24248: my_print_defaults is not taking all the values when using -e > option which is called from mysql.server (extra_args). > > Fix: change mysql.server script to use --defaults-extra-file instead of -e > > diff --git a/mysql-test/main/my_print_defaults.test b/mysql-test/main/my_print_defaults.test > --- /dev/null > +++ b/mysql-test/main/my_print_defaults.test > @@ -0,0 +1,29 @@ > +--echo # > +--echo # Beginning of 10.5 Test > +--echo # > +--echo # > +--echo # MDEV-24248: my_print_defaults is not taking all the values when using > +--echo # -e option which is called from mysql.server (extra_args). > +--echo # > + > +--echo # checking for mysql.server > + > +--write_file $MYSQLTEST_VARDIR/tmp/tmp1.cnf > + > +[mariadb-10.5] don't use the version number in the test - you're forcing some poor soul to update the test on merges for every new version. > +key_buffer_size=20M > +max_allowed_packet=250M > +table_open_cache=1000 > +table_definition_cache=2000 > +read_buffer_size=1M > +thread_cache_size=8 > +max_connections=1024 > +long_query_time=60 > +slow_query_log=1 > +EOF > + > +--exec $MYSQL_MY_PRINT_DEFAULTS --defaults-extra-file=$MYSQLTEST_VARDIR/tmp/tmp1.cnf --mysqld mysql.server > + > +--remove_file $MYSQLTEST_VARDIR/tmp/tmp1.cnf > + > +--echo # End of 10.5 test > diff --git a/support-files/mysql.server.sh b/support-files/mysql.server.sh > index cefb3225e9e..8504e4def4b 100644 > --- a/support-files/mysql.server.sh > +++ b/support-files/mysql.server.sh > @@ -200,11 +200,11 @@ su_kill() { > extra_args="" > if test -r "$basedir/my.cnf" > then > - extra_args="-e $basedir/my.cnf" > + extra_args="--defaults-file-name=$basedir/my.cnf" eh. -e is --defaults-extra-file same below. > else > if test -r "$datadir/my.cnf" > then > - extra_args="-e $datadir/my.cnf" > + extra_args="--defaults-file-name=$datadir/my.cnf" > fi > fi > > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] b29ec61724d: MDEV-25906: SIGSEGV in flush_tables_with_read_lock on FTWRL or FTFE | SIGSEGV in ha_maria::extra
by Sergei Golubchik 01 Jul '21

01 Jul '21

Hi, Oleksandr! On Jul 01, Oleksandr Byelkin wrote: > revision-id: b29ec61724d (mariadb-10.6.2-41-gb29ec61724d) > parent(s): 83684fc9a4d > author: Oleksandr Byelkin <sanja(a)mariadb.com> > committer: Oleksandr Byelkin <sanja(a)mariadb.com> > timestamp: 2021-07-01 11:20:29 +0200 > message: > > MDEV-25906: SIGSEGV in flush_tables_with_read_lock on FTWRL or FTFE | SIGSEGV in ha_maria::extra > > Fixed check of derived tables of all kinds (view, derived, information schema). > > diff --git a/sql/sql_reload.cc b/sql/sql_reload.cc > index 0fa2fa10df8..e41da9680b0 100644 > --- a/sql/sql_reload.cc > +++ b/sql/sql_reload.cc > @@ -592,10 +592,19 @@ bool flush_tables_with_read_lock(THD *thd, TABLE_LIST *all_tables) > for (TABLE_LIST *table_list= all_tables; table_list; > table_list= table_list->next_global) > { > - if (!(table_list->is_view() || > - table_list->table->file->ha_table_flags() & HA_CAN_EXPORT)) > + /* > + We allow all deriveds and view here, but prohibit information_schema > + tables (explicitly) and performance_schema tables (implicetly: > + engine do not suport export) > + */ > + if (table_list->is_non_derived() && > + (!table_list->is_base_table() || > + !(table_list->table->file->ha_table_flags() & HA_CAN_EXPORT))) > { > - my_error(ER_ILLEGAL_HA, MYF(0),table_list->table->file->table_type(), > + my_error(ER_ILLEGAL_HA, MYF(0), > + (table_list->is_base_table() ? > + table_list->table->file->table_type(): > + "information_schema"), I'd say if you want to detect I_S tables, you should test for table_list->schema_table. It's not clear what your condition actually does. And there is no such engine "information_schema", so the message is misleading. In fact, this should be access denied. For both. Like: mysqltest: At line 12: query 'flush table information_schema.collations for export' failed: ER_DBACCESS_DENIED_ERROR (1044): Access denied for user 'foo'@'localhost' to database 'information_schema' this is from a test file that only does flush table information_schema.collations for export. > table_list->db.str, table_list->table_name.str); > goto error_reset_bits; > } > @@ -607,7 +616,8 @@ bool flush_tables_with_read_lock(THD *thd, TABLE_LIST *all_tables) > for (auto table_list= all_tables; table_list; > table_list= table_list->next_global) > { > - if (!table_list->is_view() && > + if (table_list->table && why do you check for table_list->table ? Can it happen that table_list->table == NULL and table_list->is_base_table() == true? > + table_list->is_base_table() && > table_list->table->file->extra(HA_EXTRA_FLUSH)) > goto error_reset_bits; > } Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

[Maria-developers] MDEV-25325 built-in documentation for performance_schema tables
by Haidong Ji 24 Jun '21

24 Jun '21

Hi, I've grabbed this JIRA story, https://jira.mariadb.org/browse/MDEV-25325 . The coding and test cases creation were all done. I've also submitted a PR here: https://github.com/MariaDB/server/pull/1830 The PR was submitted more than 1 month ago. I'm curious about next steps. Thanks, Haidong

1 0

[Maria-developers] JSON_TABLE: on default values
by Sergey Petrunia 16 Jun '21

16 Jun '21

Hi Alexey, At the moment MariaDB requires that the values in DEFAULT clauses are quoted. Example: select * from json_table( '{"intval": 1000}', '$' columns( col1 int path '$.intval_' default '100' on empty ) ) as T; here, "100" must be quoted, otherwise one gets a parse error. However, the quoted value is interpreted as an SQL literal. This looks puzzling. MySQL-8 also requires that the default value is quoted, but they have a (very odd) reason for it: they interpret the default value as JSON: https://docs.oracle.com/cd/E17952_01/mysql-8.0-en/json-table-functions.html says: DEFAULT json_string ON EMPTY: the provided json_string is parsed as JSON, as long as it is valid, and stored instead of the missing value. Column type rules also apply to the default value. I am not sure why MySQL chose to do this. Looking into the SQL Standard, one can see: <JSON table regular column definition> ::= <column name> <data type> [ PATH <JSON table column path specification> ] [ <JSON table column empty behavior> ON EMPTY ] [ <JSON table column error behavior> ON ERROR ] <JSON table column empty behavior> ::= ERROR | NULL | DEFAULT <value expression> ... This doesn't say whether the <value expression> should be interepreted as JSON or just as a value. But one can find this passage: <quote> Without Feature T826, “General value expression in ON ERROR or ON EMPTY clauses”, the <value expression> contained in <JSON table column empty behavior> or <JSON table column error behavior> contained in a <JSON table regular column definition> JTRCD shall be a <literal> that can be cast to the data type specified by the <data type> contained in JTRCD without raising an exception condition according to the General Rules of Subclause 6.13, “<cast specification>”. </quote> The important part is: ... shall be a <literal> that can be cast to the data type specified ... which means it is not JSON. It is just a literal, and literal can be a string literal (in quotes, 'string') or an integer literal (without quotes) or other kind of literal. Btw, Oracle Database allows non-string literals in the default clause: https://dbfiddle.uk/?rdbms=oracle_18&fiddle=9af7e43ede77ee285e1a65f1f419d3bd What are your thoughts on this? Is MariaDB's behavior intentional? Should we follow the standard and allow all kinds of literals? What was the reason for the limitation that default values are quoted? BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://petrunia.net

2 4

[Maria-developers] MDEV-25875: RE 02469b: MDEV-17399 JSON_TABLE Accept JSON values for the JSON fields
by Sergey Petrunia 16 Jun '21

16 Jun '21

Hi Alexey, This is review input re https://github.com/MariaDB/server/commit/02469bdead5753eccb5d70c98a158a0702…. First, our workflow dictates that this patch should have its own MDEV entry. I've filed it https://jira.mariadb.org/browse/MDEV-25875. Please use it in commit descriptions. == Updates mixed with another patch == The patch has .result changes like: > @@ -551,10 +551,12 @@ JSON_TABLE('{}', '$' COLUMNS (x INT PATH '$.x' DEFAULT NULL ON ERROR)) jt; > ERROR 42000: You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'NULL ON ERROR)) jt' at line 2 > SELECT * FROM > JSON_TABLE('{}', '$' COLUMNS (x INT PATH '$.x' DEFAULT 0 ON EMPTY)) jt; > -ERROR 42000: You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near '0 ON EMPTY)) jt' at line 2 > +x > +0 > SELECT * FROM > JSON_TABLE('{}', '$' COLUMNS (x INT PATH '$.x' DEFAULT 0 ON ERROR)) jt; > -ERROR 42000: You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near '0 ON ERROR)) jt' at line 2 > +x > +NULL > SELECT * FROM > JSON_TABLE('{}', '$' COLUMNS (x DATE > PATH '$.x' These are obviously from another patch (the one about "accepting non-string literals as default"). Please move them to that patch. == Comments == > @@ -377,6 +378,25 @@ static void store_json_in_field(Field *f, const json_engine_t *je) > } > > > +static int store_json_in_json(Field *f, json_engine_t *je) > +{ Please add a comment for the function. Something like "Store the current JSON element pointed by je into field f, as a separate JSON document" The name "store_json_in_json" doesn't look good to me. Do you think store_element_as_json_doc() would be better? > + const uchar *from= je->value_begin; > + const uchar *to; > + > + if (json_value_scalar(je)) > + to= je->value_end; > + else > + { > + int error; > + if ((error= json_skip_level(je))) > + return error; > + to= je->s.c_str; > + } > + f->store((const char *) from, (uint32) (to - from), je->s.cs); > + return 0; > +} > + > + > bool Json_table_nested_path::check_error(const char *str) > { > if (m_engine.s.error) > @@ -541,7 +561,12 @@ int ha_json_table::fill_column_values(THD *thd, uchar * buf, uchar *pos) > } > else > { > - if (!(error= !json_value_scalar(&je))) > + if (jc->m_format_json) > + { > + if (!(error= store_json_in_json(*f, &je))) > + error= er_handler.errors; Can there be a case when this assignment assigns any value other than 0? (My logic - er_handler.errors already had some error status, why did we call store_json_in_json? If there was an error inside store_json_in_json, we do not get to this assignment, as store_json_in_json() returned non-zero error. Is this correct?) > + } > + else if (!(error= !json_value_scalar(&je))) > { > store_json_in_field(*f, &je); > error= er_handler.errors; > @@ -868,6 +893,10 @@ int Json_table_column::set(THD *thd, enum_type ctype, const LEX_CSTRING &path) > anctual content. Not sure though if we should. > */ > m_path.s.c_str= (const uchar *) path.str; > + > + if (ctype == PATH) > + m_format_json= m_field->type_handler() == &type_handler_json_longtext; This will probably cause warnings on some compiler. Please wrap into "MY_TEST(...)" or just brackets, "z= (x == y)". == A discrepancy between this and MySQL == I've found this case: select * from json_table('{"foo": null}', '$' columns( jscol json path '$.foo') ) as T; with the above patch, it produces a JSON null value: +-------+ | jscol | +-------+ | null | +-------+ while in MySQL-8 it produces an SQL NULL: +-------+ | jscol | +-------+ | NULL | +-------+ I'm not yet sure what we should do about this. Any thoughts? BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://petrunia.net

2 1

[Maria-developers] New Member (GSoC MDEV-25015)
by Alan Cueva 15 Jun '21

15 Jun '21

Hello everyone, I am Alan Cueva, during the summer I will be working on the Custom formatting of strings in MariaDB queries (MDEV-25015 <https://jira.mariadb.org/browse/MDEV-25015>) project, but I would like to continue contributing in this community. I hope to be useful and learn from everyone. Thanks!

1 0

Re: [Maria-developers] 27060eb6ba5: MDEV-21916: COM_STMT_BULK_EXECUTE with RETURNING insert wrong values
by Sergei Golubchik 10 Jun '21

10 Jun '21

Hi, Oleksandr! On Oct 12, Oleksandr Byelkin wrote: > revision-id: 27060eb6ba5 (mariadb-10.5.4-220-g27060eb6ba5) > parent(s): 861cd4ce286 > author: Oleksandr Byelkin <sanja(a)mariadb.com> > committer: Oleksandr Byelkin <sanja(a)mariadb.com> > timestamp: 2020-10-07 15:22:41 +0200 > message: > > MDEV-21916: COM_STMT_BULK_EXECUTE with RETURNING insert wrong values > > To allocate new net buffer to avoid changing bufer we are reading. You still didn't clarify the commit comment > diff --git a/sql/sql_delete.cc b/sql/sql_delete.cc > index 7280236e43f..5aaff3cf623 100644 > --- a/sql/sql_delete.cc > +++ b/sql/sql_delete.cc > @@ -685,8 +685,14 @@ bool mysql_delete(THD *thd, TABLE_LIST *table_list, COND *conds, > !table->prepare_triggers_for_delete_stmt_or_event()) > will_batch= !table->file->start_bulk_delete(); > > - if (returning) > + /* > + thd->get_stmt_da()->is_set() means first iteration of prepared statement > + with array binding operation execution (non optimized so it is not > + INSERT) > + */ > + if (returning && !thd->get_stmt_da()->is_set()) > { > + DBUG_ASSERT(thd->lex->sql_command != SQLCOM_INSERT); a strange assert to see in sql_delete.cc :) can one even reach mysql_delete() with SQLCOM_INSERT? not just with returning and !thd->get_stmt_da()->is_set(), anywhere? > if (result->send_result_set_metadata(returning->item_list, > Protocol::SEND_NUM_ROWS | Protocol::SEND_EOF)) > goto cleanup; > diff --git a/sql/sql_error.cc b/sql/sql_error.cc > index b3ef0d89a98..a753af2b34d 100644 > --- a/sql/sql_error.cc > +++ b/sql/sql_error.cc > @@ -380,16 +380,33 @@ Diagnostics_area::set_eof_status(THD *thd) > if (unlikely(is_error() || is_disabled())) > return; > > + if (m_status == DA_EOF_BULK) > + { > /* > If inside a stored procedure, do not return the total > number of warnings, since they are not available to the client > anyway. > */ > + if (!thd->spcont) > + m_statement_warn_count+= current_statement_warn_count(); > + } > + else > + { > + /* > + If inside a stored procedure, do not return the total > + number of warnings, since they are not available to the client > + anyway. > + */ I don't think it helps to duplicate the comment. You could just put it once before the if. > if (thd->spcont) > { > m_statement_warn_count= 0; > + m_affected_rows= 0; why do you reset m_affected_rows too? > } > else > m_statement_warn_count= current_statement_warn_count(); > + m_status= (is_bulk_op() ? DA_EOF_BULK : DA_EOF); > + } do we have tests for bulk operations and CALL? > > - m_status= DA_EOF; > DBUG_VOID_RETURN; > } > > diff --git a/sql/sql_prepare.cc b/sql/sql_prepare.cc > index ecb56e70f88..c144d3a8d7e 100644 > --- a/sql/sql_prepare.cc > +++ b/sql/sql_prepare.cc > @@ -4357,24 +4361,37 @@ Prepared_statement::execute_bulk_loop(String *expanded_query, > > + /* > + Here second buffer for not optimized commands, > + optimized commands do it inside thier internal loop. > + */ > + if (!(sql_command_flags[lex->sql_command] & CF_SP_BULK_OPTIMIZED) && why "SP" and not "PS" ? > + this->lex->has_returning()) what about CALL? It won't have lex->has_returning(). What about SELECT? Can you bind an array to a parameter in SELECT or CALL? > + { > + // Above check can be true for SELECT in future > + DBUG_ASSERT(lex->sql_command != SQLCOM_SELECT); how can SQLCOM_SELECT have lex->has_returning() ? > + readbuff= thd->net.buff; // old buffer > + if (net_allocate_new_packet(&thd->net, thd, MYF(MY_THREAD_SPECIFIC))) > + { > + readbuff= NULL; // failure, net_allocate_new_packet keeps old buffer > + goto err; > + } > } > > #ifndef EMBEDDED_LIBRARY Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 2

Re: [Maria-developers] 557ab2992dd: MDEV-21117 Incremental patch for a few review notes addressed.
by Sergei Golubchik 09 Jun '21

09 Jun '21

Hi, Andrei! Again, this is a review of the git diff f09d33f521b 557ab2992dd, not only 557ab2992dd. On Jun 02, Andrei Elkin wrote: > revision-id: 557ab2992dd (mariadb-10.6.0-67-g557ab2992dd) > parent(s): 4cc5a3074da > author: Andrei Elkin <andrei.elkin(a)mariadb.com> > committer: Andrei Elkin <andrei.elkin(a)mariadb.com> > timestamp: 2021-05-27 17:37:22 +0300 > message: > > MDEV-21117 Incremental patch for a few review notes addressed. > diff --git a/mysql-test/suite/binlog/t/binlog_truncate_active_log.inc b/mysql-test/suite/binlog/t/binlog_truncate_active_log.inc > new file mode 100644 > --- /dev/null > +++ b/mysql-test/suite/binlog/t/binlog_truncate_active_log.inc > @@ -0,0 +1,65 @@ > +connect(master1,localhost,root,,); > +connect(master2,localhost,root,,); > +connect(master3,localhost,root,,); > + > +--connection default > + > +# First to commit few transactions > +INSERT INTO t VALUES (10); > +INSERT INTO tm VALUES (10); > + > +--connection master1 > +# Hold insert after write to binlog and before "run_commit_ordered" in engine > +SET DEBUG_SYNC= "commit_before_get_LOCK_commit_ordered SIGNAL master1_ready WAIT_FOR signal_never_arrives"; > +--send_eval $query1 > + > +--connection master2 > +SET DEBUG_SYNC= "now WAIT_FOR master1_ready"; > +SET DEBUG_SYNC= "commit_before_get_LOCK_after_binlog_sync SIGNAL master2_ready"; > +if ($pre_q2) > +{ > + --send_eval $pre_q2; > +} > +# To binlog non-xid transactional group which will be truncated all right > +if (!$pre_q2) > +{ > + --send_eval $query2 > +} eh, this makes no sense. In the previous version of the patch you've run pre_q2 before sending query2. now you send either pre_q2 or query2. so you don't need pre_q2 anymore now, just set query2 to the what used to be pre_q2. > + > +--connection master3 > +SET DEBUG_SYNC= "now WAIT_FOR master2_ready"; > +SELECT @@global.gtid_binlog_pos as 'Before the crash'; > + > +--connection default > +--source include/kill_mysqld.inc > +--disconnect master1 > +--disconnect master2 > +--disconnect master3 > + > +# > +# Server restart > +# > +--let $restart_parameters= --rpl-semi-sync-slave-enabled=1 > +--source include/start_mysqld.inc > + > +# Check error log for a successful truncate message. > +--let $log_error_ = $MYSQLTEST_VARDIR/log/mysqld.1.err > + > +--let SEARCH_FILE=$log_error_ > +--let SEARCH_PATTERN=Successfully truncated.*to remove transactions starting from GTID $truncate_gtid_pos > +--replace_regex /FOUND [0-9]+/FOUND #/ can it be found an unpredictable number of times? Why would binlog be truncated more than once? > +--source include/search_pattern_in_file.inc > + > +--echo Pre-crash binlog file content: > +--let $binlog_file= query_get_value(show binary logs, Log_name, $binlog_file_index) > +--source include/show_binlog_events.inc > + > +SELECT @@global.gtid_binlog_pos as 'After the crash'; > +--echo "One row should be present in table 't'" > +SELECT * FROM t; > + > +# prepare binlog file index for the next test > +--inc $binlog_file_index > + > +# Local cleanup > +DELETE FROM t; > diff --git a/mysql-test/suite/binlog/t/binlog_truncate_active_log.test b/mysql-test/suite/binlog/t/binlog_truncate_active_log.test > new file mode 100644 > --- /dev/null > +++ b/mysql-test/suite/binlog/t/binlog_truncate_active_log.test > @@ -0,0 +1,104 @@ > +# ==== Purpose ==== > +# > +# Test verifies the truncation of single binary log file. > +# > +# ==== References ==== > +# > +# MDEV-21117: recovery for --rpl-semi-sync-slave-enabled server > + > +--source include/have_innodb.inc > +--source include/have_aria.inc > +# File: binlog_truncate_active_log.inc included in test makes use of > +# 'debug_sync' facility. > +--source include/have_debug_sync.inc > +--source include/have_binlog_format_statement.inc > + > +call mtr.add_suppression("Can.t init tc log"); > +call mtr.add_suppression("Aborting"); > + > +# The following cases are tested: > +# A. 2pc transaction is followed by a blank "zero-engines" one > +# B. 2pc transaction follows the blank one > +# C. Similarly to A, with the XA blank transaction > + > +RESET MASTER; > +CREATE TABLE t (f INT) ENGINE=INNODB; > +CREATE TABLE t2 (f INT) ENGINE=INNODB; > +CREATE TABLE tm (f INT) ENGINE=Aria; > + > +# Old (pre-crash) binlog file index initial value. > +# It keeps incremented at the end of each case. > +--let $binlog_file_index=1 > + > +--echo # Case A. > +# Using 'debug_sync' hold 'query1' execution after 'query1' is flushed and > +# synced to binary log but not yet committed. In an another connection hold > +# 'query2' execution after 'query2' is flushed and synced to binlog. > +# Crash and restart server with --rpl-semi-sync-slave-enabled=1 > +# > +# During recovery of binary log 'query1' status is checked with InnoDB engine, > +# it will be in prepared but not yet commited. All transactions starting from > +# 'query1' onwards will be removed from the binary log. > +# Show-binlog-events is to prove that. > + > +--let $truncate_gtid_pos = 0-1-6 > +--let $query1 = INSERT INTO t VALUES (20) > +--let $query2 = DELETE FROM t2 WHERE f = 0 /* no such record */ > +--source binlog_truncate_active_log.inc > + > +--echo # Case B. > +# The inverted sequence ends up to truncate starting from $query2 > +--let $truncate_gtid_pos = 0-1-10 > +--let $query1 = DELETE FROM t2 WHERE f = 0 > +--let $query2 = INSERT INTO t VALUES (20) > +--source binlog_truncate_active_log.inc > + > + > +--echo # Case C. > +delimiter |; > +CREATE PROCEDURE sp_blank_xa() > +BEGIN > + XA START 'blank'; > + DELETE FROM t2 WHERE f = 0 /* no such record */; > + XA END 'blank'; > + XA PREPARE 'blank'; > +END| > +delimiter ;| > + > +# The same as in A with $query2 being the zero-engine XA transaction. > +# Both $query1 and $query2 are going to be truncated. > +--let $truncate_gtid_pos = 0-1-14 > +--let $query1 = INSERT INTO t VALUES (20) > +--let $pre_q2 = CALL sp_blank_xa > +--let $query2 = XA COMMIT 'blank' hm.. now your binlog_truncate_active_log.inc only runs query2 is pre_q2 is empty. So this XA COMMIT isn't done at all. Very confusing. > +--source binlog_truncate_active_log.inc > + > +DROP PROCEDURE sp_blank_xa; > + > + > +--echo # Case D. > +delimiter |; > +CREATE PROCEDURE sp_xa() > +BEGIN > + XA START 'xid'; > + DELETE FROM t WHERE f = 10; > + XA END 'xid'; > + XA PREPARE 'xid'; > +END| > +delimiter ;| > + > +# The same as in B with $query1 being the prepared XA transaction. > +# Truncation must occurs at $query2. > +--let $truncate_gtid_pos = 0-1-20 > +--let $query1 = CALL sp_xa > +--let $pre_q2 = > +--let $query2 = INSERT INTO t2 VALUES (20) > +--source binlog_truncate_active_log.inc > + > +DROP PROCEDURE sp_xa; > + > + > +--echo # Cleanup > +DROP TABLE t,t2,tm; > + > +--echo # End of the tests > diff --git a/sql/handler.h b/sql/handler.h > --- a/sql/handler.h > +++ b/sql/handler.h > @@ -931,6 +932,32 @@ struct xid_t { > }; > typedef struct xid_t XID; > > +/* > + Enumerates a sequence in the order of > + their creation that is in the top-down order of the index file. > + Ranges from zero through MAX_binlog_id. > + Not confuse the value with the binlog file numerical suffix, > + neither with the binlog file line in the binlog index file. Ah! Good, thanks. I thought it was a binlog file numerical suffix, indeed. > +*/ > +typedef uint Binlog_file_id; > +const Binlog_file_id MAX_binlog_id= UINT_MAX; > +/* > + Compound binlog-id and byte offset of transaction's first event > + in a sequence (e.g the recovery sequence) of binlog files. > + Binlog_offset(0,0) is the minimum value to mean > + the first byte of the first binlog file. > +*/ > +typedef std::pair<Binlog_file_id, my_off_t> Binlog_offset; > + > +/* binlog-based recovery transaction descriptor */ > +struct xid_recovery_member > +{ > + my_xid xid; > + uint in_engine_prepare; // number of engines that have xid prepared > + bool decided_to_commit; > + Binlog_offset binlog_coord; // semisync recovery binlog offset > +}; > + > /* for recover() handlerton call */ > #define MIN_XID_LIST_SIZE 128 > #define MAX_XID_LIST_SIZE (1024*128) > diff --git a/sql/log.cc b/sql/log.cc > --- a/sql/log.cc > +++ b/sql/log.cc > @@ -1692,17 +1698,13 @@ int binlog_init(void *p) > binlog_hton->savepoint_rollback= binlog_savepoint_rollback; > binlog_hton->savepoint_rollback_can_release_mdl= > binlog_savepoint_rollback_can_release_mdl; > - binlog_hton->commit= binlog_commit; > + binlog_hton->commit= binlog_commit_dummy; you could use a lambda for that too, [](handlerton*,THD*,bool) { return 0; } > binlog_hton->rollback= binlog_rollback; > binlog_hton->drop_table= [](handlerton *, const char*) { return -1; }; Oh did you actually need it? Or it's for completeness only? I mean, was there some code that tried to drop a "binlog" table? > if (WSREP_ON || opt_bin_log) > { > binlog_hton->prepare= binlog_prepare; > binlog_hton->start_consistent_snapshot= binlog_start_consistent_snapshot; > - binlog_hton->commit_by_xid= binlog_commit_by_xid; > - binlog_hton->rollback_by_xid= binlog_rollback_by_xid; > - // recover needs to be set to make xa{commit,rollback}_handlerton effective > - binlog_hton->recover= binlog_xa_recover_dummy; > } > binlog_hton->flags= HTON_NOT_USER_SELECTABLE | HTON_HIDDEN | HTON_NO_ROLLBACK; > return 0; > @@ -9975,6 +9977,154 @@ int TC_LOG::using_heuristic_recover() > /****** transaction coordinator log for 2pc - binlog() based solution ******/ > #define TC_LOG_BINLOG MYSQL_BIN_LOG > > +/** > + Truncates the current binlog to specified position. Removes the rest of binlogs > + which are present after this binlog file. > + > + @param truncate_file Holds the binlog name to be truncated > + @param truncate_pos Position within binlog from where it needs to > + truncated. > + > + @retval true ok > + @retval false error > + > +*/ > +bool MYSQL_BIN_LOG::truncate_and_remove_binlogs(const char *file_name, > + my_off_t pos, > + rpl_gtid *ptr_gtid) > +{ > + int error= 0; > +#ifdef HAVE_REPLICATION > + LOG_INFO log_info; > + THD *thd= current_thd; > + my_off_t index_file_offset= 0; > + File file= -1; > + MY_STAT s; > + my_off_t old_size; > + > + if ((error= find_log_pos(&log_info, file_name, 1))) > + { > + sql_print_error("Failed to locate binary log file:%s." > + "Error:%d", file_name, error); > + goto end; > + } > + > + while (!(error= find_next_log(&log_info, 1))) > + { > + if (!index_file_offset) > + { > + index_file_offset= log_info.index_file_start_offset; > + if ((error= open_purge_index_file(TRUE))) > + { > + sql_print_error("Failed to open purge index " > + "file:%s. Error:%d", purge_index_file_name, error); > + goto end; > + } > + } > + if ((error= register_purge_index_entry(log_info.log_file_name))) > + { > + sql_print_error("Failed to copy %s to purge index" > + " file. Error:%d", log_info.log_file_name, error); > + goto end; > + } > + } > + > + if (error != LOG_INFO_EOF) > + { > + sql_print_error("Failed to find the next binlog to " > + "add to purge index register. Error:%d", error); > + goto end; > + } > + > + if (is_inited_purge_index_file()) > + { > + if (!index_file_offset) > + index_file_offset= log_info.index_file_start_offset; > + > + if ((error= sync_purge_index_file())) > + { > + sql_print_error("Failed to flush purge index " > + "file. Error:%d", error); > + goto end; > + } > + > + // Trim index file > + if ((error= > + mysql_file_chsize(index_file.file, index_file_offset, '\n', > + MYF(MY_WME))) || > + (error= > + mysql_file_sync(index_file.file, MYF(MY_WME|MY_SYNC_FILESIZE)))) > + { > + sql_print_error("Failed to trim binlog index " > + "file:%s to offset:%llu. Error:%d", index_file_name, > + index_file_offset, error); > + goto end; > + } > + > + /* Reset data in old index cache */ > + if ((error= reinit_io_cache(&index_file, READ_CACHE, (my_off_t) 0, 0, 1))) > + { > + sql_print_error("Failed to reinit binlog index " > + "file. Error:%d", error); > + goto end; > + } > + > + /* Read each entry from purge_index_file and delete the file. */ > + if ((error= purge_index_entry(thd, NULL, TRUE))) > + { > + sql_print_error("Failed to process registered " > + "files that would be purged."); > + goto end; > + } > + } > + > + DBUG_ASSERT(pos); > + > + if ((file= mysql_file_open(key_file_binlog, file_name, > + O_RDWR | O_BINARY, MYF(MY_WME))) < 0) > + { > + error= 1; > + sql_print_error("Failed to open binlog file:%s for " > + "truncation.", file_name); > + goto end; > + } > + my_stat(file_name, &s, MYF(0)); > + old_size= s.st_size; > + clear_inuse_flag_when_closing(file); > + /* Change binlog file size to truncate_pos */ > + if ((error= > + mysql_file_chsize(file, pos, 0, MYF(MY_WME))) || > + (error= mysql_file_sync(file, MYF(MY_WME|MY_SYNC_FILESIZE)))) I'd rather not to overload if() conditions with function calls like this, they're rather annoying to single-step through and impossible to put a breakpoint on a specific function call. > + { > + sql_print_error("Failed to trim the " > + "binlog file:%s to size:%llu. Error:%d", > + file_name, pos, error); > + goto end; > + } > + else > + { > + char buf[21]; > + my_stat(file_name, &s, MYF(0)); > + my_off_t new_size= s.st_size; do you need it? new_size == pos, isn't it? You yourself use the word "size" to describe pos in sql_print_error just few lines above :) > + > + longlong10_to_str(ptr_gtid->seq_no, buf, 10); > + sql_print_information("Successfully truncated binlog file:%s " > + "to pos:%llu to remove transactions starting from " > + "GTID %u-%u-%s; " > + "former file size shrunk from %llu to %llu.", This isn't quite English. Better just "previous (or old) file size: %llu", that is not "former" and no need to specify the new size twice. Also, you say that you "truncated binlog file" here, but on failure you fail to "trim the binlog file". I suggest to use consistent wording, either always "trim" or always "truncate" (and always with or without "the") - both on success and on failure. Personally, I favor "truncate the binlog file" > + file_name, pos, > + ptr_gtid->domain_id, ptr_gtid->server_id, buf, > + old_size, new_size); > + } > + > +end: > + if (file >= 0) > + mysql_file_close(file, MYF(MY_WME)); > + > + error= error || close_purge_index_file(); > +#endif > + return error > 0; > +} > int TC_LOG_BINLOG::open(const char *opt_name) > { > int error= 1; > @@ -10449,7 +10599,528 @@ start_binlog_background_thread() ... > + > +Recovery_context::Recovery_context() : > + prev_event_pos(0), > + last_gtid_standalone(false), last_gtid_valid(false), last_gtid_no2pc(false), > + last_gtid_engines(0), > + do_truncate(rpl_semi_sync_slave_enabled), > + truncate_validated(false), truncate_reset_done(false), > + truncate_set_in_1st(false), id_binlog(MAX_binlog_id), > + checksum_alg(BINLOG_CHECKSUM_ALG_UNDEF), gtid_maybe_to_truncate(NULL) > +{ > + last_gtid_coord= Binlog_offset(0,0); > + binlog_truncate_coord= binlog_truncate_coord_1st_round= Binlog_offset(0,0); > + binlog_unsafe_coord= Binlog_offset(0,0); Okay, I admit, just typedef'ing std::pair is a huge improvement already > + binlog_truncate_file_name[0]= 0; > + binlog_unsafe_file_name [0]= 0; > + binlog_unsafe_gtid= truncate_gtid= truncate_gtid_1st_round= rpl_gtid(); > + if (do_truncate) > + gtid_maybe_to_truncate= new Dynamic_array<rpl_gtid>(16, 16); > +} > + Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 1

[Maria-developers] Tips on Writing/Reading JSON
by Michael Okoko 09 Jun '21

09 Jun '21

Hi Alexey, I'm Michael Okoko, a GSoC '21 student working on Using JSON as the on-disk format for MariaDB histograms(https://jira.mariadb.org/browse/MDEV-21130). The project requires writing and reading/parsing JSON, and I was wondering if you could provide some tips and pointers regarding a proper JSON writer/JSON parser to use. Regards, Michael.

2 1

Re: [Maria-developers] ef2519fee4e: MDEV-16546 System versioning setting to allow history modification
by Aleksey Midenkov 08 Jun '21

08 Jun '21

Hello, Sergei! On Fri, May 3, 2019 at 8:43 PM Sergei Golubchik <serg(a)mariadb.com> wrote: > > Hi, Aleksey! > > On May 03, Aleksey Midenkov wrote: > > revision-id: ef2519fee4e (versioning-1.0.5-17-gef2519fee4e) > > parent(s): 56145be2951 > > author: Aleksey Midenkov <midenok(a)gmail.com> > > committer: Aleksey Midenkov <midenok(a)gmail.com> > > timestamp: 2018-06-28 13:42:09 +0300 > > message: > > > > MDEV-16546 System versioning setting to allow history modification > > > > 1. Add server variable system_versioning_modify_history which will > > allow to set values for row_start, row_end in DML operations. > > > > 2. If secure_timestamp is YES or REPLICATION, > > system_versioning_modify_history does not have effect. If > > secure_timestamp is SUPER, system_versioning_modify_history requires > > special privilege (same as for setting current timestamp). > > I thought more about this idea. We don't really want to have the history > editable, do we? Well, I'm thinking about rollback table data to specific point in time. That could be a useful feature. > But it's needed for replication, to keep the master and > slave identical. That's what secure_timestamp is for. > > The idea was that this new variable, system_versioning_modify_history, > will be just a convenience feature, it will not allow history editing > any more than one can do without it. > > But now I suspect that even with secure_timestamp=NO one cannot truly > edit history. One can only insert new rows with arbitrary timestamps. > For example, to insert a row with row_start=1000 and row_end=2000, one > needs to do (if secure_timestamp=NO): > > set timestamp=1000; > insert row; > set timestamp=2000; > delete row; > > But I don't see how one can update or delete a history row with > secure_timestamp=NO. > > Now, with a SUPER privilege and secure_timestamp=NO or SUPER, one can > use the BINLOG command and truly edit the history arbitrarily, by faking > row events. I don't really get it why this is so important: since there is some limitation by configuration and privilege, we are just fine. Everything can be changed at filesystem level after all. > > The conclusion, I believe, is that system_versioning_modify_history > should allow INSERTs when secure_timestamp=NO, and it should allow > UPDATE/DELETE only for a SUPER user when secure_timestamp=NO or SUPER. I don't see a reason to argue on that. The only thing that is not clear, why we don't allow INSERTs when secure_timestamp=SUPER? > > The second thing I don't like at all, is when a table is created like > > CREATE TABLE t1 (a int) WITH SYSTEM VERSIONING > > with row_start/row_end implicit. You don't have it in the test, but > anyway one should be able to load history into such a table, while the > table does not have row_start and row_end columns. From the user point > of view these columns don't exist, they're pseudo-columns, like ROWID. > They just cannot be insertable-into, conceptually. But a user will want > to restore the history, right? I don't have a solution for this yet :( > Any ideas? We don't have to follow the conception if it doesn't help us. Since we have physical row_start/row_end, we don't have to pretend they don't exist. Who will win from that? > > See below a couple of minor comments about the patch itself. > > ... These are going to be fixed. > > Regards, > Sergei > Chief Architect MariaDB > and security(a)mariadb.org -- All the best, Aleksey Midenkov @midenok

2 15

[Maria-developers] Request for Discussion: Structure for JSON histogram objects
by Michael Okoko 04 Jun '21

04 Jun '21

Hi, MariaDB community! I'm implementing the patch for Using JSON as the on-disk format for MariaDB histograms(https://jira.mariadb.org/browse/MDEV-21130). as part of GSoC '21 and hoping to get insights and ideas from the community on how the JSON format for histograms should be structured. We are planning to change the columns of `mysql.column_stats` into something like: CREATE TABLE `column_stats` ( `db_name` varchar(64) COLLATE utf8mb3_bin NOT NULL, `table_name` varchar(64) COLLATE utf8mb3_bin NOT NULL, `column_name` varchar(64) COLLATE utf8mb3_bin NOT NULL, `histogram` blob ); The histogram column above will then be responsible for storing the JSON document for that row. Currently, the JSON structure we are looking at is based on the existing columns in `column_stats` table as well MySQL's JSON histogram implementation( https://dev.mysql.com/doc/refman/8.0/en/optimizer-statistics.html) Here's an example: { "min_value": 0, "max_value": 0, "hist_type": "singleton", "bucket": [ [1,0.3333333333333333], [2, 0.6666666666666666], [3,1] ], "nulls_ratios": 0, "last_updated": "2021-06-02 13:32:40.000000", "sampling_rate": 1, } - The min_value and max_values retain the meanings described at ( https://mariadb.com/kb/en/mysqlcolumn_stats-table/) - hist_type can be one of `SINGLETON` (where the number of buckets is >= the number of distinct values in the specified column) or `EQUI-HEIGHT` (where the number of buckets is < the number of distinct values found in the column). - nulls_ratio is a value between 0.0 and 1.0 indicating the fraction of column values that are SQL NULL values. If 0, the column contains no NULL values. - `last_updated` indicates when the histogram was generated, as a UTC value in YYYY-MM-DD hh:mm:ss.uuuuuu format. - `sampling_rate` is also a value between 0.0 and 1.0 indicating the fraction of column values that were sampled to create the histogram. A value of 1 means that all of the data was read (no sampling).

3 2

Re: [Maria-developers] 557ab2992dd: MDEV-21117 Incremental patch for a few review notes addressed.
by Sergei Golubchik 03 Jun '21

03 Jun '21

Hi, Andrei! On Jun 03, Andrei Elkin wrote: > >> - binlog_hton->commit= binlog_commit; > >> + binlog_hton->commit= binlog_commit_dummy; > >> binlog_hton->rollback= binlog_rollback; > >> binlog_hton->drop_table= [](handlerton *, const char*) { return -1; }; > > > > Oh did you actually need it? Or it's for completeness only? > > I am confused a bit, > 6c52931680a4 (Sergei Golubchik 2020-06-22 1703) binlog_hton->drop_table= [](handlerton *, const char*) { return -1; }; > > And I never reviewed the patch. Grrr. Sorry! When reviewing I diffed this patch with a previous version that I reviewed (a useful speedup technique at the late phase, when commits only change slightly) and this drop_table line was in the new patch and not in the old. Of course, it was because the new patch was in 10.6 and the old in 10.2, there were quite a few changes in the diff because of that. I tried to ignore them, but somehow missed this one. Note, there's no + before the drop_table line, it's not yours, it's indeed something that already was in 10.6, I should've noticed, my mistake :( Why I added it in 6c52931680a4? It was for DROP TABLE FORCE, that simply does plugin_foreach() and tries to hton->drop_table() in every hton available. In binlog_hton too. The goal was to solve cases like a table in InnoDB data dictionary but with no frm, or Aria table where MAD file exists, but no MAI - cases when DROP TABLE doesn't see the table, but CREATE TABLE still fails as some remnants of a table still exist. > >> @@ -9975,6 +9977,154 @@ int TC_LOG::using_heuristic_recover() > >> + else > >> + { > >> + char buf[21]; > >> + my_stat(file_name, &s, MYF(0)); > >> + my_off_t new_size= s.st_size; > > > > do you need it? new_size == pos, isn't it? You yourself use the word > > "size" to describe pos in sql_print_error just few lines above :) > > Sure. Let turn it into an > > assert( pos == new_size ) > > [x] and this is, what I think, you'd call "too much checks" :) I'd personally would've dropped my_stat and new_size, especially as I also suggest to remove them from the sql_print_information below. > >> + > >> + longlong10_to_str(ptr_gtid->seq_no, buf, 10); > >> + sql_print_information("Successfully truncated binlog file:%s " > >> + "to pos:%llu to remove transactions starting from " > >> + "GTID %u-%u-%s; " > >> + "former file size shrunk from %llu to %llu.", > > > > This isn't quite English. Better just "previous (or old) file > > size: %llu", that is not "former" and no need to specify the new size twice. > > > > Also, you say that you "truncated binlog file" here, but on failure > > you fail to "trim the binlog file". I suggest to use consistent > > wording, either always "trim" or always "truncate" (and always with > > or without "the") - both on success and on failure. Personally, I favor > > "truncate the binlog file" > > some two styles of two writers of these blocks mixed up :-). (Partly > relates to the size in above). > > [x] > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 8e79d168614: MDEV-25672 table alias from previous statement interferes later commands.
by Sergei Golubchik 02 Jun '21

02 Jun '21

Hi, Aleksey! On Jun 02, Aleksey Midenkov wrote: > revision-id: 8e79d168614 (mariadb-10.2.31-989-g8e79d168614) > parent(s): 433dd490d33 > author: Aleksey Midenkov <midenok(a)gmail.com> > committer: Aleksey Midenkov <midenok(a)gmail.com> > timestamp: 2021-05-31 16:20:38 +0300 > message: > > MDEV-25672 table alias from previous statement interferes later commands. I'm sorry. I don't understand what all this new code for. To make sure that vcol's field->table_name is refreshed for every statement? What is vcol's field->table_name used for besides your CREATE TABLE check? If it's not used for anything then a much simpler fix would be narrow the check - it only needs to be run for new vcols in CREATE TABLE, disabling it for old vcols in ALTER TABLE will do the trick. Like - if (p.table_name.length && table_name) + if (!field && p.table_name.length && table_name) Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] e92037989f7: MDEV-21117: refine the server binlog-based recovery for semisync
by Sergei Golubchik 01 Jun '21

01 Jun '21

Hi, Andrei! On Jun 01, Andrei Elkin wrote: > >>>> > >>>> It's actually turns out not to be easy. A sequence of execution events > >>>> > >>>> E1.prepare(trx), E2.prepare(trx), E1.commit(trx), *crash* > >>>> > >>>> does not make E1 such as Innodb committed persistently in file. > >>>> At recovery in E1 trx may be found in prepared state, unless > >>>> before the crash I'd do something like Binlog CheckPoint (BCP) > >>>> notification request and wait of it. > >>> > The committed state is done asynchronously (by virtue of MDEV-232 that > implements a part of BCP). Oh. Indeed. Sorry, I've missed that. Still a prepare of another transaction can force this commit to disk, as far as I understand. So what about this: you stop the thread on a debug sync after the first commit then another connection starts committing in InnoDB, it stops on a debug sync after the prepare - before anything is written into binlog then you crash. looks like it'll create the case you need > >> No doubt it's technically possible to wait for the flushing and > >> crash, but still does not look easy. > >> Consider if we defer this sort of mtr test to MDEV-18959? > > Sujatha just contributed `74a13b4de2` with an mtr test that crashes > after the asynchronous fsync is done to the partially (2 engine case) > committed state. ... > Aslo now that `74a13b4de2` provides with a method to crash after the > "commmit" fsync we might also employ it for RQG. No, this is very different. This is a crash happening where Sujatha expects it to happen, there's no value in that now, it's needed for regression testing, to make sure the recovery won't be broken in the future. But it makes no sense to keep testing now how recovery works in case of a crash happening at exactly the place where Sujatha wanted it to happen and after she verified that crash at that exactly line does not break the recovery. The whole point of RQG tests is to try crashes where you did not expect them to happen. Adding DBUG_EXECUTE_IF() doesn't help this goal at all. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] e92037989f7: MDEV-21117: refine the server binlog-based recovery for semisync
by Sergei Golubchik 31 May '21

31 May '21

Hi, Andrei! On May 27, Andrei Elkin wrote: >>> >>> I had a rather sobering followup, slipped communicating to you >>> though, sorry. >>> >>> It's actually turns out not to be easy. A sequence of execution events >>> >>> E1.prepare(trx), E2.prepare(trx), E1.commit(trx), *crash* >>> >>> does not make E1 such as Innodb committed persistently in file. At >>> recovery in E1 trx may be found in prepared state, unless before the >>> crash I'd do something like Binlog CheckPoint (BCP) notification request >>> and wait of it. >> >> I don't understand, what a binlog checkpoint has to do with it? > > So E1=Innodb. > In the above case in order to create partially committed trx > Innodb.commit(trx) > must be followed with flushing trx to disk, which is typically > asynchronous. > > I mentioned BCP 'cos it ensures a trx (actually the last in a binlog > file) is flushed. With --innodb-flush-log-at-trx-commit=1 it should be perfectly synnchronous, should it not? > >> In this just a selected engine would be requested, unlike in BCP > >> event case that is generated when both (in this case) have > >> persistently committed. (That is we even can't involve BCP event to > >> wait for, and the involvement would require some simulation as > >> well). > >> > >> So I propose to keep the current method leaving the "honest" crash > >> to RQG that Alice has been training the patch with. > > > > Alice was loading the server with transactions and doing `kill -9` > > at some point. Many interesting cases that we discuss (for example, > > the one above) happen in a very small time window, so I don't know > > whether random killing could provide an adequate test coverage. > > No doubt it's technically possible to wait for the flushing and crash, > but still does not look easy. > Consider if we defer this sort of mtr test to MDEV-18959? But we've spent numerous hours and emails discussing and you've spend countless days and weeks implementing various tricky corner cases are are and won't be ever tested. Here's an idea of how to test it. We're only interested in crashes during commit. So * when starting a commit - atomically increment some in_commit counter * on leaving the commit - decrement it * on signal (some unused one) - crash the server if in_commit > 0 * after the crash - get the stack traces from the core to see where in the commit did it crash. I've attached a small patch to illustrate this idea. Hopefully, it'll cover everything, but if not, then after analyzing stack traces we'll see where it did not crash, and move atomic increment/decrements to be around this area only. > >> >> diff --git a/sql/handler.h b/sql/handler.h > >> >> --- a/sql/handler.h > >> >> +++ b/sql/handler.h > >> >> @@ -873,6 +874,15 @@ typedef struct xid_t XID; > >> >> /* The 'buf' has to have space for at least SQL_XIDSIZE bytes. */ > >> >> uint get_sql_xid(XID *xid, char *buf); > >> >> > >> >> +/* struct for semisync slave binlog truncate recovery */ > >> >> +struct xid_recovery_member > >> >> +{ > >> >> + my_xid xid; > >> >> + uint in_engine_prepare; // number of engines that have xid prepared > >> >> + bool decided_to_commit; > >> >> + std::pair<uint, my_off_t> binlog_coord; // semisync recovery binlog offset > >> > > >> > wouldn't it be clearer to have a struct with named members? > >> > >> I am not sure if there's a need for naming. But we do need > >> comparison that std::pair provides for granted. > >> > >> I could consider (a) new typedef(s) to represent the binlog id and > >> offset in the file to make it more readable. > >> > >> You say? > >> > >> > in fact, I'm somewhat surprised there's no such struct for binlog coords > >> > already. > >> > >> I had started that back at HB implementation actually, but that struct > >> did not attact much of attention. > > > > > I'd still suggest a structure, like > > > > struct binlog_coords_t { > > uint binlog_num; > > my_off_t offset; > > }; > > Well, with this, like I said about std:pair, we'd also have to have > bool operator < (const binlog_coords_t) like bool operator <(const binlog_coords_t c) { return binlog_num < c.binlog_num || binlog_num == c.binlog_num && offset < c.offset; } > I seriously rate refuse to reuse as a deadly sin of programming. > (Unlike normal sins it may be soften by realization though :-)). > > Let's waive it away, right :-)? > > Hoping on that `72a59e4b068` already defined the following: ... > +/* > + Compound binlog-id and byte offset of transaction's first event > + in a sequence (e.g the recovery sequence) of binlog files. > + Binlog_offset(0,0) is the minimum value to mean > + the first byte of the first binlog file. > +*/ > +typedef std::pair<Binlog_file_id, my_off_t> Binlog_offset; This is better than writing std::pair<uint, my_off_t> everywhere. Not as clear as a binlog_coords_t from above, but _perhaps_ it'll do, let's see how it'll be used. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

[Maria-developers] Using JSON_TABLE to create JSON columns?
by Sergey Petrunia 31 May '21

31 May '21

Hi Alexey and everyone, I was looking at whether it's possible use JSON_TABLE to extract portions of JSON document. Apparently it is possible in MySQL with JSON datatype: Q1: select * from json_table('[{"color": "blue", "price": { "high": 10, "low": 5}}, {"color": "red", "price": { "high": 20, "low": 8}}]', '$[*]' columns(color varchar(100) path '$.color', price json path '$.price' ) ) as T; +-------+------------------------+ | color | price | +-------+------------------------+ | blue | {"low": 5, "high": 10} | | red | {"low": 8, "high": 20} | +-------+------------------------+ Note that if one uses any datatype other than JSON, they get NULLs: Q2: select * from json_table('[{"color": "blue", "price": { "high": 10, "low": 5}}, {"color": "red", "price": { "high": 20, "low": 8}}]', '$[*]' columns(color varchar(100) path '$.color', price text path '$.price' ) ) as T; +-------+-------+ | color | price | +-------+-------+ | blue | NULL | | red | NULL | +-------+-------+ Oracle-the-database doesn't yet(*) have a JSON datatype. So I can only run Q2 and then I get NULLs in the price column. MariaDB accepts JSON as datatype so query Q1 is accepted. However the logic in MDEV-17399 code doesn't have support for dumping a portion of JSON document, so one gets empty strings in the price column. Should we support Q1 with JSON output in the price column? If yes, should we do it within the scope of MDEV-17399 or create another task for this? (*) I see this: https://docs.oracle.com/en/database/oracle/oracle-database/20/newft/new-jso… BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

3 2

[Maria-developers] GSoC Student Introduction for MDEV-24676
by Владислав Какурин 31 May '21

31 May '21

Hello, my name is Vladislav Kakurin, and this year I'm a GSOC 2021 participant for MariaDB. I will work on concurrent ring-buffer for improvement of IO_CACHE (https://jira.mariadb.org/browse/MDEV-24676) I hope I will succeed in this matter, as well as become a part of the MariaDB community

1 0

[Maria-developers] GSoC Student Introduction for MDEV-21130
by Michael Okoko 27 May '21

27 May '21

Hi, I'm Michael Okoko, a GSOC 2021 mentee for MariaDB and I'd be working on switching the on-disk storage format for MariaDB histograms to JSON ( https://jira.mariadb.org/browse/MDEV-21130) I'm looking forward to working with everyone in the community and contributing more to the MariaDB during and after GSOC.

1 0

Re: [Maria-developers] e92037989f7: MDEV-21117: refine the server binlog-based recovery for semisync
by Sergei Golubchik 26 May '21

26 May '21

Hi, Andrei! On Apr 27, Andrei Elkin wrote: > >> +--let $case = B: one engine has committed its transaction branch > >> +# Hold off after one engine has committed. > >> +--let $shutdown_timeout=0 > >> +--let $debug_sync_action = "commit_after_run_commit_ordered SIGNAL con1_ready WAIT_FOR signal_no_signal" > >> +# Both debug_sync and debug-dbug are required to make sure Engines remember the commit state > >> +# debug_sync alone will not help. > >> +--let $restart_parameters = --rpl-semi-sync-slave-enabled=1 --debug-dbug=d,binlog_truncate_partial_commit > > > > in the first review I wrote > > > > this seems to be a rather crude way of faking a partially committed > > transaction. better to crash after the first engine has committed, > > that'd be much more natural. > > > > and you replied > > > > This simulation aimed at (allows for) more complicated recovery time > > event sequences. > > In this case, indeed, crashing by demand is about of the same efforts. > > I can convert to that. > > > > [x] > > I had a rather sobering followup, slipped communicating to you though, sorry. > > It's actually turns out not to be easy. A sequence of execution events > > E1.prepare(trx), E2.prepare(trx), E1.commit(trx), *crash* > > does not make E1 such as Innodb committed persistently in file. At > recovery in E1 trx may be found in prepared state, unless before the > crash I'd do something like Binlog CheckPoint (BCP) notification request > and wait of it. I don't understand, what a binlog checkpoint has to do with it? > In this just a selected engine would be requested, unlike in BCP event > case that is generated when both (in this case) have persistently committed. > (That is we even can't involve BCP event to wait for, and the involvement > would require some simulation as well). > > So I propose to keep the current method leaving the "honest" crash to > RQG that Alice has been training the patch with. Alice was loading the server with transactions and doing `kill -9` at some point. Many interesting cases that we discuss (for example, the one above) happen in a very small time window, so I don't know whether random killing could provide an adequate test coverage. > >> diff --git a/mysql-test/suite/rpl/t/rpl_semi_sync_fail_over.test b/mysql-test/suite/rpl/t/rpl_semi_sync_fail_over.test > >> --- /dev/null > >> +++ b/mysql-test/suite/rpl/t/rpl_semi_sync_fail_over.test > >> @@ -0,0 +1,143 @@ > >> + > >> +--source include/have_innodb.inc > >> +--source include/have_debug_sync.inc > >> +--source include/have_binlog_format_row.inc > >> +--let $rpl_topology=1->2 > >> +--source include/rpl_init.inc > > > > why not to source master-slave.inc if you're using a standard master-slave > > topology anyway? > > Using numbers it's easier to manipulate with failover operatations. > Besides it's truly uncomfortable both read and write > when `master` would be serving the slave role as it happens in the test :-). I believe I already replied to that earlier. master-slave.inc creates server_1 and server_2 connections all right. > >> diff --git a/sql/handler.h b/sql/handler.h > >> --- a/sql/handler.h > >> +++ b/sql/handler.h > >> @@ -873,6 +874,15 @@ typedef struct xid_t XID; > >> /* The 'buf' has to have space for at least SQL_XIDSIZE bytes. */ > >> uint get_sql_xid(XID *xid, char *buf); > >> > >> +/* struct for semisync slave binlog truncate recovery */ > >> +struct xid_recovery_member > >> +{ > >> + my_xid xid; > >> + uint in_engine_prepare; // number of engines that have xid prepared > >> + bool decided_to_commit; > >> + std::pair<uint, my_off_t> binlog_coord; // semisync recovery binlog offset > > > > wouldn't it be clearer to have a struct with named members? > > I am not sure if there's a need for naming. But we do need > comparison that std::pair provides for granted. > > I could consider (a) new typedef(s) to represent the binlog id and > offset in the file to make it more readable. > > You say? > > > in fact, I'm somewhat surprised there's no such struct for binlog coords > > already. > > I had started that back at HB implementation actually, but that struct > did not attact much of attention. I'd still suggest a structure, like struct binlog_coords_t { uint binlog_num; my_off_t offset; }; > > > >> +}; > >> + > >> /* for recover() handlerton call */ > >> #define MIN_XID_LIST_SIZE 128 > >> #define MAX_XID_LIST_SIZE (1024*128) > >> @@ -4820,7 +4830,8 @@ int ha_commit_one_phase(THD *thd, bool all); > >> int ha_commit_trans(THD *thd, bool all); > >> int ha_rollback_trans(THD *thd, bool all); > >> int ha_prepare(THD *thd); > >> -int ha_recover(HASH *commit_list); > >> +int ha_recover(HASH *commit_list, MEM_ROOT *mem_root= NULL); > >> +uint ha_recover_complete(HASH *commit_list, std::pair<uint, my_off_t> *coord= NULL); > > > > is coord a truncation position? > > So NULL is for the normal recovery. > Non-NULL is an estimate of the 1st to rollback. > Trx with lesser offsets are to commit, greater of equal - to rollback. > > Why it's only an estimate is 'cos the function also handles > no-truncation cases, it then passes as `coord` the last committed > trx's coordinate. > The last committed can't be decided to rollback because of higher precedence of > > xid_recovery_member::decided_to_commit > > which is `true` under current circumstance, in > > +static bool xarecover_decide(xid_recovery_member* member, > + xid_t x, std::pair<uint, my_off_t> *coord_ptr) > +{ > + return > + member->decided_to_commit ? true : > + !coord_ptr ? false : > + (member->binlog_coord < *coord_ptr > > I did not reflect the subtlety of the above in function header comments. > To be done, if you're sastified here. yes, please, reflect it. thanks :) > >> diff --git a/sql/log_event.h b/sql/log_event.h > >> --- a/sql/log_event.h > >> +++ b/sql/log_event.h > >> @@ -482,6 +482,16 @@ class String; > >> */ > >> #define LOG_EVENT_IGNORABLE_F 0x80 > >> > >> +/** > >> + @def LOG_EVENT_ACCEPT_OWN_F > >> + > >> + Flag sets by the semisync slave for accepting > >> + the same server_id ("own") events which the slave must not have > >> + in its state. Typically such events were never committed by > >> + their originator (this server) and discared at its semisync-slave recovery. > >> +*/ > >> +#define LOG_EVENT_ACCEPT_OWN_F 0x4000 > > > > may be, add an assert on all received events that such a flag is not set? > > it can only be set on events in relay log. > > There's no place the flag gets logged. It'd be way too fool-proof in my > opinion :-)! > Or if you have a case I'll do. too fool-proof is when you do a=1; assert(a == 1); if you write a binlog event to disk, another thread reads it and sends over the network to another server, which also writes it to disk first, then another thread reads it again and executes - I don't think it's "too" to check that the event doesn't have this flag. Also, I personally value a lot a documentation role of asserts. Assert, like DBUG_ASSERT(ev->flags & LOG_EVENT_ACCEPT_OWN_F == 0); is basically saying /* LOG_EVENT_ACCEPT_OWN_F must be not set here */ but it's a self-fulfilling comment, it documents the code and at the same time guarantees that the documentation is up-to-date. > >> @@ -3357,6 +3367,12 @@ class Gtid_log_event: public Log_event > >> uint64 commit_id; > >> uint32 domain_id; > >> uchar flags2; > >> + uint flags_extra; // more flags area placed after the regular flags2's one > >> + /* > >> + Extra to a "base" engine recoverable engines participating > >> + in the transaction. Zero, when the base engine only is present. > > > > what's a "base engine"? > > The 1st ha_info one that is not binlog. why not to say "number of participating engines minus one" ? this won't puzzle anyone with a new concept of a "base engine" > > > >> + */ > >> + uint8 extra_engines; > >> > >> /* Flags2. */ > >> > >> diff --git a/sql/handler.cc b/sql/handler.cc > >> --- a/sql/handler.cc > >> +++ b/sql/handler.cc > >> @@ -1637,9 +1672,17 @@ commit_one_phase_2(THD *thd, bool all, THD_TRANS *trans, bool is_real_trans) > >> DEBUG_SYNC(thd, "commit_one_phase_2"); > >> if (ha_info) > >> { > >> + int err; > >> + > > >> + if (has_binlog_hton(ha_info) && > > > > can you replace has_binlog_hton() with, like, if trx cache is not empty or > > binlog enabled or something like that? > > I see you're driving at removing binlog ha_info from the list, are not > you? I'm not sure :) Yes, I'd like to get rid of binlog_hton, as we discussed. But here, perhaps, I just didn't like that you scan the list for something that most likely could be checked much easier. > I already made some experiments to have observed various complications > in the way of implementing an equivalent of > has_binlog_hton(). > The closest would be > non-empty cache. > > But `binlog_commit` may need invoking even in this case, as its branch > shows > > if (cache_mngr->trx_cache.empty()) > { > /* > we're here because cache_log was flushed in MYSQL_BIN_LOG::log_xid() > */ > cache_mngr->reset(false, true); > > > Overall, I'd stay with the current approach to defer its refinement to > time binlog_hton stops registering for transaction. > > Your comment? Okay. I just don't like scanning the list, aesthetically, when one can check a value in some fixed location or a variable. But it's not a performance concern, the list has usually only two elements and hardly ever can have more than three. So okay, let's stay with the current approach. > > @@ -1962,8 +2008,177 @@ struct xarecover_st > > >> sql_print_warning("%s transaction with xid %llu", > > > > may be not a sql_print_warning, but a sql_print_information? > > it's just an informational message > > [->] I presume, it means [x] > >> + member->decided_to_commit ? > >> + "Committed" : "Rolled back", (ulonglong) member->xid); > >> + } > >> + > >> + return false; > >> +} Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 308c28e8a60: MDEV-15888 Implement FLUSH TABLES tbl_name [, tbl_name] ... WITH READ LOCK for views.
by Sergei Golubchik 23 May '21

23 May '21

Hi, Alexey! On Mar 21, Alexey Botchkov wrote: > revision-id: 308c28e8a60 (mariadb-10.5.2-393-g308c28e8a60) > parent(s): 786bc312b85 > author: Alexey Botchkov <holyfoot(a)mariadb.com> > committer: Alexey Botchkov <holyfoot(a)mariadb.com> > timestamp: 2021-02-09 00:59:55 +0400 > message: > > MDEV-15888 Implement FLUSH TABLES tbl_name [, tbl_name] ... WITH READ LOCK for views. > > Kind of 'naive' solution that surprisingly worked so trying to > understand why. > > diff --git a/mysql-test/main/flush-innodb.result b/mysql-test/main/flush-innodb.result > index 21e5bda7785..2c886e4f9fc 100644 > --- a/mysql-test/main/flush-innodb.result > +++ b/mysql-test/main/flush-innodb.result > @@ -60,7 +60,7 @@ DROP TABLE export; > CREATE VIEW v1 AS SELECT 1; > CREATE TEMPORARY TABLE t1 (a INT); > FLUSH TABLES v1 FOR EXPORT; > -ERROR HY000: 'test.v1' is not of type 'BASE TABLE' > +UNLOCK TABLES; good, but it only tests that there's no error. may be you need to add tests to verify that tables are actually locked? > FLUSH TABLES t1 FOR EXPORT; > ERROR 42S02: Table 'test.t1' doesn't exist > FLUSH TABLES non_existent FOR EXPORT; > diff --git a/sql/sql_reload.cc b/sql/sql_reload.cc > index 8f87d633d19..c146b7693c1 100644 > --- a/sql/sql_reload.cc > +++ b/sql/sql_reload.cc > @@ -543,6 +543,7 @@ bool flush_tables_with_read_lock(THD *thd, TABLE_LIST *all_tables) > > if (thd->lex->type & REFRESH_READ_LOCK) > { > +#ifdef MDEV15887 > /* > Acquire SNW locks on tables to be flushed. Don't acquire global > IX and database-scope IX locks on the tables as this will make > @@ -559,6 +560,7 @@ bool flush_tables_with_read_lock(THD *thd, TABLE_LIST *all_tables) > for (auto table_list= all_tables; table_list; > table_list= table_list->next_global) > table_list->mdl_request.ticket= NULL; > +#endif /*MDEV15887*/ what's that? btw, the number is MDEV15888 > } > > thd->variables.option_bits|= OPTION_TABLE_LOCK; Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] Request for a discusison: A fine-grained concurrent ring buffer mode for IO_CACHE
by Nikita Malyavin 22 May '21

22 May '21

Welcome to the thread, Andrei! For everybody: we just had a productive chat with Andrei, and I'd like to outline the results. Let's take a look at binlog_commit, or more exactly MYSQL_BIN_LOG::trx_group_commit_leader called from there (follow binlog_commit_flush_trx_cache -> THD::binlog_flush_pending_rows_event -> MYSQL_BIN_LOG::flush_and_set_pending_rows_event -> MYSQL_BIN_LOG::write_transaction_to_binlog call chain). The committing is currently strongly sequenced. The transactions are organized into groups, and when the latest transaction is acknowledged on the first commit phase, they all are being committed by leader in a chosen order. However, even here we can write the transactions in parallel, preserving the order. Andrei also claims, that the order can be potentially restored on the replication side. Anyway we technically can't send the transaction to the replication slave before the binlog flush&fsync, nevertheless the data will still be preserved in the volatile append cache. There was MDEV-20925 <https://jira.mariadb.org/browse/MDEV-20925> to store the transaction length in the event, but it unfortunately was rejected: > I will be closing this issue because we have COMMIT/ROLLBACK query log event > in the end of transaction , whose size is difficult to determine , > So current plan is to do MDEv-19687 without transaction length. The replication team decided to calculate the transaction sizes during receiving the data form the io, and then to store it in the hash, and no protocol modifications would be required then. I suggested to buffer the transaction separately, and then push it into the relay log in the data frame, storing the length. Then, we have MDEV-19687 <https://jira.mariadb.org/browse/MDEV-19687>, which was the supertask for MDEV-20925 <https://jira.mariadb.org/browse/MDEV-20925>. The aim is to parallelize the replication on the slave side. They are implementing their own parallel circular buffer with single-writer, multiple-reader use case: the parallel workers are going to pick out the transaction data, and resolve the commit order later. Sergei petrunya, you've been questioning: > I see SEQ_READ_APPEND is only used for Relay Log on the slave. Afaiu, the > relay > log has only only one producer, the network io thread. Andrei clarified, that there can actually be multiple sources. However they are just going to create a working queue for each source. =============================================================== To underline, the rationale is changed for this IO_CACHE improvement: * The replication relay log is not the use-case anymore, since a separate, much simplified circular buffer is going to be implemented (AFAIU, Sachin is in the middle of the progress of MDEV-19687 <https://jira.mariadb.org/browse/MDEV-19687>). * The binlog commit is instead a good case! Most likely I have forgotten about it, when I was writing the rationale. But anyway it wasn't clear for me, can the transactions be written in parallel there. On the reader side, the network replication sender should not block the committing process by reading from the binlog. Regards, Nikita

1 0

[Maria-developers] Request for a discusison: A fine-grained concurrent ring buffer mode for IO_CACHE
by Nikita Malyavin 19 May '21

19 May '21

Hello, dear MariaDB community! I am glad to present the design for IO_CACHE's SEQ_READ_APPEND mode concurrency improvements. Jira task is MDEV-24676. A student, Vladislav Kakurin, has applied to GSoC for this task, and has passed, so he will be engaged in implementing it this summer. The discussion is very important here, since I could make the mistakes, and Vladislav can miss them, but the success of the task completion may depend on it. We're hoping to succeed in implementing the multi-producer till GSoC end. So please don't hesitate to ask for clarifications. Note that reader/writer and producer/consumer pairs are used interchangeably in the text. The text is attached as PDF, but for the sake of convenient quoting it is additionally embodied in the email with shrinked markup and no illustrations. Sincerely, Nikita Malyavin Software Engineer of MariaDB ==================================================================== A fine-grained concurrent ring buffer mode for IO_CACHE 1. Rationale. --------------------------------------------------- Current implementation of IO_CACHE’s SEQ_READ_APPEND mode behaves coarsely grained on its write buffer: every read and every write is protected by append_buffer_lock. int _my_b_seq_read(IO_CACHE *info, ...) { lock_append_buffer(info); ... // read logic unlock_append_buffer(info); return Count ? 1 : 0; } int my_b_append(IO_CACHE *info, ...) { lock_append_buffer(info); ... // append logic unlock_append_buffer(info); return 0; } Despite the separate read buffer is read-only, and therefore is accessed wait-free, the write buffer can have a contention with medium-sized transactions. The design described hereafter is going to solve this issue, and an extension for a parallel multi-producer workflow is additionally provided. Furthermore, the API extension for multi-producer approach support is proposed, and the multi-consumerness is discussed. 2. The single-producer, single-consumer case. --------------------------------------------------- Idea. The memcpy operations of consumer and producer never overlap, therefore they can be freed of locks. Overflow and emptiness We cannot begin writing in the area still involved in reading. Therefore, reader should not update the pointers before it finishes reading. This means that we should lock in the beginning to atomically read the data, and in the end, to write the new reader data. The same for vice-versa, we cannot read from the area still involved into writing, therefore a read should finish with EMPTY error (currently _my_b_seq_read just returns 1) When we reach a “buffer is full” condition, we can flip the read and write (append) buffers, if we were reading from an append buffer. Otherwise, the append buffer is flushed. The algorithm. The following pseudocode will describe the single-consumer, single-producer approach. It is assumed that reading from the read buffer is handled in the usual way. io->total_size and io->read_buffer are considered to be accessed atomically. io_err_t read(IO_CACHE *io, uchar *buffer, size_t sz) { if (sz > io->total_size) return E_IO_EMPTY; uchar *read_buffer = io->read_buffer; if (io->read_pos points to read_buffer) sz_read = read_from_read_buffer(io, buffer, sz); buffer += sz_read; sz -= sz_read; io->total_size -= sz_read; if (sz == 0) return 0; // else copy from append buffer lock(io->append_buffer_lock); // copy the local variables uchar *read_pos = io->read_pos; uchar *read_buffer = io->read_buffer; uchar *append_start_pos = i->append_start_pos; uchar *append_size = io->append_size; uchar *append_pos = io->append_pos; // etc, if needed unlock(io->append_buffer_lock); read from append buffer; lock(io->append_buffer_lock); // update the variables io->append_size -= sz; io->append_start_pos += sz; if (i->append_start_pos >= io->append_buf + io->cache_size) io->append_start_pos -= io->cache_size; unlock(io->append_buffer_lock); io->total_size -= sz; } The first read()’s part tries to read from a read-only buffer. If it’s empty, it moves the effort to a volatile append buffer. All the metadata is copied in the first critical section, before the data copying, to the stack. It is updated back in the second critical section, after the data copying. io_err_t write(IO_CACHE *io, uchar *buffer, size_t sz) { lock(io->append_buffer_lock); if (append_buffer is full and io->total_size <= io->append_size) swap(io->append_buffer, io->read_buffer); else flush the append buffer if needed; write to disk directly, if the data is too large; uchar *write_pos = io->write_pos; unlock(io->append_buffer_lock); write to append buffer; lock(io->append_buffer_lock); io->write_pos = new_write_pos; unlock(io->append_buffer_lock); io->total_size += sz; } The important note here is that we access io->read_buffer in the reader’s thread without the lock (the accesses are marked bold). However this access happens only once in the beginning and is safe: Only writer changes read_buffer. The writer can change it only once during one read() if io->read_buffer is considered reads-only, then it will not flip again, and continue to be consistent, until io->total_size is changed: io->total_size -= sz_read; Then the lock happens. It should be fine to read from a flipped buffer on that stage. 3. Multi-producer concurrency --------------------------------------------------- Idea. Writes start from io->write_start, which is to update immediately. Reads are possible only until io->read_end, which is updated, as soon as writes are finished. Medium-grained approach io->write_start is updated immediately to allow parallel writes. However, we cannot update io->read_end immediately after this thread’s write ends, because earlier writes can still be in progress. We should wait for them i.e. we wait while (io->read_end != local_read_end) Algorithm (medium-grained). Medium-grained approach will modify write() function as follows (the changed lines and locks are bolded): io_err_t write(IO_CACHE *io, uchar *buffer, size_t sz) { lock(io->append_buffer_lock); if (buffer flip of flush is needed) wait until all the writes are finished; if (append_buffer is full && io->write_total_size <= io->append_size) swap(io->append_buffer, io->read_buffer); else flush the append buffer if needed; write to disk directly, if the data is too large; uchar *local_write_start = io->write_start; io->write_total_size += sz; io->write_start += sz; if (io->write_start > io->append_buffer + io->cache_size) io->write_start -= io->cache_size; unlock(io->append_buffer_lock); write to append buffer; lock(io->write_event_lock) while(local_write_start != io->read_end) cond_wait(io->write_event, io->write_event_lock); unlock(io->write_event_lock) lock(io->append_buffer_lock); io->read_end = new_read_end; unlock(io->append_buffer_lock); cond_signal(io->write_event); io->total_size += sz; } The read function should be modified mostly cosmetically. Fine graining. The writers are still waiting for each other’s finish. The approach described here defers waiting through helping pattern by introducing progress slots. Each time a writer begins progress it allocates a slot in the dedicated (fixed size) array. When the writer finishes its job, it checks whether it is the leftmost one (relative to its read_end value. If it is, it updates read_end for itself, and for all the consecutive writers already finished. The slot allocation will be controlled by a semaphore to prevent overflow. Therefore, only a fixed number of producers can work simultaneously. The slot array is made of elements of private cache_slot_t structure: struct cache_slot_t { bool vacant: 1; bool finished: 1; uint next: size_bits(uint) - 2; uint pos; } The slot is acquired whenever a write begins by searching an array cell with vacant=1. When it’s found, vacant = 0, finished = 0 is set. The last_slot variable holds the slot index for the latest write. slots[ last_slot].next is set to a new index, and last_slot itself is updated. The following example demonstrates how the slots work: +-------------------------------------------------------------+ | | write1 | write2 | write3 | | +-------------------------------------------------------------+ A B C D +-------------------------------------------------+ | vacant=0 | vacant=1 | vacant=0 | vacant=0 | | finished=0 | | finished=1 | finished=1 | | next=2 | | next=3 | next=? | | pos=A | | pos=B | pos=C | +-------------------------------------------------+ there were three writes currently running in parallel. write2 and write3 are finished, but write1 is still running. When it finishes, it will hop through slot.next while vacant==0 and finished==1 and pos != io->write_start. Therefore, read_end will be updated to C if no other write will begin in parallel. If another write begins in parallel before write1 finishes, it allocates slots[1] and sets pos=D. slots[3].next would be set to 1, and last_slot will be updated from 3 to 1. The slot run through expected complexity is O(1). The proof for acquisition is however not that obvious to prove the same, and no effort was spent for proving it (It’s only obvious that it’s O(slots)). 4. Arbitrary data sources support --------------------------------------------------- The widely spread use-case is pouring from another IO_CACHE source (like a statement or transaction cache). The operation may require several consecutive write() calls with an external lock: lock(write_lock); uchar buffer[SIZE]; while(cache_out is not empty) { read(ceche_out, buffer, SIZE); write(cache_in, buffer, SIZE); } unlock(write_lock); This case destroys all the parallel design described. However, let’s make api changes to allow blocks of predicted size be written in parallel: /** Allocates the slot of a requested size for a writer. Returns new slot id. */ slot_id_t append_allocate(IO_CAHCE*, size_t block_size); /** Frees the slot and propogates the data to be available for reading */ void append_commit(IO_CACHE*, slot_id_t); These two functions just decompose our write() function: append_allocate would include the first critical section and append_commit would include the second one. The use-case will be changed slightly: slot_id_t slot = append_allocate(cache_out, append_tell(cache_in)); uchar buffer[SIZE]; while(cache_out is not empty) { read(ceche_out, buffer, SIZE); write(cache_in, buffer, SIZE); } append_commit(cache_out, slot); 5. Multi-consumerness --------------------------------------------------- We currently have no cases with several readers working in parallel in SEQ_READ_APPEND mode. It is only used by the replication thread to read out the log, where it is delegated to a dedicated worker. The first problem is that parallel readout would require additional coordination -- the order of event application can be important. Another problem is that a variable-sized blocks require at least two consecutive reads if the structure is not known. If the length is stored, it can be read out with exactly two reads (first reads length, second reads the body). The slot allocation strategy can be applied, and api can be added similar to a new write api: /** lock the cache and allocate the read slot */ slot_id_t read_allocate_lock(IO_CACHE*); /** Allocate a read zone of the requested size and unlock the cache */ void read_allocate_unlock(IO_CACHE*, slot_id_t, size_t size); /** Finish reading; deallocate the read slot */ viod read_commit(IO_CACHE*, slot_id_t); Reading api needs one function more than writing api -- the allocation is split on two phases: locking phase (to compute the block length), and the actual requesting phase. This approach has several disadvantages: 1. The read buffer access is no longer lock-free 2. read_allocate_lock leaves the IO_CACHE in a locked state, which can be potentially misused. Additionally, two SX locks can be used (one for readers and one for writers) for extra parallelism.

2 2

[Maria-developers] Request for a discusison: A fine-grained concurrent ring buffer mode for IO_CACHE
by Nikita Malyavin 18 May '21

18 May '21

Hello, dear MariaDB community! I am glad to present the design for IO_CACHE's SEQ_READ_APPEND mode concurrency improvements. Jira task is MDEV-24676. A student, Vladislav Kakurin, has applied to GSoC for this task, and has passed, so he will be engaged in implementing it this summer. The discussion is very important here, since I could make the mistakes, and Vladislav can miss them, but the success of the task completion may depend on it. We're hoping to succeed in implementing the multi-producer till GSoC end. So please don't hesitate to ask for clarifications. Note that reader/writer and producer/consumer pairs are used interchangeably in the text. The text is attached as PDF, but for the sake of convenient quoting it is additionally embodied in the email with shrinked markup and no illustrations. Sincerely, Nikita Malyavin Software Engineer of MariaDB ==================================================================== A fine-grained concurrent ring buffer mode for IO_CACHE 1. Rationale. --------------------------------------------------- Current implementation of IO_CACHE’s SEQ_READ_APPEND mode behaves coarsely grained on its write buffer: every read and every write is protected by append_buffer_lock. int _my_b_seq_read(IO_CACHE *info, ...) { lock_append_buffer(info); ... // read logic unlock_append_buffer(info); return Count ? 1 : 0; } int my_b_append(IO_CACHE *info, ...) { lock_append_buffer(info); ... // append logic unlock_append_buffer(info); return 0; } Despite the separate read buffer is read-only, and therefore is accessed wait-free, the write buffer can have a contention with medium-sized transactions. The design described hereafter is going to solve this issue, and an extension for a parallel multi-producer workflow is additionally provided. Furthermore, the API extension for multi-producer approach support is proposed, and the multi-consumerness is discussed. 2. The single-producer, single-consumer case. --------------------------------------------------- Idea. The memcpy operations of consumer and producer never overlap, therefore they can be freed of locks. Overflow and emptiness We cannot begin writing in the area still involved in reading. Therefore, reader should not update the pointers before it finishes reading. This means that we should lock in the beginning to atomically read the data, and in the end, to write the new reader data. The same for vice-versa, we cannot read from the area still involved into writing, therefore a read should finish with EMPTY error (currently _my_b_seq_read just returns 1) When we reach a “buffer is full” condition, we can flip the read and write (append) buffers, if we were reading from an append buffer. Otherwise, the append buffer is flushed. The algorithm. The following pseudocode will describe the single-consumer, single-producer approach. It is assumed that reading from the read buffer is handled in the usual way. io->total_size and io->read_buffer are considered to be accessed atomically. io_err_t read(IO_CACHE *io, uchar *buffer, size_t sz) { if (sz > io->total_size) return E_IO_EMPTY; uchar *read_buffer = io->read_buffer; if (io->read_pos points to read_buffer) sz_read = read_from_read_buffer(io, buffer, sz); buffer += sz_read; sz -= sz_read; io->total_size -= sz_read; if (sz == 0) return 0; // else copy from append buffer lock(io->append_buffer_lock); // copy the local variables uchar *read_pos = io->read_pos; uchar *read_buffer = io->read_buffer; uchar *append_start_pos = i->append_start_pos; uchar *append_size = io->append_size; uchar *append_pos = io->append_pos; // etc, if needed unlock(io->append_buffer_lock); read from append buffer; lock(io->append_buffer_lock); // update the variables io->append_size -= sz; io->append_start_pos += sz; if (i->append_start_pos >= io->append_buf + io->cache_size) io->append_start_pos -= io->cache_size; unlock(io->append_buffer_lock); io->total_size -= sz; } The first read()’s part tries to read from a read-only buffer. If it’s empty, it moves the effort to a volatile append buffer. All the metadata is copied in the first critical section, before the data copying, to the stack. It is updated back in the second critical section, after the data copying. io_err_t write(IO_CACHE *io, uchar *buffer, size_t sz) { lock(io->append_buffer_lock); if (append_buffer is full and io->total_size <= io->append_size) swap(io->append_buffer, io->read_buffer); else flush the append buffer if needed; write to disk directly, if the data is too large; uchar *write_pos = io->write_pos; unlock(io->append_buffer_lock); write to append buffer; lock(io->append_buffer_lock); io->write_pos = new_write_pos; unlock(io->append_buffer_lock); io->total_size += sz; } The important note here is that we access io->read_buffer in the reader’s thread without the lock (the accesses are marked bold). However this access happens only once in the beginning and is safe: Only writer changes read_buffer. The writer can change it only once during one read() if io->read_buffer is considered reads-only, then it will not flip again, and continue to be consistent, until io->total_size is changed: io->total_size -= sz_read; Then the lock happens. It should be fine to read from a flipped buffer on that stage. 3. Multi-producer concurrency --------------------------------------------------- Idea. Writes start from io->write_start, which is to update immediately. Reads are possible only until io->read_end, which is updated, as soon as writes are finished. Medium-grained approach io->write_start is updated immediately to allow parallel writes. However, we cannot update io->read_end immediately after this thread’s write ends, because earlier writes can still be in progress. We should wait for them i.e. we wait while (io->read_end != local_read_end) Algorithm (medium-grained). Medium-grained approach will modify write() function as follows (the changed lines and locks are bolded): io_err_t write(IO_CACHE *io, uchar *buffer, size_t sz) { lock(io->append_buffer_lock); if (buffer flip of flush is needed) wait until all the writes are finished; if (append_buffer is full && io->write_total_size <= io->append_size) swap(io->append_buffer, io->read_buffer); else flush the append buffer if needed; write to disk directly, if the data is too large; uchar *local_write_start = io->write_start; io->write_total_size += sz; io->write_start += sz; if (io->write_start > io->append_buffer + io->cache_size) io->write_start -= io->cache_size; unlock(io->append_buffer_lock); write to append buffer; lock(io->write_event_lock) while(local_write_start != io->read_end) cond_wait(io->write_event, io->write_event_lock); unlock(io->write_event_lock) lock(io->append_buffer_lock); io->read_end = new_read_end; unlock(io->append_buffer_lock); cond_signal(io->write_event); io->total_size += sz; } The read function should be modified mostly cosmetically. Fine graining. The writers are still waiting for each other’s finish. The approach described here defers waiting through helping pattern by introducing progress slots. Each time a writer begins progress it allocates a slot in the dedicated (fixed size) array. When the writer finishes its job, it checks whether it is the leftmost one (relative to its read_end value. If it is, it updates read_end for itself, and for all the consecutive writers already finished. The slot allocation will be controlled by a semaphore to prevent overflow. Therefore, only a fixed number of producers can work simultaneously. The slot array is made of elements of private cache_slot_t structure: struct cache_slot_t { bool vacant: 1; bool finished: 1; uint next: size_bits(uint) - 2; uint pos; } The slot is acquired whenever a write begins by searching an array cell with vacant=1. When it’s found, vacant = 0, finished = 0 is set. The last_slot variable holds the slot index for the latest write. slots[ last_slot].next is set to a new index, and last_slot itself is updated. The following example demonstrates how the slots work: +-------------------------------------------------------------+ | | write1 | write2 | write3 | | +-------------------------------------------------------------+ A B C D +-------------------------------------------------+ | vacant=0 | vacant=1 | vacant=0 | vacant=0 | | finished=0 | | finished=1 | finished=1 | | next=2 | | next=3 | next=? | | pos=A | | pos=B | pos=C | +-------------------------------------------------+ there were three writes currently running in parallel. write2 and write3 are finished, but write1 is still running. When it finishes, it will hop through slot.next while vacant==0 and finished==1 and pos != io->write_start. Therefore, read_end will be updated to C if no other write will begin in parallel. If another write begins in parallel before write1 finishes, it allocates slots[1] and sets pos=D. slots[3].next would be set to 1, and last_slot will be updated from 3 to 1. The slot run through expected complexity is O(1). The proof for acquisition is however not that obvious to prove the same, and no effort was spent for proving it (It’s only obvious that it’s O(slots)). 4. Arbitrary data sources support --------------------------------------------------- The widely spread use-case is pouring from another IO_CACHE source (like a statement or transaction cache). The operation may require several consecutive write() calls with an external lock: lock(write_lock); uchar buffer[SIZE]; while(cache_out is not empty) { read(ceche_out, buffer, SIZE); write(cache_in, buffer, SIZE); } unlock(write_lock); This case destroys all the parallel design described. However, let’s make api changes to allow blocks of predicted size be written in parallel: /** Allocates the slot of a requested size for a writer. Returns new slot id. */ slot_id_t append_allocate(IO_CAHCE*, size_t block_size); /** Frees the slot and propogates the data to be available for reading */ void append_commit(IO_CACHE*, slot_id_t); These two functions just decompose our write() function: append_allocate would include the first critical section and append_commit would include the second one. The use-case will be changed slightly: slot_id_t slot = append_allocate(cache_out, append_tell(cache_in)); uchar buffer[SIZE]; while(cache_out is not empty) { read(ceche_out, buffer, SIZE); write(cache_in, buffer, SIZE); } append_commit(cache_out, slot); 5. Multi-consumerness --------------------------------------------------- We currently have no cases with several readers working in parallel in SEQ_READ_APPEND mode. It is only used by the replication thread to read out the log, where it is delegated to a dedicated worker. The first problem is that parallel readout would require additional coordination -- the order of event application can be important. Another problem is that a variable-sized blocks require at least two consecutive reads if the structure is not known. If the length is stored, it can be read out with exactly two reads (first reads length, second reads the body). The slot allocation strategy can be applied, and api can be added similar to a new write api: /** lock the cache and allocate the read slot */ slot_id_t read_allocate_lock(IO_CACHE*); /** Allocate a read zone of the requested size and unlock the cache */ void read_allocate_unlock(IO_CACHE*, slot_id_t, size_t size); /** Finish reading; deallocate the read slot */ viod read_commit(IO_CACHE*, slot_id_t); Reading api needs one function more than writing api -- the allocation is split on two phases: locking phase (to compute the block length), and the actual requesting phase. This approach has several disadvantages: 1. The read buffer access is no longer lock-free 2. read_allocate_lock leaves the IO_CACHE in a locked state, which can be potentially misused. Additionally, two SX locks can be used (one for readers and one for writers) for extra parallelism.

1 0

Re: [Maria-developers] eebebe8ef75: MDEV-23842 Atomic RENAME TABLE
by Sergei Golubchik 16 May '21

16 May '21

Hi, Monty! On May 10, Michael Widenius wrote: > revision-id: eebebe8ef75 (mariadb-10.6.0-77-geebebe8ef75) > parent(s): 5a602e2ce7f > author: Michael Widenius <michael.widenius(a)gmail.com> > committer: Michael Widenius <michael.widenius(a)gmail.com> > timestamp: 2021-05-04 21:09:10 +0300 > message: > > MDEV-23842 Atomic RENAME TABLE > diff --git a/client/mysqltest.cc b/client/mysqltest.cc > index 350c55edee2..133d1f76839 100644 > --- a/client/mysqltest.cc > +++ b/client/mysqltest.cc > @@ -8095,9 +8095,10 @@ void handle_error(struct st_command *command, > const char *err_sqlstate, DYNAMIC_STRING *ds) > { > int i; > - > DBUG_ENTER("handle_error"); > > + var_set_int("$errno", err_errno); there was $mysql_errno already, was is not something you could've used? > + > command->used_replace= 1; > if (command->require_file) > { > diff --git a/mysql-test/suite/atomic/rename_case.result b/mysql-test/suite/atomic/rename_case.result > new file mode 100644 > index 00000000000..4b58c555cf2 > --- /dev/null > +++ b/mysql-test/suite/atomic/rename_case.result > @@ -0,0 +1,52 @@ > +create database test2; > +# > +# Testing rename error in different places > +# > +create table t1 (a int); > +create table T2 (b int); > +create table t3 (c int); > +create table T4 (d int); > +insert into t1 values(1); > +insert into T2 values(2); > +insert into t3 values(3); > +insert into T4 values(4); > +create temporary table tmp1 (a int); > +create temporary table tmp2 (b int); > +create temporary table tmp3 (c int); > +create temporary table tmp4 (d int); > +insert into tmp1 values(11); > +insert into tmp2 values(22); > +insert into tmp3 values(33); > +insert into tmp4 values(44); > +rename table t3 to T4, t1 to t5, T2 to t1, t5 to T2; > +ERROR 42S01: Table 'T4' already exists > +rename table t1 to t5, t3 to T4, T2 to t1, t5 to T2; > +ERROR 42S01: Table 'T4' already exists > +rename table t1 to t5, T2 to t1, t3 to T4, t5 to T2; > +ERROR 42S01: Table 'T4' already exists > +rename table t1 to t5, T2 to t1, t5 to T2, t3 to T4; > +ERROR 42S01: Table 'T4' already exists > +# Try failed rename using two databases > +rename table test.t1 to test2.t5, test.T2 to test.t1, t5 to test.T2; > +ERROR 42S02: Table 'test.t5' doesn't exist > +select t1.a+T2.b+t3.c+T4.d from t1,T2,t3,T4; better do select t1.a,T2.b,t3.c,T4.d from t1,T2,t3,T4; same one row, but it can detect when tables are swapped. or if you need one scalar value (you do below) it could ne select CONCAT(t1.a,T2.b,t3.c,T4.d) from t1,T2,t3,T4; > +t1.a+T2.b+t3.c+T4.d > +10 > +select * from t5; > +ERROR 42S02: Table 'test.t5' doesn't exist > +T2.MYD > +T2.MYI > +T2.frm > +T4.MYD > +T4.MYI > +T4.frm > +db.opt > +t1.MYD > +t1.MYI > +t1.frm > +t3.MYD > +t3.MYI > +t3.frm > +# Cleanup > +drop table t1,T2,t3,T4; > +drop database test2; > diff --git a/sql/sql_class.h b/sql/sql_class.h > index 0dcb2e04ba4..5ae0866672d 100644 > --- a/sql/sql_class.h > +++ b/sql/sql_class.h > @@ -2835,6 +2835,7 @@ class THD: public THD_count, /* this must be first */ > > #ifndef MYSQL_CLIENT > binlog_cache_mngr * binlog_setup_trx_data(); > + ulonglong binlog_xid; /* binlog request storing of xid */ "Tell binlog to store thd->query_id in the next Query_log_event" > > /* > Public interface to write RBR events to the binlog > diff --git a/sql/log_event_client.cc b/sql/log_event_client.cc > index 6b5d71348e1..733fc14c697 100644 > --- a/sql/log_event_client.cc > +++ b/sql/log_event_client.cc > @@ -1822,7 +1822,7 @@ bool Query_log_event::print_query_header(IO_CACHE* file, > my_b_printf(file, > "\t%s\tthread_id=%lu\texec_time=%lu\terror_code=%d\n", > get_type_str(), (ulong) thread_id, (ulong) exec_time, > - error_code)) > + error_code, (ulong) xid)) huh? you didn't add "\nxid=%lu" to the format string > goto err; > } > > diff --git a/sql/debug_sync.h b/sql/debug_sync.h > index 3b8aa8815e1..4e3e10fcc51 100644 > --- a/sql/debug_sync.h > +++ b/sql/debug_sync.h > @@ -53,4 +53,12 @@ static inline bool debug_sync_set_action(THD *, const char *, size_t) > { return false; } > #endif /* defined(ENABLED_DEBUG_SYNC) */ > > +#ifndef DBUG_OFF > +void debug_crash_here(const char *keyword); > +bool debug_simulate_error(const char *keyword, uint error); > +#else > +#define debug_crash_here(A) do { } while(0) > +#define debug_simulate_error(A, B) 0 > +#endif this doesn't belong to debug_sync.h or debug_sync.cc put it, hmm, may be in sql_priv.h ? or sql_class.h/sql_class.cc > + > #endif /* DEBUG_SYNC_INCLUDED */ > diff --git a/sql/debug_sync.cc b/sql/debug_sync.cc > index 1fbb15592a4..f523e22f6ce 100644 > --- a/sql/debug_sync.cc > +++ b/sql/debug_sync.cc > @@ -1628,3 +1628,74 @@ bool debug_sync_set_action(THD *thd, const char *action_str, size_t len) > /* prevent linker/lib warning about file without public symbols */ > int debug_sync_dummy; > #endif /* defined(ENABLED_DEBUG_SYNC) */ > + > + > +/** > + Debug utility to do crash after a set number of executions > + > + The user variable, either @debug_crash_counter or @debug_error_counter, this is very hackish. better to use @@debug_dbug_counter, something like that. also it could be global for an extra benefit if you'd need to crash a non-user thread. > + is decremented each time debug_crash() or debug_simulate_error is called > + if the keyword is set with @@debug_push, like > + @@debug_push="d+frm_data_type_info_emulate" > + > + If the variable is not set or is not an integer it will be ignored. > +*/ > + > +#ifndef DBUG_OFF > + > +static const LEX_CSTRING debug_crash_counter= > +{ STRING_WITH_LEN("debug_crash_counter") }; > +static const LEX_CSTRING debug_error_counter= > +{ STRING_WITH_LEN("debug_error_counter") }; > + > +static bool debug_decrement_counter(const LEX_CSTRING *name) > +{ > + THD *thd= current_thd; > + user_var_entry *entry= (user_var_entry*) > + my_hash_search(&thd->user_vars, (uchar*) name->str, name->length); > + if (!entry || entry->type != INT_RESULT || ! entry->value) > + return 0; > + (*(ulonglong*) entry->value)= (*(ulonglong*) entry->value)-1; > + return !*(ulonglong*) entry->value; > +} > + > +void debug_crash_here(const char *keyword) > +{ > + DBUG_ENTER("debug_crash_here"); > + DBUG_PRINT("enter", ("keyword: %s", keyword)); > + > + DBUG_EXECUTE_IF(keyword, > + if (debug_decrement_counter(&debug_crash_counter)) > + { > + my_printf_error(ER_INTERNAL_ERROR, > + "Crashing at %s", > + MYF(ME_ERROR_LOG | ME_NOTE), keyword); > + DBUG_SUICIDE(); > + }); > + DBUG_VOID_RETURN; > +} > + > +/* > + This can be used as debug_counter to simulate an error at a specific > + position. > + > + Typical usage would be > + if (debug_simualte_error("keyword")) > + error= 1; > +*/ > + > +bool debug_simulate_error(const char *keyword, uint error) > +{ > + DBUG_ENTER("debug_crash_here"); > + DBUG_PRINT("enter", ("keyword: %s", keyword)); > + DBUG_EXECUTE_IF(keyword, > + if (debug_decrement_counter(&debug_error_counter)) > + { > + my_printf_error(error, > + "Simulating error for '%s'", > + MYF(ME_ERROR_LOG), keyword); > + DBUG_RETURN(1); > + }); > + DBUG_RETURN(0); > +} > +#endif /* DBUG_OFF */ > diff --git a/sql/log_event_server.cc b/sql/log_event_server.cc > index cca05f1fcca..0cf037e3a6f 100644 > --- a/sql/log_event_server.cc > +++ b/sql/log_event_server.cc > @@ -1294,6 +1294,15 @@ bool Query_log_event::write() > int3store(start, when_sec_part); > start+= 3; > } > + > + /* xid's is used with ddl_log handling */ it's not really an XID, it's a query_id or the DDL statetent. XID is a combination of a 64-byte global transaction identifier and a 64-byte branch qualifier. It would be much less confusing not to pretend that the query_id is an "xid" is some shape or form. Better to rename it to, say ddl_query_id. Or even ddl_id, if you like it short. > + if (thd && thd->binlog_xid) > + { > + *start++= Q_XID; > + int8store(start, thd->query_id); > + start+= 8; > + } > + > /* > NOTE: When adding new status vars, please don't forget to update > the MAX_SIZE_LOG_EVENT_STATUS in log_event.h and update the function > diff --git a/sql/log.cc b/sql/log.cc > index 17c599627cb..6e0a9706ec8 100644 > --- a/sql/log.cc > +++ b/sql/log.cc > @@ -10654,6 +10669,8 @@ int TC_LOG_BINLOG::recover(LOG_INFO *linfo, const char *last_log_name, > { > if (ha_recover(&xids)) > goto err2; > + if (ddl_log_close_binlogged_events(&xids)) your xids and normal Xid_log_event xids are from two different non-intersecting sets and have even different semantics. It'd be safer not to have them in the same hash. > + goto err2; > > free_root(&mem_root, MYF(0)); > my_hash_free(&xids); > diff --git a/sql/sql_view.cc b/sql/sql_view.cc > index 5df5ecb76a9..e6d726b30d7 100644 > --- a/sql/sql_view.cc > +++ b/sql/sql_view.cc > @@ -2231,16 +2234,25 @@ mysql_rename_view(THD *thd, > (uchar*)&view_def, view_parameters)) > { > /* restore renamed view in case of error */ > - rename_in_schema_file(thd, new_db->str, new_name->str, view->db.str, > - view->table_name.str); > + rename_in_schema_file(thd, new_db->str, new_name->str, old_db->str, > + old_name->str); > goto err; > } > - } else > + } > + else > DBUG_RETURN(1); > > /* remove cache entries */ > - query_cache_invalidate3(thd, view, 0); > - sp_cache_invalidate(); > + { > + char key[NAME_LEN*2+1], *ptr; > + memcpy(key, old_db->str, old_db->length); > + ptr= key+ old_db->length; > + *ptr++= 0; > + memcpy(key, old_name->str, old_name->length); > + ptr= key+ old_db->length; > + *ptr++= 0; there's tdc_create_key for that. > + query_cache.invalidate(thd, key, (size_t) (ptr-key), 0); > + } > error= FALSE; > > err: > diff --git a/sql/ddl_log.h b/sql/ddl_log.h > index 0747699cd85..316e6708f22 100644 > --- a/sql/ddl_log.h > +++ b/sql/ddl_log.h > @@ -24,6 +24,9 @@ > enum ddl_log_entry_code > { > /* > + DDL_LOG_UNKOWN s/UNKOWN/UNKNOWN/ > + Here mainly to detect blocks that are all zero > + > DDL_LOG_EXECUTE_CODE: > This is a code that indicates that this is a log entry to > be executed, from this entry a linked list of log entries > @@ -34,48 +37,95 @@ enum ddl_log_entry_code > DDL_IGNORE_LOG_ENTRY_CODE: > An entry that is to be ignored > */ > - DDL_LOG_EXECUTE_CODE = 'e', > - DDL_LOG_ENTRY_CODE = 'l', > - DDL_IGNORE_LOG_ENTRY_CODE = 'i' > + DDL_LOG_UNKNOWN= 0, > + DDL_LOG_EXECUTE_CODE= 1, > + DDL_LOG_ENTRY_CODE= 2, > + DDL_IGNORE_LOG_ENTRY_CODE= 3, > + DDL_LOG_ENTRY_CODE_LAST= 4 > }; > > + > +/* > + When adding things below, also add an entry to ddl_log_entry_phases in > + ddl_log.cc > +*/ > + > enum ddl_log_action_code > { > /* > The type of action that a DDL_LOG_ENTRY_CODE entry is to > perform. > - DDL_LOG_DELETE_ACTION: > - Delete an entity > - DDL_LOG_RENAME_ACTION: > - Rename an entity > - DDL_LOG_REPLACE_ACTION: > - Rename an entity after removing the previous entry with the > - new name, that is replace this entry. > - DDL_LOG_EXCHANGE_ACTION: > - Exchange two entities by renaming them a -> tmp, b -> a, tmp -> b. > */ > - DDL_LOG_DELETE_ACTION = 'd', > - DDL_LOG_RENAME_ACTION = 'r', > - DDL_LOG_REPLACE_ACTION = 's', > - DDL_LOG_EXCHANGE_ACTION = 'e' > + DDL_LOG_UNKNOWN_ACTION= 0, > + > + /* Delete a .frm file or a table in the partition engine */ > + DDL_LOG_DELETE_ACTION= 1, > + > + /* Rename a .frm fire a table in the partition engine */ > + DDL_LOG_RENAME_ACTION= 2, > + > + /* > + Rename an entity after removing the previous entry with the > + new name, that is replace this entry. > + */ > + DDL_LOG_REPLACE_ACTION= 3, > + > + /* Exchange two entities by renaming them a -> tmp, b -> a, tmp -> b */ > + DDL_LOG_EXCHANGE_ACTION= 4, > + /* > + log do_rename(): Rename of .frm file, table, stat_tables and triggers > + */ > + DDL_LOG_RENAME_TABLE_ACTION= 5, > + DDL_LOG_RENAME_VIEW_ACTION= 6, > + DDL_LOG_LAST_ACTION /* End marker */ > }; > > + > +/* Number of phases for each ddl_log_action_code */ > +extern const uchar ddl_log_entry_phases[DDL_LOG_LAST_ACTION]; > + > + > enum enum_ddl_log_exchange_phase { > EXCH_PHASE_NAME_TO_TEMP= 0, > EXCH_PHASE_FROM_TO_NAME= 1, > EXCH_PHASE_TEMP_TO_FROM= 2 > }; > > +enum enum_ddl_log_rename_table_phase { > + DDL_RENAME_PHASE_TRIGGER= 0, > + DDL_RENAME_PHASE_STAT, > + DDL_RENAME_PHASE_TABLE, > +}; > + > +/* > + Setting ddl_log_entry.phase to this has the same effect as setting > + action_type to DDL_IGNORE_LOG_ENTRY_CODE this doesn't make much sense. May be you can add a line or two about why it's needed and one cannot simply set action_type to DDL_IGNORE_LOG_ENTRY_CODE instead? > +*/ > + > +#define DDL_LOG_FINAL_PHASE ((uchar) 0xff) > > typedef struct st_ddl_log_entry > { > - const char *name; > - const char *from_name; > - const char *handler_name; > - const char *tmp_name; > + LEX_CSTRING name; > + LEX_CSTRING from_name; > + LEX_CSTRING handler_name; > + LEX_CSTRING tmp_name; > + LEX_CSTRING db; > + LEX_CSTRING from_db; > + LEX_CSTRING from_handler_name; > + uchar uuid[MY_UUID_SIZE]; // UUID for new frm file > + > + ulonglong xid; // Xid stored in the binary log > uint next_entry; > - uint entry_pos; > - enum ddl_log_entry_code entry_type; > + uint entry_pos; // Set by write_dll_log_entry() > + /* > + unique_id can be used to store a unique number to check current state. > + Currently it is used to store new size of frm file or link to a ddl log > + entry. is that so? I don't see it storing a file version, a new size of the frm or a link, I see it stores some kind of a counter to abort recovery when it was repeated more than DDL_LOG_MAX_RETRY times > + */ > + uint32 unique_id; // Unique id for file version > + uint16 flags; // Flags unique for each command > + enum ddl_log_entry_code entry_type; // Set automatically > enum ddl_log_action_code action_type; > /* > Most actions have only one phase. REPLACE does however have two > @@ -95,17 +145,52 @@ typedef struct st_ddl_log_memory_entry > } DDL_LOG_MEMORY_ENTRY; > > > +typedef struct st_ddl_log_state > +{ > + /* List of ddl log entries */ > + DDL_LOG_MEMORY_ENTRY *list; > + /* One execute entry per list */ > + DDL_LOG_MEMORY_ENTRY *execute_entry; > +} DDL_LOG_STATE; would be nice to have a small comment, explaining what is a "state" and how two lists can represent it. is execute_entry a pointer to a some element from the list? > + > + > +/* These functions are for recovery */ > +bool ddl_log_initialize(); > +void ddl_log_release(); > +bool ddl_log_close_binlogged_events(HASH *xids); > +int ddl_log_execute_recovery(); > + > +/* functions for updating the ddl log */ > bool ddl_log_write_entry(DDL_LOG_ENTRY *ddl_log_entry, > DDL_LOG_MEMORY_ENTRY **active_entry); > + > bool ddl_log_write_execute_entry(uint first_entry, > - bool complete, > - DDL_LOG_MEMORY_ENTRY **active_entry); > -bool ddl_log_increment_phase(uint entry_no); > + DDL_LOG_MEMORY_ENTRY **active_entry); > +bool ddl_log_disable_execute_entry(DDL_LOG_MEMORY_ENTRY **active_entry); > + > +void ddl_log_complete(DDL_LOG_STATE *ddl_log_state); > +void ddl_log_revert(THD *thd, DDL_LOG_STATE *ddl_log_state); > + > +bool ddl_log_update_phase(DDL_LOG_STATE *entry, uchar phase); > +bool ddl_log_update_xid(DDL_LOG_STATE *state, ulonglong xid); > +bool ddl_log_disable_entry(DDL_LOG_STATE *state); > +bool ddl_log_increment_phase(uint entry_pos); > void ddl_log_release_memory_entry(DDL_LOG_MEMORY_ENTRY *log_entry); > bool ddl_log_sync(); > -void ddl_log_release(); > -void ddl_log_execute_recovery(); > bool ddl_log_execute_entry(THD *thd, uint first_entry); > > +void ddl_log_release_entries(DDL_LOG_STATE *ddl_log_state); > +bool ddl_log_rename_table(THD *thd, DDL_LOG_STATE *ddl_state, > + handlerton *hton, > + const LEX_CSTRING *org_db, > + const LEX_CSTRING *org_alias, > + const LEX_CSTRING *new_db, > + const LEX_CSTRING *new_alias); > +bool ddl_log_rename_view(THD *thd, DDL_LOG_STATE *ddl_state, > + const LEX_CSTRING *org_db, > + const LEX_CSTRING *org_alias, > + const LEX_CSTRING *new_db, > + const LEX_CSTRING *new_alias); > + > extern mysql_mutex_t LOCK_gdl; > #endif /* DDL_LOG_INCLUDED */ > diff --git a/sql/sql_rename.cc b/sql/sql_rename.cc > index b7aed97a8a2..2b2f32cf126 100644 > --- a/sql/sql_rename.cc > +++ b/sql/sql_rename.cc > @@ -186,49 +172,43 @@ bool mysql_rename_tables(THD *thd, TABLE_LIST *table_list, bool silent, > /* Add IF EXISTS to binary log */ > thd->variables.option_bits|= OPTION_IF_EXISTS; > } > + > + debug_crash_here("ddl_log_rename_before_binlog"); > + /* > + Store xid in ddl log and binary log so that we can check on ddl recovery > + if the item is in the binary log (and thus the operation was complete > + */ > + thd->binlog_xid= thd->query_id; > + ddl_log_update_xid(&ddl_log_state, thd->binlog_xid); > binlog_error= write_bin_log(thd, TRUE, thd->query(), thd->query_length()); I'd prefer to avoid blowing up THD with arbitrary flags that are only needed rarely in one specific place in the code to pass some argument few stack frames down. Perhaps you can do instead something like bool Query_log_event::write() { ... /* xid's is used with ddl_log handling */ if (ddl_log_entry_incomplete()) { *start++= Q_XID; int8store(start, thd->query_id); start+= 8; } or at least you could replace ulonglong binlog_xid with a one bit state, like thd->server_status & SERVER_STATUS_IN_DDL or with some other bitmap of flags. > + if (binlog_error) > + error= 1; > + thd->binlog_xid= 0; > thd->variables.option_bits= save_option_bits; > + debug_crash_here("ddl_log_rename_after_binlog"); > > if (likely(!binlog_error)) > my_ok(thd); > } > > if (likely(!error)) > + { > query_cache_invalidate3(thd, table_list, 0); > + ddl_log_complete(&ddl_log_state); > + } > + else > + { > + /* Revert the renames of normal tables with the help of the ddl log */ > + ddl_log_revert(thd, &ddl_log_state); > + } > > err: > DBUG_RETURN(error || binlog_error); > } > > > -/* > - reverse table list > - > - SYNOPSIS > - reverse_table_list() > - table_list pointer to table _list > - > - RETURN > - pointer to new (reversed) list > -*/ > -static TABLE_LIST *reverse_table_list(TABLE_LIST *table_list) > -{ > - TABLE_LIST *prev= 0; > - > - while (table_list) > - { > - TABLE_LIST *next= table_list->next_local; > - table_list->next_local= prev; > - prev= table_list; > - table_list= next; > - } > - return (prev); > -} > - > - > static bool > -do_rename_temporary(THD *thd, TABLE_LIST *ren_table, TABLE_LIST *new_table, > - bool skip_error) > +do_rename_temporary(THD *thd, TABLE_LIST *ren_table, TABLE_LIST *new_table) I wonder why we never got any warnings about it > { > LEX_CSTRING *new_alias; > DBUG_ENTER("do_rename_temporary"); > diff --git a/sql/ddl_log.cc b/sql/ddl_log.cc > index bad786b3799..777dfdddbc7 100644 > --- a/sql/ddl_log.cc > +++ b/sql/ddl_log.cc > @@ -22,176 +22,333 @@ > #include "log.h" // sql_print_error() > #include "ddl_log.h" > #include "ha_partition.h" // PAR_EXT > +#include "sql_table.h" // build_table_filename > +#include "sql_statistics.h" // rename_table_in_stats_tables > +#include "sql_view.h" // mysql_rename_view() > +#include "strfunc.h" // strconvert > +#include <mysys_err.h> // EE_LINK > > > /*-------------------------------------------------------------------------- > > MODULE: DDL log > ----------------- > > - This module is used to ensure that we can recover from crashes that occur > - in the middle of a meta-data operation in MySQL. E.g. DROP TABLE t1, t2; > - We need to ensure that both t1 and t2 are dropped and not only t1 and > - also that each table drop is entirely done and not "half-baked". > + This module is used to ensure that we can recover from crashes that > + occur in the middle of a meta-data operation in MySQL. E.g. DROP > + TABLE t1, t2; We need to ensure that both t1 and t2 are dropped and > + not only t1 and also that each table drop is entirely done and not > + "half-baked". > > - To support this we create log entries for each meta-data statement in the > - ddl log while we are executing. These entries are dropped when the > - operation is completed. > + To support this we create log entries for each meta-data statement > + in the ddl log while we are executing. These entries are dropped > + when the operation is completed. > > At recovery those entries that were not completed will be executed. > > There is only one ddl log in the system and it is protected by a mutex > and there is a global struct that contains information about its current > state. > > + DDl recovery after a crash works the following way: DDL or ddl > + > + - ddl_log_initialize() initializes the global global_ddl_log variable > + and opens the binary log if it exists. If it doesn't exists a new one > + is created. it doesn't seem to be opening a binary log. Did you mean ddl log? > + - ddl_log_close_binlogged_events() loops over all log events and checks if > + their xid (stored in the EXECUTE_CODE event) is in the binary log. If xid > + exists in the binary log the entry is marked as finished in the ddl log. > + - After a new binary log is created and is open for new entries, > + ddl_log_execute_recovery() is executed on remaining open events: > + - Loop over all events > + - For each entry with DDL_LOG_ENTRY_CODE execute the remaining phases > + in ddl_log_execute_entry_no_lock() > + > + The ddl_log.log file is created at startup and deleted when server goes down. > + After the final recovery phase is done, the file is truncated. > + > History: > First version written in 2006 by Mikael Ronstrom > - Second version written in 2020 by Monty > + Second version in 2020 by Monty > --------------------------------------------------------------------------*/ > > +#define DDL_LOG_MAGIC_LENGTH 4 > +/* How many times to try to execute a ddl log entry that causes crashes */ > +#define DDL_LOG_MAX_RETRY 3 > + > +uchar ddl_log_file_magic[]= > +{ (uchar) 254, (uchar) 254, (uchar) 11, (uchar) 2 }; > + > +/* Action names for ddl_log_action_code */ > + > +const char *ddl_log_action_name[DDL_LOG_LAST_ACTION]= > +{ > + "Unknown", "partitioning delete", "partitioning rename", > + "partitioning replace", "partitioning exchange", > + "rename table", "rename view" > +}; > + > +/* Number of phases per entry */ > +const uchar ddl_log_entry_phases[DDL_LOG_LAST_ACTION]= > +{ > + 1, 1, 2, 3, 4, 1 > +}; > + > + > struct st_global_ddl_log > { > - /* > - We need to adjust buffer size to be able to handle downgrades/upgrades > - where IO_SIZE has changed. We'll set the buffer size such that we can > - handle that the buffer size was upto 4 times bigger in the version > - that wrote the DDL log. > - */ > - char file_entry_buf[4*IO_SIZE]; > - char file_name_str[FN_REFLEN]; > - char *file_name; > + uchar *file_entry_buf; > DDL_LOG_MEMORY_ENTRY *first_free; > DDL_LOG_MEMORY_ENTRY *first_used; > File file_id; > - uint name_len; > uint num_entries; > + uint name_pos; > uint io_size; > - bool inited; > - bool do_release; > - bool recovery_phase; > - st_global_ddl_log() : inited(false), do_release(false) {} > + bool initialized; > + bool open; > }; > > st_global_ddl_log global_ddl_log; static? > > mysql_mutex_t LOCK_gdl; > > +/* Positions to different data in a ddl log block */ > #define DDL_LOG_ENTRY_TYPE_POS 0 > +/* > + Note that ACTION_TYPE and PHASE_POS must be after each other. > + See update_phase() > +*/ > #define DDL_LOG_ACTION_TYPE_POS 1 > #define DDL_LOG_PHASE_POS 2 > #define DDL_LOG_NEXT_ENTRY_POS 4 > -#define DDL_LOG_NAME_POS 8 > +/* Flags to remember something unique about the query, like if .frm was used */ > +#define DDL_LOG_FLAG_POS 8 > +/* Used to store XID entry that was written to binary log */ > +#define DDL_LOG_XID_POS 10 > +/* Used to store unique uuid from the .frm file */ "unique uuid" is a weird choice of words > +#define DDL_LOG_UUID_POS 18 > +/* ID_POS can be used to store something unique, like file size (4 bytes) */ > +#define DDL_LOG_ID_POS DDL_LOG_UUID_POS + MY_UUID_SIZE > +#define DDL_LOG_END_POS DDL_LOG_ID_POS + 8 > > -#define DDL_LOG_NUM_ENTRY_POS 0 > -#define DDL_LOG_NAME_LEN_POS 4 > -#define DDL_LOG_IO_SIZE_POS 8 > +/* > + Position to where names are stored in the ddl log blocks. The current > + value is stored in the header and can thus be changed if we need more > + space for constants in the header than what is between DDL_LOG_ID_POS and > + DDL_LOG_TMP_NAME_POS. > +*/ > +#define DDL_LOG_TMP_NAME_POS 56 > > +/* Definitions for the ddl log header, the first block in the file */ > +/* IO_SIZE is stored in the header and can thus be changed */ > +#define DDL_LOG_IO_SIZE IO_SIZE > + > +/* Header is stored in positions 0-3 */ > +#define DDL_LOG_IO_SIZE_POS 4 > +#define DDL_LOG_NAME_OFFSET_POS 6 > +/* Sum of the above variables */ > +#define DDL_LOG_HEADER_SIZE 4+2+2 > + > /** > + Sync the ddl log file. > + > + @return Operation status > + @retval FALSE Success > + @retval TRUE Error > +*/ > + > +static bool ddl_log_sync_file() > +{ > + DBUG_ENTER("ddl_log_sync_file"); > + DBUG_RETURN(mysql_file_sync(global_ddl_log.file_id, MYF(MY_WME))); > +} I don't quite understand that, are there cases when you want to sync the ddl log and you *don't know* whether LOCK_gdl is locked or not? because if you know there should be no lock, you'd better use mysql_mutex_assert_not_owner here. > + > +/* Same as above, but ensure we have the LOCK_gdb locked */ that's an interesting Freudian slip > + > +static bool ddl_log_sync_no_lock() > +{ > + DBUG_ENTER("ddl_log_sync_no_lock"); > + > + mysql_mutex_assert_owner(&LOCK_gdl); > + DBUG_RETURN(ddl_log_sync_file()); > +} ... > +/* > + Store and read strings in ddl log buffers > + > + Format is: > + 2 byte: length (not counting end \0) > + X byte: string value of length 'length' > + 1 byte: \0 a bit redundant, don't you think? Why did you do it? > +*/ > + > +static uchar *store_string(uchar *pos, uchar *end, const LEX_CSTRING *str) > { > + uint32 length= (uint32) str->length; > + if (unlikely(pos + 2 + length + 1 > end)) > + { > + DBUG_ASSERT(0); > + return end; // Overflow > } > + > + int2store(pos, length); > + if (likely(length)) > + memcpy(pos+2, str->str, length); > + pos[2+length]= 0; // Store end \0 > + return pos + 2 + length +1; > +} > + > + > +static LEX_CSTRING get_string(uchar **pos, const uchar *end) > { > + LEX_CSTRING tmp; > + uint32 length; > + if (likely(*pos + 3 <= end)) > + { > + length= uint2korr(*pos); > + if (likely(*pos + 2 + length + 1 <= end)) > + { > + char *str= (char*) *pos+2; > + *pos= *pos + 2 + length + 1; > + tmp.str= str; > + tmp.length= length; > + return tmp; > } > + } > + /* > + Overflow on read, should never happen > + Set *pos to end to ensure any future calls also returns empty string > + */ > + DBUG_ASSERT(0); > + *pos= (uchar*) end; > + tmp.str= ""; > + tmp.length= 0; > + return tmp; > } > > > @@ -256,38 +484,32 @@ static uint read_ddl_log_header() > > static void set_global_from_ddl_log_entry(const DDL_LOG_ENTRY *ddl_log_entry) it'd be good to have some comment explaining why you need this function. Couldn't you write directly from the ddl_log_entry? or use a local buffer? why to serialize into a buffer inside global_ddl_log? > { > + uchar *file_entry_buf= global_ddl_log.file_entry_buf, *pos, *end; > + > mysql_mutex_assert_owner(&LOCK_gdl); it would be logical to have LOCK_gdl inside the global_ddl_log struct > - global_ddl_log.file_entry_buf[DDL_LOG_ENTRY_TYPE_POS]= > - (char)DDL_LOG_ENTRY_CODE; > - global_ddl_log.file_entry_buf[DDL_LOG_ACTION_TYPE_POS]= > - (char)ddl_log_entry->action_type; > - global_ddl_log.file_entry_buf[DDL_LOG_PHASE_POS]= 0; > - int4store(&global_ddl_log.file_entry_buf[DDL_LOG_NEXT_ENTRY_POS], > - ddl_log_entry->next_entry); > - DBUG_ASSERT(strlen(ddl_log_entry->name) < FN_REFLEN); > - strmake(&global_ddl_log.file_entry_buf[DDL_LOG_NAME_POS], > - ddl_log_entry->name, FN_REFLEN - 1); > - if (ddl_log_entry->action_type == DDL_LOG_RENAME_ACTION || > - ddl_log_entry->action_type == DDL_LOG_REPLACE_ACTION || > - ddl_log_entry->action_type == DDL_LOG_EXCHANGE_ACTION) > - { > - DBUG_ASSERT(strlen(ddl_log_entry->from_name) < FN_REFLEN); > - strmake(&global_ddl_log.file_entry_buf[DDL_LOG_NAME_POS + FN_REFLEN], > - ddl_log_entry->from_name, FN_REFLEN - 1); > - } > - else > - global_ddl_log.file_entry_buf[DDL_LOG_NAME_POS + FN_REFLEN]= 0; > - DBUG_ASSERT(strlen(ddl_log_entry->handler_name) < FN_REFLEN); > - strmake(&global_ddl_log.file_entry_buf[DDL_LOG_NAME_POS + (2*FN_REFLEN)], > - ddl_log_entry->handler_name, FN_REFLEN - 1); > - if (ddl_log_entry->action_type == DDL_LOG_EXCHANGE_ACTION) > - { > - DBUG_ASSERT(strlen(ddl_log_entry->tmp_name) < FN_REFLEN); > - strmake(&global_ddl_log.file_entry_buf[DDL_LOG_NAME_POS + (3*FN_REFLEN)], > - ddl_log_entry->tmp_name, FN_REFLEN - 1); > - } > - else > - global_ddl_log.file_entry_buf[DDL_LOG_NAME_POS + (3*FN_REFLEN)]= 0; > + > + file_entry_buf[DDL_LOG_ENTRY_TYPE_POS]= (uchar) ddl_log_entry->entry_type; > + file_entry_buf[DDL_LOG_ACTION_TYPE_POS]= (uchar) ddl_log_entry->action_type; > + file_entry_buf[DDL_LOG_PHASE_POS]= (uchar) ddl_log_entry->phase; > + int4store(file_entry_buf+DDL_LOG_NEXT_ENTRY_POS, ddl_log_entry->next_entry); > + int2store(file_entry_buf+DDL_LOG_FLAG_POS, ddl_log_entry->flags); > + int8store(file_entry_buf+DDL_LOG_XID_POS, ddl_log_entry->xid); > + memcpy(file_entry_buf+DDL_LOG_UUID_POS, ddl_log_entry->uuid, MY_UUID_SIZE); > + int8store(file_entry_buf+DDL_LOG_ID_POS, ddl_log_entry->unique_id); > + bzero(file_entry_buf+DDL_LOG_END_POS, > + global_ddl_log.name_pos - DDL_LOG_END_POS); > + > + pos= file_entry_buf + global_ddl_log.name_pos; > + end= file_entry_buf + global_ddl_log.io_size; > + > + pos= store_string(pos, end, &ddl_log_entry->handler_name); > + pos= store_string(pos, end, &ddl_log_entry->db); > + pos= store_string(pos, end, &ddl_log_entry->name); > + pos= store_string(pos, end, &ddl_log_entry->from_handler_name); > + pos= store_string(pos, end, &ddl_log_entry->from_db); > + pos= store_string(pos, end, &ddl_log_entry->from_name); > + pos= store_string(pos, end, &ddl_log_entry->tmp_name); you have DBUG_ASSERT(0) that all those strings won't overflow io_size. How can you guarantee that? > + bzero(pos, global_ddl_log.io_size - (pos - file_entry_buf)); > } > > > @@ -442,63 +691,120 @@ static bool ddl_log_sync_no_lock() ... > + > + > +/* > + Build a filename for a table, trigger file or .frm > + Delete also any temporary file suffixed with ~ Where do you get these *~ files from? What creates them? > +*/ > + > +static void build_filename_and_delete_tmp_file(char *path, size_t path_length, > + const LEX_CSTRING *db, > + const LEX_CSTRING *name, > + const char *ext, > + PSI_file_key psi_key) > +{ > + uint length= build_table_filename(path, path_length-1, > + db->str, name->str, ext, 0); > + > + path[length]= '~'; > + path[length+1]= 0; > + (void) mysql_file_delete(psi_key, path, MYF(0)); > + path[length]= 0; > } > > > @@ -806,154 +1296,132 @@ static bool ddl_log_execute_entry_no_lock(THD *thd, uint first_entry) > bool ddl_log_write_entry(DDL_LOG_ENTRY *ddl_log_entry, > DDL_LOG_MEMORY_ENTRY **active_entry) > { > - bool error, write_header; > + bool error; > + uchar *pos, *end; > DBUG_ENTER("ddl_log_write_entry"); > > mysql_mutex_assert_owner(&LOCK_gdl); > - if (init_ddl_log()) > + if (!global_ddl_log.open) > DBUG_RETURN(TRUE); > + > + ddl_log_entry->entry_type= DDL_LOG_ENTRY_CODE; > set_global_from_ddl_log_entry(ddl_log_entry); > - if (get_free_ddl_log_entry(active_entry, &write_header)) > + if (ddl_log_get_free_entry(active_entry)) > DBUG_RETURN(TRUE); > + > error= FALSE; > + pos= global_ddl_log.file_entry_buf + global_ddl_log.name_pos; > + end= global_ddl_log.file_entry_buf + global_ddl_log.io_size; > DBUG_PRINT("ddl_log", > - ("write type %c next %u name '%s' from_name '%s' handler '%s'" > - " tmp_name '%s'", > + ("type: %c next: %u handler: %s " > + "to_name: '%s.%s' from_name: '%s.%s' " > + "tmp_name: '%s'", > (char) global_ddl_log.file_entry_buf[DDL_LOG_ACTION_TYPE_POS], > ddl_log_entry->next_entry, > - (char*) &global_ddl_log.file_entry_buf[DDL_LOG_NAME_POS], > - (char*) &global_ddl_log.file_entry_buf[DDL_LOG_NAME_POS > - + FN_REFLEN], > - (char*) &global_ddl_log.file_entry_buf[DDL_LOG_NAME_POS > - + (2*FN_REFLEN)], > - (char*) &global_ddl_log.file_entry_buf[DDL_LOG_NAME_POS > - + (3*FN_REFLEN)])); > + get_string(&pos, end).str, // Handler > + get_string(&pos, end).str, // to db.table > + get_string(&pos, end).str, > + get_string(&pos, end).str, // From db.table > + get_string(&pos, end).str, > + get_string(&pos, end).str)); // Tmp name This is not a critical bug, because it's in a DBUG_PRINT, but it's still a bug. C++ (and C) standard both say that the order of evaluation of function arguments is *unspecified*. > + > if (unlikely(write_ddl_log_file_entry((*active_entry)->entry_pos))) > { > + ddl_log_release_memory_entry(*active_entry); > + *active_entry= 0; > error= TRUE; > - sql_print_error("Failed to write entry_no = %u", > + sql_print_error("DDL_LOG: Failed to write entry %u", > (*active_entry)->entry_pos); > } > - if (write_header && likely(!error)) > - { > - (void) ddl_log_sync_no_lock(); > - if (write_ddl_log_header()) > - error= TRUE; > - } > - if (unlikely(error)) > - ddl_log_release_memory_entry(*active_entry); > DBUG_RETURN(error); > } > > > /** > - @brief Write final entry in the ddl log. > + @brief Write or update execute entry in the ddl log. > > - @details This is the last write in the ddl log. The previous log entries > - have already been written but not yet synched to disk. > - We write a couple of log entries that describes action to perform. > - This entries are set-up in a linked list, however only when a first > - execute entry is put as the first entry these will be executed. > - This routine writes this first. > + @details An execute entry points to the first entry that should > + be excuted during recovery. In some cases it's only written once, > + in other cases it's updated for each log entry to point to the new > + header for the list. > > + When called, the previous log entries have already been written but not yet > + synched to disk. We write a couple of log entries that describes > + action to perform. This entries are set-up in a linked list, > + however only when an execute entry is put as the first entry these will be > + executed during recovery. > + > @param first_entry First entry in linked list of entries > - to execute, if 0 = NULL it means that > - the entry is removed and the entries > - are put into the free list. > - @param complete Flag indicating we are simply writing > - info about that entry has been completed > + to execute. > @param[in,out] active_entry Entry to execute, 0 = NULL if the entry > is written first time and needs to be > returned. In this case the entry written > is returned in this parameter > - > @return Operation status > @retval TRUE Error > @retval FALSE Success > */ > > bool ddl_log_write_execute_entry(uint first_entry, > - bool complete, > DDL_LOG_MEMORY_ENTRY **active_entry) > { > - bool write_header= FALSE; > - char *file_entry_buf= (char*)global_ddl_log.file_entry_buf; > + uchar *file_entry_buf= global_ddl_log.file_entry_buf; > + bool got_free_entry= 0; > DBUG_ENTER("ddl_log_write_execute_entry"); > > mysql_mutex_assert_owner(&LOCK_gdl); > - if (init_ddl_log()) > - { > - DBUG_RETURN(TRUE); > - } > - if (!complete) > - { > /* > We haven't synched the log entries yet, we synch them now before > writing the execute entry. If complete is true we haven't written there's no 'complete' anymore > any log entries before, we are only here to write the execute > entry to indicate it is done. > */ > (void) ddl_log_sync_no_lock(); Why do you need to sync previous log entries? As far as I understand, they're useless without DDL_LOG_EXECUTE_CODE entry, so why would it matter whether they're on disk or not after a crash? > - file_entry_buf[DDL_LOG_ENTRY_TYPE_POS]= (char)DDL_LOG_EXECUTE_CODE; > - } > - else > - file_entry_buf[DDL_LOG_ENTRY_TYPE_POS]= (char)DDL_IGNORE_LOG_ENTRY_CODE; > - file_entry_buf[DDL_LOG_ACTION_TYPE_POS]= 0; /* Ignored for execute entries */ > - file_entry_buf[DDL_LOG_PHASE_POS]= 0; > - int4store(&file_entry_buf[DDL_LOG_NEXT_ENTRY_POS], first_entry); > - file_entry_buf[DDL_LOG_NAME_POS]= 0; > - file_entry_buf[DDL_LOG_NAME_POS + FN_REFLEN]= 0; > - file_entry_buf[DDL_LOG_NAME_POS + 2*FN_REFLEN]= 0; > + bzero(file_entry_buf, global_ddl_log.io_size); > + > + file_entry_buf[DDL_LOG_ENTRY_TYPE_POS]= (uchar)DDL_LOG_EXECUTE_CODE; > + int4store(file_entry_buf + DDL_LOG_NEXT_ENTRY_POS, first_entry); > + > if (!(*active_entry)) > { > - if (get_free_ddl_log_entry(active_entry, &write_header)) > - { > + if (ddl_log_get_free_entry(active_entry)) > DBUG_RETURN(TRUE); > + got_free_entry= TRUE; > } > - write_header= TRUE; > - } > if (write_ddl_log_file_entry((*active_entry)->entry_pos)) > { > - sql_print_error("Error writing execute entry in ddl log"); > - ddl_log_release_memory_entry(*active_entry); > - DBUG_RETURN(TRUE); > - } > - (void) ddl_log_sync_no_lock(); > - if (write_header) > + if (got_free_entry) > { > - if (write_ddl_log_header()) > - { > ddl_log_release_memory_entry(*active_entry); > + *active_entry= 0; > + } > + sql_print_error("DDL_LOG: Error writing execute entry %u", > + (*active_entry)->entry_pos); > DBUG_RETURN(TRUE); > } > - } > + (void) ddl_log_sync_no_lock(); > DBUG_RETURN(FALSE); > } > > - > /** > - Deactivate an individual entry. > + Increment phase for enty. Will deactivate entry after all phases are done > > @details see ddl_log_increment_phase_no_lock. > > - @param entry_no Entry position of record to change > + @param entry_pos Entry position of record to change > > @return Operation status > @retval TRUE Error > @retval FALSE Success > */ > > -bool ddl_log_increment_phase(uint entry_no) > +bool ddl_log_increment_phase(uint entry_pos) > { > bool error; > DBUG_ENTER("ddl_log_increment_phase"); > > mysql_mutex_lock(&LOCK_gdl); > - error= ddl_log_increment_phase_no_lock(entry_no); > + error= ddl_log_increment_phase_no_lock(entry_pos); > mysql_mutex_unlock(&LOCK_gdl); > DBUG_RETURN(error); > } > @@ -1042,87 +1484,157 @@ static void close_ddl_log() > (void) mysql_file_close(global_ddl_log.file_id, MYF(MY_WME)); > global_ddl_log.file_id= (File) -1; > } > + global_ddl_log.open= 0; > DBUG_VOID_RETURN; > } > > > /** > + Loop over ddl log excute entries and mark those that are already stored > + in the binary log as completed > + > + @return > + @retval 0 ok > + @return 1 fail (write error) > + > +*/ > + > +bool ddl_log_close_binlogged_events(HASH *xids) > +{ > + uint i; > + DDL_LOG_ENTRY ddl_log_entry; > + DBUG_ENTER("ddl_log_close_binlogged_events"); > + > + if (global_ddl_log.num_entries == 0 || xids->records == 0) > + DBUG_RETURN(0); > + > + mysql_mutex_lock(&LOCK_gdl); > + for (i= 1; i <= global_ddl_log.num_entries; i++) > + { > + if (read_ddl_log_entry(i, &ddl_log_entry)) > + break; // Read error. Ignore it's not really "ignore", it's more like "stop reading" > + if (ddl_log_entry.entry_type == DDL_LOG_EXECUTE_CODE && > + ddl_log_entry.xid != 0 && > + my_hash_search(xids, (uchar*) &ddl_log_entry.xid, > + sizeof(ddl_log_entry.xid))) > + { > + if (disable_execute_entry(i)) > + { > + mysql_mutex_unlock(&LOCK_gdl); > + DBUG_RETURN(1); // Write error. Fatal! > + } > + } > + } > + (void) ddl_log_sync_no_lock(); > + mysql_mutex_unlock(&LOCK_gdl); > + DBUG_RETURN(0); > +} > + > + > +/** > Execute the ddl log at recovery of MySQL Server. > + > + @return > + @retval 0 Ok. > + @retval > 0 Fatal error. We have to abort (can't create ddl log) > + @return < -1 Recovery failed, but new log exists and is usable > + > */ > > -void ddl_log_execute_recovery() > +int ddl_log_execute_recovery() > { > - uint num_entries, i; > + uint i, count= 0; > + int error= 0; > THD *thd; > DDL_LOG_ENTRY ddl_log_entry; > - char file_name[FN_REFLEN]; > static char recover_query_string[]= "INTERNAL DDL LOG RECOVER IN PROGRESS"; > DBUG_ENTER("ddl_log_execute_recovery"); > > - /* > - Initialise global_ddl_log struct > - */ > - bzero(global_ddl_log.file_entry_buf, sizeof(global_ddl_log.file_entry_buf)); > - global_ddl_log.inited= FALSE; > - global_ddl_log.recovery_phase= TRUE; > - global_ddl_log.io_size= IO_SIZE; > - global_ddl_log.file_id= (File) -1; > + if (global_ddl_log.num_entries == 0) > + DBUG_RETURN(0); > > /* > To be able to run this from boot, we allocate a temporary THD > */ > if (!(thd=new THD(0))) > - DBUG_VOID_RETURN; > + { > + DBUG_ASSERT(0); // Fatal error > + DBUG_RETURN(1); > + } > thd->thread_stack= (char*) &thd; > thd->store_globals(); > + thd->log_all_errors= (global_system_variables.log_warnings >= 3); > > thd->set_query(recover_query_string, strlen(recover_query_string)); > > - /* this also initialize LOCK_gdl */ > - num_entries= read_ddl_log_header(); > mysql_mutex_lock(&LOCK_gdl); > - for (i= 1; i < num_entries + 1; i++) > + for (i= 1; i <= global_ddl_log.num_entries; i++) > { > if (read_ddl_log_entry(i, &ddl_log_entry)) > { > - sql_print_error("Failed to read entry no = %u from ddl log", > - i); > + error= -1; > continue; > } > if (ddl_log_entry.entry_type == DDL_LOG_EXECUTE_CODE) > { > + /* purecov: begin tested */ really? you tested with purify? > + if (ddl_log_entry.unique_id > DDL_LOG_MAX_RETRY) > + { > + error= -1; > + continue; > + } > + update_unique_id(i, ++ddl_log_entry.unique_id); > + if (ddl_log_entry.unique_id > DDL_LOG_MAX_RETRY) > + { > + sql_print_error("DDL_LOG: Aborting executing entry %u after %llu " > + "retries", i, ddl_log_entry.unique_id); > + error= -1; > + continue; > + } > + /* purecov: end tested */ > if (ddl_log_execute_entry_no_lock(thd, ddl_log_entry.next_entry)) > { > - /* Real unpleasant scenario but we continue anyways. */ > + /* Real unpleasant scenario but we have to continue anyway */ why can this happen? do you have an example? besides hardware failure, like a disk cannot be read > + error= -1; > continue; > } > + count++; > } > } > close_ddl_log(); > - create_ddl_log_file_name(file_name); > - (void) mysql_file_delete(key_file_global_ddl_log, file_name, MYF(0)); > - global_ddl_log.recovery_phase= FALSE; > mysql_mutex_unlock(&LOCK_gdl); > thd->reset_query(); > delete thd; > - DBUG_VOID_RETURN; > + > + /* > + Create a new ddl_log to get rid of old stuff and ensure that header matches > + the current source version > + */ because here you delete/overwrite the old ddl log, it'd make sense to log all details of every entry that you failed to execute but decided to continue anyway. Now you log only the entry number on failure, which is rather useless, because it's a number in a ddl log that you overwrite. > + if (create_ddl_log()) > + DBUG_RETURN(1); > + if (count > 0) > + sql_print_information("DDL_LOG: Crash recovery executed %u entries", > + count); > + DBUG_RETURN(error); > } > > > /** > - Release all memory allocated to the ddl log. > + Release all memory allocated to the ddl log and delete the ddl log > */ > > void ddl_log_release() > { > + char file_name[FN_REFLEN]; > DDL_LOG_MEMORY_ENTRY *free_list; > DDL_LOG_MEMORY_ENTRY *used_list; > DBUG_ENTER("ddl_log_release"); > > - if (!global_ddl_log.do_release) > + if (!global_ddl_log.initialized) > DBUG_VOID_RETURN; > > - mysql_mutex_lock(&LOCK_gdl); > + global_ddl_log.initialized= 0; > + > free_list= global_ddl_log.first_free; > used_list= global_ddl_log.first_used; > while (used_list) Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] c80f867b0df: MDEV-23844 Atomic DROP TABLE
by Sergei Golubchik 16 May '21

16 May '21

Hi, Michael! On May 12, Michael Widenius wrote: > revision-id: c80f867b0df (mariadb-10.6.0-78-gc80f867b0df) > parent(s): eebebe8ef75 > author: Michael Widenius <michael.widenius(a)gmail.com> > committer: Michael Widenius <michael.widenius(a)gmail.com> > timestamp: 2021-05-04 21:09:11 +0300 > message: > > MDEV-23844 Atomic DROP TABLE ... > diff --git a/sql/sql_view.cc b/sql/sql_view.cc > index e6d726b30d7..ff8aa55adef 100644 > --- a/sql/sql_view.cc > +++ b/sql/sql_view.cc > @@ -1861,8 +1868,18 @@ bool mysql_drop_view(THD *thd, TABLE_LIST *views, enum_drop_mode drop_mode) > not_exists_count++; > continue; > } > + if (!first_table++) yuck. so your "first_table" actually means "not_first_view", because when it's FALSE, it's the first view. why not to rename it to "not_first_view" ? > + { > + if (ddl_log_drop_view_init(thd, &ddl_log_state)) > + DBUG_RETURN(TRUE); what about a crash here? the first entry is written, the second is not. > + } > + if (ddl_log_drop_view(thd, &ddl_log_state, &cpath, &view->db, > + &view->table_name)) > + DBUG_RETURN(TRUE); > + debug_crash_here("ddl_log_drop_before_delete_view"); > if (unlikely(mysql_file_delete(key_file_frm, path, MYF(MY_WME)))) > delete_error= TRUE; > + debug_crash_here("ddl_log_drop_after_delete_view"); > > some_views_deleted= TRUE; > > diff --git a/sql/log.cc b/sql/log.cc > index 6e0a9706ec8..dba999054e4 100644 > --- a/sql/log.cc > +++ b/sql/log.cc > @@ -10471,16 +10483,16 @@ int TC_LOG_BINLOG::recover(LOG_INFO *linfo, const char *last_log_name, > bool last_gtid_standalone= false; > bool last_gtid_valid= false; > #endif > + DBUG_ENTER("TC_LOG_BINLOG::recover"); > > if (! fdle->is_valid() || > - (do_xa && my_hash_init(key_memory_binlog_recover_exec, &xids, > - &my_charset_bin, TC_LOG_PAGE_SIZE/3, 0, > - sizeof(my_xid), 0, 0, MYF(0)))) > + (my_hash_init(key_memory_binlog_recover_exec, &xids, > + &my_charset_bin, TC_LOG_PAGE_SIZE/3, 0, > + sizeof(my_xid), 0, 0, MYF(0)))) > goto err1; > > - if (do_xa) > - init_alloc_root(key_memory_binlog_recover_exec, &mem_root, > - TC_LOG_PAGE_SIZE, TC_LOG_PAGE_SIZE, MYF(0)); > + init_alloc_root(key_memory_binlog_recover_exec, &mem_root, > + TC_LOG_PAGE_SIZE, TC_LOG_PAGE_SIZE, MYF(0)); I don't think these two changes are correct. do_xa is TRUE on crash recovery and FALSE on gtid state recovery on 5.5->10.0 upgade or if gtid state file was deleted. There is no point to initialize the xid hash or mem_root, as it's not a crash recovery and there can be no incomplete transactions or ddl statements. Yes, I've seen the commit comment: > - TC_LOG_BINLOG::recover() was changed to always collect Xid's from the > binary log and always call ddl_log_close_binlogged_events(). This was > needed to be able to collect DROP TABLE events with embedded Xid's > (used by ddl log). but I cannot understand how it applies here. > fdle->flags&= ~LOG_EVENT_BINLOG_IN_USE_F; // abort on the first error > > diff --git a/sql/sql_table.cc b/sql/sql_table.cc > index 7a88ffdfd1b..0a83396ae14 100644 > --- a/sql/sql_table.cc > +++ b/sql/sql_table.cc > @@ -1363,6 +1373,17 @@ int mysql_rm_table_no_locks(THD *thd, TABLE_LIST *tables, bool if_exists, > > thd->replication_flags= 0; > was_view= table_type == TABLE_TYPE_VIEW; > + > + if (!first_table++) first_table -> not_first_table > + { > + LEX_CSTRING comment= {comment_start, (size_t) comment_len}; > + if (ddl_log_drop_table_init(thd, &ddl_log_state, &comment)) > + { > + error= 1; > + goto err; > + } > + } > + > if ((table_type == TABLE_TYPE_UNKNOWN) || (was_view && !drop_view) || > (table_type != TABLE_TYPE_SEQUENCE && drop_sequence)) > { > @@ -1565,6 +1611,7 @@ int mysql_rm_table_no_locks(THD *thd, TABLE_LIST *tables, bool if_exists, > table_name.str, (uint)table_name.length); > mysql_audit_drop_table(thd, table); > } > + ddl_log_update_phase(&ddl_log_state, DDL_DROP_PHASE_COLLECT); better call it DDL_DROP_PHASE_BINLOG. It would've saved me ~10 minutes reading the code back and forth, as ""DDL_DROP_PHASE_BINLOG" immediately explains what this phase is for. > > if (!dont_log_query && > (!error || table_dropped || non_existing_table_error(error))) > diff --git a/sql/ddl_log.h b/sql/ddl_log.h > index 316e6708f22..b843404fc3c 100644 > --- a/sql/ddl_log.h > +++ b/sql/ddl_log.h > @@ -97,6 +101,14 @@ enum enum_ddl_log_rename_table_phase { > DDL_RENAME_PHASE_TABLE, > }; > > +enum enum_ddl_log_drop_table_phase { > + DDL_DROP_PHASE_TABLE=0, > + DDL_DROP_PHASE_TRIGGER, no phase for deleting data from stat tables? > + DDL_DROP_PHASE_COLLECT, > + DDL_DROP_PHASE_RESET, /* Reset found list of dropped tables */ this one is unused (also unused in further patches, I've checked) > + DDL_DROP_PHASE_END > +}; > + > /* > Setting ddl_log_entry.phase to this has the same effect as setting > action_type to DDL_IGNORE_LOG_ENTRY_CODE > diff --git a/sql/ddl_log.cc b/sql/ddl_log.cc > index 777dfdddbc7..6145b2f7318 100644 > --- a/sql/ddl_log.cc > +++ b/sql/ddl_log.cc > @@ -109,6 +112,7 @@ struct st_global_ddl_log > }; > > st_global_ddl_log global_ddl_log; > +String ddl_drop_query; // Used during startup recovery static > > mysql_mutex_t LOCK_gdl; > > @@ -1132,6 +1149,122 @@ static int ddl_log_execute_action(THD *thd, MEM_ROOT *mem_root, > (void) update_phase(entry_pos, DDL_LOG_FINAL_PHASE); > } > break; > + case DDL_LOG_DROP_TABLE_INIT_ACTION: > + { > + LEX_CSTRING *comment= &ddl_log_entry->tmp_name; > + ddl_drop_query.length(0); > + ddl_drop_query.set_charset(system_charset_info); > + ddl_drop_query.append(STRING_WITH_LEN("DROP TABLE IF EXISTS ")); > + if (comment->length) > + { > + ddl_drop_query.append(comment); > + ddl_drop_query.append(' '); > + } > + /* We don't increment phase as we want to retry this in case of crash */ > + break; > + } > + case DDL_LOG_DROP_TABLE_ACTION: > + { > + LEX_CSTRING db, table, path; > + db= ddl_log_entry->db; > + table= ddl_log_entry->name; > + /* Note that path is without .frm extension */ > + path= ddl_log_entry->tmp_name; > + > + switch (ddl_log_entry->phase) { > + case DDL_DROP_PHASE_TABLE: > + if (hton) > + { > + if ((error= hton->drop_table(hton, path.str))) > + { > + if (!non_existing_table_error(error)) > + break; > + error= -1; > + } > + } > + else > + error= ha_delete_table_force(thd, path.str, &db, &table); > + if (error <= 0) > + { > + /* Not found or already deleted. Delete .frm if it exists */ > + strxnmov(to_path, sizeof(to_path)-1, path.str, reg_ext, NullS); > + mysql_file_delete(key_file_frm, to_path, MYF(MY_WME|MY_IGNORE_ENOENT)); this should rather be inside the previous if(). no need to do it for ha_delete_table_force(), as it doesn't leave any traces and removes all remnants of a table. > + } > + if (ddl_log_increment_phase_no_lock(entry_pos)) > + break; > + (void) ddl_log_sync_no_lock(); No need to, ddl_log_increment_phase_no_lock() syncs internally > + /* Fall through */ > + case DDL_DROP_PHASE_TRIGGER: > + Table_triggers_list::drop_all_triggers(thd, &db, &table, > + MYF(MY_WME | MY_IGNORE_ENOENT)); > + if (ddl_log_increment_phase_no_lock(entry_pos)) > + break; > + (void) ddl_log_sync_no_lock(); same. and in other places too, I'd expect. > + /* Fall through */ > + > + case DDL_DROP_PHASE_COLLECT: indentation? > + append_identifier(thd, &ddl_drop_query, &db); > + ddl_drop_query.append('.'); > + append_identifier(thd, &ddl_drop_query, &table); > + ddl_drop_query.append(','); > + /* We don't increment phase as we want to retry this in case of crash */ > + > + if (!ddl_log_entry->next_entry && mysql_bin_log.is_open()) > + { > + /* Last drop table. Write query to binlog */ > + LEX_CSTRING end_comment= > + { STRING_WITH_LEN(" /* generated by ddl log */")}; "generated by ddl log" ? a "log" cannot "generate" a statement, may be "generated during ddl recovery" ? > + ddl_drop_query.length(ddl_drop_query.length()-1); > + ddl_drop_query.append(&end_comment); > + > + mysql_mutex_unlock(&LOCK_gdl); > + (void) thd->binlog_query(THD::STMT_QUERY_TYPE, ddl_drop_query.ptr(), > + ddl_drop_query.length(), TRUE, FALSE, > + FALSE, 0); > + mysql_mutex_lock(&LOCK_gdl); > + } > + break; > + } > + break; > + } > + case DDL_LOG_DROP_VIEW_INIT_ACTION: > + { > + ddl_drop_query.length(0); > + ddl_drop_query.set_charset(system_charset_info); > + ddl_drop_query.append(STRING_WITH_LEN("DROP VIEW IF EXISTS ")); > + /* We don't increment phase as we want to retry this in case of crash */ > + break; > + } > + case DDL_LOG_DROP_VIEW_ACTION: > + { > + LEX_CSTRING db, table, path; > + db= ddl_log_entry->db; > + table= ddl_log_entry->name; > + /* Note that for views path is WITH .frm extension */ > + path= ddl_log_entry->tmp_name; > + > + mysql_file_delete(key_file_frm, path.str, MYF(MY_WME|MY_IGNORE_ENOENT)); > + append_identifier(thd, &ddl_drop_query, &db); > + ddl_drop_query.append('.'); > + append_identifier(thd, &ddl_drop_query, &table); > + ddl_drop_query.append(','); > + > + if (!ddl_log_entry->next_entry) > + { > + /* Last drop view. Write query to binlog */ > + LEX_CSTRING end_comment= > + { STRING_WITH_LEN(" /* generated by ddl log */")}; > + ddl_drop_query.length(ddl_drop_query.length()-1); > + ddl_drop_query.append(&end_comment); > + > + mysql_mutex_unlock(&LOCK_gdl); > + (void) thd->binlog_query(THD::STMT_QUERY_TYPE, ddl_drop_query.ptr(), > + ddl_drop_query.length(), TRUE, FALSE, > + FALSE, 0); > + mysql_mutex_lock(&LOCK_gdl); > + } > + break; > + } > default: > DBUG_ASSERT(0); > break; > @@ -1682,6 +1823,7 @@ void ddl_log_release_entries(DDL_LOG_STATE *ddl_log_state) > next= log_entry->next_active_log_entry; > ddl_log_release_memory_entry(log_entry); > } > + ddl_log_state->list= 0; may be you should also clear ddl_log_state->execute_entry below? > > if (ddl_log_state->execute_entry) > ddl_log_release_memory_entry(ddl_log_state->execute_entry); > @@ -1776,6 +1918,33 @@ bool ddl_log_update_xid(DDL_LOG_STATE *state, ulonglong xid) > } > > > +/* > + Write ddl_log_entry and write or update ddl_execute_entry > +*/ > + > +static bool ddl_log_write(DDL_LOG_STATE *ddl_state, > + DDL_LOG_ENTRY *ddl_log_entry) could you use a more descriptive name, please? > +{ > + int error; > + DDL_LOG_MEMORY_ENTRY *log_entry; > + DBUG_ENTER("ddl_log_write"); > + > + mysql_mutex_lock(&LOCK_gdl); > + error= ((ddl_log_write_entry(ddl_log_entry, &log_entry)) || > + ddl_log_write_execute_entry(log_entry->entry_pos, > + &ddl_state->execute_entry)); > + mysql_mutex_unlock(&LOCK_gdl); > + if (error) > + { > + if (log_entry) > + ddl_log_release_memory_entry(log_entry); > + DBUG_RETURN(1); > + } > + add_log_entry(ddl_state, log_entry); > + DBUG_RETURN(0); > +} > + > + > /** > Logging of rename table > */ Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] 2be9b69f4ff: MDEV-23001 Precreate static Item_bool() to simplify code
by Sergei Golubchik 14 May '21

14 May '21

Hi, Monty! On Mar 25, Michael Widenius wrote: > revision-id: 2be9b69f4ff (mariadb-10.5.2-510-g2be9b69f4ff) > parent(s): cb545f11169 > author: Michael Widenius <michael.widenius(a)gmail.com> > committer: Michael Widenius <michael.widenius(a)gmail.com> > timestamp: 2021-03-24 14:05:15 +0200 > message: > > MDEV-23001 Precreate static Item_bool() to simplify code > > The following changes where done: > - Create global Item: Item_false and Item_true > - Replace all creation if 'FALSE' and 'TRUE' top level items used for > WHERE/HAVING/ON clauses to use Item_false and Item_true. > > The benefit are: > - No test needed if we where able to create the new item. > - Fixed possible errors if 'new' would have failed for the Item_bool's agree > - Less and faster code I don't quite agree with that, it's not less code: > 7 files changed, 65 insertions(+), 43 deletions(-) and I doubt that speedup can be measured in any benchmarks whatsoever. > diff --git a/sql/item.cc b/sql/item.cc > index 1a86b8b3114..0f160fa257b 100644 > --- a/sql/item.cc > +++ b/sql/item.cc > @@ -55,7 +55,8 @@ const char *item_empty_name=""; > const char *item_used_name= "\0"; > > static int save_field_in_field(Field *, bool *, Field *, bool); > - > +const Item_bool_static Item_false("FALSE", 0); > +const Item_bool_static Item_true("TRUE", 1); The main question - we have tons of modifiable Item members. How do you know none of them will modified in your Item_bool_static objects? > @@ -421,6 +421,9 @@ Item::Item(THD *thd): > /* Initially this item is not attached to any JOIN_TAB. */ > join_tab_idx= MAX_TABLES; > > + /* thd is NULL in case of static items like Item_true */ > + if (!thd) > + return; No, please, don't. thd can be NULL only twice, for statically initialized Item_true and Item_false. For that you removed a useful assert and added an if() that is completely unnecessary for billions of items created runtime. Better to have special contructor just for these two very special items. > /* Put item in free list so that we can free all items at end */ > next= thd->free_list; > thd->free_list= this; > @@ -3637,6 +3641,19 @@ void Item_int::print(String *str, enum_query_type query_type) > } > > > +/* > + This function is needed to ensure that Item_bool_static doesn't change > + the value of the member str_value. > +*/ > + > +void Item_bool::print(String *str, enum_query_type query_type) > +{ > + // my_charset_bin is good enough for numbers > + String tmp(value ? (char*) "1" : (char*) "0" , 1, &my_charset_bin); > + str->append(tmp); > +} This is not needed anymore. Item_bool inherits from Item_int, and Item_int::print is void Item_int::print(String *str, enum_query_type query_type) { StringBuffer<LONGLONG_BUFFER_SIZE> buf; // my_charset_bin is good enough for numbers buf.set_int(value, unsigned_flag, &my_charset_bin); str->append(buf); } You removed str_value from Item_int::print in July. > Item *Item_bool::neg_transformer(THD *thd) > { > value= !value; > diff --git a/sql/item.h b/sql/item.h > index a1c288ab1f0..1087c08869e 100644 > --- a/sql/item.h > +++ b/sql/item.h > @@ -1976,7 +1976,7 @@ class Item: public Value_source, > virtual bool limit_index_condition_pushdown_processor(void *arg) { return 0; } > virtual bool exists2in_processor(void *arg) { return 0; } > virtual bool find_selective_predicates_list_processor(void *arg) { return 0; } > - bool cleanup_is_expensive_cache_processor(void *arg) > + virtual bool cleanup_is_expensive_cache_processor(void *arg) You've added a new virtual method? First you try so hard to remove virtual methods, then you add a new one specifically for these two static items. Why does it matter what cleanup_is_expensive_cache_processor will do with them? You overwrite is_expensive() anyway. > { > is_expensive_cache= (int8)(-1); > return 0; > @@ -3220,9 +3220,11 @@ class Item_literal: public Item_basic_constant > Item_literal(THD *thd): Item_basic_constant(thd) > { } > Type type() const override { return CONST_ITEM; } > - bool check_partition_func_processor(void *) override { return false;} > + bool check_partition_func_processor(void *int_arg) override { return false;} > bool const_item() const override { return true; } > bool basic_const_item() const override { return true; } > + bool is_expensive() override { return false; } > + bool cleanup_is_expensive_cache_processor(void *arg) override { return 0; } > }; > > > diff --git a/sql/opt_index_cond_pushdown.cc b/sql/opt_index_cond_pushdown.cc > index 15bc2074e1f..e950ad1b7ca 100644 > --- a/sql/opt_index_cond_pushdown.cc > +++ b/sql/opt_index_cond_pushdown.cc > @@ -185,8 +185,8 @@ bool uses_index_fields_only(Item *item, TABLE *tbl, uint keyno, > static Item *make_cond_for_index(THD *thd, Item *cond, TABLE *table, uint keyno, > bool other_tbls_ok) > { > - if (!cond) > - return NULL; > + if (!cond || cond->basic_const_item()) > + return cond; The effect of this change is that you don't set cond->marker= ICP_COND_USES_INDEX_ONLY for basic_const_item's anymore. This looked suspicious at first. But then I noticed (and Sergey Petrunia confirmed) that ICP_COND_USES_INDEX_ONLY is not used at all. Since 5.3. So here I'd suggest to revert this your change and completely remove any ICP_COND_USES_INDEX_ONLY manipulations together with the ICP_COND_USES_INDEX_ONLY definition. > if (cond->type() == Item::COND_ITEM) > { > uint n_marked= 0; > diff --git a/sql/sql_parse.cc b/sql/sql_parse.cc > index 119c7360f07..e63e120a02b 100644 > --- a/sql/sql_parse.cc > +++ b/sql/sql_parse.cc > @@ -2395,6 +2395,7 @@ dispatch_command_return dispatch_command(enum enum_server_command command, THD * > thd->update_server_status(); > thd->protocol->end_statement(); > query_cache_end_of_result(thd); > + /* Check that the query didn't change const items */ what does that mean? Is it a TODO comment? Or it documents what happens below? > } > if (drop_more_results) > thd->server_status&= ~SERVER_MORE_RESULTS_EXISTS; > diff --git a/sql/sql_select.cc b/sql/sql_select.cc > index 93f5d3591ed..894d511adf4 100644 > --- a/sql/sql_select.cc > +++ b/sql/sql_select.cc > @@ -2364,7 +2364,7 @@ int JOIN::optimize_stage2() > if (!conds && outer_join) > { > /* Handle the case where we have an OUTER JOIN without a WHERE */ > - conds= new (thd->mem_root) Item_bool(thd, true); // Always true > + conds= (Item*) &Item_true; Hmm, does casting const away even work in the Intel compiler? > } > > if (impossible_where) > @@ -22743,6 +22743,8 @@ make_cond_for_table_from_pred(THD *thd, Item *root_cond, Item *cond, > return new_cond; > } > } > + else if (cond->basic_const_item()) > + return cond; > > if (is_top_and_level && used_table == rand_table_bit && > (cond->used_tables() & ~OUTER_REF_TABLE_BIT) != rand_table_bit) this is rather fragile, isn't it? You skip item->set_join_tab_idx(join_tab_idx_arg), quite understandably. And currently, as far as I can see, item->get_join_tab_idx() is always used under if (!item->const_item()), so what you did is safe. But can it change in the future? Or is join_tab_idx something that conceptually doesn't make sense for const items? Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 2

Re: [Maria-developers] 9083627c50f: Make rename atomic/repeatable in MyISAM and Aria
by Sergei Golubchik 14 May '21

14 May '21

Hi, Michael! On May 04, Michael Widenius wrote: > revision-id: 9083627c50f (mariadb-10.5.2-562-g9083627c50f) > parent(s): 105cda3b1cb > author: Michael Widenius <michael.widenius(a)gmail.com> > committer: Michael Widenius <michael.widenius(a)gmail.com> > timestamp: 2021-03-25 12:06:34 +0200 > message: > > Make rename atomic/repeatable in MyISAM and Aria > > This is required to make Atomic RENAME TABLE work for these engines > > The requirement is that if we have a server crash in the middle of a > storage engine rename call, the upcoming ddl log recovery should be able > to finalize it by re-execute the rename. > > diff --git a/storage/maria/ma_rename.c b/storage/maria/ma_rename.c > index a4388596f6b..64dac32a51c 100644 > --- a/storage/maria/ma_rename.c > +++ b/storage/maria/ma_rename.c > @@ -106,28 +106,37 @@ int maria_rename(const char *old_name, const char *new_name) > _ma_reset_state(info); > maria_close(info); > > + /* > + This code is written so that it should be possible to re-run a > + failed rename (even if there is a server crash in between the > + renames) and complete it. > + */ > fn_format(from,old_name,"",MARIA_NAME_IEXT,MY_UNPACK_FILENAME|MY_APPEND_EXT); > fn_format(to,new_name,"",MARIA_NAME_IEXT,MY_UNPACK_FILENAME|MY_APPEND_EXT); > if (mysql_file_rename_with_symlink(key_file_kfile, from, to, > MYF(MY_WME | sync_dir))) > - DBUG_RETURN(my_errno); > + index_file_rename_error= my_errno; Shouldn't you continue only if index_file_rename_error == ENOENT ? also, I only very quickly looked at mysql_file_rename_with_symlink, but it didn't look atomic to me when symlinks are present. > fn_format(from,old_name,"",MARIA_NAME_DEXT,MY_UNPACK_FILENAME|MY_APPEND_EXT); > fn_format(to,new_name,"",MARIA_NAME_DEXT,MY_UNPACK_FILENAME|MY_APPEND_EXT); ... > diff --git a/storage/myisam/mi_rename.c b/storage/myisam/mi_rename.c > index 19df2e54213..13cf8c60897 100644 > --- a/storage/myisam/mi_rename.c > +++ b/storage/myisam/mi_rename.c > @@ -22,6 +22,7 @@ > int mi_rename(const char *old_name, const char *new_name) > { > char from[FN_REFLEN],to[FN_REFLEN]; > + int save_errno= 0; > DBUG_ENTER("mi_rename"); > > #ifdef EXTRA_DEBUG > @@ -32,10 +33,13 @@ int mi_rename(const char *old_name, const char *new_name) > fn_format(from,old_name,"",MI_NAME_IEXT,MY_UNPACK_FILENAME|MY_APPEND_EXT); > fn_format(to,new_name,"",MI_NAME_IEXT,MY_UNPACK_FILENAME|MY_APPEND_EXT); > if (mysql_file_rename_with_symlink(mi_key_file_kfile, from, to, MYF(MY_WME))) > - DBUG_RETURN(my_errno); > + save_errno= my_errno; > fn_format(from,old_name,"",MI_NAME_DEXT,MY_UNPACK_FILENAME|MY_APPEND_EXT); > fn_format(to,new_name,"",MI_NAME_DEXT,MY_UNPACK_FILENAME|MY_APPEND_EXT); > - DBUG_RETURN(mysql_file_rename_with_symlink(mi_key_file_dfile, > - from, to, > - MYF(MY_WME)) ? my_errno : 0); > + if (mysql_file_rename_with_symlink(mi_key_file_dfile, > + from, to, > + MYF(MY_WME))) > + if (save_errno) > + save_errno= my_errno; > + DBUG_RETURN(save_errno); Why mi_rename isn't trying to undo a half-failed rename like ma_rename does? Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] 32c73138ff4: Remove some usage of Check_level_instant_set and Sql_mode_save
by Sergei Golubchik 14 May '21

14 May '21

Hi, Monty! On Apr 19, Michael Widenius wrote: > revision-id: 32c73138ff4 (mariadb-10.5.2-559-g32c73138ff4) > parent(s): 07708eb9b28 > author: Michael Widenius <michael.widenius(a)gmail.com> > committer: Michael Widenius <michael.widenius(a)gmail.com> > timestamp: 2021-03-25 12:06:34 +0200 > message: > > Remove some usage of Check_level_instant_set and Sql_mode_save > > The reason for the removal are: > - Generates more code > - Storing and retreving THD > - Causes extra code and daata to be generated to handle possible throw > exceptions (which never happens in MariaDB code) > - Uses more stack space No, Monty. We've discussed it already, this makes code more complex, fragile and bug prone. And it does *not* generates extra code or data or stack - before writing this email I've compiled mariadbd both ways and compared the generated code for the function in question, it was *identical*. > Other things: > - Changed convert_const_to_int() to use item->save_in_field_no_warnings(), > which made the code shorter and simpler. > - Added thd as argument to store_key.copy() to make function simpler > - Added thd as argument to some subselect* constructor that inherites > from Item_subselect. these changes are good, thanks Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] 6b06a99e4a4: Avoid creating the .frm file twice in some cases
by Sergei Golubchik 14 May '21

14 May '21

Hi, Monty! On Apr 19, Michael Widenius wrote: > revision-id: 6b06a99e4a4 (mariadb-10.5.2-557-g6b06a99e4a4) > parent(s): 9c1e36696f3 > author: Michael Widenius <michael.widenius(a)gmail.com> > committer: Michael Widenius <michael.widenius(a)gmail.com> > timestamp: 2021-03-25 12:06:34 +0200 > message: > > Avoid creating the .frm file twice in some cases > > diff --git a/sql/handler.cc b/sql/handler.cc > index c286aef3a9e..fe2fe6e9426 100644 > --- a/sql/handler.cc > +++ b/sql/handler.cc > @@ -5536,8 +5536,9 @@ int ha_create_table(THD *thd, const char *path, > > if (frm) > { > - bool write_frm_now= !create_info->db_type->discover_table && > - !create_info->tmp_table(); > + bool write_frm_now= (!create_info->db_type->discover_table && > + !create_info->tmp_table() && > + !create_info->frm_is_created); Is this the real problem? frm created twice? Why ha_create_table() is invoked twice at all? > > share.frm_image= frm; > > diff --git a/sql/sql_table.cc b/sql/sql_table.cc > index e00d3ee1ed2..73642a5bd97 100644 > --- a/sql/sql_table.cc > +++ b/sql/sql_table.cc > @@ -547,8 +547,13 @@ uint build_table_filename(char *buff, size_t bufflen, const char *db, > > (void) tablename_to_filename(db, dbbuff, sizeof(dbbuff)); > > - /* Check if this is a temporary table name. Allow it if a corresponding .frm file exists */ > - if (is_prefix(table_name, tmp_file_prefix) && strlen(table_name) < NAME_CHAR_LEN && > + /* > + Check if this is a temporary table name. Allow it if a corresponding .frm > + file exists. > + */ > + if (!(flags & FN_IS_TMP) && > + is_prefix(table_name, tmp_file_prefix) && > + strlen(table_name) < NAME_CHAR_LEN && good idea! > check_if_frm_exists(tbbuff, dbbuff, table_name)) > flags|= FN_IS_TMP; > > @@ -9817,7 +9827,7 @@ do_continue:; > > if (table->s->tmp_table != NO_TMP_TABLE) > { > - /* Close lock if this is a transactional table */ > + /* Unlock lock if this is a transactional temporary table */ strictly speaking, you don't "unlock lock", you "unlock a resource" (e.g. a table) or you "release a lock" > if (thd->lock) > { > if (thd->locked_tables_mode != LTM_LOCK_TABLES && Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] 9c1e36696f3: MDEV-20025: ADD_MONTHS() Oracle function
by Sergei Golubchik 14 May '21

14 May '21

Hi, Monty! On Apr 16, Michael Widenius wrote: > commit 9c1e36696f3 > Author: Michael Widenius <michael.widenius(a)gmail.com> > Date: Fri Feb 5 17:10:43 2021 +0200 > > diff --git a/mysql-test/suite/compat/oracle/r/func_add_months.result b/mysql-test/suite/compat/oracle/r/func_add_months.result > new file mode 100644 > index 00000000000..0e8ca1eef94 > --- /dev/null > +++ b/mysql-test/suite/compat/oracle/r/func_add_months.result > @@ -0,0 +1,85 @@ > +Test for ADD_MONTHS > +CREATE TABLE t1(c1 int, c2 datetime, c3 date, c4 time, c5 timestamp); > +INSERT INTO t1 VALUES (1, '2011-11-12 12:10:11', '2011-11-12', '12:10:11', '2011-11-12 12:10:11'); > +INSERT INTO t1 VALUES (2, '2021-11-12 00:23:12', '2021-11-12', '00:23:12', '2021-11-12 00:23:12'); > +INSERT INTO t1 VALUES (3, '2011-01-22 16:45:45', '2011-01-22', '16:45:45', '2011-01-22 16:45:45'); > +INSERT INTO t1 VALUES (4, '2031-05-12 04:11:34', '2031-05-12', '04:11:34', '2031-05-12 04:11:34'); > +INSERT INTO t1 VALUES (5, '2031-09-02 08:15:22', '2031-09-02', '08:15:22', '2031-09-02 08:15:22'); > +INSERT INTO t1 VALUES (6, '0000-09-02 00:00:00', '0000-09-02', '00:00:00', '1980-09-02 00:00:00'); > +INSERT INTO t1 VALUES (7, '9999-09-02', '9999-09-02', '00:00:00', '1980-09-02'); there's no last-day test here. Like ADD_MONTHS('2012-01-31', 1) = '2012-02-29' should be here, as Oracle manual specifically documents this case. Regards, Sergei

2 1

Re: [Maria-developers] fbe1dad15a1: Ensure that we do not allocate strings bigger than 4G in String objects.
by Sergei Golubchik 14 May '21

14 May '21

Hi, Monty! On Apr 16, Michael Widenius wrote: > revision-id: fbe1dad15a1 (mariadb-10.5.2-555-gfbe1dad15a1) > parent(s): 57c19902326 > author: Michael Widenius <michael.widenius(a)gmail.com> > committer: Michael Widenius <michael.widenius(a)gmail.com> > timestamp: 2021-03-25 12:06:34 +0200 > message: > > Ensure that we do not allocate strings bigger than 4G in String objects. > > This is needed as we are using uint32 for allocated and current length. > > diff --git a/sql/sql_string.cc b/sql/sql_string.cc > index 9c57bb22085..fedf5f4a48a 100644 > --- a/sql/sql_string.cc > +++ b/sql/sql_string.cc > @@ -37,6 +37,8 @@ bool Binary_string::real_alloc(size_t length) > DBUG_ASSERT(arg_length > length); > if (arg_length <= length) > return TRUE; /* Overflow */ > + if (arg_length >= UINT_MAX32) > + return FALSE; > str_length=0; > if (Alloced_length < arg_length) > { > @@ -45,7 +47,6 @@ bool Binary_string::real_alloc(size_t length) > arg_length,MYF(MY_WME | (thread_specific ? > MY_THREAD_SPECIFIC : 0))))) > return TRUE; > - DBUG_ASSERT(length < UINT_MAX32); > Alloced_length=(uint32) arg_length; > alloced=1; I liked assert more. It meant that we should never under any circumstances request more than 4G from Binary_string::real_alloc. You're saying that it's ok to do it and Binary_string::real_alloc should be ready to deal with it. I'd say the first approach was better, >4G should just never happen. Regards, Sergei

2 1

Re: [Maria-developers] c157c285db9: Optimize Sql_alloc
by Sergei Golubchik 14 May '21

14 May '21

Hi, Michael! On Mar 27, Michael Widenius wrote: > revision-id: c157c285db9 (mariadb-10.5.2-514-gc157c285db9) > parent(s): 16e38888c06 > author: Michael Widenius <michael.widenius(a)gmail.com> > committer: Michael Widenius <michael.widenius(a)gmail.com> > timestamp: 2021-03-24 14:31:53 +0200 > message: > > Optimize Sql_alloc > > - Remove 'dummy_for_valgrind' overrun marker as this doesn't help much. > The element also distorts the sizes of objects a bit, which makes it > harder to calculate gain in object sizes when doing size optimizations. > - Avoid one extra call indirection when using thd_get_current_thd(), which > is used by Sql_alloc. > > diff --git a/sql/mysqld.cc b/sql/mysqld.cc > index 4f522d96a9f..1ab52401705 100644 > --- a/sql/mysqld.cc > +++ b/sql/mysqld.cc > @@ -662,7 +662,15 @@ static std::atomic<char*> shutdown_user; > > static thread_local THD *THR_THD; > > +/** > + Get current THD object from thread local data > + > + @retval The THD object for the thread, NULL if not connection thread > +*/ > + > MYSQL_THD _current_thd() { return THR_THD; } > +THD *thd_get_current_thd() { return THR_THD; } I'd rather remove thd_get_current_thd() completely, why do we need two identical functions? Sql_alloc can use _current_thd() just fine, I've tried (also rocksdb uses thd_get_current_thd in one place, but it can use current_thd like the rest of the server code). Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] 21eb8969ce9: Improved storage size for Item, Field and some other classes
by Sergei Golubchik 14 May '21

14 May '21

Hi, Monty! Looks ok, but again it doesn't seem you've squashed intermediate commits as you said you did. Bit fields and non-existent commits in columnstore - it's is clearly an intermediate work-in-progress state, all fixed in your later commits. On Sep 08, Michael Widenius wrote: > revision-id: 21eb8969ce9 (mariadb-10.5.2-270-g21eb8969ce9) > parent(s): c3ecf0d6243 > author: Michael Widenius <michael.widenius(a)gmail.com> > committer: Michael Widenius <michael.widenius(a)gmail.com> > timestamp: 2020-09-02 20:58:34 +0300 > message: > > Improved storage size for Item, Field and some other classes > > - Changed order of class fields to remove dead alignment space. > - Changed bool fields in Item to bit fields. > - Used packed enum's for some fields in common classes > - Removed not used Item::rsize. > - Changed some class variables from uint/int to smaller type int's. > - Ensured that field_index is uint16 in all classes and functions. Fixed > also that we proparly compare with NO_CACHED_FIELD_INDEX when checking > if variable is not set. > - Removed checking of highest bit of unireg_check (has not been used in > a long time) > - Fixed wrong arguments to make_cond_for_table() for join_tab_idx_arg > from false to 0. > > One of the result was reducing the size if class Item with ~24 bytes ... > @@ -929,18 +904,48 @@ class Item: public Value_source, > + > + /* > + str_values's main purpose is to be used to cache the value in > + save_in_field > + */ > + String str_value; > + > + LEX_CSTRING name; /* Name of item */ > + /* Original item name (if it was renamed)*/ > + const char *orig_name; > + > + uint32 /* All common bool variables for Item stored here */ > + maybe_null:1, /* If item may be null */ > + in_rollup:1, /* If used in GROUP BY list of a query with ROLLUP */ > + with_param:1, /* True if Item contains an SP parameter */ > + with_window_func:1, /* True if item contains a window func */ > + with_field:1, /* True if any item except Item_sum contains a field. > + Set during parsing. */ > + fixed:1; /* If item was fixed with fix_fields */ > + > + int16 marker; > + > diff --git a/storage/columnstore/columnstore b/storage/columnstore/columnstore > index b6b02ed516f..f606e76fb77 160000 > --- a/storage/columnstore/columnstore > +++ b/storage/columnstore/columnstore > @@ -1 +1 @@ > -Subproject commit b6b02ed516f92055127d416370799d91a82754ea > +Subproject commit f606e76fb779e40f3376693fff9969e4f2b7669a Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 3

Re: [Maria-developers] 4088269c578d: Improved storage size for Item, Field and some other classes
by Sergei Golubchik 14 May '21

14 May '21

Hi, Michael! On Dec 03, Michael Widenius wrote: > revision-id: 4088269c578d (mariadb-10.5.2-270-g4088269c578d) > parent(s): 994ea2af3973 > author: Michael Widenius <michael.widenius(a)gmail.com> > committer: Michael Widenius <michael.widenius(a)gmail.com> > timestamp: 2020-09-17 12:25:44 +0300 > message: > > Improved storage size for Item, Field and some other classes > > diff --git a/storage/columnstore/columnstore b/storage/columnstore/columnstore > index b6b02ed516f9..f606e76fb779 160000 > --- a/storage/columnstore/columnstore > +++ b/storage/columnstore/columnstore > @@ -1 +1 @@ > -Subproject commit b6b02ed516f92055127d416370799d91a82754ea > +Subproject commit f606e76fb779e40f3376693fff9969e4f2b7669a Again. You should not push invalid commit hashes into the main branch: storage/columnstore/columnstore $ git fetch origin storage/columnstore/columnstore $ git show f606e76fb779e40f3376693fff9969e4f2b7669a fatal: bad object f606e76fb779e40f3376693fff9969e4f2b7669a Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] 16e38888c06: Improved storage size for Item, Field and some other classes
by Sergei Golubchik 14 May '21

14 May '21

Hi, Michael! Did you consider trying -Wpadded? '-Wpadded' Warn if padding is included in a structure, either to align an element of the structure or to align the whole structure. Sometimes when this happens it is possible to rearrange the fields of the structure to reduce the padding and so make the structure smaller. Might reveal more useful cases where fields could be rearranged. On Mar 27, Michael Widenius wrote: > revision-id: 16e38888c06 (mariadb-10.5.2-513-g16e38888c06) > parent(s): 4cfb680b685 > author: Michael Widenius <michael.widenius(a)gmail.com> > committer: Michael Widenius <michael.widenius(a)gmail.com> > timestamp: 2021-03-24 14:31:53 +0200 > message: > > Improved storage size for Item, Field and some other classes > > - Changed order of class fields to remove dead alignment space. > - Changed bool fields in Item to bit fields. > - Used packed enum's for some fields in common classes > - Removed not used Item::rsize. > - Changed some class variables from uint/int to smaller type int's. > - Ensured that field_index is uint16 in all classes and functions. Fixed > also that we proparly compare with NO_CACHED_FIELD_INDEX when checking > if variable is not set. > - Removed checking of highest bit of unireg_check (has not been used in > a long time) > - Fixed wrong arguments to make_cond_for_table() for join_tab_idx_arg > from false to 0. > > One of the result was reducing the size if class Item with ~24 bytes Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] 094ae6bbcc0: MDEV-24395 Atomic DROP TRIGGER
by Sergei Golubchik 13 May '21

13 May '21

Hi, Monty! On May 13, Michael Widenius wrote: > revision-id: 094ae6bbcc0 (mariadb-10.6.0-79-g094ae6bbcc0) > parent(s): c80f867b0df > author: Michael Widenius <michael.widenius(a)gmail.com> > committer: Michael Widenius <michael.widenius(a)gmail.com> > timestamp: 2021-05-04 21:09:11 +0300 > message: > > MDEV-24395 Atomic DROP TRIGGER > diff --git a/sql/ddl_log.cc b/sql/ddl_log.cc > index 6145b2f7318..cf65f8ad876 100644 > --- a/sql/ddl_log.cc > +++ b/sql/ddl_log.cc > @@ -1265,6 +1290,70 @@ static int ddl_log_execute_action(THD *thd, MEM_ROOT *mem_root, > } > break; > } > + case DDL_LOG_DROP_TRIGGER_ACTION: > + { > + MY_STAT stat_info; > + off_t frm_length= 1; // Impossible length > + LEX_CSTRING thd_db= thd->db; > + > + /* Delete trigger temporary file if it still exists */ > + if (!build_filename_and_delete_tmp_file(to_path, sizeof(to_path) - 1, > + &ddl_log_entry->db, > + &ddl_log_entry->name, > + TRG_EXT, > + key_file_fileparser)) > + { > + /* Temporary file existed and was deleted, nothing left to do */ you didn't answer my question in one of the previous reviews, what is a temporary file *~ ? What creates it and where? > + (void) update_phase(entry_pos, DDL_LOG_FINAL_PHASE); > + break; > + } > + /* > + We can use length of TRG file as an indication if trigger was removed. > + If there is no file, then it means that this was the last trigger > + and the file was removed. > + */ > + if (my_stat(to_path, &stat_info, MYF(0))) > + frm_length= (off_t) stat_info.st_size; > + if (frm_length != (off_t) ddl_log_entry->unique_id && > + mysql_bin_log.is_open()) > + { > + /* > + File size changed and it was not binlogged (as this entry was > + executed) > + */ > + (void) rm_trigname_file(to_path, &ddl_log_entry->db, > + &ddl_log_entry->from_name, > + MYF(0)); > + > + ddl_drop_query.length(0); > + ddl_drop_query.set_charset(system_charset_info); > + if (ddl_log_entry->tmp_name.length) I'm not sure it's a good idea. It'll have some rather confusing conditional behavior when sometimes a query is logged with the original user's comment. And adding a couple of# characters to the comment makes the comment to disappear because the query will be replaced with a generated one. It might've been more predictable and less confusing, if the query would be always generated. After all, it's only for the ddl recovery case, it's reasonable to see a generated query for redo actions that happen during the crash recovery. > + { > + /* We can use the original query */ > + ddl_drop_query.append(&ddl_log_entry->tmp_name); > + } > + else > + { > + /* Generate new query */ > + ddl_drop_query.append(STRING_WITH_LEN("DROP TRIGGER IF EXISTS ")); > + append_identifier(thd, &ddl_drop_query, &ddl_log_entry->from_name); > + ddl_drop_query.append(&end_comment); > + } > + if (mysql_bin_log.is_open()) > + { > + mysql_mutex_unlock(&LOCK_gdl); > + thd->db= ddl_log_entry->db; > + (void) thd->binlog_query(THD::STMT_QUERY_TYPE, > + ddl_drop_query.ptr(), > + ddl_drop_query.length(), TRUE, FALSE, > + FALSE, 0); > + thd->db= thd_db; > + mysql_mutex_lock(&LOCK_gdl); > + } > + } > + (void) update_phase(entry_pos, DDL_LOG_FINAL_PHASE); > + break; > + } > default: > DBUG_ASSERT(0); > break; Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] c76ac1e6de8: MDEV-24089 support oracle syntax: rownum
by Sergei Golubchik 13 May '21

13 May '21

Hi, Michael! On Apr 12, Michael Widenius wrote: > revision-id: c76ac1e6de8 (mariadb-10.5.2-551-gc76ac1e6de8) > parent(s): a507eb17708 > author: Michael Widenius <michael.widenius(a)gmail.com> > committer: Michael Widenius <michael.widenius(a)gmail.com> > timestamp: 2021-03-24 19:42:24 +0200 > message: > diff --git a/mysql-test/main/rownum.test b/mysql-test/main/rownum.test > new file mode 100644 > index 00000000000..8d89eed2f86 > --- /dev/null > +++ b/mysql-test/main/rownum.test > @@ -0,0 +1,542 @@ > +--source include/have_sequence.inc > +# > +# Test of basic rownum() functionallity > +# Test are done with Aria to ensure that row order is stable what does that mean? > +# > + > +CREATE OR REPLACE TABLE t1(a int, b int) engine=aria; > +CREATE OR REPLACE TABLE t2(a int, b int) engine=aria; > +insert into t1 values (1,10),(2,20),(3,30); > +insert into t2 values (2,21),(3,31),(4,41); > + > +--echo # > +--echo # Simple selects > +--echo # > + > +select a,b,rownum() from t1; > +select a,b,rownum() from t1 where rownum() < 2; > +select a,b from t1 where rownum() <= 2; > +select a,b from t1 where rownum() > 2; > + > +--echo # > +--echo # Subqueries > +--echo # > + > +select t1.a,rownum(),t3.a,t3.t2_rownum from t1, (select t2.a,rownum() as t2_rownum from t2 where rownum() <=2) t3; > +select t1.a, (select t2.b from t2 where t1.a=t2.a and rownum() <= 1) 'b' from t1; > +select t1.a, t3.a from t1, (select * from t2 where rownum() <= 2) t3; > +select * from (select tt.*,rownum() as id from (select * from t1) tt) t3 where id>2; > + > +--echo # > +--echo # Joins > +--echo # > + > +select t1.a,t1.b,t2.a,t2.b,rownum() from t1,t2 where rownum() <= 4; > +select *,rownum() from t1,t2 where t1.a=t2.a and rownum()<=2; > +select * from t1 left join t2 on (t2.a=t1.a and rownum()=0); > +select * from t1 left join t2 on (t2.a=t1.a and rownum()>1); > +select * from t1 left join t2 on (t2.a=t1.a and rownum()<1); > +select * from t1 left join t2 on (t2.a=t1.a and rownum()=1); > +select * from t1 left join t2 on (t2.a=t1.a and rownum()>=1); > + > +--echo # > +--echo # Union > +--echo # > + > +select * from t1 where rownum() <=2 union select * from t2 where rownum()<= 1; > + > +--echo # > +--echo # Order by > +--echo # > + > +select * from t1 where rownum() <= 2 order by a desc; > +explain select * from t1 where rownum() <= 2 order by a desc; > +select t1.a,t1.b,rownum() from t1 where rownum() <= 2 order by a desc; > +explain select t1.a,t1.b,rownum() from t1 where rownum() <= 2 order by a desc; > +select *,rownum() from t1,t2; > +select *,rownum() from t1,t2 order by t2.a desc, t1.a desc; > +select * from (select * from t1 order by a desc) as t where rownum() <= 2; > +select * from t1,t2 where t1.a=t2.a and rownum()<=2 order by t1.a,t2.a; > + > +--echo # > +--echo # Having > +--echo # > + > +select t1.a, sum(t1.b), rownum() as 'r' from t1 group by t1.a having r <= 2; > +--error ER_FUNCTION_CANNOT_BE_USED_IN_CLAUSE > +select * from t1 having rownum() <= 2; > +select t1.a, sum(t1.b), rownum() from t1 group by t1.a; > +--error ER_FUNCTION_CANNOT_BE_USED_IN_CLAUSE > +select t1.a, sum(t1.b), rownum() as 'r' from t1 group by t1.a having rownum() <= 2; > + > +--echo # > +--echo # Sum functions > +--echo # > + > +select max(rownum()),min(rownum()) from t1; > +select sum(rownum()),avg(rownum()) from t1; > + > +--echo # > +--echo # Group by > +--echo # > + > +select t1.a,sum(t1.b) from t1 where rownum() < 2 group by t1.a; > +select t1.a,sum(t2.b) from t1 JOIN t2 ON (t1.a=t2.a) where rownum() <= 2 group by t1.a; > +select * from (select t1.a,sum(t2.b) from t1 JOIN t2 ON (t1.a=t2.a) group by t1.a) as t where rownum() <= 1; > +select t1.a,sum(rownum()),count(*) from t1 where rownum() <= 2 group by t1.a; > +select * from (select t1.a,sum(t1.b) from t1 group by t1.a) as t3 where rownum() < 2; > + > +create table t3 (a int) engine=myisam; > +insert into t3 values (3),(5),(5),(3); > +select a, max(rownum()) from t3 group by a; > +drop table t3; > + > +CREATE TABLE t3 ( > + a int(11) DEFAULT NULL, > + b varchar(1024) DEFAULT NULL > +); > +insert into t3 select mod(seq*3,20)+1, repeat(char(33+mod(seq,90)),mod(seq,10)*100) from seq_1_to_23; > +SELECT sq.a,length(sq.f) FROM (SELECT a, GROUP_CONCAT(b,b) AS f FROM t3 GROUP BY a ORDER BY a desc) as sq WHERE ROWNUM() <= 10; > +drop table t3; > + > +--echo # > +--echo # Prepared statements > +--echo # > + > +PREPARE stmt1 from "select a,b,rownum() from t1 where rownum() <= 2"; > +execute stmt1; > +execute stmt1; > +deallocate prepare stmt1; > + > +--echo # > +--echo # Views > +--echo # > + > +create view v1 as select t1.a,rownum() from t1; > +select * from v1; > +select t1.a,v1.* from t1,v1 where t1.a=v1.a; > +drop view v1; > + > +CREATE TABLE t3 (a INT); > +INSERT INTO t3 VALUES (1),(2),(3); > +CREATE VIEW v1 AS SELECT a FROM t3 WHERE ROWNUM() <= 2; > +SELECT * FROM v1; > +drop view v1; > +drop table t3; > + > +--echo # > +--echo # Reserved words > +--echo # > + > +create table t4 (a int, rownum int); > +insert into t4 (a,rownum) values (1,2); > +select t4.a,t4.rownum from t4; > +drop table t4; > + > +--echo # > +--echo # Test Oracle mode > +--echo # > + > +set SQL_MODE=ORACLE; > +select t1.a,rownum from t1 where rownum<=2; > +select t1.a,rownum() from t1 where rownum()<=2; > +--error ER_PARSE_ERROR > +create table t4 (a int, rownum int); > + > +DELIMITER |; > +DECLARE > + CURSOR c_cursor > + IS select a,b,rownum from t1 where rownum <= 2; > + v_a t1.a%TYPE; > + v_b t1.b%TYPE; > + v_rn t1.a%TYPE; > +BEGIN > + OPEN c_cursor; > + FETCH c_cursor INTO v_a, v_b, v_rn; > + WHILE c_cursor%FOUND LOOP > + SELECT concat(v_a,'--',v_b,'--',v_rn); > + FETCH c_cursor INTO v_a, v_b, v_rn; > + END LOOP; > + CLOSE c_cursor; > +END;| > +DELIMITER ;| > + > +--error ER_FUNCTION_CANNOT_BE_USED_IN_CLAUSE > +select a, rownum from t1 group by a, rownum having rownum < 3; > +select a, rownum from t1 group by a, rownum having "rownum" < 3; what's the difference between these two? Why the first is an error and the second is not? > +select a, rownum as r from t1 group by a, rownum having r < 3; And the this one. Why is it not an error either? > + > +set SQL_MODE=DEFAULT; > + > +--echo # Cleanup > + > +drop table t1,t2; > + > +--echo # > +--echo # INSERT > +--echo # > + > +create table t1 (a int not null primary key, b int); > +insert into t1 values (1,rownum()),(2,rownum()),(3,rownum()); > +select * from t1; > +drop table t1; > + > +--echo # > +--echo # INSERT DELAYED > +--echo # > + > +create table t1 (a int not null primary key, b int); > +insert delayed into t1 values (1,rownum()),(2,rownum()),(3,rownum()); > +select * from t1; > +drop table t1; > + > +--echo # > +--echo # INSERT ... RETURNING > +--echo # > + > +create or replace table t1 (a int); > +insert into t1 values (1),(2) returning a, rownum(); > +drop table t1; > + > +--echo # > +--echo # UPDATE > +--echo # > + > +create table t1 (a int not null primary key, b int); > +insert into t1 values (1,1),(2,2),(3,3); > +update t1 set b=0; > +update t1 set b=rownum()+1; > +select * from t1; > + > +update t1 set b=0; > +update t1 set b=rownum() where a < 10 and rownum() < 2; > +select * from t1; > +drop table t1; > + > +create table t1 (a int); > +insert into t1 values (10),(20),(30); > +update t1 set a = rownum(); > +select * from t1; > +update t1 set a = rownum(); > +select * from t1; > +drop table t1; > + > +--echo # > +--echo # DELETE > +--echo # > + > +create table t1 (a int not null primary key, b int); > +insert into t1 values (1,1),(2,0),(3,0); > +delete from t1 where a < 10 and rownum() < 2; > +select * from t1; > +drop table t1; > + > +--echo # > +--echo # MULTI-TABLE-DELETE > +--echo # > + > +create table t1 (a int not null primary key); > +insert into t1 values (1),(2),(3); > +create table t2 (a int not null primary key); > +insert into t2 values (1),(2),(3); > + > +delete t1,t2 from t1,t2 where t1.a=t2.a and rownum() <= 2; > +select * from t1; > +select * from t2; > +drop table t1,t2; > + > +--echo # > +--echo # MULTI-TABLE-UPDATE > + > +CREATE TABLE t1 (ID INT); > +CREATE TABLE t2 (ID INT, > + s1 TEXT, s2 TEXT, s3 VARCHAR(10), s4 TEXT, s5 VARCHAR(10)); > + > +INSERT INTO t1 VALUES (1),(2); > +INSERT INTO t2 VALUES (1,'test', 'test', 'test', 'test', 'test'), > + (2,'test', 'test', 'test', 'test', 'test'); > + > +SELECT * FROM t1 LEFT JOIN t2 USING(ID); > +UPDATE t1 LEFT JOIN t2 USING(ID) SET s1 = 'changed'; > +select * from t2; > +update t2 set s1=""; > +UPDATE t1 LEFT JOIN t2 USING(ID) SET s1 = 'changed' where rownum() <=1; > +select * from t2; > +drop table t1,t2; > + > +--echo # > +--echo # LOAD DATA > +--echo # > + > +create table t1 (a int, b int, c int); > +load data infile '../../std_data/loaddata7.dat' into table t1 fields terminated by ',' lines terminated by "\r\n" (a,b) set c=rownum(); > +select * from t1; > +drop table t1; > + > +--echo # > +--echo # LIMIT OPTIMIZATION > +--echo # > + > +create table t1 (a int); > +insert into t1 select seq from seq_1_to_100; > + > +flush status; > +select * from t1 where rownum() <= 3; > +show status like "Rows_read"; > +flush status; > +flush status; > +select * from t1 where rownum() <= 4 and rownum() <= 3; > +show status like "Rows_read"; > +flush status; > +select * from t1 where rownum() < 4 and a > 10; > +show status like "Rows_read"; > +flush status; > +select * from t1 where 3 >= rownum(); > +show status like "Rows_read"; > +flush status; > +select * from t1 where 4 > rownum() and a > 20; > +show status like "Rows_read"; > +flush status; > +select * from t1 where rownum() = 1 and a > 10; > +show status like "Rows_read"; > +flush status; > +select * from t1 where a > 30 && 1 = rownum(); > +show status like "Rows_read"; > +flush status; > + > +--echo # No limit optimization > + > +select * from t1 where rownum() > 10; > +show status like "Rows_read"; > +flush status; > +select * from t1 where 10 < rownum(); > +show status like "Rows_read"; > +flush status; > +select * from t1 where rownum() >= 10; > +show status like "Rows_read"; > +flush status; > +select * from t1 where 10 < rownum(); > +show status like "Rows_read"; > +flush status; > +select * from t1 where 10 <= rownum(); > +show status like "Rows_read"; > +flush status; > +select * from t1 where 2 = rownum(); > +show status like "Rows_read"; > +flush status; > +select * from t1 where rownum() = 2; > +show status like "Rows_read"; > +flush status; > +select * from t1 where rownum() <= 0; > +show status like "Rows_read"; > +flush status; > +select * from t1 where rownum() < 10 limit 4, 4; > +show status like "Rows_read"; > +flush status; > +select * from t1 where rownum() < 10 order by a; > +show status like "Rows_read"; > +flush status; > + > + > +--echo # rownum and limit > + > +select * from t1 where rownum() < 4 limit 10; > +show status like "Rows_read"; > +flush status; > +select * from t1 where rownum() < 10 limit 4; > +show status like "Rows_read"; > + > +drop table t1; > + > +--echo # > +--echo # Rownum examples from Woqutech > +--echo # > + > +set SQL_MODE=ORACLE; > +create table t1 (c1 int ,c2 varchar(20)) engine=myisam; > +insert into t1 values (1, 'aaa'),(2, 'bbb'),(3, 'ccc'),(4, 'ddd'),(5, 'eee'); > +update t1 set c2 = 'xxx' where rownum = 2; > +select * from t1 where c2='xxx'; > +update t1 set c2 = 'xxx' where rownum < 3; > +select * from t1 where c2='xxx'; > +delete from t1 where rownum = 2; > +select count(*) from t1; > +delete from t1 where rownum < 3; > +select count(*) from t1; > +delete from t1 where c1=rownum ; > +select count(*) from t1; > +delete from t1 where c1=rownum+2 ; > +select count(*) from t1; > +set SQL_MODE=DEFAULT; > +drop table t1; > + > +--echo # > +--echo # Rownum() used in not supported places (returns 0 or gives an error) > +--echo # > + > +set @a=rownum(); > +select @a; > + > +--error ER_GENERATED_COLUMN_FUNCTION_IS_NOT_ALLOWED > +create or replace table t (a int, b int as (rownum()) virtual); > + > +create table t1 (a int); > +insert into t1 values (3),(1),(5),(8),(4); > +handler t1 open; > +--error ER_GENERATED_COLUMN_FUNCTION_IS_NOT_ALLOWED > +handler t1 read next where rownum() < 1; > +handler t1 close; > +drop table t1; > + > +# rownum() executed in a function will be run in the function context. > + > +create table t1 (a int not null primary key, b int); > +insert into t1 values (1,1),(2,2),(3,3); > +create function f() returns int return rownum(); > +select a, rownum(), f() from t1; > +drop function f; > +drop table t1; > + > +# rownum() executed in a trigger will be run in the function context. > + > +create or replace table t1 (a int, r int); > +create trigger tr before update on t1 for each row set NEW.r = rownum(); > +insert into t1 (a) values (1),(2); > +select * from t1; > +update t1 set a=a+10; > +select * from t1; > +drop trigger tr; > +drop table t1; > + > +--echo # > +--echo # LIMIT optimisation > +--echo # > + > +create table t1 (a int); > +insert into t1 values (1),(2),(3),(4),(5); > + > +let $query= > +select * from (select a from t1 where a < 1000) as tt where rownum() <= 2; > +flush status; > +eval $query; > +show status like "Rows_read"; > +eval explain extended $query; > +eval prepare stmt from "$query"; > +flush status; > +execute stmt; > +show status like "Rows_read"; > +flush status; > +execute stmt; > +show status like "Rows_read"; > +deallocate prepare stmt; > + > + > +let $query= > +select * from (select a from t1 where a < 1000 group by a) as tt where rownum() <= 2; > +flush status; > +eval $query; > +show status like "Rows_read"; > +eval explain extended $query; > +eval prepare stmt from "$query"; > +execute stmt; > +execute stmt; > +deallocate prepare stmt; > + > +let $query= > +select * from (select a from t1 where a < 1000 group by a order by 1) as tt where rownum() <= 2; > +flush status; > +eval $query; > +show status like "Rows_read"; > +eval explain extended $query; > +eval prepare stmt from "$query"; > +execute stmt; > +execute stmt; > +deallocate prepare stmt; > + > +let $query= > +select * from (select a from t1 where a < 1000 union select 10) as tt where rownum() <= 2; > +flush status; > +eval $query; > +show status like "Rows_read"; > +eval explain extended $query; > +eval prepare stmt from "$query"; > +execute stmt; > +execute stmt; > +deallocate prepare stmt; > + > +--echo # Other limit > + > +let $query= > +select * from (select a from t1 where a < 1000 group by a order by 1 limit 1) as tt where rownum() <= 2; > +eval $query; > +eval explain extended $query; > +eval prepare stmt from "$query"; > +execute stmt; > +execute stmt; > +deallocate prepare stmt; > + > +--echo # Other limit less > + > +let $query= > +select * from (select a from t1 where a < 1000 group by a order by 1 limit 10) as tt where rownum() <= 2; > +eval $query; > +eval explain extended $query; > +eval prepare stmt from "$query"; > +execute stmt; > +execute stmt; > +deallocate prepare stmt; > + > +let $query= > +select * from (select a from t1 where a < 1000 union select 10 limit 1) as tt where rownum() <= 2; > +eval $query; > +eval explain extended $query; > +eval prepare stmt from "$query"; > +execute stmt; > +execute stmt; > +deallocate prepare stmt; > + > +--echo # < rownum > + > +let $query= > +select * from (select a from t1 where a < 1000) as tt where rownum() < 2; > +eval $query; > +eval explain extended $query; > +eval prepare stmt from "$query"; > +execute stmt; > +execute stmt; > +deallocate prepare stmt; > + > +--echo # Simple expression > + > +let $query= > +select * from (select a from t1 where a < 1000) as tt where rownum() <= 1+1; > +eval $query; > +eval explain extended $query; > +eval prepare stmt from "$query"; > +execute stmt; > +execute stmt; > +deallocate prepare stmt; > + > +--echo # Simple expression reversed > + > +let $query= > +select * from (select a from t1 where a < 1000) as tt where 1+1 >= rownum(); > +eval $query; > +eval explain extended $query; > +eval prepare stmt from "$query"; > +execute stmt; > +execute stmt; > +deallocate prepare stmt; > + > +--echo # expensive (no opt) > + > +let $query= > +select * from (select a from t1 where a < 1000) as tt where (select max(a) from t1) >= rownum(); > +eval $query; > +eval explain extended $query; > +eval prepare stmt from "$query"; > +execute stmt; > +execute stmt; > +deallocate prepare stmt; > + > +drop table t1; > diff --git a/mysql-test/main/rownum.result b/mysql-test/main/rownum.result > new file mode 100644 > index 00000000000..e7ea92e5dde > --- /dev/null > +++ b/mysql-test/main/rownum.result > @@ -0,0 +1,925 @@ > +CREATE OR REPLACE TABLE t1(a int, b int) engine=aria; > +CREATE OR REPLACE TABLE t2(a int, b int) engine=aria; > +insert into t1 values (1,10),(2,20),(3,30); > +insert into t2 values (2,21),(3,31),(4,41); > +# > +# Simple selects > +# > +select a,b,rownum() from t1; > +a b rownum() > +1 10 1 > +2 20 2 > +3 30 3 > +select a,b,rownum() from t1 where rownum() < 2; > +a b rownum() > +1 10 1 > +select a,b from t1 where rownum() <= 2; > +a b > +1 10 > +2 20 > +select a,b from t1 where rownum() > 2; > +a b > +# > +# Subqueries > +# > +select t1.a,rownum(),t3.a,t3.t2_rownum from t1, (select t2.a,rownum() as t2_rownum from t2 where rownum() <=2) t3; > +a rownum() a t2_rownum > +1 1 2 1 > +1 2 3 2 > +2 3 2 1 > +2 4 3 2 > +3 5 2 1 > +3 6 3 2 > +select t1.a, (select t2.b from t2 where t1.a=t2.a and rownum() <= 1) 'b' from t1; > +a b > +1 NULL > +2 21 > +3 31 > +select t1.a, t3.a from t1, (select * from t2 where rownum() <= 2) t3; > +a a > +1 2 > +1 3 > +2 2 > +2 3 > +3 2 > +3 3 > +select * from (select tt.*,rownum() as id from (select * from t1) tt) t3 where id>2; > +a b id > +3 30 3 > +# > +# Joins > +# > +select t1.a,t1.b,t2.a,t2.b,rownum() from t1,t2 where rownum() <= 4; > +a b a b rownum() > +1 10 2 21 1 > +2 20 2 21 2 > +3 30 2 21 3 > +1 10 3 31 4 > +select *,rownum() from t1,t2 where t1.a=t2.a and rownum()<=2; > +a b a b rownum() > +2 20 2 21 1 > +3 30 3 31 2 > +select * from t1 left join t2 on (t2.a=t1.a and rownum()=0); > +a b a b > +1 10 NULL NULL > +2 20 NULL NULL > +3 30 NULL NULL > +select * from t1 left join t2 on (t2.a=t1.a and rownum()>1); > +a b a b > +1 10 NULL NULL > +2 20 NULL NULL > +3 30 NULL NULL > +select * from t1 left join t2 on (t2.a=t1.a and rownum()<1); > +a b a b > +1 10 NULL NULL > +2 20 NULL NULL > +3 30 NULL NULL > +select * from t1 left join t2 on (t2.a=t1.a and rownum()=1); > +a b a b > +2 20 2 21 > +1 10 NULL NULL > +3 30 NULL NULL > +select * from t1 left join t2 on (t2.a=t1.a and rownum()>=1); > +a b a b > +2 20 2 21 > +3 30 3 31 > +1 10 NULL NULL > +# > +# Union > +# > +select * from t1 where rownum() <=2 union select * from t2 where rownum()<= 1; > +a b > +1 10 > +2 20 > +2 21 > +# > +# Order by > +# > +select * from t1 where rownum() <= 2 order by a desc; > +a b > +2 20 > +1 10 > +explain select * from t1 where rownum() <= 2 order by a desc; > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 ALL NULL NULL NULL NULL 3 Using where; Using filesort > +select t1.a,t1.b,rownum() from t1 where rownum() <= 2 order by a desc; > +a b rownum() > +2 20 2 > +1 10 1 > +explain select t1.a,t1.b,rownum() from t1 where rownum() <= 2 order by a desc; > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 ALL NULL NULL NULL NULL 3 Using where; Using temporary; Using filesort > +select *,rownum() from t1,t2; > +a b a b rownum() > +1 10 2 21 1 > +2 20 2 21 2 > +3 30 2 21 3 > +1 10 3 31 4 > +2 20 3 31 5 > +3 30 3 31 6 > +1 10 4 41 7 > +2 20 4 41 8 > +3 30 4 41 9 > +select *,rownum() from t1,t2 order by t2.a desc, t1.a desc; > +a b a b rownum() > +3 30 4 41 9 > +2 20 4 41 8 > +1 10 4 41 7 > +3 30 3 31 6 > +2 20 3 31 5 > +1 10 3 31 4 > +3 30 2 21 3 > +2 20 2 21 2 > +1 10 2 21 1 > +select * from (select * from t1 order by a desc) as t where rownum() <= 2; > +a b > +3 30 > +2 20 > +select * from t1,t2 where t1.a=t2.a and rownum()<=2 order by t1.a,t2.a; > +a b a b > +2 20 2 21 > +3 30 3 31 > +# > +# Having > +# > +select t1.a, sum(t1.b), rownum() as 'r' from t1 group by t1.a having r <= 2; > +a sum(t1.b) r > +1 10 1 > +2 20 2 > +select * from t1 having rownum() <= 2; > +ERROR HY000: Function 'rownum()' cannot be used in the HAVING clause > +select t1.a, sum(t1.b), rownum() from t1 group by t1.a; > +a sum(t1.b) rownum() > +1 10 1 > +2 20 2 > +3 30 3 > +select t1.a, sum(t1.b), rownum() as 'r' from t1 group by t1.a having rownum() <= 2; > +ERROR HY000: Function 'rownum()' cannot be used in the HAVING clause > +# > +# Sum functions > +# > +select max(rownum()),min(rownum()) from t1; > +max(rownum()) min(rownum()) > +3 1 > +select sum(rownum()),avg(rownum()) from t1; > +sum(rownum()) avg(rownum()) > +6 2.0000 > +# > +# Group by > +# > +select t1.a,sum(t1.b) from t1 where rownum() < 2 group by t1.a; > +a sum(t1.b) > +1 10 > +select t1.a,sum(t2.b) from t1 JOIN t2 ON (t1.a=t2.a) where rownum() <= 2 group by t1.a; > +a sum(t2.b) > +2 21 > +3 31 > +select * from (select t1.a,sum(t2.b) from t1 JOIN t2 ON (t1.a=t2.a) group by t1.a) as t where rownum() <= 1; > +a sum(t2.b) > +2 21 > +select t1.a,sum(rownum()),count(*) from t1 where rownum() <= 2 group by t1.a; > +a sum(rownum()) count(*) > +1 1 1 > +2 2 1 > +select * from (select t1.a,sum(t1.b) from t1 group by t1.a) as t3 where rownum() < 2; > +a sum(t1.b) > +1 10 > +create table t3 (a int) engine=myisam; > +insert into t3 values (3),(5),(5),(3); > +select a, max(rownum()) from t3 group by a; > +a max(rownum()) > +3 4 > +5 3 > +drop table t3; > +CREATE TABLE t3 ( > +a int(11) DEFAULT NULL, > +b varchar(1024) DEFAULT NULL > +); > +insert into t3 select mod(seq*3,20)+1, repeat(char(33+mod(seq,90)),mod(seq,10)*100) from seq_1_to_23; > +SELECT sq.a,length(sq.f) FROM (SELECT a, GROUP_CONCAT(b,b) AS f FROM t3 GROUP BY a ORDER BY a desc) as sq WHERE ROWNUM() <= 10; > +a length(sq.f) > +20 600 > +19 1200 > +18 1800 > +17 400 > +16 1000 > +15 1600 > +14 200 > +13 800 > +12 1400 > +11 0 > +drop table t3; > +# > +# Prepared statements > +# > +PREPARE stmt1 from "select a,b,rownum() from t1 where rownum() <= 2"; > +execute stmt1; > +a b rownum() > +1 10 1 > +2 20 2 > +execute stmt1; > +a b rownum() > +1 10 1 > +2 20 2 > +deallocate prepare stmt1; > +# > +# Views > +# > +create view v1 as select t1.a,rownum() from t1; > +select * from v1; > +a rownum() > +1 1 > +2 2 > +3 3 > +select t1.a,v1.* from t1,v1 where t1.a=v1.a; > +a a rownum() > +1 1 1 > +2 2 2 > +3 3 3 > +drop view v1; > +CREATE TABLE t3 (a INT); > +INSERT INTO t3 VALUES (1),(2),(3); > +CREATE VIEW v1 AS SELECT a FROM t3 WHERE ROWNUM() <= 2; > +SELECT * FROM v1; > +a > +1 > +2 > +drop view v1; > +drop table t3; > +# > +# Reserved words > +# > +create table t4 (a int, rownum int); > +insert into t4 (a,rownum) values (1,2); > +select t4.a,t4.rownum from t4; > +a rownum > +1 2 > +drop table t4; > +# > +# Test Oracle mode > +# > +set SQL_MODE=ORACLE; > +select t1.a,rownum from t1 where rownum<=2; > +a rownum > +1 1 > +2 2 > +select t1.a,rownum() from t1 where rownum()<=2; > +a rownum() > +1 1 > +2 2 > +create table t4 (a int, rownum int); > +ERROR 42000: You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'rownum int)' at line 1 > +DECLARE > +CURSOR c_cursor > +IS select a,b,rownum from t1 where rownum <= 2; > +v_a t1.a%TYPE; > +v_b t1.b%TYPE; > +v_rn t1.a%TYPE; > +BEGIN > +OPEN c_cursor; > +FETCH c_cursor INTO v_a, v_b, v_rn; > +WHILE c_cursor%FOUND LOOP > +SELECT concat(v_a,'--',v_b,'--',v_rn); > +FETCH c_cursor INTO v_a, v_b, v_rn; > +END LOOP; > +CLOSE c_cursor; > +END;| > +concat(v_a,'--',v_b,'--',v_rn) > +1--10--1 > +concat(v_a,'--',v_b,'--',v_rn) > +2--20--2 > +select a, rownum from t1 group by a, rownum having rownum < 3; > +ERROR HY000: Function 'rownum()' cannot be used in the HAVING clause > +select a, rownum as r from t1 group by a, rownum having r < 3; > +a r > +1 1 > +2 2 > +select a, rownum from t1 group by a, rownum having "rownum" < 3; > +a rownum > +1 1 > +2 2 > +set SQL_MODE=DEFAULT; > +# Cleanup > +drop table t1,t2; > +# > +# INSERT > +# > +create table t1 (a int not null primary key, b int); > +insert into t1 values (1,rownum()),(2,rownum()),(3,rownum()); > +select * from t1; > +a b > +1 1 > +2 2 > +3 3 > +drop table t1; > +# > +# INSERT DELAYED > +# > +create table t1 (a int not null primary key, b int); > +insert delayed into t1 values (1,rownum()),(2,rownum()),(3,rownum()); > +select * from t1; > +a b > +1 1 > +2 2 > +3 3 > +drop table t1; > +# > +# INSERT ... RETURNING > +# > +create or replace table t1 (a int); > +insert into t1 values (1),(2) returning a, rownum(); > +a rownum() > +1 1 > +2 2 > +drop table t1; > +# > +# UPDATE > +# > +create table t1 (a int not null primary key, b int); > +insert into t1 values (1,1),(2,2),(3,3); > +update t1 set b=0; > +update t1 set b=rownum()+1; > +select * from t1; > +a b > +1 2 > +2 3 > +3 4 > +update t1 set b=0; > +update t1 set b=rownum() where a < 10 and rownum() < 2; > +select * from t1; > +a b > +1 1 > +2 0 > +3 0 > +drop table t1; > +create table t1 (a int); > +insert into t1 values (10),(20),(30); > +update t1 set a = rownum(); > +select * from t1; > +a > +1 > +2 > +3 > +update t1 set a = rownum(); > +select * from t1; > +a > +1 > +2 > +3 > +drop table t1; > +# > +# DELETE > +# > +create table t1 (a int not null primary key, b int); > +insert into t1 values (1,1),(2,0),(3,0); > +delete from t1 where a < 10 and rownum() < 2; > +select * from t1; > +a b > +2 0 > +3 0 > +drop table t1; > +# > +# MULTI-TABLE-DELETE > +# > +create table t1 (a int not null primary key); > +insert into t1 values (1),(2),(3); > +create table t2 (a int not null primary key); > +insert into t2 values (1),(2),(3); > +delete t1,t2 from t1,t2 where t1.a=t2.a and rownum() <= 2; > +select * from t1; > +a > +3 > +select * from t2; > +a > +3 > +drop table t1,t2; > +# > +# MULTI-TABLE-UPDATE > +CREATE TABLE t1 (ID INT); > +CREATE TABLE t2 (ID INT, > +s1 TEXT, s2 TEXT, s3 VARCHAR(10), s4 TEXT, s5 VARCHAR(10)); > +INSERT INTO t1 VALUES (1),(2); > +INSERT INTO t2 VALUES (1,'test', 'test', 'test', 'test', 'test'), > +(2,'test', 'test', 'test', 'test', 'test'); > +SELECT * FROM t1 LEFT JOIN t2 USING(ID); > +ID s1 s2 s3 s4 s5 > +1 test test test test test > +2 test test test test test > +UPDATE t1 LEFT JOIN t2 USING(ID) SET s1 = 'changed'; > +select * from t2; > +ID s1 s2 s3 s4 s5 > +1 changed test test test test > +2 changed test test test test > +update t2 set s1=""; > +UPDATE t1 LEFT JOIN t2 USING(ID) SET s1 = 'changed' where rownum() <=1; > +select * from t2; > +ID s1 s2 s3 s4 s5 > +1 changed test test test test > +2 test test test test > +drop table t1,t2; > +# > +# LOAD DATA > +# > +create table t1 (a int, b int, c int); > +load data infile '../../std_data/loaddata7.dat' into table t1 fields terminated by ',' lines terminated by "\r\n" (a,b) set c=rownum(); > +select * from t1; > +a b c > +2 2 1 > +3 3 2 > +4 4 3 > +5 5 4 > +6 6 5 > +drop table t1; > +# > +# LIMIT OPTIMIZATION > +# > +create table t1 (a int); > +insert into t1 select seq from seq_1_to_100; > +flush status; > +select * from t1 where rownum() <= 3; > +a > +1 > +2 > +3 > +show status like "Rows_read"; > +Variable_name Value > +Rows_read 3 > +flush status; > +flush status; why twice? > +select * from t1 where rownum() <= 4 and rownum() <= 3; > +a > +1 > +2 > +3 > +show status like "Rows_read"; > +Variable_name Value > +Rows_read 3 > +flush status; > +select * from t1 where rownum() < 4 and a > 10; > +a > +11 > +12 > +13 > +show status like "Rows_read"; > +Variable_name Value > +Rows_read 13 > +flush status; > +select * from t1 where 3 >= rownum(); > +a > +1 > +2 > +3 > +show status like "Rows_read"; > +Variable_name Value > +Rows_read 3 > +flush status; > +select * from t1 where 4 > rownum() and a > 20; > +a > +21 > +22 > +23 > +show status like "Rows_read"; > +Variable_name Value > +Rows_read 23 > +flush status; > +select * from t1 where rownum() = 1 and a > 10; > +a > +11 > +show status like "Rows_read"; > +Variable_name Value > +Rows_read 11 > +flush status; > +select * from t1 where a > 30 && 1 = rownum(); > +a > +31 > +show status like "Rows_read"; > +Variable_name Value > +Rows_read 31 > +flush status; > +# No limit optimization > +select * from t1 where rownum() > 10; > +a > +show status like "Rows_read"; > +Variable_name Value > +Rows_read 100 You could've optimized it to `limit 0` but I guess it's not a particularly common use case > +flush status; > +select * from t1 where 10 < rownum(); > +a > +show status like "Rows_read"; > +Variable_name Value > +Rows_read 100 > +flush status; > +select * from t1 where rownum() >= 10; > +a > +show status like "Rows_read"; > +Variable_name Value > +Rows_read 100 > +flush status; > +select * from t1 where 10 < rownum(); > +a > +show status like "Rows_read"; > +Variable_name Value > +Rows_read 100 > +flush status; > +select * from t1 where 10 <= rownum(); > +a > +show status like "Rows_read"; > +Variable_name Value > +Rows_read 100 > +flush status; > +select * from t1 where 2 = rownum(); > +a > +show status like "Rows_read"; > +Variable_name Value > +Rows_read 100 > +flush status; > +select * from t1 where rownum() = 2; > +a > +show status like "Rows_read"; > +Variable_name Value > +Rows_read 100 > +flush status; > +select * from t1 where rownum() <= 0; > +a > +show status like "Rows_read"; > +Variable_name Value > +Rows_read 100 > +flush status; > +select * from t1 where rownum() < 10 limit 4, 4; can you add a test for `select a, rownum() from t1 limit 4, 4` please ? > +a > +5 > +6 > +7 > +8 > +show status like "Rows_read"; > +Variable_name Value > +Rows_read 8 > +flush status; > +select * from t1 where rownum() < 10 order by a; > +a > +1 > +2 > +3 > +4 > +5 > +6 > +7 > +8 > +9 > +show status like "Rows_read"; > +Variable_name Value > +Rows_read 100 > +flush status; > +# rownum and limit > +select * from t1 where rownum() < 4 limit 10; > +a > +1 > +2 > +3 > +show status like "Rows_read"; > +Variable_name Value > +Rows_read 3 > +flush status; > +select * from t1 where rownum() < 10 limit 4; > +a > +1 > +2 > +3 > +4 > +show status like "Rows_read"; > +Variable_name Value > +Rows_read 4 > +drop table t1; > +# > +# Rownum examples from Woqutech > +# > +set SQL_MODE=ORACLE; > +create table t1 (c1 int ,c2 varchar(20)) engine=myisam; > +insert into t1 values (1, 'aaa'),(2, 'bbb'),(3, 'ccc'),(4, 'ddd'),(5, 'eee'); > +update t1 set c2 = 'xxx' where rownum = 2; > +select * from t1 where c2='xxx'; > +c1 c2 > +update t1 set c2 = 'xxx' where rownum < 3; > +select * from t1 where c2='xxx'; > +c1 c2 > +1 xxx > +2 xxx > +delete from t1 where rownum = 2; > +select count(*) from t1; > +count(*) > +5 > +delete from t1 where rownum < 3; > +select count(*) from t1; > +count(*) > +3 > +delete from t1 where c1=rownum ; > +select count(*) from t1; > +count(*) > +3 > +delete from t1 where c1=rownum+2 ; > +select count(*) from t1; > +count(*) > +0 > +set SQL_MODE=DEFAULT; > +drop table t1; > +# > +# Rownum() used in not supported places (returns 0 or gives an error) > +# > +set @a=rownum(); > +select @a; > +@a > +0 > +create or replace table t (a int, b int as (rownum()) virtual); > +ERROR HY000: Function or expression 'rownum()' cannot be used in the GENERATED ALWAYS AS clause of `b` > +create table t1 (a int); > +insert into t1 values (3),(1),(5),(8),(4); > +handler t1 open; > +handler t1 read next where rownum() < 1; > +ERROR HY000: Function or expression 'rownum()' cannot be used in the WHERE clause of `HANDLER` why is that? `handler t1 read next limit 2` works just fine. > +handler t1 close; > +drop table t1; > +create table t1 (a int not null primary key, b int); > +insert into t1 values (1,1),(2,2),(3,3); > +create function f() returns int return rownum(); > +select a, rownum(), f() from t1; > +a rownum() f() > +1 1 0 > +2 2 0 > +3 3 0 > +drop function f; > +drop table t1; > +create or replace table t1 (a int, r int); > +create trigger tr before update on t1 for each row set NEW.r = rownum(); > +insert into t1 (a) values (1),(2); > +select * from t1; > +a r > +1 NULL > +2 NULL > +update t1 set a=a+10; > +select * from t1; > +a r > +11 0 > +12 0 > +drop trigger tr; > +drop table t1; > +# > +# LIMIT optimisation > +# > +create table t1 (a int); > +insert into t1 values (1),(2),(3),(4),(5); > +flush status; > +select * from (select a from t1 where a < 1000) as tt where rownum() <= 2; > +a > +1 > +2 > +show status like "Rows_read"; > +Variable_name Value > +Rows_read 2 > +explain extended select * from (select a from t1 where a < 1000) as tt where rownum() <= 2; > +id select_type table type possible_keys key key_len ref rows filtered Extra > +1 PRIMARY <derived2> ALL NULL NULL NULL NULL 2 100.00 Using where > +2 DERIVED t1 ALL NULL NULL NULL NULL 5 100.00 Using where > +Warnings: > +Note 1003 /* select#1 */ select `tt`.`a` AS `a` from (/* select#2 */ select `test`.`t1`.`a` AS `a` from `test`.`t1` where `test`.`t1`.`a` < 1000 limit 2) `tt` where rownum() <= 2 > +prepare stmt from "select * from (select a from t1 where a < 1000) as tt where rownum() <= 2"; > +flush status; > +execute stmt; > +a > +1 > +2 > +show status like "Rows_read"; > +Variable_name Value > +Rows_read 2 > +flush status; > +execute stmt; > +a > +1 > +2 > +show status like "Rows_read"; > +Variable_name Value > +Rows_read 2 > +deallocate prepare stmt; > +flush status; > +select * from (select a from t1 where a < 1000 group by a) as tt where rownum() <= 2; > +a > +1 > +2 > +show status like "Rows_read"; > +Variable_name Value > +Rows_read 5 > +explain extended select * from (select a from t1 where a < 1000 group by a) as tt where rownum() <= 2; > +id select_type table type possible_keys key key_len ref rows filtered Extra > +1 PRIMARY <derived2> ALL NULL NULL NULL NULL 2 100.00 Using where > +2 DERIVED t1 ALL NULL NULL NULL NULL 5 100.00 Using where; Using temporary; Using filesort > +Warnings: > +Note 1003 /* select#1 */ select `tt`.`a` AS `a` from (/* select#2 */ select `test`.`t1`.`a` AS `a` from `test`.`t1` where `test`.`t1`.`a` < 1000 group by `test`.`t1`.`a` limit 2) `tt` where rownum() <= 2 > +prepare stmt from "select * from (select a from t1 where a < 1000 group by a) as tt where rownum() <= 2"; > +execute stmt; > +a > +1 > +2 > +execute stmt; > +a > +1 > +2 > +deallocate prepare stmt; > +flush status; > +select * from (select a from t1 where a < 1000 group by a order by 1) as tt where rownum() <= 2; > +a > +1 > +2 > +show status like "Rows_read"; > +Variable_name Value > +Rows_read 5 > +explain extended select * from (select a from t1 where a < 1000 group by a order by 1) as tt where rownum() <= 2; > +id select_type table type possible_keys key key_len ref rows filtered Extra > +1 PRIMARY <derived2> ALL NULL NULL NULL NULL 2 100.00 Using where > +2 DERIVED t1 ALL NULL NULL NULL NULL 5 100.00 Using where; Using temporary; Using filesort > +Warnings: > +Note 1003 /* select#1 */ select `tt`.`a` AS `a` from (/* select#2 */ select `test`.`t1`.`a` AS `a` from `test`.`t1` where `test`.`t1`.`a` < 1000 group by `test`.`t1`.`a` order by 1 limit 2) `tt` where rownum() <= 2 > +prepare stmt from "select * from (select a from t1 where a < 1000 group by a order by 1) as tt where rownum() <= 2"; > +execute stmt; > +a > +1 > +2 > +execute stmt; > +a > +1 > +2 > +deallocate prepare stmt; > +flush status; > +select * from (select a from t1 where a < 1000 union select 10) as tt where rownum() <= 2; > +a > +1 > +2 > +show status like "Rows_read"; > +Variable_name Value > +Rows_read 5 > +explain extended select * from (select a from t1 where a < 1000 union select 10) as tt where rownum() <= 2; > +id select_type table type possible_keys key key_len ref rows filtered Extra > +1 PRIMARY <derived2> ALL NULL NULL NULL NULL 2 100.00 Using where > +2 DERIVED t1 ALL NULL NULL NULL NULL 5 100.00 Using where > +3 UNION NULL NULL NULL NULL NULL NULL NULL NULL No tables used > +NULL UNION RESULT <union2,3> ALL NULL NULL NULL NULL NULL NULL > +Warnings: > +Note 1003 /* select#1 */ select `tt`.`a` AS `a` from (/* select#2 */ select `test`.`t1`.`a` AS `a` from `test`.`t1` where `test`.`t1`.`a` < 1000 union /* select#3 */ select 10 AS `10` limit 2) `tt` where rownum() <= 2 > +prepare stmt from "select * from (select a from t1 where a < 1000 union select 10) as tt where rownum() <= 2"; > +execute stmt; > +a > +1 > +2 > +execute stmt; > +a > +1 > +2 > +deallocate prepare stmt; > +# Other limit > +select * from (select a from t1 where a < 1000 group by a order by 1 limit 1) as tt where rownum() <= 2; > +a > +1 > +explain extended select * from (select a from t1 where a < 1000 group by a order by 1 limit 1) as tt where rownum() <= 2; > +id select_type table type possible_keys key key_len ref rows filtered Extra > +1 PRIMARY <derived2> ALL NULL NULL NULL NULL 2 100.00 Using where > +2 DERIVED t1 ALL NULL NULL NULL NULL 5 100.00 Using where; Using temporary; Using filesort > +Warnings: > +Note 1003 /* select#1 */ select `tt`.`a` AS `a` from (/* select#2 */ select `test`.`t1`.`a` AS `a` from `test`.`t1` where `test`.`t1`.`a` < 1000 group by `test`.`t1`.`a` order by 1 limit 1) `tt` where rownum() <= 2 > +prepare stmt from "select * from (select a from t1 where a < 1000 group by a order by 1 limit 1) as tt where rownum() <= 2"; > +execute stmt; > +a > +1 > +execute stmt; > +a > +1 > +deallocate prepare stmt; > +# Other limit less > +select * from (select a from t1 where a < 1000 group by a order by 1 limit 10) as tt where rownum() <= 2; > +a > +1 > +2 > +explain extended select * from (select a from t1 where a < 1000 group by a order by 1 limit 10) as tt where rownum() <= 2; > +id select_type table type possible_keys key key_len ref rows filtered Extra > +1 PRIMARY <derived2> ALL NULL NULL NULL NULL 2 100.00 Using where > +2 DERIVED t1 ALL NULL NULL NULL NULL 5 100.00 Using where; Using temporary; Using filesort > +Warnings: > +Note 1003 /* select#1 */ select `tt`.`a` AS `a` from (/* select#2 */ select `test`.`t1`.`a` AS `a` from `test`.`t1` where `test`.`t1`.`a` < 1000 group by `test`.`t1`.`a` order by 1 limit 2) `tt` where rownum() <= 2 > +prepare stmt from "select * from (select a from t1 where a < 1000 group by a order by 1 limit 10) as tt where rownum() <= 2"; > +execute stmt; > +a > +1 > +2 > +execute stmt; > +a > +1 > +2 > +deallocate prepare stmt; > +select * from (select a from t1 where a < 1000 union select 10 limit 1) as tt where rownum() <= 2; > +a > +1 > +explain extended select * from (select a from t1 where a < 1000 union select 10 limit 1) as tt where rownum() <= 2; > +id select_type table type possible_keys key key_len ref rows filtered Extra > +1 PRIMARY <derived2> ALL NULL NULL NULL NULL 5 100.00 Using where > +2 DERIVED t1 ALL NULL NULL NULL NULL 5 100.00 Using where > +3 UNION NULL NULL NULL NULL NULL NULL NULL NULL No tables used > +NULL UNION RESULT <union2,3> ALL NULL NULL NULL NULL NULL NULL > +Warnings: > +Note 1003 /* select#1 */ select `tt`.`a` AS `a` from (/* select#2 */ select `test`.`t1`.`a` AS `a` from `test`.`t1` where `test`.`t1`.`a` < 1000 union /* select#3 */ select 10 AS `10` limit 1) `tt` where rownum() <= 2 > +prepare stmt from "select * from (select a from t1 where a < 1000 union select 10 limit 1) as tt where rownum() <= 2"; > +execute stmt; > +a > +1 > +execute stmt; > +a > +1 > +deallocate prepare stmt; > +# < rownum > +select * from (select a from t1 where a < 1000) as tt where rownum() < 2; > +a > +1 > +explain extended select * from (select a from t1 where a < 1000) as tt where rownum() < 2; > +id select_type table type possible_keys key key_len ref rows filtered Extra > +1 PRIMARY <derived2> ALL NULL NULL NULL NULL 2 100.00 Using where > +2 DERIVED t1 ALL NULL NULL NULL NULL 5 100.00 Using where > +Warnings: > +Note 1003 /* select#1 */ select `tt`.`a` AS `a` from (/* select#2 */ select `test`.`t1`.`a` AS `a` from `test`.`t1` where `test`.`t1`.`a` < 1000 limit 1) `tt` where rownum() < 2 > +prepare stmt from "select * from (select a from t1 where a < 1000) as tt where rownum() < 2"; > +execute stmt; > +a > +1 > +execute stmt; > +a > +1 > +deallocate prepare stmt; > +# Simple expression > +select * from (select a from t1 where a < 1000) as tt where rownum() <= 1+1; > +a > +1 > +2 > +explain extended select * from (select a from t1 where a < 1000) as tt where rownum() <= 1+1; > +id select_type table type possible_keys key key_len ref rows filtered Extra > +1 PRIMARY <derived2> ALL NULL NULL NULL NULL 2 100.00 Using where > +2 DERIVED t1 ALL NULL NULL NULL NULL 5 100.00 Using where > +Warnings: > +Note 1003 /* select#1 */ select `tt`.`a` AS `a` from (/* select#2 */ select `test`.`t1`.`a` AS `a` from `test`.`t1` where `test`.`t1`.`a` < 1000 limit 2) `tt` where rownum() <= <cache>(1 + 1) > +prepare stmt from "select * from (select a from t1 where a < 1000) as tt where rownum() <= 1+1"; > +execute stmt; > +a > +1 > +2 > +execute stmt; > +a > +1 > +2 > +deallocate prepare stmt; > +# Simple expression reversed > +select * from (select a from t1 where a < 1000) as tt where 1+1 >= rownum(); > +a > +1 > +2 > +explain extended select * from (select a from t1 where a < 1000) as tt where 1+1 >= rownum(); > +id select_type table type possible_keys key key_len ref rows filtered Extra > +1 PRIMARY <derived2> ALL NULL NULL NULL NULL 2 100.00 Using where > +2 DERIVED t1 ALL NULL NULL NULL NULL 5 100.00 Using where > +Warnings: > +Note 1003 /* select#1 */ select `tt`.`a` AS `a` from (/* select#2 */ select `test`.`t1`.`a` AS `a` from `test`.`t1` where `test`.`t1`.`a` < 1000 limit 2) `tt` where <cache>(1 + 1) >= rownum() > +prepare stmt from "select * from (select a from t1 where a < 1000) as tt where 1+1 >= rownum()"; > +execute stmt; > +a > +1 > +2 > +execute stmt; > +a > +1 > +2 > +deallocate prepare stmt; > +# expensive (no opt) > +select * from (select a from t1 where a < 1000) as tt where (select max(a) from t1) >= rownum(); > +a > +1 > +2 > +3 > +4 > +5 > +explain extended select * from (select a from t1 where a < 1000) as tt where (select max(a) from t1) >= rownum(); > +id select_type table type possible_keys key key_len ref rows filtered Extra > +1 PRIMARY <derived2> ALL NULL NULL NULL NULL 5 100.00 Using where > +3 SUBQUERY t1 ALL NULL NULL NULL NULL 5 100.00 > +2 DERIVED t1 ALL NULL NULL NULL NULL 5 100.00 Using where > +Warnings: > +Note 1003 /* select#1 */ select `tt`.`a` AS `a` from (/* select#2 */ select `test`.`t1`.`a` AS `a` from `test`.`t1` where `test`.`t1`.`a` < 1000) `tt` where (/* select#3 */ select max(`test`.`t1`.`a`) from `test`.`t1`) >= rownum() > +prepare stmt from "select * from (select a from t1 where a < 1000) as tt where (select max(a) from t1) >= rownum()"; > +execute stmt; > +a > +1 > +2 > +3 > +4 > +5 > +execute stmt; > +a > +1 > +2 > +3 > +4 > +5 > +deallocate prepare stmt; > +drop table t1; > diff --git a/sql/filesort.h b/sql/filesort.h > index 29ae5e20cc6..806a4a8b027 100644 > --- a/sql/filesort.h > +++ b/sql/filesort.h > @@ -44,24 +44,21 @@ class Filesort: public Sql_alloc > /** Number of records to return */ > ha_rows limit; > /** ORDER BY list with some precalculated info for filesort */ > + ha_rows *accepted_rows; /* For ROWNUM */ > SORT_FIELD *sortorder; first, I suspect that you want to put your accepted_rows declaration before the comment that applies to sortorder. second, could you do a bit more than just "For ROWNUM"? why do you need something "for ROWNUM" inside Filesort? > /** select to use for getting records */ > SQL_SELECT *select; > + > /** TRUE <=> free select on destruction */ > bool own_select; > - /** true means we are using Priority Queue for order by with limit. */ > + /** TRUE means we are using Priority Queue for order by with limit. */ > bool using_pq; > - > /* > TRUE means sort operation must produce table rowids. > FALSE means that it halso has an option of producing {sort_key, > addon_fields} pairs. > */ > bool sort_positions; > - > - Filesort_tracker *tracker; > - Sort_keys *sort_keys; > - > /* > TRUE means all the fields of table of whose bitmap read_set is set > need to be read while reading records in the sort buffer. > diff --git a/sql/item_func.h b/sql/item_func.h > index 53c236bbce7..c237fcd9ecc 100644 > --- a/sql/item_func.h > +++ b/sql/item_func.h > @@ -2125,6 +2125,63 @@ class Item_func_rand :public Item_real_func > }; > > > +class Item_func_rownum :public Item_longlong_func > +{ > + /* > + This points to a variable that contains the number of rows > + accpted so far in the result set > + */ > + ha_rows *accepted_rows; > + SELECT_LEX *select; > +public: > + Item_func_rownum(THD *thd); > + longlong val_int() override; > + LEX_CSTRING func_name_cstring() const override > + { > + static LEX_CSTRING name= {STRING_WITH_LEN("rownum") }; > + return name; > + } > + enum Functype functype() const override { return ROWNUM_FUNC; } > + void update_used_tables() override {} > + bool const_item() const override { return 0; } > + void fix_after_optimize(THD *thd) override; > + bool fix_fields(THD* thd, Item **ref) override; > + bool fix_length_and_dec() override > + { > + unsigned_flag= 1; > + used_tables_cache= RAND_TABLE_BIT; > + const_item_cache=0; > + set_maybe_null(); how can rownum be NULL? > + return FALSE; > + } > + void cleanup() override > + { > + Item_longlong_func::cleanup(); > + /* Ensure we don't point to freed memory */ > + accepted_rows= 0; > + } > + bool check_vcol_func_processor(void *arg) override > + { > + return mark_unsupported_function(func_name(), "()", arg, > + VCOL_IMPOSSIBLE); > + } > + bool check_handler_func_processor(void *arg) override > + { > + return mark_unsupported_function(func_name(), "()", arg, > + VCOL_IMPOSSIBLE); first, as I wrote above, I think that there's no logical reason why rownum should not with with handler. but even if it doesn't, it's much better to check for SQLCOM_HA_READ from Item_func_rownum::fix_fields instead of traversing the whole item tree for every single HANDLER READ command looking for Item_func_rownum that nobody (literally, not a single user out of all our millions) will use there. > + } > + Item *get_copy(THD *thd) override > + { > + return this; It looks wrong. We don't have any other Item that returns `this` here. I think rownum cannot be pushed down because of RAND_TABLE_BIT, So better return NULL here, meaning "cannot be pushed down". Instead of `this` meaning "if you push it down, it'll surreptitiously start corrupting the memory" > + } > + /* This function is used in insert, update and delete */ > + void store_pointer_to_row_counter(ha_rows *row_counter) So this isn't a virtual method, and to use it you downcast after checking type() == FUNC_ITEM and functype() == ROWNUM_FUNC. To avoid adding a new virtual method thus blowing up the vtable, I presume. And fix_after_optimize() is virtual. To avoid error-prone downcasting, would be my guess. Can you explain when you use what approach, please? > + { > + accepted_rows= row_counter; > + } > +}; > + > + > class Item_func_sign :public Item_long_func > { > bool check_arguments() const override > diff --git a/sql/item_func.cc b/sql/item_func.cc > index 25714dda03c..1165b87356a 100644 > --- a/sql/item_func.cc > +++ b/sql/item_func.cc > @@ -7206,3 +7206,72 @@ void Item_func_setval::print(String *str, enum_query_type query_type) > str->append_ulonglong(round); > str->append(')'); > } > + > + > +/* > + Return how many row combinations has accepted so far + 1 > + > + The + 1 is to ensure that 'WHERE ROWNUM <=1' returns one row this is a very weird explanation, it sounds as if it's a heuristical a hack for ROWNUM <=1 to work Why not to write "+1 takes into account the currently computed row" > +*/ > + > +longlong Item_func_rownum::val_int() > +{ > + if (!accepted_rows) > + { > + /* > + Rownum is not properly set up. Probably used in wrong context when > + it should not be used. In this case returning 0 is probably the best > + solution. > + */ DBUG_ASSERT(0); here? > + return 0; > + } > + return (longlong) *accepted_rows+1; > +} > + > + > +Item_func_rownum::Item_func_rownum(THD *thd): > + Item_longlong_func(thd),accepted_rows(0) > +{ > + /* > + Remember the select context. > + Add the function to the list fix_after_optimize in the select context > + so that we can easily initializef all rownum functions with the pointers > + to the row counters. > + */ > + select= thd->lex->current_select; > + select->fix_after_optimize.push_back(this, thd->mem_root); > + > + /* > + Mark that query is using rownum() and ensure that this select is > + not merged with other selects > + */ > + select->with_rownum= 1; > + thd->lex->with_rownum= 1; > + thd->lex->uncacheable(UNCACHEABLE_RAND); Why? ROWNUM doesn't make a query uncacheable, it's perfectly deterministic and repeatable. > + > + /* If this command changes data, mark it as unsafe for statement logging */ > + if (sql_command_flags[thd->lex->sql_command] & > + (CF_UPDATES_DATA | CF_DELETES_DATA)) > + thd->lex->set_stmt_unsafe(LEX::BINLOG_STMT_UNSAFE_SYSTEM_FUNCTION); Why? It's the same as LIMIT. unsafe without ORDER BY and perfectly safe with. > +} > + > + > +bool Item_func_rownum::fix_fields(THD* thd, Item **ref) > +{ > + if (select->having_fix_field) > + { > + my_error(ER_FUNCTION_CANNOT_BE_USED_IN_CLAUSE, MYF(0), > + "rownum()", "HAVING"); > + return 1; > + } > + return Item_longlong_func::fix_fields(thd, ref); > +} > + > +/* > + Store a reference to the variable that contains number of accepted rows > +*/ > + > +void Item_func_rownum::fix_after_optimize(THD *thd) > +{ > + accepted_rows= &select->join->accepted_rows; why do you need both that and fix_rownum_pointers()? I'd think that the latter alone would be sufficient > +} > diff --git a/sql/share/errmsg-utf8.txt b/sql/share/errmsg-utf8.txt > index badaaa17716..aecb00563f7 100644 > --- a/sql/share/errmsg-utf8.txt > +++ b/sql/share/errmsg-utf8.txt > @@ -7973,3 +7973,5 @@ ER_DATA_WAS_COMMITED_UNDER_ROLLBACK > eng "Engine %s does not support rollback. Changes were committed during rollback call" > ER_PK_INDEX_CANT_BE_IGNORED > eng "A primary key cannot be marked as IGNORE" > +ER_FUNCTION_CANNOT_BE_USED_IN_CLAUSE > + eng "Function '%s' cannot be used in the %s clause" Is it the first function we've got that cannot be used in HAVING? > diff --git a/sql/sql_derived.cc b/sql/sql_derived.cc > index 35b4294707f..da6c567b87c 100644 > --- a/sql/sql_derived.cc > +++ b/sql/sql_derived.cc > @@ -656,14 +656,22 @@ static > bool mysql_derived_prepare(THD *thd, LEX *lex, TABLE_LIST *derived) > { > SELECT_LEX_UNIT *unit= derived->get_unit(); > - bool res= FALSE; > + SELECT_LEX *first_select; > + bool res= FALSE, keep_row_order; > DBUG_ENTER("mysql_derived_prepare"); > DBUG_PRINT("enter", ("unit: %p table_list: %p alias: '%s'", > unit, derived, derived->alias.str)); > if (!unit) > DBUG_RETURN(FALSE); > > - SELECT_LEX *first_select= unit->first_select(); > + first_select= unit->first_select(); > + /* > + If rownum() is used we have to preserve the insert row order > + to make group by with filesort to work. it'd help if you'd add an example of a SELECT here > + */ > + keep_row_order= (thd->lex->with_rownum && > + (first_select->group_list.elements || > + first_select->order_list.elements)); > > if (derived->is_recursive_with_table() && > !derived->is_with_table_recursive_reference() && > diff --git a/sql/sql_insert.cc b/sql/sql_insert.cc > index 806049b4465..5c4dff33914 100644 > --- a/sql/sql_insert.cc > +++ b/sql/sql_insert.cc > @@ -746,6 +746,9 @@ bool mysql_insert(THD *thd, TABLE_LIST *table_list, > DBUG_RETURN(TRUE); > } > + > + if (thd->lex->with_rownum && table_list->lock_type == TL_WRITE_DELAYED) > + table_list->lock_type= TL_WRITE; why? do you have a test case when it's needed? > > if (table_list->lock_type == TL_WRITE_DELAYED) > { > if (open_and_lock_for_insert_delayed(thd, table_list)) > @@ -968,11 +971,16 @@ bool mysql_insert(THD *thd, TABLE_LIST *table_list, > */ > if (returning && > result->send_result_set_metadata(returning->item_list, > - Protocol::SEND_NUM_ROWS | Protocol::SEND_EOF)) > + Protocol::SEND_NUM_ROWS | > + Protocol::SEND_EOF)) > goto values_loop_end; > > THD_STAGE_INFO(thd, stage_update); > thd->decide_logging_format_low(table); > + fix_rownum_pointers(thd, thd->lex->current_select, &info.copied); > + if (returning) > + fix_rownum_pointers(thd, thd->lex->returning(), &info.copied); > + Note that info.copied can be incremented more than once per row (in INSERT...ON DUPLICATE KEY UPDATE, for example). This will probably break rownum logic. > do > { > DBUG_PRINT("info", ("iteration %llu", iteration)); > @@ -2133,8 +2141,14 @@ int write_record(THD *thd, TABLE *table, COPY_INFO *info, select_result *sink) > autoinc values (generated inside the handler::ha_write()) and > values updated in ON DUPLICATE KEY UPDATE. > */ > - if (sink && sink->send_data(thd->lex->returning()->item_list) < 0) > - trg_error= 1; > + if (sink) > + { > + /* We have to adjust 'copied' to make ROWNUM() work in RETURNING */ > + info->copied--; > + if (sink->send_data(thd->lex->returning()->item_list) < 0) > + trg_error= 1; > + info->copied++; this is quite silly. Can you simply increment info->copied after this sink->send_data call? > + } > > after_trg_or_ignored_err: > if (key) > diff --git a/sql/sql_limit.h b/sql/sql_limit.h > index a4fcedac14a..62fd0546af0 100644 > --- a/sql/sql_limit.h > +++ b/sql/sql_limit.h > @@ -67,6 +67,15 @@ class Select_limit_counters > { return select_limit_cnt; } > ha_rows get_offset_limit() > { return offset_limit_cnt; } > + > + void set_limit_if_lower(ha_rows limit) > + { > + if (!offset_limit_cnt) I suppose, alternatively, you could've had if (offset_limit_cnt + select_limit_cnt > limit) select_limit_cnt= limit - offset_limit_cnt; > + { > + if (select_limit_cnt > limit) > + select_limit_cnt= limit; > + } > + } > }; > > #endif // INCLUDES_MARIADB_SQL_LIMIT_H > diff --git a/sql/sql_yacc.yy b/sql/sql_yacc.yy > index 2bd2575b882..57ba9df42c0 100644 > --- a/sql/sql_yacc.yy > +++ b/sql/sql_yacc.yy > @@ -10286,6 +10287,22 @@ function_call_nonkeyword: > if (unlikely($$ == NULL)) > MYSQL_YYABORT; > } > +/* Start SQL_MODE_ORACLE_SPECIFIC > + | ROWNUM_SYM optional_braces > + { > + $$= new (thd->mem_root) Item_func_rownum(thd); > + if (unlikely($$ == NULL)) > + MYSQL_YYABORT; > + } > +End SQL_MODE_ORACLE_SPECIFIC */ > +/* Start SQL_MODE_DEFAULT_SPECIFIC */ > + | ROWNUM_SYM '(' ')' > + { > + $$= new (thd->mem_root) Item_func_rownum(thd); > + if (unlikely($$ == NULL)) > + MYSQL_YYABORT; > + } > +/* End SQL_MODE_DEFAULT_SPECIFIC */ You have two nearly identical rules compiled conditionally. I can pretty much guarantee that they'll go out of sync eventually. Better do something like /* Start SQL_MODE_ORACLE_SPECIFIC oracle_optional_braces: optional_braces ; End SQL_MODE_ORACLE_SPECIFIC */ /* Start SQL_MODE_DEFAULT_SPECIFIC */ oracle_optional_braces: '(' ')' ; /* End SQL_MODE_DEFAULT_SPECIFIC */ or even | ROWNUM_SYM /* Start SQL_MODE_ORACLE_SPECIFIC optional_braces End SQL_MODE_ORACLE_SPECIFIC */ /* Start SQL_MODE_DEFAULT_SPECIFIC */ '(' ')' /* End SQL_MODE_DEFAULT_SPECIFIC */ { $$= new (thd->mem_root) Item_func_rownum(thd); if (unlikely($$ == NULL)) MYSQL_YYABORT; } > | SUBDATE_SYM '(' expr ',' expr ')' > { > $$= new (thd->mem_root) Item_date_add_interval(thd, $3, $5, > diff --git a/sql/table.cc b/sql/table.cc > index 7f3a6468aa2..100813e1f95 100644 > --- a/sql/table.cc > +++ b/sql/table.cc > @@ -9331,6 +9331,10 @@ bool TABLE_LIST::init_derived(THD *thd, bool init_view) > { > /* A subquery might be forced to be materialized due to a side-effect. */ > if (!is_materialized_derived() && first_select->is_mergeable() && > + (unit->outer_select() && !unit->outer_select()->with_rownum) && no merging if *outer select* uses rownum? why? example? > + (!thd->lex->with_rownum || > + (!first_select->group_list.elements && > + !first_select->order_list.elements)) && no merging if no rownum but there's group by or order by? group by - this is obvious, and is already checked earlier in this if() order by - why? > optimizer_flag(thd, OPTIMIZER_SWITCH_DERIVED_MERGE) && > !thd->lex->can_not_use_merged(1) && > !is_recursive_with_table()) > diff --git a/sql/sql_select.cc b/sql/sql_select.cc > index 4bb0bc661bd..b002e27799b 100644 > --- a/sql/sql_select.cc > +++ b/sql/sql_select.cc > @@ -1789,10 +1803,18 @@ JOIN::optimize_inner() > { > DBUG_ENTER("JOIN::optimize_inner"); > subq_exit_fl= false; > - do_send_rows = (unit->lim.get_select_limit()) ? 1 : 0; > > DEBUG_SYNC(thd, "before_join_optimize"); > > + // rownum used somewhere in query, no limits and it is derived > + if (unlikely(thd->lex->with_rownum && > + select_lex->first_cond_optimization && > + select_lex->master_unit()->derived)) > + optimize_upper_rownum_func(); > + > + do_send_rows = (unit->lim.get_select_limit()) ? 1 : 0; Is there any particular reason why you put that between DEBUG_SYNC(thd, "before_join_optimize"); and THD_STAGE_INFO(thd, stage_optimizing); ? I'd expect them logically to go directly one after the other > + > + > THD_STAGE_INFO(thd, stage_optimizing); > > set_allowed_join_cache_types(); > @@ -14210,7 +14236,24 @@ static ORDER * > remove_const(JOIN *join,ORDER *first_order, COND *cond, > bool change_list, bool *simple_order) > { > - *simple_order= join->rollup.state == ROLLUP::STATE_NONE; > + /* > + We can't do ORDER BY using filesort if the select list contains a non > + deterministic value like RAND() or ROWNUM(). > + For example: > + SELECT RAND() as 'r' from t1 order by r; This one uses temporary for me. Can you put here a correct example please? > + > + If we would first sort the table 't1', the RAND() column would be generated > + during end_send() and the order would be wrong. > + > + Previously we had here also a test of ROLLUP: > + 'join->rollup.state == ROLLUP::STATE_NONE' > + I deleted this because > + the ROLLUP was never enforced because of a bug where the inital value > + of simple_order was ignored. Having ROLLUP tested now when the code > + is fixed causes many test failure and some wrong results so better to > + leave the code as it was related to ROLLUP. No need to explain in a comment why something that is no longer here was removed. Use git for that, commit comments. The code comment should explain the code as it is. > + */ > + *simple_order= !join->rand_table_in_field_list; why do you need rand_table_in_field_list instead of writing *simple_order= join->select_lex->select_list_tables & RAND_TABLE_BIT; ? > if (join->only_const_tables()) > return change_list ? 0 : first_order; // No need to sort > > @@ -29587,6 +29640,273 @@ void unpack_to_base_table_fields(TABLE *table) > (*cp->do_copy)(cp); > } > > +/* > + Call item->fix_after_optimize for all items registered in > + lex->fix_after_optimize > + > + This is needed for items like ROWNUM(), which needs access to structures > + created by the early optimizer pass, like JOIN > +*/ > + > +static void fix_items_after_optimize(THD *thd, SELECT_LEX *select_lex) > +{ > + List_iterator<Item> li(select_lex->fix_after_optimize); > + > + while (Item *item= li++) > + item->fix_after_optimize(thd); > +} > + > + here you could've had a comment, like /* callbacks for process_direct_rownum_comparison() */ > +static bool is_const(Item *item) > +{ > + /* > + Constant expression in WHERE can be also subquery or other expensive > + thing we shoud not evaluate on OPTIMIZATION phase > + */ > + return item->const_item() && !item->is_expensive(); > +} > + > + > +// Check if this is an unexpensive functional expression over basic constants > + > +static bool is_basic_const(Item *item) > +{ > + if (item->basic_const_item()) > + return true; > + if (!item->const_item() || item->is_expensive()) > + return false; > + if (item->type() == Item::FUNC_ITEM) > + { > + Item_func *func= ((Item_func *)item); > + if (func->argument_count() == 0) > + return false; > + bool result= true; > + Item **args= func->arguments(); > + for (uint i= 0; i < func->argument_count() && result; i++) > + result= result && is_basic_const(args[i]); > + return result; > + } > + return false; > +} is_basic_const() is a misleading name, a function of basic consts isn't a basic const itself. what's an example of an expression has is_const() == true, but is_basic_const() == false? > + > + > +inline bool is_simple_rownum_exp(Item *rownum, Item *constant) unused > +{ > + return (constant->basic_const_item() && > + rownum->type() == Item::FUNC_ITEM && > + ((Item_func*) rownum)->functype() == Item_func::ROWNUM_FUNC); > +} > + > + > +static bool temporary_set_limit(THD *thd __attribute__((unused)), > + SELECT_LEX_UNIT *unit, > + ha_rows lim) > +{ > + unit->set_limit_if_lower(lim); why is it temporary? in the PS/SP sense, just for one execution? > + return false; > +} > + > + > +static bool permanent_limit(THD *thd, SELECT_LEX_UNIT *unit, ha_rows lim) > +{ > + SELECT_LEX *gpar= unit->global_parameters(); > + if (gpar->select_limit != 0 && > + // limit can not be an expression but can be parameter > + (!gpar->select_limit->basic_const_item() || > + ((ha_rows)gpar->select_limit->val_int()) < lim)) > + return false; > + > + Query_arena *arena, backup; > + arena= thd->activate_stmt_arena_if_needed(&backup); > + > + gpar->select_limit= > + new (thd->mem_root)Item_int(thd, lim, 20 /* max 64 bit length*/ ); we have defines for that, MAX_BIGINT_WIDTH, MY_INT64_NUM_DECIMAL_DIGITS, LONGLONG_BUFFER_SIZE - whatever you prefer. > + if (gpar->select_limit == 0) > + return true; // EOM > + > + unit->set_limit(gpar); > + > + gpar->explicit_limit= true; // to show in EXPLAIN > + > + if (arena) > + thd->restore_active_arena(arena, &backup); > + > + return false; > +} > + > + > +/** > + Check posibility of LIMIT setting by rownum() of upper SELECT and do it > + > + @note Ideal is to convert something like > + SELECT ... > + FROM (SELECT ...) table > + WHERE rownum() < <CONSTANT>; > + to > + SELECT ... > + FROM (SELECT ... LIMIT <CONSTANT>) table > + WHERE rownum() < <CONSTANT>; 1. what if the outer select uses LIMIT, not ROWNUM? why not to push that down too? 2. why are you not removing rownum from the outer select? if you remove rownum and reset with_rownum=false, it'll generally give optimizer more freedom. > + > + @retval true EOM > + @retval false no errors > +*/ > + > +bool JOIN::optimize_upper_rownum_func() > +{ > + DBUG_ASSERT(select_lex->master_unit()->derived); > + > + if (select_lex->master_unit()->first_select() != select_lex) > + return false; // first will set parameter > + > + if (select_lex->master_unit()->global_parameters()->offset_limit != NULL) > + return false; // offset is set, we cannot set limit > + > + SELECT_LEX *outer_select= select_lex->master_unit()->outer_select(); > + /* > + Check that the outer query (the one that has 'WHERE rownum()...') is using > + to see if the can be use it to limit this sub query > + */ > + if (outer_select == NULL || > + !outer_select->with_rownum || > + (outer_select->options & SELECT_DISTINCT) || > + outer_select->table_list.elements != 1 || > + outer_select->where == NULL || > + outer_select->where->type() != Item::FUNC_ITEM) > + return false; > + > + return process_direct_rownum_comparison(thd, unit, outer_select->where, > + &is_basic_const, > + &permanent_limit); > +} > + > +/** > + Test if the predicate compares rownum() with a constant > + > + @return 1 No or invalid rownum() compare > + @return 0 rownum() is compared with a constant. > + In this case *args contains the constant and > + *inv_order constains 1 if the rownum() was the right > + argument, like in 'WHERE 2 >= rownum()'. > +*/ > + > +static bool check_rownum_usage(Item_func *func_item, longlong *limit, > + bool *inv_order, const_probe probe) > +{ > + Item *arg1, *arg2; > + *inv_order= 0; > + DBUG_ASSERT(func_item->argument_count() == 2); > + > + /* 'rownum op const' or 'const op field' */ > + arg1= func_item->arguments()[0]->real_item(); > + if (arg1->type() == Item::FUNC_ITEM && > + ((Item_func*) arg1)->functype() == Item_func::ROWNUM_FUNC) > + { > + arg2= func_item->arguments()[1]->real_item(); > + if ((*probe)(arg2)) > + { > + *limit= arg2->val_int(); > + return *limit <= 0 || (ulonglong) *limit >= HA_POS_ERROR; > + } > + } > + else if ((*probe)(arg1)) > + { > + arg2= func_item->arguments()[1]->real_item(); why not to assign arg2 outside of the if() ? > + if (arg2->type() == Item::FUNC_ITEM && > + ((Item_func*) arg2)->functype() == Item_func::ROWNUM_FUNC) > + { > + *limit= arg1->val_int(); > + *inv_order= 1; > + return *limit <= 0 || (ulonglong) *limit >= HA_POS_ERROR; > + } > + } > + return 1; > +} > + > + > +/* > + Limit optimization for ROWNUM() > + > + Go through the WHERE clause and find out if there are any of the following > + constructs on the top level: > + rownum() <= integer_constant > + rownum() < integer_constant > + rownum() = 1 > + > + If yes, then threat the select as if 'LIMIT integer_constant' would > + have been used > +*/ > + > +static void optimize_rownum(THD *thd, SELECT_LEX_UNIT *unit, Item *cond) > +{ > + DBUG_ENTER("optimize_rownum"); > + > + if (cond->type() == Item::COND_ITEM) > + { > + if (((Item_cond*) cond)->functype() == Item_func::COND_AND_FUNC) > + { > + List_iterator<Item> li(*((Item_cond*) cond)->argument_list()); > + Item *item; > + while ((item= li++)) > + optimize_rownum(thd, unit, item); > + } > + DBUG_VOID_RETURN; > + } > + > + process_direct_rownum_comparison(thd, unit, cond, &is_const, > + &temporary_set_limit); please, explain why it's is_const/temporary_set_limit here and is_basic_const/permanent_limit in optimize_upper_rownum_func(). > + DBUG_VOID_RETURN; > +} > + > + > +static bool process_direct_rownum_comparison(THD *thd, > + SELECT_LEX_UNIT *unit, > + Item *cond, > + const_probe probe, > + set_limit_callback set_limit) > +{ > + DBUG_ENTER("process_direct_rownum_comparison"); > + if (cond->real_type() == Item::FUNC_ITEM) > + { > + Item_func *pred= (Item_func*) cond; > + longlong limit; > + bool inv; > + > + if (pred->argument_count() != 2) > + DBUG_RETURN(false); // Not a compare functions > + if (check_rownum_usage(pred, &limit, &inv, probe)) > + DBUG_RETURN(false); > + > + Item_func::Functype pred_type= pred->functype(); > + > + if (inv && pred_type != Item_func::EQ_FUNC) > + { > + if (pred_type == Item_func::GT_FUNC) // # > rownum() > + pred_type= Item_func::LT_FUNC; > + else if (pred_type == Item_func::GE_FUNC) // # >= rownum() > + pred_type= Item_func::LE_FUNC; > + else > + DBUG_RETURN(false); > + } > + switch (pred_type) { > + case Item_func::LT_FUNC: // rownum() < # > + { > + if (limit <= 0) > + DBUG_RETURN(false); > + DBUG_RETURN((*set_limit)(thd, unit, limit - 1)); > + case Item_func::LE_FUNC: > + DBUG_RETURN((*set_limit)(thd, unit, limit)); > + case Item_func::EQ_FUNC: > + if (limit == 1) > + DBUG_RETURN((*set_limit)(thd, unit, limit)); > + break; > + default: > + break; > + } > + } what if it's rownum() > 5 ? why not to mark it as always false right away? > + } > + DBUG_RETURN(false); > +} > + > > /** > @} (end of group Query_Optimizer) Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 4

Re: [Maria-developers] a94502ab674: Added typedef decimal_digits_t (uint16) for number of decimals
by Sergei Golubchik 11 May '21

11 May '21

Hi, Michael! On Mar 26, Michael Widenius wrote: > revision-id: a94502ab674 (mariadb-10.5.2-511-ga94502ab674) > parent(s): 2be9b69f4ff > author: Michael Widenius <michael.widenius(a)gmail.com> > committer: Michael Widenius <michael.widenius(a)gmail.com> > timestamp: 2021-03-24 14:19:55 +0200 > message: > > Added typedef decimal_digits_t (uint16) for number of decimals > For fields and Item's uint8 should be good enough. After > discussions with Alexander Barkov we choose uint16 (for now) > as some format functions may accept +256 digits. > > The reason for this patch was to make the storage of decimal > digits simlar. Before this patch decimals was stored/used as > uint8, int and uint. > > Changed most decimal variables and functions to use the new > typedef. > > diff --git a/include/decimal.h b/include/decimal.h > index cab18f99348..f713409ede1 100644 > --- a/include/decimal.h > +++ b/include/decimal.h > @@ -53,10 +53,11 @@ int decimal2double(const decimal_t *from, double *to); > int double2decimal(double from, decimal_t *to); > int decimal_actual_fraction(const decimal_t *from); > int decimal2bin(const decimal_t *from, uchar *to, int precision, int scale); > -int bin2decimal(const uchar *from, decimal_t *to, int precision, int scale); > +int bin2decimal(const uchar *from, decimal_t *to, int precision, > + decimal_digits_t scale); > > -int decimal_size(int precision, int scale); > -int decimal_bin_size(int precision, int scale); > +int decimal_size(int precision, decimal_digits_t scale); > +int decimal_bin_size(int precision, decimal_digits_t scale); this is *very* confusing. "decimal" in the context of an Item mean "number of digits after a dot" "decimal" in decimal.h means the whole number, and after a dot is "scale" decimal_digits_t in decimal.h does *not* mean what you want, I'm reviewing this your patch for the third time (first was in Sept, then in Dec). And only now I finally understood what you mean (I think). Please, please, don't use decimal_digits_t in decimal.h FYI, more confusing terminology thay I might need below "precision" for decimal numbers is the total number of digits in a number "precision" for temporal types means the number of digits after a dot > int decimal_result_size(decimal_t *from1, decimal_t *from2, char op, > int param); > > diff --git a/include/m_ctype.h b/include/m_ctype.h > index 5fa8f28ff7a..66ca2bf4537 100644 > --- a/include/m_ctype.h > +++ b/include/m_ctype.h > @@ -79,6 +79,7 @@ typedef const struct my_collation_handler_st MY_COLLATION_HANDLER; > typedef const struct unicase_info_st MY_UNICASE_INFO; > typedef const struct uni_ctype_st MY_UNI_CTYPE; > typedef const struct my_uni_idx_st MY_UNI_IDX; > +typedef uint16 decimal_digits_t; This is Item (and Field) specific declaration, please put it in field.h or item.h > typedef struct unicase_info_char_st > { > diff --git a/sql/field.cc b/sql/field.cc > index 52074417046..08c168e0e21 100644 > --- a/sql/field.cc > +++ b/sql/field.cc > @@ -3281,10 +3281,11 @@ Field *Field_decimal::make_new_field(MEM_ROOT *root, TABLE *new_table, > ** Field_new_decimal > ****************************************************************************/ > > -static uint get_decimal_precision(uint len, uint8 dec, bool unsigned_val) > +static decimal_digits_t get_decimal_precision(uint len, decimal_digits_t dec, > + bool unsigned_val) > { > uint precision= my_decimal_length_to_precision(len, dec, unsigned_val); > - return MY_MIN(precision, DECIMAL_MAX_PRECISION); > + return (decimal_digits_t) MY_MIN(precision, DECIMAL_MAX_PRECISION); No, this is wrong (or, rather, inconsistent with your other changes). "precision" for DECIMAL is the total number of digits, not a number of digits after the dot. Judging by your edits in decimal.h you don't want that to be decimal_digits_t. > } > > Field_new_decimal::Field_new_decimal(uchar *ptr_arg, > @@ -10390,7 +10391,8 @@ void Column_definition::create_length_to_internal_length_bit() > void Column_definition::create_length_to_internal_length_newdecimal() > { > DBUG_ASSERT(length < UINT_MAX32); > - uint prec= get_decimal_precision((uint)length, decimals, flags & UNSIGNED_FLAG); > + decimal_digit_t prec= get_decimal_precision((uint)length, decimals, > + flags & UNSIGNED_FLAG); same as above, you should decide whether you want decimal precision to be decimal_digit_t or not. Currently this change contraditcs your decimal.h changes. > pack_length= my_decimal_get_binary_size(prec, decimals); > } > > diff --git a/sql/field.h b/sql/field.h > index 4a4f7cee2a5..5b6a69d0075 100644 > --- a/sql/field.h > +++ b/sql/field.h > @@ -2331,7 +2332,7 @@ class Field_decimal final :public Field_real { > class Field_new_decimal final :public Field_num { > public: > /* The maximum number of decimal digits can be stored */ > - uint precision; > + decimal_digits_t precision; oops. And here again you use decimal_digits_t for decimal precision. > uint bin_size; > /* > Constructors take max_length of the field as a parameter - not the > diff --git a/sql/item.h b/sql/item.h > index 1087c08869e..6753474f2dd 100644 > --- a/sql/item.h > +++ b/sql/item.h > @@ -1649,14 +1649,14 @@ class Item: public Value_source, > return type_handler()->Item_decimal_precision(this); > } > /* Returns the number of integer part digits only */ > - inline int decimal_int_part() const > - { return my_decimal_int_part(decimal_precision(), decimals); } > + inline decimal_digits_t decimal_int_part() const > + { return (decimal_digits_t) my_decimal_int_part(decimal_precision(), decimals); } and here you use decimal_digits_t for precision-scale. > /* > Returns the number of fractional digits only. > NOT_FIXED_DEC is replaced to the maximum possible number > of fractional digits, taking into account the data type. > */ > - uint decimal_scale() const > + decimal_digits_t decimal_scale() const for example, this is consistent with your other changes. scale is decimal_digits_t, it's clear. > { > return type_handler()->Item_decimal_scale(this); > } > @@ -4853,7 +4853,7 @@ class Item_temporal_literal :public Item_literal > Item_literal(thd) > { > collation= DTCollation_numeric(); > - decimals= dec_arg; > + decimals= (decimal_digits_t) dec_arg; why do you cast? I'd expect dec_arg to be decimal_digits_t already > } > > int save_in_field(Field *field, bool no_conversions) override > diff --git a/sql/item_func.cc b/sql/item_func.cc > index a0ef4020aae..db51639a5af 100644 > --- a/sql/item_func.cc > +++ b/sql/item_func.cc > @@ -2710,7 +2710,7 @@ my_decimal *Item_func_round::decimal_op(my_decimal *decimal_value) > dec= INT_MIN; > > if (!(null_value= (value.is_null() || args[1]->null_value || > - value.round_to(decimal_value, (uint) dec, > + value.round_to(decimal_value, (int) dec, > truncate ? TRUNCATE : HALF_UP) > 1))) I don't think you need to cast decimal_digits_t (aka uint16) to int explicitly. A compiler can handle it, it's not lossy. > return decimal_value; > return 0; > diff --git a/sql/item_func.h b/sql/item_func.h > index e774d9c53bd..ae94698ff96 100644 > --- a/sql/item_func.h > +++ b/sql/item_func.h > @@ -1383,10 +1383,10 @@ class Item_decimal_typecast :public Item_func > { > my_decimal decimal_value; > public: > - Item_decimal_typecast(THD *thd, Item *a, uint len, uint dec) > + Item_decimal_typecast(THD *thd, Item *a, uint len, decimal_digits_t dec) > :Item_func(thd, a) > { > - decimals= (uint8) dec; > + decimals= (decimal_digits_t) dec; not needed, dec is decimal_digits_t already > collation= DTCollation_numeric(); > fix_char_length(my_decimal_precision_to_length_no_truncation(len, dec, > unsigned_flag)); > diff --git a/sql/sql_type.cc b/sql/sql_type.cc > index 5a31b39c7b6..1c5e03f34bd 100644 > --- a/sql/sql_type.cc > +++ b/sql/sql_type.cc > @@ -1218,9 +1218,10 @@ uint32 Type_numeric_attributes::find_max_octet_length(Item **item, uint nitems) > } > > > -int Type_numeric_attributes::find_max_decimal_int_part(Item **item, uint nitems) > +decimal_digits_t Type_numeric_attributes:: > +find_max_decimal_int_part(Item **item, uint nitems) > { > - int max_int_part= 0; > + decimal_digits_t max_int_part= 0; again, "int parts" = precision-scale. it's not clear from your patch whether it should be decimal_digits_t > for (uint i=0 ; i < nitems ; i++) > set_if_bigger(max_int_part, item[i]->decimal_int_part()); > return max_int_part; > @@ -1237,11 +1238,12 @@ Type_numeric_attributes::aggregate_numeric_attributes_decimal(Item **item, > uint nitems, > bool unsigned_arg) > { > - int max_int_part= find_max_decimal_int_part(item, nitems); > + decimal_digits_t max_int_part= find_max_decimal_int_part(item, nitems); same here > decimals= find_max_decimals(item, nitems); > - int precision= MY_MIN(max_int_part + decimals, DECIMAL_MAX_PRECISION); > + decimal_digits_t precision= (decimal_digits_t) > + MY_MIN(max_int_part + decimals, DECIMAL_MAX_PRECISION); and here > max_length= my_decimal_precision_to_length_no_truncation(precision, > - (uint8) decimals, > + decimals, > unsigned_flag); > } > > @@ -6955,20 +6957,20 @@ const Vers_type_handler* Type_handler_blob_common::vers() const > > /***************************************************************************/ > > -uint Type_handler::Item_time_precision(THD *thd, Item *item) const > +decimal_digits_t Type_handler::Item_time_precision(THD *thd, Item *item) const But here it's correct. "precision" in the temporal context is the same as "scale" in the decimal context and "decimals" in the Item/Field context. So, decimal_digits_t, all right. > { > return MY_MIN(item->decimals, TIME_SECOND_PART_DIGITS); > } > > > -uint Type_handler::Item_datetime_precision(THD *thd, Item *item) const > +decimal_digits_t Type_handler::Item_datetime_precision(THD *thd, Item *item) const > { > return MY_MIN(item->decimals, TIME_SECOND_PART_DIGITS); > } > > > -uint Type_handler_string_result::Item_temporal_precision(THD *thd, Item *item, > - bool is_time) const > +decimal_digits_t Type_handler_string_result:: > +Item_temporal_precision(THD *thd, Item *item, bool is_time) const > { > StringBuffer<64> buf; > String *tmp; > @@ -7020,7 +7022,7 @@ uint Type_handler_temporal_result:: > > /***************************************************************************/ > > -uint Type_handler_string_result::Item_decimal_precision(const Item *item) const > +decimal_digits_t Type_handler_string_result::Item_decimal_precision(const Item *item) const but here it's "decimal precision", so wrong/inconsistent again. > { > uint res= item->max_char_length(); > /* Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] 58df9e3446b: MDEV-22010: use executables MariaDB named in scripts
by Sergei Golubchik 11 May '21

11 May '21

Hi, Rucha! It's ok to push, with a couple of changes, see below On May 11, Rucha Deodhar wrote: > revision-id: 58df9e3446b (mariadb-10.5.2-386-g58df9e3446b) > parent(s): de407e7cb4d > author: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > committer: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > timestamp: 2021-04-06 16:40:42 +0530 > message: > > MDEV-22010: use executables MariaDB named in scripts > diff --git a/scripts/CMakeLists.txt b/scripts/CMakeLists.txt > index 8d6a486a29d..6f3393a9c22 100644 > --- a/scripts/CMakeLists.txt > +++ b/scripts/CMakeLists.txt > @@ -142,11 +142,11 @@ IF(UNIX) > # FIND_PROC and CHECK_PID are used by mysqld_safe > IF(CMAKE_SYSTEM_NAME MATCHES "Linux") > SET (FIND_PROC > - "ps wwwp $PID | grep -v mysqld_safe | grep -- $MYSQLD > /dev/null") > + "ps wwwp $PID | grep -v mariadbd-safe | grep -- $MYSQLD > /dev/null") I think you need to filter out both mysqld_safe and mariadbd-safe here. because the user can start this script under either name, you need to ignore both. Like grep -v '$mysqld_safe\|mariadbd-safe$' > ENDIF() > IF(NOT FIND_PROC AND CMAKE_SYSTEM_NAME MATCHES "SunOS") > SET (FIND_PROC > - "ps -p $PID | grep -v mysqld_safe | grep -- $MYSQLD > /dev/null") > + "ps -p $PID | grep -v mariadbd-safe | grep -- $MYSQLD > /dev/null") > ENDIF() > > IF(NOT FIND_PROC) > diff --git a/scripts/mysql_install_db.sh b/scripts/mysql_install_db.sh > index 5f183afe8fc..05ce4f31a69 100644 > --- a/scripts/mysql_install_db.sh > +++ b/scripts/mysql_install_db.sh > @@ -613,8 +613,8 @@ then > echo "PLEASE REMEMBER TO SET A PASSWORD FOR THE MariaDB root USER !" > echo "To do so, start the server, then issue the following commands:" > echo > - echo "'$bindir/mysqladmin' -u root password 'new-password'" > - echo "'$bindir/mysqladmin' -u root -h $hostname password 'new-password'" > + echo "'$bindir/mariadb-admin' -u root password 'new-password'" > + echo "'$bindir/mariadb-admin' -u root -h $hostname password 'new-password'" > echo > echo "Alternatively you can run:" > echo "'$bindir/mysql_secure_installation'" ^^^ mariadb-secure-installation Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] e7762b3a725: MDEV-8334: Rename utf8 to utf8mb3
by Sergei Golubchik 05 May '21

05 May '21

Hi, Rucha! Looks great! Just one question below re. using global vs session old_behavior variable. On May 05, Rucha Deodhar wrote: > revision-id: e7762b3a725 (mariadb-10.5.2-583-ge7762b3a725) > parent(s): bfedf1eb4b6 > author: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > committer: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > timestamp: 2021-04-20 12:50:32 +0530 > message: > > MDEV-8334: Rename utf8 to utf8mb3 > > This patch changes the main name of 3 byte character set from utf8 to > utf8mb3. New old_mode UTF8_IS_UTF8MB3 is added and set TRUE by default, > so that utf8 would mean utf8mb3. If not set, utf8 would mean utf8mb4. > diff --git a/libmariadb b/libmariadb > --- a/libmariadb > +++ b/libmariadb > @@ -1 +1 @@ > -Subproject commit b6f8883d9687936a50a7ed79bd9e5af2340efccd > +Subproject commit 03d983b287f8a1fe855cb5ed479a3f7ab4f922ab when rebasing and pushing into 10.6 take care not to rollback C/C changes. That is, only update C/C submodule reference if it's earlier than your 03d983b287f8a1fe855cb5ed479a3f7ab4f922ab > Binary files a/mysql-test/suite/sys_vars/r/character_set_results_basic.result and b/mysql-test/suite/sys_vars/r/character_set_results_basic.result differ > diff --git a/sql/item.cc b/sql/item.cc > --- a/sql/item.cc > +++ b/sql/item.cc > @@ -2359,6 +2359,9 @@ left_is_superset(const DTCollation *left, const DTCollation *right) > > bool DTCollation::aggregate(const DTCollation &dt, uint flags) > { > + > + myf utf8_flag= global_system_variables.old_behavior & > + OLD_MODE_UTF8_IS_UTF8MB3 ? MY_UTF8_IS_UTF8MB3 : 0; if old_behavior is a session variable, then you should use session value here. > if (!my_charset_same(collation, dt.collation)) > { > /* > diff --git a/sql/mysqld.cc b/sql/mysqld.cc > --- a/sql/mysqld.cc > +++ b/sql/mysqld.cc > @@ -4099,6 +4098,8 @@ static int init_common_variables() > test purposes, to be able to start "mysqld" even if > the requested character set is not available (see bug#18743). > */ > + myf utf8_flag= global_system_variables.old_behavior & > + OLD_MODE_UTF8_IS_UTF8MB3 ? MY_UTF8_IS_UTF8MB3 : 0; ok, here there's no "session" yet, so global value is correct > for (;;) > { > char *next_character_set_name= strchr(default_character_set_name, ','); > diff --git a/sql/set_var.cc b/sql/set_var.cc > --- a/sql/set_var.cc > +++ b/sql/set_var.cc > @@ -533,11 +533,12 @@ static my_old_conv old_conv[]= > CHARSET_INFO *get_old_charset_by_name(const char *name) > { > my_old_conv *conv; > - > + myf utf8_flag= global_system_variables.old_behavior & > + OLD_MODE_UTF8_IS_UTF8MB3 ? MY_UTF8_IS_UTF8MB3 : 0; technically, you should use a session value here too. but see the old_conv array, it doesn't have "utf8" anywhere, so it doesn't matter what the flag is, you can use 0 there. > for (conv= old_conv; conv->old_name; conv++) > { > if (!my_strcasecmp(&my_charset_latin1, name, conv->old_name)) > - return get_charset_by_csname(conv->new_name, MY_CS_PRIMARY, MYF(0)); > + return get_charset_by_csname(conv->new_name, MY_CS_PRIMARY, MYF(utf8_flag)); > } > return NULL; > } > diff --git a/sql/sp.cc b/sql/sp.cc > --- a/sql/sp.cc > +++ b/sql/sp.cc > @@ -291,7 +291,8 @@ bool load_charset(MEM_ROOT *mem_root, > CHARSET_INFO **cs) > { > LEX_CSTRING cs_name; > - > + myf utf8_flag= global_system_variables.old_behavior & > + OLD_MODE_UTF8_IS_UTF8MB3 ? MY_UTF8_IS_UTF8MB3 : 0; but this needs a session value, I suppose > if (field->val_str_nopad(mem_root, &cs_name)) > { > *cs= dflt_cs; > @@ -324,9 +325,10 @@ bool load_collation(MEM_ROOT *mem_root, > *cl= dflt_cl; > return TRUE; > } > + myf utf8_flag= thd->get_utf8_flag(); Hmm, here you do use a session value. So, I suppose your using global value above was not an oversight, but you intentionally did it. What were your reasons? > > DBUG_ASSERT(cl_name.str[cl_name.length] == 0); > - *cl= get_charset_by_name(cl_name.str, MYF(0)); > + *cl= get_charset_by_name(cl_name.str, MYF(utf8_flag)); > > if (*cl == NULL) > { > diff --git a/sql/sql_class.h b/sql/sql_class.h > --- a/sql/sql_class.h > +++ b/sql/sql_class.h > @@ -1051,14 +1052,16 @@ static inline void update_global_memory_status(int64 size) > @retval NULL on error > @retval Pointter to CHARSET_INFO with the given name on success > */ > -static inline CHARSET_INFO * > -mysqld_collation_get_by_name(const char *name, > +inline CHARSET_INFO * static inline, please > +mysqld_collation_get_by_name(const char *name, bool utf8_is_utf8mb3, up to you, but wouldn't it be more convenient to pass the flag here? Then you can invoke it with thd->utf8_flag() > CHARSET_INFO *name_cs= system_charset_info) > { > CHARSET_INFO *cs; > MY_CHARSET_LOADER loader; > + myf utf8_flag= utf8_is_utf8mb3 ? MY_UTF8_IS_UTF8MB3 : 0; > my_charset_loader_init_mysys(&loader); > - if (!(cs= my_collation_get_by_name(&loader, name, MYF(0)))) > + > + if (!(cs= my_collation_get_by_name(&loader, name, MYF(utf8_flag)))) > { > ErrConvString err(name, name_cs); > my_error(ER_UNKNOWN_COLLATION, MYF(0), err.ptr()); > diff --git a/sql/sql_parse.cc b/sql/sql_parse.cc > --- a/sql/sql_parse.cc > +++ b/sql/sql_parse.cc > @@ -10449,7 +10449,10 @@ merge_charset_and_collation(CHARSET_INFO *cs, CHARSET_INFO *cl) > CHARSET_INFO *find_bin_collation(CHARSET_INFO *cs) > { > const char *csname= cs->csname; > - cs= get_charset_by_csname(csname, MY_CS_BINSORT, MYF(0)); > + myf utf8_flag= global_system_variables.old_behavior & > + OLD_MODE_UTF8_IS_UTF8MB3 ? > + MY_UTF8_IS_UTF8MB3 : 0; Why not a session value here? > + cs= get_charset_by_csname(csname, MY_CS_BINSORT, MYF(utf8_flag)); > if (!cs) > { > char tmp[65]; Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 2bceae199bb: MDEV-14974: --port ignored for --host=localhost
by Sergei Golubchik 05 May '21

05 May '21

Hi, Brandon! Just a couple of minor issues: On May 05, Brandon Nesterenko wrote: > revision-id: 2bceae199bb (mariadb-10.6.0-24-g2bceae199bb) > parent(s): 4ff4df3232f > author: Brandon Nesterenko <brandon.nesterenko(a)mariadb.com> > committer: Brandon Nesterenko <brandon.nesterenko(a)mariadb.com> > timestamp: 2021-05-05 01:01:01 +0000 > message: > > MDEV-14974: --port ignored for --host=localhost > > diff --git a/client/client_priv.h b/client/client_priv.h > index 64818d2ab8d..606b629a4d5 100644 > --- a/client/client_priv.h > +++ b/client/client_priv.h > @@ -136,3 +136,37 @@ enum options_client > Name of the sys schema database. > */ > #define SYS_SCHEMA_DB_NAME "sys" > + > + > +/** > + Utility function to implicitly change the connection protocol to a > + consistent value given the command line arguments. Additionally, > + warns the user that the protocol has been changed. > + > + Arguments: if you do `git show 2bceae199bb` you'll see git highlighting invisible spaces at line ends. Could you, please, remove them? (everywhere in your commit, not only in the comment above) > + @param [in] host Name of the host to connect to > + @param [in, out] opt_protocol Location of the protocol option > + variable to update > + @param [in] new_protocol New protocol to force > diff --git a/client/mysqlcheck.c b/client/mysqlcheck.c > index fb3103a318d..4f8891817e3 100644 > --- a/client/mysqlcheck.c > +++ b/client/mysqlcheck.c > @@ -285,10 +287,14 @@ static void usage(void) > > static my_bool > get_one_option(const struct my_option *opt, > - const char *argument, > - const char *filename __attribute__((unused))) > + const char *argument, > + const char *filename) indentation went wrong here. (only here. in other files where you removed the __attribute__ it was fine) > { > int orig_what_to_do= what_to_do; > + > + /* Track when protocol is set via CLI to not force overrides */ > + static my_bool ignore_protocol_override = FALSE; > + > DBUG_ENTER("get_one_option"); > > switch(opt->id) { > diff --git a/man/mysql.1 b/man/mysql.1 > index 03f23df3660..27a7e4d4d70 100644 > --- a/man/mysql.1 > +++ b/man/mysql.1 > @@ -1199,7 +1199,8 @@ Do not write line numbers for errors\&. Useful when you want to compare result f > \fB\-S \fR\fB\fIpath\fR\fR > .sp > For connections to > -localhost, the Unix socket file to use, or, on Windows, the name of the named pipe to use\&. > +localhost, the Unix socket file to use, or, on Windows, the name of the named pipe to use\&. > +Forces --protocol=socket when specified without other connection properties\&. here and everywhere, for -P and -S: I'd clarify that "... when specified on the command line ..." > .RE > .sp > .RS 4 > diff --git a/mysql-test/main/cli_options_force_protocol.result b/mysql-test/main/cli_options_force_protocol.result > new file mode 100644 > index 00000000000..c69a2b4f578 > --- /dev/null > +++ b/mysql-test/main/cli_options_force_protocol.result > @@ -0,0 +1,25 @@ > +# > +# MDEV-14974: --port ignored for --host=localhost > +# > +# > +# The following group of tests should produce no warnings > +# > +# exec MYSQL --host=localhost -e "status" 2>&1 | grep "Connection:\|WARNING:" > +Connection: Localhost via UNIX socket > +# exec MYSQL --port=MASTER_MYPORT --protocol=tcp -e "status" 2>&1 | grep "Connection:\|WARNING:" > +Connection: localhost via TCP/IP > +# exec MYSQL --host=localhost --port=MASTER_MYPORT --protocol=socket -e "status" 2>&1 | grep "Connection:\|WARNING:" > +Connection: Localhost via UNIX socket > +# exec MYSQL --host=127.0.0.1 --port=MASTER_MYPORT -e "status" 2>&1 | grep "Connection:\|WARNING:" > +Connection: 127.0.0.1 via TCP/IP > +# exec MYSQL --host=localhost --socket=MASTER_MYSOCK --port=MASTER_MYPORT -e "status" 2>&1 | grep "Connection:\|WARNING:" > +Connection: Localhost via UNIX socket > +# > +# The remaining tests should produce warnings > +# I now have some reservations about it, see below: > +# exec MYSQL --host=localhost --port=MASTER_MYPORT -e "status" 2>&1 | grep "Connection:\|WARNING:" > +WARNING: Forcing protocol to TCP due to option specification. Please explicitly state intended protocol. > +Connection: localhost via TCP/IP old behavior was "localhost via UNIX socket", you've changed it to TCP/IP and issued a warning. Good so far. > +# exec MYSQL --host=localhost --socket=MASTER_MYSOCK -e "status" 2>&1 | grep "Connection:\|WARNING:" > +WARNING: Forcing protocol to SOCKET due to option specification. Please explicitly state intended protocol. > +Connection: Localhost via UNIX socket here the behavior isn't changed, but you still issue a warning. Is it justified? may be it'd be better only to issue a warning when the behavior changes? Regards, Sergei

1 0

Re: [Maria-developers] 2a5663ae524: MDEV-14974: --port ignored for --host=localhost
by Sergei Golubchik 04 May '21

04 May '21

Hi, Brandon! On May 04, Brandon Nesterenko wrote: > revision-id: 2a5663ae524 (mariadb-10.6.0-18-g2a5663ae524) > parent(s): 9fe681c9e4d > author: Brandon Nesterenko <brandon.nesterenko(a)mariadb.com> > committer: Brandon Nesterenko <brandon.nesterenko(a)mariadb.com> > timestamp: 2021-05-03 19:47:07 +0000 > message: > > MDEV-14974: --port ignored for --host=localhost > > Problem: > ======= > MariaDB's command line utilities (e.g., mysql, > mysqldump, etc) silently ignore connection > property options (e.g., --port and --socket) > when protocol is not explicitly set via the > command-line for localhost connections. > > Fix: > === > If connection properties are specified without a > protocol, override the protocol to be consistent. > For example, if --port is specified, automatically > set protocol=tcp. > > Caveats: > ======= > * When multiple connection properties are > specified, nothing is overridden > * If protocol is is set via the command-line, > its value is used > > diff --git a/client/client_priv.h b/client/client_priv.h > index 64818d2ab8d..94e1fc8bb08 100644 > --- a/client/client_priv.h > +++ b/client/client_priv.h > @@ -136,3 +136,57 @@ enum options_client > Name of the sys schema database. > */ > #define SYS_SCHEMA_DB_NAME "sys" > + > + > +/** > + Utility function to implicitly change the connection protocol to a > + consistent value given the command line arguments. Additionally, > + warns the user that the protocol has been changed. > + > + Arguments: > + @param [in] warn_to The file to write the warning to > + @param [in] host Name of the host to connect to > + @param [in, out] opt_protocol Location of the protocol option > + variable to update > + @param [in] new_protocol New protocol to force > +*/ > +static inline void warn_protocol_override(FILE *warn_to, it's always stderr, isn't it? Why did you want it as an argument? > + char *host, > + uint *opt_protocol, > + uint new_protocol) > +{ > + if ((host == NULL > + || strncmp(host, LOCAL_HOST, sizeof(LOCAL_HOST)-1) == 0)) > + { > + const char *TCP_NAME = "TCP"; > + const char *SOCKET_NAME = "SOCKET"; > + char *protocol_name; > + > + if(new_protocol == MYSQL_PROTOCOL_TCP) > + protocol_name = (char *) TCP_NAME; better not to hard-code protocol names here, you can do protocol_name= sql_protocol_typelib.type_names[new_protocol-1]; > + else if(new_protocol == MYSQL_PROTOCOL_SOCKET) > + protocol_name = (char *) SOCKET_NAME; > + else > + { > + /* > + This should never be entered, but just in case we are called incorrectly > + */ > + > + fprintf(warn_to, "%s %d %s %d\n", > + "WARNING: Protocol ID ", > + new_protocol, > + " cannot override connection type. " > + "Using the configuration value of ", > + *opt_protocol); > + return; for things that cannot happen code-wise (some code guarantees that it can never happen), better add an assert (DBUG_ASSERT, in fact) and there's no need to do an warning or anything. An assert is like a self-maintaining documentation. Its main purpose is to document that something cannot happen. But unlike your comment it also checks that it doesn't become outdated. so, this could become DBUG_ASSERT(new_protocol == MYSQL_PROTOCOL_SOCKET || new_protocol == MYSQL_PROTOCOL_TCP); protocol_name= sql_protocol_typelib.type_names[new_protocol-1]; > + } > + > + fprintf(warn_to, "%s %s %s\n", > + "WARNING: Forcing protocol to ", > + protocol_name, > + " due to option specification. " > + "Please explicitly state intended protocol."); > + > + *opt_protocol = new_protocol; > + } > +} > diff --git a/client/mysql.cc b/client/mysql.cc > index 433fbd281b9..5ca9b4393ec 100644 > --- a/client/mysql.cc > +++ b/client/mysql.cc > @@ -206,6 +206,8 @@ static uint opt_protocol=0; > static const char *opt_protocol_type= ""; > static CHARSET_INFO *charset_info= &my_charset_latin1; > > +static uint protocol_to_force= MYSQL_PROTOCOL_DEFAULT; > + > #include "sslopt-vars.h" > > const char *default_dbug_option="d:t:o,/tmp/mariadb.trace"; > @@ -1162,6 +1164,9 @@ int main(int argc,char *argv[]) > close(stdout_fileno_copy); /* Clean up dup(). */ > } > > + /* We need to know if protocol-related options originate from CLI args */ > + my_defaults_mark_files = TRUE; > + > load_defaults_or_exit("my", load_default_groups, &argc, &argv); > defaults_argv=argv; > if ((status.exit_status= get_options(argc, (char **) argv))) > @@ -1171,6 +1176,14 @@ int main(int argc,char *argv[]) > exit(status.exit_status); > } > > + /* Command line options override configured protocol */ > + if (protocol_to_force > MYSQL_PROTOCOL_DEFAULT > + && protocol_to_force != opt_protocol) > + { > + warn_protocol_override(stderr, current_host, &opt_protocol, protocol_to_force); > + } > + > + > if (status.batch && !status.line_buff && > !(status.line_buff= batch_readline_init(MAX_BATCH_BUFFER_SIZE, stdin))) > { > @@ -1715,8 +1728,11 @@ static void usage(int version) > > > my_bool > -get_one_option(const struct my_option *opt, const char *argument, const char *) > +get_one_option(const struct my_option *opt, const char *argument, const char *filename) > { > + /* Track when protocol is set via CLI to not force port TCP protocol override */ > + static my_bool ignore_protocol_override = FALSE; > + > switch(opt->id) { > case OPT_CHARSETS_DIR: > strmake_buf(mysql_charsets_dir, argument); > @@ -1781,6 +1797,14 @@ get_one_option(const struct my_option *opt, const char *argument, const char *) > opt->name)) <= 0) > exit(1); > #endif > + > + /* Specification of protocol via CLI trumps implicit overrides */ > + if (filename[0] == '\0') > + { > + ignore_protocol_override = TRUE; > + protocol_to_force = MYSQL_PROTOCOL_DEFAULT; > + } > + > break; > case OPT_SERVER_ARG: > #ifdef EMBEDDED_LIBRARY > @@ -1872,6 +1896,13 @@ get_one_option(const struct my_option *opt, const char *argument, const char *) > #ifdef __WIN__ > opt_protocol = MYSQL_PROTOCOL_PIPE; > opt_protocol_type= "pipe"; > + > + /* Prioritize pipe if explicit via command line */ > + if (filename[0] == '\0') > + { > + ignore_protocol_override = TRUE; > + protocol_to_force = MYSQL_PROTOCOL_DEFAULT; > + } > #endif > break; > #include <sslopt-case.h> > @@ -1883,6 +1914,40 @@ get_one_option(const struct my_option *opt, const char *argument, const char *) > status.exit_status= 0; > mysql_end(-1); > break; > + case 'P': > + /* If port and socket are set, fall back to default behavior */ > + if (protocol_to_force == MYSQL_PROTOCOL_SOCKET) > + { > + ignore_protocol_override = TRUE; > + protocol_to_force = MYSQL_PROTOCOL_DEFAULT; > + } > + > + /* If port is set via CLI, try to force protocol to TCP */ > + if (filename[0] == '\0' && > + !ignore_protocol_override && > + MYSQL_PROTOCOL_TCP > protocol_to_force) that's a strange condition. I believe it's practically identical to protocol_to_force == MYSQL_PROTOCOL_DEFAULT but a lot more confusing. Why not to write it as protocol_to_force == MYSQL_PROTOCOL_DEFAULT ? > + { > + protocol_to_force = MYSQL_PROTOCOL_TCP; > + } > + break; > + case 'S': > +#ifndef __WIN__ > + /* If port and socket are set, fall back to default behavior */ > + if (protocol_to_force == MYSQL_PROTOCOL_TCP) > + { > + ignore_protocol_override = TRUE; > + protocol_to_force = MYSQL_PROTOCOL_DEFAULT; > + } > + > + /* Prioritize socket if set via command line */ > + if (filename[0] == '\0' && > + !ignore_protocol_override && > + MYSQL_PROTOCOL_SOCKET > protocol_to_force) same here > + { > + protocol_to_force = MYSQL_PROTOCOL_SOCKET; > + } > +#endif > + break; > case 'I': > case '?': > usage(0); > diff --git a/client/mysqladmin.cc b/client/mysqladmin.cc > index e40e82f8038..08ec14815bd 100644 > --- a/client/mysqladmin.cc > +++ b/client/mysqladmin.cc > @@ -54,6 +54,8 @@ static bool sql_log_bin_off= false; > static uint opt_protocol=0; > static myf error_flags; /* flags to pass to my_printf_error, like ME_BELL */ > > +static uint protocol_to_force= MYSQL_PROTOCOL_DEFAULT; > + > /* > When using extended-status relatively, ex_val_max_len is the estimated > maximum length for any relative value printed by extended-status. The > @@ -241,8 +243,12 @@ static const char *load_default_groups[]= > 0 }; > > my_bool > -get_one_option(const struct my_option *opt, const char *argument, const char *) > +get_one_option(const struct my_option *opt, const char *argument, const char *filename) > { > + > + /* Track when protocol is set via CLI to not force overrides */ > + static my_bool ignore_protocol_override = FALSE; > + > switch(opt->id) { > case 'c': > opt_count_iterations= 1; > @@ -274,6 +280,13 @@ get_one_option(const struct my_option *opt, const char *argument, const char *) > case 'W': > #ifdef __WIN__ > opt_protocol = MYSQL_PROTOCOL_PIPE; > + > + /* Prioritize pipe if explicit via command line */ > + if (filename[0] == '\0') > + { > + ignore_protocol_override = TRUE; > + protocol_to_force = MYSQL_PROTOCOL_DEFAULT; > + } > #endif > break; > case '#': > @@ -309,6 +322,48 @@ get_one_option(const struct my_option *opt, const char *argument, const char *) > sf_leaking_memory= 1; /* no memory leak reports here */ > exit(1); > } > + > + /* Specification of protocol via CLI trumps implicit overrides */ > + if (filename[0] == '\0') > + { > + ignore_protocol_override = TRUE; > + protocol_to_force = MYSQL_PROTOCOL_DEFAULT; > + } > + > + break; > + case 'P': > + /* If port and socket are set, fall back to default behavior */ > + if (protocol_to_force == MYSQL_PROTOCOL_SOCKET) > + { > + ignore_protocol_override = TRUE; > + protocol_to_force = MYSQL_PROTOCOL_DEFAULT; > + } > + > + /* If port is set via CLI, try to force protocol to TCP */ > + if (filename[0] == '\0' && > + !ignore_protocol_override && > + MYSQL_PROTOCOL_TCP > protocol_to_force) > + { > + protocol_to_force = MYSQL_PROTOCOL_TCP; > + } > + break; > + case 'S': > +#ifndef __WIN__ > + /* If port and socket are set, fall back to default behavior */ > + if (protocol_to_force == MYSQL_PROTOCOL_TCP) > + { > + ignore_protocol_override = TRUE; > + protocol_to_force = MYSQL_PROTOCOL_DEFAULT; > + } > + > + /* Prioritize socket if set via command line */ > + if (filename[0] == '\0' && > + !ignore_protocol_override && > + MYSQL_PROTOCOL_SOCKET > protocol_to_force) > + { > + protocol_to_force = MYSQL_PROTOCOL_SOCKET; > + } > +#endif > break; > } > return 0; > @@ -323,6 +378,10 @@ int main(int argc,char *argv[]) > > MY_INIT(argv[0]); > sf_leaking_memory=1; /* don't report memory leaks on early exits */ > + > + /* We need to know if protocol-related options originate from CLI args */ > + my_defaults_mark_files = TRUE; > + > load_defaults_or_exit("my", load_default_groups, &argc, &argv); > save_argv = argv; /* Save for free_defaults */ > > @@ -331,6 +390,13 @@ int main(int argc,char *argv[]) > temp_argv= mask_password(argc, &argv); > temp_argc= argc; > > + /* Command line options override configured protocol */ > + if (protocol_to_force > MYSQL_PROTOCOL_DEFAULT > + && protocol_to_force != opt_protocol) > + { > + warn_protocol_override(stderr, host, &opt_protocol, protocol_to_force); > + } > + > if (debug_info_flag) > my_end_arg= MY_CHECK_ERROR | MY_GIVE_INFO; > if (debug_check_flag) > diff --git a/client/mysqlbinlog.cc b/client/mysqlbinlog.cc > index fd31ab6694e..1ff6048e5ad 100644 > --- a/client/mysqlbinlog.cc > +++ b/client/mysqlbinlog.cc > @@ -98,6 +98,8 @@ static const char *output_prefix= ""; > static char **defaults_argv= 0; > static MEM_ROOT glob_root; > > +static uint protocol_to_force= MYSQL_PROTOCOL_DEFAULT; > + > #ifndef DBUG_OFF > static const char *default_dbug_option = "d:t:o,/tmp/mariadb-binlog.trace"; > const char *current_dbug_option= default_dbug_option; > @@ -1959,9 +1961,13 @@ static my_time_t convert_str_to_timestamp(const char* str) > > > extern "C" my_bool > -get_one_option(const struct my_option *opt, const char *argument, const char *) > +get_one_option(const struct my_option *opt, const char *argument, const char *filename) > { > bool tty_password=0; > + > + /* Track when protocol is set via CLI to not force overrides */ > + static my_bool ignore_protocol_override = FALSE; > + > switch (opt->id) { > #ifndef DBUG_OFF > case '#': > @@ -2011,6 +2017,14 @@ get_one_option(const struct my_option *opt, const char *argument, const char *) > sf_leaking_memory= 1; /* no memory leak reports here */ > die(); > } > + > + /* Specification of protocol via CLI trumps implicit overrides */ > + if (filename[0] == '\0') > + { > + ignore_protocol_override = TRUE; > + protocol_to_force = MYSQL_PROTOCOL_DEFAULT; > + } > + > break; > #ifdef WHEN_FLASHBACK_REVIEW_READY > case opt_flashback_review: > @@ -2092,6 +2106,40 @@ get_one_option(const struct my_option *opt, const char *argument, const char *) > case OPT_PRINT_ROW_EVENT_POSITIONS: > print_row_event_positions_used= 1; > break; > + case 'P': > + /* If port and socket are set, fall back to default behavior */ > + if (protocol_to_force == MYSQL_PROTOCOL_SOCKET) > + { > + ignore_protocol_override = TRUE; > + protocol_to_force = MYSQL_PROTOCOL_DEFAULT; > + } > + > + /* If port is set via CLI, try to force protocol to TCP */ > + if (filename[0] == '\0' && > + !ignore_protocol_override && > + MYSQL_PROTOCOL_TCP > protocol_to_force) > + { > + protocol_to_force = MYSQL_PROTOCOL_TCP; > + } > + break; > + case 'S': > +#ifndef __WIN__ > + /* If port and socket are set, fall back to default behavior */ > + if (protocol_to_force == MYSQL_PROTOCOL_TCP) > + { > + ignore_protocol_override = TRUE; > + protocol_to_force = MYSQL_PROTOCOL_DEFAULT; > + } > + > + /* Prioritize socket if set via command line */ > + if (filename[0] == '\0' && > + !ignore_protocol_override && > + MYSQL_PROTOCOL_SOCKET > protocol_to_force) > + { > + protocol_to_force = MYSQL_PROTOCOL_SOCKET; > + } > +#endif > + break; > case 'v': > if (argument == disabled_my_option) > verbose= 0; > @@ -3049,6 +3097,9 @@ int main(int argc, char** argv) > my_init_time(); // for time functions > tzset(); // set tzname > > + /* We need to know if protocol-related options originate from CLI args */ > + my_defaults_mark_files = TRUE; > + > load_defaults_or_exit("my", load_groups, &argc, &argv); > defaults_argv= argv; > > @@ -3062,6 +3113,13 @@ int main(int argc, char** argv) > > parse_args(&argc, (char***)&argv); > > + /* Command line options override configured protocol */ > + if (protocol_to_force > MYSQL_PROTOCOL_DEFAULT > + && protocol_to_force != opt_protocol) > + { > + warn_protocol_override(stderr, host, &opt_protocol, protocol_to_force); > + } > + > if (!argc || opt_version) > { > if (!opt_version) > diff --git a/client/mysqlcheck.c b/client/mysqlcheck.c > index fb3103a318d..54f8d2eb4f3 100644 > --- a/client/mysqlcheck.c > +++ b/client/mysqlcheck.c > @@ -56,6 +56,8 @@ static char *opt_skip_database; > DYNAMIC_ARRAY tables4repair, tables4rebuild, alter_table_cmds; > DYNAMIC_ARRAY views4repair; > static uint opt_protocol=0; > + > +static uint protocol_to_force= MYSQL_PROTOCOL_DEFAULT; > > enum operations { DO_CHECK=1, DO_REPAIR, DO_ANALYZE, DO_OPTIMIZE, DO_FIX_NAMES }; > const char *operation_name[]= > @@ -289,6 +291,10 @@ get_one_option(const struct my_option *opt, > const char *filename __attribute__((unused))) not __attribute__((unused)) anymore here and in other files too, I won't repeat this comment for every file > { > int orig_what_to_do= what_to_do; > + > + /* Track when protocol is set via CLI to not force overrides */ > + static my_bool ignore_protocol_override = FALSE; > + > DBUG_ENTER("get_one_option"); > > switch(opt->id) { > diff --git a/mysql-test/main/cli_options_force_protocol.result b/mysql-test/main/cli_options_force_protocol.result > --- /dev/null > +++ b/mysql-test/main/cli_options_force_protocol.result > @@ -0,0 +1,27 @@ > +# > +# MDEV-14974: --port ignored for --host=localhost > +# > +# > +# The following tests until the first cat_file should produce no warnings > +# > +# /home/buildbot/workspace/server/build-mariadb-server-debug/client/mysql --defaults-file=/home/buildbot/workspace/server/build-mariadb-server-debug/mysql-test/var/my.cnf --host=localhost -e "status" | grep "Connection:" oh-oh. You cannot hard-code your local path into the result file. Not the port number either. That's why in my previous review I wrote --echo # --host=localhost --port=MASTER_MYPORT and not --echo # $MYSQL --host=localhost --port=$MASTER_MYPORT -e "status" | grep "Connection:" (actually I had --port=$MASTER_MYPORT in my email, but that was wrong, should be no $-sign in --echo :) > +Connection: Localhost via UNIX socket > +# /home/buildbot/workspace/server/build-mariadb-server-debug/client/mysql --defaults-file=/home/buildbot/workspace/server/build-mariadb-server-debug/mysql-test/var/my.cnf --port=16000 --protocol=tcp -e "status" | grep "Connection:" > +Connection: localhost via TCP/IP > +# /home/buildbot/workspace/server/build-mariadb-server-debug/client/mysql --defaults-file=/home/buildbot/workspace/server/build-mariadb-server-debug/mysql-test/var/my.cnf --host=localhost --port=16000 --protocol=socket -e "status" | grep "Connection:" > +Connection: Localhost via UNIX socket > +# /home/buildbot/workspace/server/build-mariadb-server-debug/client/mysql --defaults-file=/home/buildbot/workspace/server/build-mariadb-server-debug/mysql-test/var/my.cnf --host=127.0.0.1 --port=16000 -e "status" | grep "Connection:" > +Connection: 127.0.0.1 via TCP/IP > +# /home/buildbot/workspace/server/build-mariadb-server-debug/client/mysql --defaults-file=/home/buildbot/workspace/server/build-mariadb-server-debug/mysql-test/var/my.cnf --host=localhost --socket=/home/buildbot/workspace/server/build-mariadb-server-debug/mysql-test/var/tmp/mysqld.1.sock --port=16000 -e "status" | grep "Connection:" > +Connection: Localhost via UNIX socket > +CURRENT_TEST: main.cli_options_force_protocol > +# > +# The remaining tests should produce warnings > +# > +# /home/buildbot/workspace/server/build-mariadb-server-debug/client/mysql --defaults-file=/home/buildbot/workspace/server/build-mariadb-server-debug/mysql-test/var/my.cnf --host=localhost --port=16000 -e "status" | grep "Connection:" > +Connection: localhost via TCP/IP > +# /home/buildbot/workspace/server/build-mariadb-server-debug/client/mysql --defaults-file=/home/buildbot/workspace/server/build-mariadb-server-debug/mysql-test/var/my.cnf --host=localhost --socket=/home/buildbot/workspace/server/build-mariadb-server-debug/mysql-test/var/tmp/mysqld.1.sock -e "status" | grep "Connection:" > +Connection: Localhost via UNIX socket > +CURRENT_TEST: main.cli_options_force_protocol > +WARNING: Forcing protocol to TCP due to option specification. Please explicitly state intended protocol. > +WARNING: Forcing protocol to SOCKET due to option specification. Please explicitly state intended protocol. > diff --git a/mysql-test/main/cli_options_force_protocol.test b/mysql-test/main/cli_options_force_protocol.test > new file mode 100644 > index 00000000000..f91d7833a0d > --- /dev/null > +++ b/mysql-test/main/cli_options_force_protocol.test > @@ -0,0 +1,41 @@ > +--echo # > +--echo # MDEV-14974: --port ignored for --host=localhost > +--echo # > + > +--source include/not_embedded.inc > +--source include/not_windows.inc > + > +--echo # > +--echo # The following tests until the first cat_file should produce no warnings > +--echo # > + > +--echo # $MYSQL --host=localhost -e "status" | grep "Connection:" > +--exec $MYSQL --host=localhost -e "status" | grep "Connection:" > + > +--echo # $MYSQL --port=$MASTER_MYPORT --protocol=tcp -e "status" | grep "Connection:" > +--exec $MYSQL --port=$MASTER_MYPORT --protocol=tcp -e "status" | grep "Connection:" > + > +--echo # $MYSQL --host=localhost --port=$MASTER_MYPORT --protocol=socket -e "status" | grep "Connection:" > +--exec $MYSQL --host=localhost --port=$MASTER_MYPORT --protocol=socket -e "status" | grep "Connection:" > + > +--echo # $MYSQL --host=127.0.0.1 --port=$MASTER_MYPORT -e "status" | grep "Connection:" > +--exec $MYSQL --host=127.0.0.1 --port=$MASTER_MYPORT -e "status" | grep "Connection:" > + > +--echo # $MYSQL --host=localhost --socket=$MASTER_MYSOCK --port=$MASTER_MYPORT -e "status" | grep "Connection:" > +--exec $MYSQL --host=localhost --socket=$MASTER_MYSOCK --port=$MASTER_MYPORT -e "status" | grep "Connection:" > + > +--cat_file $MYSQLTEST_VARDIR/log/current_test > + > + > +--echo # > +--echo # The remaining tests should produce warnings > +--echo # > + > +--echo # $MYSQL --host=localhost --port=$MASTER_MYPORT -e "status" | grep "Connection:" > +--exec $MYSQL --host=localhost --port=$MASTER_MYPORT -e "status" | grep "Connection:" > + > +--echo # $MYSQL --host=localhost --socket=$MASTER_MYSOCK -e "status" | grep "Connection:" > +--exec $MYSQL --host=localhost --socket=$MASTER_MYSOCK -e "status" | grep "Connection:" > + > +--cat_file $MYSQLTEST_VARDIR/log/current_test I still don't understand why you include current test name into the result file. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 242cc3fc41f: MDEV-14974: --port ignored for --host=localhost
by Sergei Golubchik 03 May '21

03 May '21

Hi, Brandon! On May 03, Brandon Nesterenko wrote: > revision-id: 242cc3fc41f (mariadb-10.6.0-17-g242cc3fc41f) > parent(s): fd8c68c7fe6 > author: Brandon Nesterenko <brandon.nesterenko(a)mariadb.com> > committer: Brandon Nesterenko <brandon.nesterenko(a)mariadb.com> > timestamp: 2021-04-30 23:17:37 +0000 > message: > > MDEV-14974: --port ignored for --host=localhost > > Problem: > ======= > MariaDB's command line utilities (e.g., mysql, mysqldump, etc) silently ignore the --port option if no host is given or it is localhost. please, try to avoid very long lines in commit comments, they're difficult to read in git console tools. > > Fix: > === > During configuration processing, force protocol to TCP if port was specified via the command line. However, if protocol is additionally specified via the command line, it will be prioritized. > > diff --git a/client/client_priv.h b/client/client_priv.h > index 64818d2ab8d..6cb1f02ba33 100644 > --- a/client/client_priv.h > +++ b/client/client_priv.h > @@ -136,3 +136,85 @@ enum options_client > Name of the sys schema database. > */ > #define SYS_SCHEMA_DB_NAME "sys" > + > + > + > +/** > + Changes flags to prepare for (and revert to the previous state) > + from handle_options. > + > + When creating the state, the original state is saved and can be later > + resumed by calling this function and setting do_revert_values to TRUE. > + > + Flags: > + my_defaults_mark_files: propagates the source of a set option > + > + @param [in] do_revert_values behavior should revert when TRUE, prepare > + when FALSE > + */ > +static inline void set_flags_for_option_handling(my_bool do_revert_values) > +{ > + static my_bool prev_mark_files; > + > + if (do_revert_values) > + { > + my_defaults_mark_files = prev_mark_files; > + } > + else > + { > + prev_mark_files = my_defaults_mark_files; > + my_defaults_mark_files = TRUE; > + } > +} this isn't very helpful. my_defaults_mark_files is part of the my_getopt API. It's supposed to be used directly by whatever tool uses my_getopt. Here you've turned one line my_defaults_mark_files= TRUE; into 27 lines, so when I (or somebody reading the code) will see set_flags_for_option_handling(...); he'd have to go to the function definition, read the comment and the function, all 27 lines, understand what they do, and then jump back. I'd rather prefer to read one line with a very clear semantics and no conditionals instead. > + > +/** > + Helper function to prepare state for handle_options (current state is saved) > + */ > +static inline void prepare_option_handling_flags() > +{ > + set_flags_for_option_handling(FALSE); > +} > + > +/** > + Helper function to revert state after handle_options to that from before > + prepare_option_handling_flags was called > + */ > +static inline void revert_option_handling_flags() > +{ > + set_flags_for_option_handling(TRUE); > +} and it keeps going on. if you remove the first function, you won't need these two functions either. Saving old value, restoring it. my_defaults_mark_files affects only my_load_defaults(). It makes no sense to restore it. > +/** > + > + Utility function to force the connection protocol to TCP when > + just the port is specified via the command line. Additionally, > + warns the user that the protocol has been changed (MDEV-14974). > + > + Notes: > + 1) This only takes effect when connecting to localhost > + 2) Windows uses TCP by default > + > + Arguments: > + @param [in] warn_to The file to write the warning to > + @param [in] host Name of the host to connect to > + @param [in, out] protocol_loc Location of the protocol option > + variable to update > +*/ > +static inline void warn_tcp_protocol_override(FILE *warn_to, > + char *host, > + uint *protocol_loc) > +{ > +#ifndef _WIN32 > + if ((host == NULL > + || strncmp(host, LOCAL_HOST, sizeof(LOCAL_HOST)-1) == 0)) > + { > + fprintf(warn_to, "WARNING: " > + "Forcing protocol to TCP due to port specification. " > + "Please explicitly state TCP protocol or remove " > + "port if unintended.\n"); > + *protocol_loc = MYSQL_PROTOCOL_TCP; > + } > +#endif > +} > diff --git a/client/mysql.cc b/client/mysql.cc > index 433fbd281b9..980daa14365 100644 > --- a/client/mysql.cc > +++ b/client/mysql.cc > @@ -206,6 +206,10 @@ static uint opt_protocol=0; > static const char *opt_protocol_type= ""; > static CHARSET_INFO *charset_info= &my_charset_latin1; > > +#ifndef _WIN32 > +static my_bool port_forcing_tcp_proto = FALSE; > +#endif > + > #include "sslopt-vars.h" > > const char *default_dbug_option="d:t:o,/tmp/mariadb.trace"; > @@ -1162,6 +1166,8 @@ int main(int argc,char *argv[]) > close(stdout_fileno_copy); /* Clean up dup(). */ > } > > + prepare_option_handling_flags(); Just do my_defaults_mark_files= TRUE; here > load_defaults_or_exit("my", load_default_groups, &argc, &argv); > defaults_argv=argv; > if ((status.exit_status= get_options(argc, (char **) argv))) > @@ -1171,6 +1177,14 @@ int main(int argc,char *argv[]) > exit(status.exit_status); > } > > + revert_option_handling_flags(); and don't revert it. > + if (port_forcing_tcp_proto) > + { > + warn_tcp_protocol_override(stderr, current_host, &opt_protocol); I'm not sure it needs a warning. But I don't have strong arguments against the warning either, so let's keep a warning, if you like it that way. Why everything is not on _WIN32 ? It has named pipes and tcp. Wouldn't it have the same issue with --port? > + } > + > + > if (status.batch && !status.line_buff && > !(status.line_buff= batch_readline_init(MAX_BATCH_BUFFER_SIZE, stdin))) > { > @@ -1715,8 +1729,11 @@ static void usage(int version) > > > my_bool > -get_one_option(const struct my_option *opt, const char *argument, const char *) > +get_one_option(const struct my_option *opt, const char *argument, const char *filename) > { > + /* Track when protocol is set via CLI to not force port TCP protocol override */ > + static my_bool ignore_port_override = FALSE; > + > switch(opt->id) { > case OPT_CHARSETS_DIR: > strmake_buf(mysql_charsets_dir, argument); > @@ -1780,6 +1797,19 @@ get_one_option(const struct my_option *opt, const char *argument, const char *) > else if ((opt_protocol= find_type_with_warning(argument, &sql_protocol_typelib, > opt->name)) <= 0) > exit(1); > +#ifndef _WIN32 > + /* MDEV-14974 > + * > + * Where specifying port implicitly will set the protocol to use TCP, if the > + * protocol is explicitly set after the port, prioritize protocol. > + */ please, see how multi-comments are usually formatted, and follow the same style. Generally there's no need to specify the MDEV, and there's no vertical line of asterisks. > + if(filename[0] == '\0') > + { > + /* Protocol is specified via command line */ > + ignore_port_override = TRUE; > + port_forcing_tcp_proto = FALSE; > + } > +#endif > #endif > break; > case OPT_SERVER_ARG: > @@ -1883,6 +1913,20 @@ get_one_option(const struct my_option *opt, const char *argument, const char *) > status.exit_status= 0; > mysql_end(-1); > break; > + case 'P': > +#ifndef _WIN32 > + /* MDEV-14974: > + * > + * If port is set via CLI, change protocol to TCP unless protocol has already > + * been specified via CLI or TCP is already used > + */ > + if (filename[0] == '\0' && !ignore_port_override > + && opt_protocol != MYSQL_PROTOCOL_TCP) > + { > + port_forcing_tcp_proto = TRUE; > + } let's also do the same treatment for --socket. If it's specified on the command line - force the socket protocol. What could be the least surprising behavior? I'd think the most intuitive would be, like, WYSIWIG thing - * if port is specified on the command line (but no socket or protocol) - it forces protocol=TCP * if socket is specified on the command line (but no port of protocol) - it forces protocol=SOCKET * if protocol is specified explicitly anywhere on the command line - port and socket lose their magic behavior * if there's no protocol, but both socket and port are specified? - I don't know, perhaps, let's keep the old behavior, no magic? hmm. How does --host affect that? > +#endif > + break; > case 'I': > case '?': > usage(0); > @@ -1916,8 +1960,10 @@ static int get_options(int argc, char **argv) > opt_max_allowed_packet= *mysql_params->p_max_allowed_packet; > opt_net_buffer_length= *mysql_params->p_net_buffer_length; > > + my_defaults_mark_files = TRUE; > if ((ho_error=handle_options(&argc, &argv, my_long_options, get_one_option))) > return(ho_error); > + my_defaults_mark_files = FALSE; my_defaults_mark_files has no effect on handle_options() > *mysql_params->p_max_allowed_packet= opt_max_allowed_packet; > *mysql_params->p_net_buffer_length= opt_net_buffer_length; > > diff --git a/mysql-test/main/port_force_tcp.result b/mysql-test/main/port_force_tcp.result > new file mode 100644 > index 00000000000..ae976bda6cc > --- /dev/null > +++ b/mysql-test/main/port_force_tcp.result > @@ -0,0 +1,14 @@ > +# > +# MDEV-14974: --port ignored for --host=localhost > +# > +Connection: Localhost via UNIX socket > +CURRENT_TEST: main.port_force_tcp > +Connection: localhost via TCP/IP > +CURRENT_TEST: main.port_force_tcp > +Connection: Localhost via UNIX socket > +CURRENT_TEST: main.port_force_tcp > +Connection: 127.0.0.1 via TCP/IP > +CURRENT_TEST: main.port_force_tcp > +Connection: localhost via TCP/IP > +CURRENT_TEST: main.port_force_tcp > +WARNING: Forcing protocol to TCP due to port specification. Please explicitly state TCP protocol or remove port if unintended. > diff --git a/mysql-test/main/port_force_tcp.test b/mysql-test/main/port_force_tcp.test > new file mode 100644 > index 00000000000..607670d16b0 > --- /dev/null > +++ b/mysql-test/main/port_force_tcp.test > @@ -0,0 +1,20 @@ > +--echo # > +--echo # MDEV-14974: --port ignored for --host=localhost > +--echo # > + > +--source include/not_embedded.inc > + > +--exec $MYSQL --host=localhost -e "status" | grep "Connection:" > +--cat_file $MYSQLTEST_VARDIR/log/current_test > + > +--exec $MYSQL --port=$MASTER_MYPORT --protocol=tcp -e "status" | grep "Connection:" > +--cat_file $MYSQLTEST_VARDIR/log/current_test > + > +--exec $MYSQL --host=localhost --port=$MASTER_MYPORT --protocol=socket -e "status" | grep "Connection:" > +--cat_file $MYSQLTEST_VARDIR/log/current_test > + > +--exec $MYSQL --host=127.0.0.1 --port=$MASTER_MYPORT -e "status" | grep "Connection:" > +--cat_file $MYSQLTEST_VARDIR/log/current_test > + > +--exec $MYSQL --host=localhost --port=$MASTER_MYPORT -e "status" | grep "Connection:" > +--cat_file $MYSQLTEST_VARDIR/log/current_test this will likely fail on windows, you need to source not_windows.inc too. why do you cat current_test after every --exec? better add --echo before every exec, like --echo # --host=localhost --port=$MASTER_MYPORT --exec $MYSQL --host=localhost --port=$MASTER_MYPORT -e "status" | grep "Connection:" Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 242cc3fc41f: MDEV-14974: --port ignored for --host=localhost
by Sergei Golubchik 03 May '21

03 May '21

Hi, Brandon! On May 03, Brandon Nesterenko wrote: > revision-id: 242cc3fc41f (mariadb-10.6.0-17-g242cc3fc41f) > parent(s): fd8c68c7fe6 > author: Brandon Nesterenko <brandon.nesterenko(a)mariadb.com> > committer: Brandon Nesterenko <brandon.nesterenko(a)mariadb.com> > timestamp: 2021-04-30 23:17:37 +0000 > message: > > MDEV-14974: --port ignored for --host=localhost > > Problem: > ======= > MariaDB's command line utilities (e.g., mysql, mysqldump, etc) silently ignore the --port option if no host is given or it is localhost. please, try to avoid very long lines in commit comments, they're difficult to read in git console tools. > > Fix: > === > During configuration processing, force protocol to TCP if port was specified via the command line. However, if protocol is additionally specified via the command line, it will be prioritized. > > diff --git a/client/client_priv.h b/client/client_priv.h > index 64818d2ab8d..6cb1f02ba33 100644 > --- a/client/client_priv.h > +++ b/client/client_priv.h > @@ -136,3 +136,85 @@ enum options_client > Name of the sys schema database. > */ > #define SYS_SCHEMA_DB_NAME "sys" > + > + > + > +/** > + Changes flags to prepare for (and revert to the previous state) > + from handle_options. > + > + When creating the state, the original state is saved and can be later > + resumed by calling this function and setting do_revert_values to TRUE. > + > + Flags: > + my_defaults_mark_files: propagates the source of a set option > + > + @param [in] do_revert_values behavior should revert when TRUE, prepare > + when FALSE > + */ > +static inline void set_flags_for_option_handling(my_bool do_revert_values) > +{ > + static my_bool prev_mark_files; > + > + if (do_revert_values) > + { > + my_defaults_mark_files = prev_mark_files; > + } > + else > + { > + prev_mark_files = my_defaults_mark_files; > + my_defaults_mark_files = TRUE; > + } > +} this isn't very helpful. my_defaults_mark_files is part of the my_getopt API. It's supposed to be used directly by whatever tool uses my_getopt. Here you've turned one line my_defaults_mark_files= TRUE; into 27 lines, so when I (or somebody reading the code) will see set_flags_for_option_handling(...); he'd have to go to the function definition, read the comment and the function, all 27 lines, understand what they do, and then jump back. I'd rather prefer to read one line with a very clear semantics and no conditionals instead. > + > +/** > + Helper function to prepare state for handle_options (current state is saved) > + */ > +static inline void prepare_option_handling_flags() > +{ > + set_flags_for_option_handling(FALSE); > +} > + > +/** > + Helper function to revert state after handle_options to that from before > + prepare_option_handling_flags was called > + */ > +static inline void revert_option_handling_flags() > +{ > + set_flags_for_option_handling(TRUE); > +} and it keeps going on. if you remove the first function, you won't need these two functions either. Saving old value, restoring it. my_defaults_mark_files affects only my_load_defaults(). It makes no sense to restore it. > +/** > + > + Utility function to force the connection protocol to TCP when > + just the port is specified via the command line. Additionally, > + warns the user that the protocol has been changed (MDEV-14974). > + > + Notes: > + 1) This only takes effect when connecting to localhost > + 2) Windows uses TCP by default > + > + Arguments: > + @param [in] warn_to The file to write the warning to > + @param [in] host Name of the host to connect to > + @param [in, out] protocol_loc Location of the protocol option > + variable to update > +*/ > +static inline void warn_tcp_protocol_override(FILE *warn_to, > + char *host, > + uint *protocol_loc) > +{ > +#ifndef _WIN32 > + if ((host == NULL > + || strncmp(host, LOCAL_HOST, sizeof(LOCAL_HOST)-1) == 0)) > + { > + fprintf(warn_to, "WARNING: " > + "Forcing protocol to TCP due to port specification. " > + "Please explicitly state TCP protocol or remove " > + "port if unintended.\n"); > + *protocol_loc = MYSQL_PROTOCOL_TCP; > + } > +#endif > +} > diff --git a/client/mysql.cc b/client/mysql.cc > index 433fbd281b9..980daa14365 100644 > --- a/client/mysql.cc > +++ b/client/mysql.cc > @@ -206,6 +206,10 @@ static uint opt_protocol=0; > static const char *opt_protocol_type= ""; > static CHARSET_INFO *charset_info= &my_charset_latin1; > > +#ifndef _WIN32 > +static my_bool port_forcing_tcp_proto = FALSE; > +#endif > + > #include "sslopt-vars.h" > > const char *default_dbug_option="d:t:o,/tmp/mariadb.trace"; > @@ -1162,6 +1166,8 @@ int main(int argc,char *argv[]) > close(stdout_fileno_copy); /* Clean up dup(). */ > } > > + prepare_option_handling_flags(); Just do my_defaults_mark_files= TRUE; here > load_defaults_or_exit("my", load_default_groups, &argc, &argv); > defaults_argv=argv; > if ((status.exit_status= get_options(argc, (char **) argv))) > @@ -1171,6 +1177,14 @@ int main(int argc,char *argv[]) > exit(status.exit_status); > } > > + revert_option_handling_flags(); and don't revert it. > + if (port_forcing_tcp_proto) > + { > + warn_tcp_protocol_override(stderr, current_host, &opt_protocol); I'm not sure it needs a warning. But I don't have strong arguments against the warning either, so let's keep a warning, if you like it that way. Why everything is not on _WIN32 ? It has named pipes and tcp. Wouldn't it have the same issue with --port? > + } > + > + > if (status.batch && !status.line_buff && > !(status.line_buff= batch_readline_init(MAX_BATCH_BUFFER_SIZE, stdin))) > { > @@ -1715,8 +1729,11 @@ static void usage(int version) > > > my_bool > -get_one_option(const struct my_option *opt, const char *argument, const char *) > +get_one_option(const struct my_option *opt, const char *argument, const char *filename) > { > + /* Track when protocol is set via CLI to not force port TCP protocol override */ > + static my_bool ignore_port_override = FALSE; > + > switch(opt->id) { > case OPT_CHARSETS_DIR: > strmake_buf(mysql_charsets_dir, argument); > @@ -1780,6 +1797,19 @@ get_one_option(const struct my_option *opt, const char *argument, const char *) > else if ((opt_protocol= find_type_with_warning(argument, &sql_protocol_typelib, > opt->name)) <= 0) > exit(1); > +#ifndef _WIN32 > + /* MDEV-14974 > + * > + * Where specifying port implicitly will set the protocol to use TCP, if the > + * protocol is explicitly set after the port, prioritize protocol. > + */ please, see how multi-comments are usually formatted, and follow the same style. Generally there's no need to specify the MDEV, and there's no vertical line of asterisks. > + if(filename[0] == '\0') > + { > + /* Protocol is specified via command line */ > + ignore_port_override = TRUE; > + port_forcing_tcp_proto = FALSE; > + } > +#endif > #endif > break; > case OPT_SERVER_ARG: > @@ -1883,6 +1913,20 @@ get_one_option(const struct my_option *opt, const char *argument, const char *) > status.exit_status= 0; > mysql_end(-1); > break; > + case 'P': > +#ifndef _WIN32 > + /* MDEV-14974: > + * > + * If port is set via CLI, change protocol to TCP unless protocol has already > + * been specified via CLI or TCP is already used > + */ > + if (filename[0] == '\0' && !ignore_port_override > + && opt_protocol != MYSQL_PROTOCOL_TCP) > + { > + port_forcing_tcp_proto = TRUE; > + } let's also do the same treatment for --socket. If it's specified on the command line - force the socket protocol. What could be the least surprising behavior? I'd think the most intuitive would be, like, WYSIWIG thing - * if port is specified on the command line (but no socket or protocol) - it forces protocol=TCP * if socket is specified on the command line (but no port of protocol) - it forces protocol=SOCKET * if protocol is specified explicitly anywhere on the command line - port and socket lose their magic behavior * if there's no protocol, but both socket and port are specified? - I don't know, perhaps, let's keep the old behavior, no magic? hmm. How does --host affect that? > +#endif > + break; > case 'I': > case '?': > usage(0); > @@ -1916,8 +1960,10 @@ static int get_options(int argc, char **argv) > opt_max_allowed_packet= *mysql_params->p_max_allowed_packet; > opt_net_buffer_length= *mysql_params->p_net_buffer_length; > > + my_defaults_mark_files = TRUE; > if ((ho_error=handle_options(&argc, &argv, my_long_options, get_one_option))) > return(ho_error); > + my_defaults_mark_files = FALSE; my_defaults_mark_files has no effect on handle_options() > *mysql_params->p_max_allowed_packet= opt_max_allowed_packet; > *mysql_params->p_net_buffer_length= opt_net_buffer_length; > > diff --git a/mysql-test/main/port_force_tcp.result b/mysql-test/main/port_force_tcp.result > new file mode 100644 > index 00000000000..ae976bda6cc > --- /dev/null > +++ b/mysql-test/main/port_force_tcp.result > @@ -0,0 +1,14 @@ > +# > +# MDEV-14974: --port ignored for --host=localhost > +# > +Connection: Localhost via UNIX socket > +CURRENT_TEST: main.port_force_tcp > +Connection: localhost via TCP/IP > +CURRENT_TEST: main.port_force_tcp > +Connection: Localhost via UNIX socket > +CURRENT_TEST: main.port_force_tcp > +Connection: 127.0.0.1 via TCP/IP > +CURRENT_TEST: main.port_force_tcp > +Connection: localhost via TCP/IP > +CURRENT_TEST: main.port_force_tcp > +WARNING: Forcing protocol to TCP due to port specification. Please explicitly state TCP protocol or remove port if unintended. > diff --git a/mysql-test/main/port_force_tcp.test b/mysql-test/main/port_force_tcp.test > new file mode 100644 > index 00000000000..607670d16b0 > --- /dev/null > +++ b/mysql-test/main/port_force_tcp.test > @@ -0,0 +1,20 @@ > +--echo # > +--echo # MDEV-14974: --port ignored for --host=localhost > +--echo # > + > +--source include/not_embedded.inc > + > +--exec $MYSQL --host=localhost -e "status" | grep "Connection:" > +--cat_file $MYSQLTEST_VARDIR/log/current_test > + > +--exec $MYSQL --port=$MASTER_MYPORT --protocol=tcp -e "status" | grep "Connection:" > +--cat_file $MYSQLTEST_VARDIR/log/current_test > + > +--exec $MYSQL --host=localhost --port=$MASTER_MYPORT --protocol=socket -e "status" | grep "Connection:" > +--cat_file $MYSQLTEST_VARDIR/log/current_test > + > +--exec $MYSQL --host=127.0.0.1 --port=$MASTER_MYPORT -e "status" | grep "Connection:" > +--cat_file $MYSQLTEST_VARDIR/log/current_test > + > +--exec $MYSQL --host=localhost --port=$MASTER_MYPORT -e "status" | grep "Connection:" > +--cat_file $MYSQLTEST_VARDIR/log/current_test this will likely fail on windows, you need to source not_windows.inc too. why do you cat current_test after every --exec? better add --echo before every exec, like --echo # --host=localhost --port=$MASTER_MYPORT --exec $MYSQL --host=localhost --port=$MASTER_MYPORT -e "status" | grep "Connection:" Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] a7032fbb32d: Bug#29363867: LOST CONNECTION TO MYSQL SERVER DURING QUERY
by Sergei Golubchik 27 Apr '21

27 Apr '21

Hi, Sanja! On Apr 27, Oleksandr Byelkin wrote: > revision-id: a7032fbb32d (mariadb-10.2.31-906-ga7032fbb32d) > parent(s): b862377c3e9 > author: Oleksandr Byelkin <sanja(a)mariadb.com> > committer: Oleksandr Byelkin <sanja(a)mariadb.com> > timestamp: 2021-04-27 16:24:43 +0200 > message: > > Bug#29363867: LOST CONNECTION TO MYSQL SERVER DURING QUERY > > The problem is that sharing default expression among set instruction > leads to attempt access result field of function created in > other instruction runtime MEM_ROOT and already freed > (a bit different then MySQL problem). > > Fix is the same as in MySQL (but no optimisation for constant), turn > DECLARE a, b, c type DEFAULT expr; > to > DECLARE a type DEFAULT expr, b type DEFAULT a, c type DEFAULT a; > > diff --git a/mysql-test/t/sp.test b/mysql-test/t/sp.test > index f13b3fbc281..007dfd1a10b 100644 > --- a/mysql-test/t/sp.test > +++ b/mysql-test/t/sp.test > @@ -10025,4 +10025,22 @@ DROP PROCEDURE p1; > DROP VIEW v1; > DROP TABLE t1; > > + > +--echo # > +--echo # forgot the bug summary (the first line of the commit comment) > +--echo # > + > +delimiter |; > +create function f1() returns bigint return now()-1| > +create procedure p1() > +begin > + declare b, c bigint default f1(); > + select b-c; > +end| > +call p1()| > +drop procedure p1| > +drop function f1| > +delimiter ;| > + > + > --echo #End of 10.2 tests > diff --git a/sql/item.cc b/sql/item.cc > index 42272fe0148..ec9f4ffb993 100644 > --- a/sql/item.cc > +++ b/sql/item.cc > @@ -2933,7 +2933,7 @@ bool Item_field::eq(const Item *item, bool binary_cmp) const > > table_map Item_field::used_tables() const > { > - if (field->table->const_table) > + if (!field || !field->table || field->table->const_table) in what cases can field or field->table be NULL here? > return 0; // const item > return (get_depended_from() ? OUTER_REF_TABLE_BIT : field->table->map); > } Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] e92037989f7: MDEV-21117: refine the server binlog-based recovery for semisync
by Sergei Golubchik 25 Apr '21

25 Apr '21

Hi, Andrei! Don't be confused by the subject, this is a review of git diff 450c017c2d 2fa526e26e that is of everything, combined. Not just one e92037989f7 commit. On Apr 25, Sujatha wrote: > revision-id: e92037989f7 (mariadb-10.3.26-128-ge92037989f7) > parent(s): 450c017c2d9 > author: Sujatha <sujatha.sivakumar(a)mariadb.com> > committer: Andrei Elkin <andrei.elkin(a)mariadb.com> > timestamp: 2021-04-13 12:26:12 +0300 > message: > > MDEV-21117: refine the server binlog-based recovery for semisync > diff --git a/libmariadb b/libmariadb > index fc431a035a2..e3824422064 160000 > --- a/libmariadb > +++ b/libmariadb > @@ -1 +1 @@ > -Subproject commit fc431a035a21ac1d4ef25d9d3cd8c4d7e64a8ee7 > +Subproject commit e38244220646a7e95c9be22576460aa7a4eb715f This is clearly a mistake, you erroneously checked in old libmaridb commit, rolling back a bunch of changes. See your commit 4bc83b2749 > diff --git a/mysql-test/suite/binlog/t/binlog_truncate_multi_engine-master.opt b/mysql-test/suite/binlog/t/binlog_truncate_multi_engine-master.opt > new file mode 100644 > index 00000000000..df675545bf9 > --- /dev/null > +++ b/mysql-test/suite/binlog/t/binlog_truncate_multi_engine-master.opt please rename this file to mysql-test/include/have_rocksdb.opt > diff --git a/mysql-test/suite/binlog/t/binlog_truncate_active_log.test b/mysql-test/suite/binlog/t/binlog_truncate_active_log.test > new file mode 100644 > index 00000000000..2b794d02dd0 > --- /dev/null > +++ b/mysql-test/suite/binlog/t/binlog_truncate_active_log.test > @@ -0,0 +1,74 @@ > +# ==== Purpose ==== > +# > +# Test verifies the truncation of single binary log file. > +# > +# ==== References ==== > +# > +# MDEV-21117: recovery for --rpl-semi-sync-slave-enabled server > + > +--source include/have_innodb.inc > +--source include/have_aria.inc > +# File: binlog_truncate_active_log.inc included in test makes use of > +# 'debug_sync' facility. > +--source include/have_debug_sync.inc you wouldn't need a comment if you'd include have_debug_sync.inc directly into binlog_truncate_active_log.inc. but ok, whatever you like > +--source include/have_binlog_format_statement.inc > + > +call mtr.add_suppression("Can.t init tc log"); > +call mtr.add_suppression("Aborting"); > + > +# The following cases are tested: > +# A. 2pc transaction is followed by a blank "zero-engines" one > +# B. 2pc transaction follows the blank one > +# C. Similarly to A, with the XA blank transaction > + > +RESET MASTER; > +CREATE TABLE t (f INT) ENGINE=INNODB; > +CREATE TABLE t2 (f INT) ENGINE=INNODB; > +CREATE TABLE tm (f INT) ENGINE=Aria; could you add a comment, explaining why you're using Aria here. (you wrote that in an email, but please add a comment too) > + > +--echo # Case A. > +# Using 'debug_sync' hold 'query1' execution after 'query1' is flushed and > +# synced to binary log but not yet committed. In an another connection hold > +# 'query2' execution after 'query2' is flushed and synced to binlog. > +# Crash and restart server with --rpl-semi-sync-slave-enabled=1 > +# > +# During recovery of binary log 'query1' status is checked with InnoDB engine, > +# it will be in prepared but not yet commited. All transactions starting from > +# 'query1' onwards will be removed from the binary log. > + > +--let $truncate_gtid_pos = 0-1-6 > +--let $query1 = INSERT INTO t VALUES (20) > +--let $query2 = DELETE FROM t2 WHERE f = 0 /* no such record */ > +--source binlog_truncate_active_log.inc > + > +--echo # Case B. > +# The inverted sequence ends up to truncate only $query2 > +--let $truncate_gtid_pos = 0-1-10 > +--let $query1 = DELETE FROM t2 WHERE f = 0 > +--let $query2 = INSERT INTO t VALUES (20) > +--source binlog_truncate_active_log.inc > + > + > +delimiter |; > +CREATE PROCEDURE sp_blank_xa() > +BEGIN > + XA START 'blank'; > + DELETE FROM t2 WHERE f = 0 /* no such record */; > + XA END 'blank'; > + XA PREPARE 'blank'; > +END| > +delimiter ;| > + > + > +--echo # Case C. > +--let $truncate_gtid_pos = 0-1-14 > +--let $query1 = INSERT INTO t VALUES (20) > +--let $pre_q2 = CALL sp_blank_xa > +--let $query2 = XA COMMIT 'blank' > +--source binlog_truncate_active_log.inc what was truncated here? a comment explains it for cases A and B, but not here. may be it'd make sense to do show binlog events after every restart, just to see the state of the binlog after truncation? > +DROP PROCEDURE sp_blank_xa; > + > +--echo # Cleanup > +DROP TABLE t,t2,tm; > + > +--echo # End of the tests > diff --git a/mysql-test/suite/binlog/t/binlog_truncate_active_log.inc b/mysql-test/suite/binlog/t/binlog_truncate_active_log.inc > new file mode 100644 > index 00000000000..bbc464066fc > --- /dev/null > +++ b/mysql-test/suite/binlog/t/binlog_truncate_active_log.inc > @@ -0,0 +1,55 @@ > +connect(master1,localhost,root,,); > +connect(master2,localhost,root,,); > +connect(master3,localhost,root,,); > + > +--connection default > + > +# First to commit few transactions > +INSERT INTO t VALUES (10); > +INSERT INTO tm VALUES (10); > + > +--connection master1 > +# Hold insert after write to binlog and before "run_commit_ordered" in engine > +SET DEBUG_SYNC= "commit_before_get_LOCK_commit_ordered SIGNAL master1_ready WAIT_FOR signal_never_arrives"; > +--send_eval $query1 > + > +--connection master2 > +SET DEBUG_SYNC= "now WAIT_FOR master1_ready"; > +if ($pre_q2) > +{ > + eval $pre_q2; > +} > +SET DEBUG_SYNC= "commit_before_get_LOCK_after_binlog_sync SIGNAL master2_ready"; > +# To binlog non-xid transactional group which will be truncated all right > +--send_eval $query2 > + > +--connection master3 > +SET DEBUG_SYNC= "now WAIT_FOR master2_ready"; > +SELECT @@global.gtid_binlog_pos as 'Before the crash'; > + > +--connection default > +--source include/kill_mysqld.inc > +--disconnect master1 > +--disconnect master2 > +--disconnect master3 > + > +# > +# Server restart > +# > +--let $restart_parameters= --rpl-semi-sync-slave-enabled=1 > +--source include/start_mysqld.inc > + > +# Check error log for a successful truncate message. > +--let $log_error_ = $MYSQLTEST_VARDIR/log/mysqld.1.err > + > +--let SEARCH_FILE=$log_error_ why not to set SEARCH_FILE directly? but ok, as you like > +--let SEARCH_PATTERN=Successfully truncated.*to remove transactions starting from GTID $truncate_gtid_pos > +--replace_regex /FOUND [0-9]+/FOUND #/ can it be found multiple times? Why would binlog be truncated more than once? > +--source include/search_pattern_in_file.inc > + > +SELECT @@global.gtid_binlog_pos as 'After the crash'; > +--echo "One row should be present in table 't'" > +SELECT * FROM t; > + > +# Local cleanup > +DELETE FROM t; > diff --git a/mysql-test/suite/binlog/t/binlog_truncate_multi_engine.test b/mysql-test/suite/binlog/t/binlog_truncate_multi_engine.test > new file mode 100644 > index 00000000000..94837e3c3ea > --- /dev/null > +++ b/mysql-test/suite/binlog/t/binlog_truncate_multi_engine.test > @@ -0,0 +1,56 @@ > +# ==== Purpose ==== > +# > +# Test verifies truncation of multiple binary logs with multiple transactional > +# storage engines > +# > +# ==== References ==== > +# > +# MDEV-21117: recovery for --rpl-semi-sync-slave-enabled server > + > +--source include/have_innodb.inc > +--source include/have_rocksdb.inc > +--source include/have_debug.inc > +--source include/have_debug_sync.inc > +--source include/have_binlog_format_row.inc > + > +--let $old_max_binlog_size= `select @@global.max_binlog_size` > +call mtr.add_suppression("Can.t init tc log"); > +call mtr.add_suppression("Aborting"); > +--let $MYSQLD_DATADIR= `SELECT @@datadir` > + > +CREATE TABLE t1 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; > +CREATE TABLE t2 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=rocksdb; > + > +--let $case = A: neither engine committed => rollback & binlog truncate > +# Hold off engine commits after write to binlog and its rotation. > +# The transaction is killed along with the server after that. > +--let $shutdown_timeout=0 > +--let $debug_sync_action = "commit_after_release_LOCK_log SIGNAL con1_ready WAIT_FOR signal_no_signal" > +--let $restart_parameters = --rpl-semi-sync-slave-enabled=1 > +--let $test_outcome= 1 row should be present in both tables; binlog is truncated; number of binlogs at reconnect - 3 > +--source binlog_truncate_multi_engine.inc > +--echo Proof of the truncated binlog file is readable (two transactions must be seen): > +--exec $MYSQL_BINLOG --short-form --skip-annotate-row-events $MYSQLD_DATADIR/master-bin.000002 > + > +--let $case = B: one engine has committed its transaction branch > +# Hold off after one engine has committed. > +--let $shutdown_timeout=0 > +--let $debug_sync_action = "commit_after_run_commit_ordered SIGNAL con1_ready WAIT_FOR signal_no_signal" > +# Both debug_sync and debug-dbug are required to make sure Engines remember the commit state > +# debug_sync alone will not help. > +--let $restart_parameters = --rpl-semi-sync-slave-enabled=1 --debug-dbug=d,binlog_truncate_partial_commit in the first review I wrote this seems to be a rather crude way of faking a partially committed transaction. better to crash after the first engine has committed, that'd be much more natural. and you replied This simulation aimed at (allows for) more complicated recovery time event sequences. In this case, indeed, crashing by demand is about of the same efforts. I can convert to that. [x] > +--let $test_outcome= 2 rows should be present in both tables; no binlog truncation; one extra binlog file compare with A; number of binlogs at reconnect - 4 > +--source binlog_truncate_multi_engine.inc > + > +--let $case = C: both engines have committed its transaction branch > +# Hold off after both engines have committed. The server is shut down. > +--let $shutdown_timeout= > +--let $restart_parameters = --rpl-semi-sync-slave-enabled=1 > +--let $test_outcome= 2 rows should be present in both tables; no binlog truncation; the same # of binlog files as in B; number of binlogs at reconnect - 4 > +--source binlog_truncate_multi_engine.inc > + > + > + > +DROP TABLE t1, t2; > + > +--echo # End of the tests > diff --git a/mysql-test/suite/binlog/t/binlog_truncate_multi_engine.inc b/mysql-test/suite/binlog/t/binlog_truncate_multi_engine.inc > new file mode 100644 > index 00000000000..41ae856dd9d > --- /dev/null > +++ b/mysql-test/suite/binlog/t/binlog_truncate_multi_engine.inc > @@ -0,0 +1,54 @@ > +# > +# Loop body of binlog_truncate_multi_engine.test what do you mean "loop body"? > +# Parameters: > +# $debug_sync_action describes debug-sync actions > +# $kill_server 1 when to crash, 0 for regular restart > +# $restart_parameters the caller may simulate partial commit at recovery > +# $test_outcome summary of extected results > +# $MYSQLD_DATADIR > + > +--echo # > +--echo # > +--echo # Case $case > +--echo # > +RESET MASTER; > +FLUSH LOGS; > +SET GLOBAL max_binlog_size= 4096; > + > +connect(con1,localhost,root,,); > +--echo List of binary logs before rotation > +--source include/show_binary_logs.inc > +INSERT INTO t1 VALUES (1, REPEAT("x", 1)); > +INSERT INTO t2 VALUES (1, REPEAT("x", 1)); I'm not sure I understand the point of REPEAT(..., 1) but sure, if you like it that way... :) > +BEGIN; > + INSERT INTO t1 VALUES (2, REPEAT("x", 4100)); > + INSERT INTO t2 VALUES (2, REPEAT("x", 4100)); > + > +--eval SET DEBUG_SYNC= $debug_sync_action > +send COMMIT; > + > +--connection default > +SET DEBUG_SYNC= "now WAIT_FOR con1_ready"; > +--echo List of binary logs after rotation > +--source include/show_binary_logs.inc > + > +--echo # restart the server with $restart_parameters > +--echo # the server is restarted > +--source include/restart_mysqld.inc > + > +--connection default > +--echo # > +--echo # *** Summary: $test_outcome: > +--echo # > +SELECT COUNT(*) FROM t1; > +SELECT COUNT(*) FROM t2; > +SELECT @@GLOBAL.gtid_binlog_state; > +SELECT @@GLOBAL.gtid_binlog_pos; > +--echo List of binary logs at the end of the tests > +--source include/show_binary_logs.inc > +--echo # *** > +# cleanup > +DELETE FROM t1; > +DELETE FROM t2; > +--disconnect con1 > +--echo # > diff --git a/mysql-test/suite/binlog/t/binlog_truncate_multi_log.test b/mysql-test/suite/binlog/t/binlog_truncate_multi_log.test > new file mode 100644 > index 00000000000..3b557bc89b8 > --- /dev/null > +++ b/mysql-test/suite/binlog/t/binlog_truncate_multi_log.test > @@ -0,0 +1,78 @@ > +# ==== Purpose ==== > +# > +# Test verifies truncation of multiple binary logs. > +# > +# ==== References ==== > +# MDEV-21117: recovery for --rpl-semi-sync-slave-enabled server > + > +--source include/have_innodb.inc > +--source include/have_debug_sync.inc > +--source include/have_binlog_format_row.inc > + > +call mtr.add_suppression("Can.t init tc log"); > +call mtr.add_suppression("Aborting"); > + > +SET @@global.max_binlog_size= 4096; > + > +RESET MASTER; > +FLUSH LOGS; > +CREATE TABLE ti (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; > +CREATE TABLE tm (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=MyISAM; > + > +connect(master1,localhost,root,,); > +--echo "List of binary logs before rotation" > +--source include/show_binary_logs.inc > + > +# Some load to either non- and transactional egines > +# that should not affect the following recovery: > +INSERT INTO ti VALUES(1,"I am gonna survive"); > +INSERT INTO tm VALUES(1,"me too!"); > + > +# hold on near engine commit > +SET DEBUG_SYNC= "commit_after_release_LOCK_after_binlog_sync SIGNAL master1_ready WAIT_FOR con1_go"; > +--send_eval INSERT INTO ti VALUES (2, REPEAT("x", 4100)) > + > +connect(master2,localhost,root,,); > +# The 2nd trx for recovery, it does not rotate binlog > +SET DEBUG_SYNC= "now WAIT_FOR master1_ready"; > +SET DEBUG_SYNC= "commit_before_get_LOCK_commit_ordered SIGNAL master2_ready WAIT_FOR master2_go"; > +--send_eval INSERT INTO ti VALUES (3, "not gonna survive") send_eval? what are you evaluating here? > + > +--connection default > +SET DEBUG_SYNC= "now WAIT_FOR master2_ready"; > +--echo "List of binary logs before crash" > +--source include/show_binary_logs.inc > +--echo # The gtid binlog state prior the crash will be truncated at the end of the test > +SELECT @@global.gtid_binlog_state; > + > +--connection default > +--source include/kill_mysqld.inc > +--disconnect master1 > +--disconnect master2 > + > +# > +# Server restart > +# > +--let $restart_parameters= --rpl-semi-sync-slave-enabled=1 > +--source include/start_mysqld.inc > + > +# Check error log for a successful truncate message. > +let $log_error_ = $MYSQLTEST_VARDIR/log/mysqld.1.err; > + > +--let SEARCH_FILE=$log_error_ > +--let SEARCH_PATTERN=truncated binlog file:.*master.*000002 > +--replace_regex /FOUND [0-9]+/FOUND #/ can it be found multiple times? Why would binlog be truncated more than once? > +--source include/search_pattern_in_file.inc > + > + > +--echo "One record should be present in table" > +SELECT * FROM ti; > + > +--echo # The truncated gtid binlog state > +SELECT @@global.gtid_binlog_state; > +SELECT @@global.gtid_binlog_pos; > + > +--echo # Cleanup > +DROP TABLE ti; > + > +--echo # End of the tests > diff --git a/mysql-test/suite/binlog/t/binlog_truncate_multi_log_unsafe.test b/mysql-test/suite/binlog/t/binlog_truncate_multi_log_unsafe.test > new file mode 100644 > index 00000000000..38a9c0832f4 > --- /dev/null > +++ b/mysql-test/suite/binlog/t/binlog_truncate_multi_log_unsafe.test > @@ -0,0 +1,120 @@ > +# ==== Purpose ==== > +# The test verifies attempt to recover by the semisync slave server whose > +# binlog is unsafe for truncation. > +# > +# ==== Implementation ==== > +# 2 binlog files are created with the 1st one destined to be the binlog > +# checkpoint file for recovery. > +# The final group of events is replication unsafe (myisam INSERT). > +# Therefore the semisync slave recovery may not. > +# > +# Steps: > +# 0 - Set max_binlog_size= 4096, to help an insert into a > +# transaction table 'ti' get binlog rotated while the > +# transaction won't be committed, being stopped at > +# a prior to commit debug_sync point > +# 1 - insert into a non-transactional 'tm' table completes with > +# binary logging as well > +# 2 - kill and attempt to restart the server as semisync slave that > +# must produce an expected unsafe-to-recover error > +# 3 - complete the test with a normal restart that successfully finds and > +# commits the transaction in doubt. > +# > +# ==== References ==== > +# > +# MDEV-21117: recovery for --rpl-semi-sync-slave-enabled server > +# > + > +--source include/have_innodb.inc > +--source include/have_debug_sync.inc > +--source include/have_binlog_format_row.inc > + > +SET @@global.max_binlog_size= 4096; > + > +call mtr.add_suppression("Table '.*tm' is marked as crashed and should be repaired"); > +call mtr.add_suppression("Got an error from unknown thread"); > +call mtr.add_suppression("Checking table: '.*tm'"); > +call mtr.add_suppression("Recovering table: '.*tm'"); > +call mtr.add_suppression("Cannot trim the binary log to file"); > +call mtr.add_suppression("Crash recovery failed"); > +call mtr.add_suppression("Can.t init tc log"); > +call mtr.add_suppression("Aborting"); > +call mtr.add_suppression("Found 1 prepared transactions"); > +call mtr.add_suppression("mysqld: Table.*tm.*is marked as crashed"); > +call mtr.add_suppression("Checking table.*tm"); > + > +RESET MASTER; > +FLUSH LOGS; > +CREATE TABLE ti (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; > +CREATE TABLE tm (f INT) ENGINE=MYISAM; > + > +--let $row_count = 5 > +--let $i = `select $row_count-2` > +--disable_query_log > +while ($i) > +{ > + --eval INSERT INTO ti VALUES ($i, REPEAT("x", 1)) > + --dec $i > +} > +--enable_query_log > +INSERT INTO tm VALUES(1); > + > +connect(master1,localhost,root,,); > +connect(master2,localhost,root,,); > +connect(master3,localhost,root,,); > + > +--connection master1 > + > +# The 1st trx binlogs, rotate binlog and hold on before committing at engine > +SET DEBUG_SYNC= "commit_after_release_LOCK_after_binlog_sync SIGNAL master1_ready WAIT_FOR master1_go"; > +--send_eval INSERT INTO ti VALUES ($row_count - 1, REPEAT("x", 4100)) > + > +--connection master2 > + > +# The 2nd trx for recovery, it does not rotate binlog > +SET DEBUG_SYNC= "commit_before_get_LOCK_commit_ordered SIGNAL master2_ready WAIT_FOR master2_go"; > +--send_eval INSERT INTO ti VALUES ($row_count, REPEAT("x", 1)) > + > +--connection master3 > +SET DEBUG_SYNC= "now WAIT_FOR master2_ready"; > +SET DEBUG_SYNC= "commit_before_get_LOCK_after_binlog_sync SIGNAL master3_ready"; > +--send INSERT INTO tm VALUES (2) > + > +--connection default > +SET DEBUG_SYNC= "now WAIT_FOR master3_ready"; > +--echo # The gtid binlog state prior the crash must be restored at the end of the test; > +SELECT @@global.gtid_binlog_state; > +--source include/kill_mysqld.inc > + > +# > +# Server restarts > +# > +--echo # Failed restart as the semisync slave > +--error 1 > +--exec $MYSQLD_LAST_CMD --rpl-semi-sync-slave-enabled=1 >> $MYSQLTEST_VARDIR/log/mysqld.1.err 2>&1 > + > +--echo # Normal restart > +--source include/start_mysqld.inc > + > +# Check error log for correct messages. > +let $log_error_ = $MYSQLTEST_VARDIR/log/mysqld.1.err; > + > +--let SEARCH_FILE=$log_error_ > +--let SEARCH_PATTERN=Cannot trim the binary log to file > +--replace_regex /FOUND [0-9]+/FOUND #/ can it be found multiple times? Why would binlog be truncated more than once? > +--source include/search_pattern_in_file.inc > + > +--echo # Proof that the in-doubt transactions are recovered by the 2nd normal server restart > +--eval SELECT COUNT(*) = $row_count as 'True' FROM ti > +# myisam table may require repair (which is not tested here) > +--disable_warnings > +SELECT COUNT(*) <= 1 FROM tm; > +--enable_warnings > + > +--echo # The gtid binlog state prior the crash is restored now > +SELECT @@GLOBAL.gtid_binlog_state; > +SELECT @@GLOBAL.gtid_binlog_pos; > + > +--echo # Cleanup > +DROP TABLE ti, tm; > +--echo End of test > diff --git a/mysql-test/suite/rpl/t/rpl_semi_sync_fail_over.cnf b/mysql-test/suite/rpl/t/rpl_semi_sync_fail_over.cnf > new file mode 100644 > index 00000000000..f8312bdc5b8 > --- /dev/null > +++ b/mysql-test/suite/rpl/t/rpl_semi_sync_fail_over.cnf > @@ -0,0 +1,11 @@ > +!include suite/rpl/rpl_1slave_base.cnf > +!include include/default_client.cnf > + > + > +[mysqld.1] > +log-slave-updates > +gtid-strict-mode=1 > + > +[mysqld.2] > +log-slave-updates > +gtid-strict-mode=1 generally opt files (rpl_semi_sync_fail_over.opt in this case) are preferred, because mtr will know what options to apply, while cnf files are more opaque > diff --git a/mysql-test/suite/rpl/t/rpl_semi_sync_fail_over.test b/mysql-test/suite/rpl/t/rpl_semi_sync_fail_over.test > new file mode 100644 > index 00000000000..a8b40d6ed05 > --- /dev/null > +++ b/mysql-test/suite/rpl/t/rpl_semi_sync_fail_over.test > @@ -0,0 +1,143 @@ > +# ==== Purpose ==== > +# > +# Test verifies replication failover scenario. > +# > +# ==== Implementation ==== > +# > +# Steps: > +# 0 - Having two servers 1 and 2 enable semi-sync replication with > +# with the master wait 'after_sync'. > +# 1 - Insert a row. While inserting second row simulate > +# a server crash at once the transaction is written to binlog, flushed > +# and synced but the binlog position is not updated. > +# 2 - Post crash-recovery on the old master execute there CHANGE MASTER > +# TO command to connect to server id 2. > +# 3 - The old master new slave server 1 must connect to the new > +# master server 2. > +# 4 - repeat the above to crash the new master and restore in role the old one > +# > +# ==== References ==== > +# > +# MDEV-21117: recovery for --rpl-semi-sync-slave-enabled server > + > + > +--source include/have_innodb.inc > +--source include/have_debug_sync.inc > +--source include/have_binlog_format_row.inc > +--let $rpl_topology=1->2 > +--source include/rpl_init.inc why not to source master-slave.inc if you're using a standard master-slave topology anyway? > + > +--connection server_2 > +--source include/stop_slave.inc > + > +--connection server_1 > +RESET MASTER; > +SET @@global.max_binlog_size= 4096; > + > +--connection server_2 > +RESET MASTER; > +SET @@global.max_binlog_size= 4096; > +set @@global.rpl_semi_sync_slave_enabled = 1; > +set @@global.gtid_slave_pos = ""; > +CHANGE MASTER TO master_use_gtid= slave_pos; > +--source include/start_slave.inc > + > + > +--connection server_1 > +ALTER TABLE mysql.gtid_slave_pos ENGINE=InnoDB; here I asked "why?" and you replied Actually not need to. There's no crash in the middle of a slave transaction. So it must be a copy-paste leftover. [x] > +set @@global.rpl_semi_sync_master_enabled = 1; > +set @@global.rpl_semi_sync_master_wait_point=AFTER_SYNC; > + > +call mtr.add_suppression("Can.t init tc log"); > +call mtr.add_suppression("Aborting"); > +call mtr.add_suppression("1 client is using or hasn.t closed the table properly"); > +call mtr.add_suppression("Table './mtr/test_suppressions' is marked as crashed and should be repaired"); > + > +CREATE TABLE t1 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; > +INSERT INTO t1 VALUES (1, 'dummy1'); > + > +# > +# CRASH the original master, and FAILOVER to the new > +# > + > +# value 1 for server id 1 -> 2 failover > +--let $failover_to_slave=1 > +--let $query_to_crash= INSERT INTO t1 VALUES (2, REPEAT("x", 4100)) > +--let $log_search_pattern=truncated binlog file:.*master.*000001 > +--source rpl_semi_sync_crash.inc > + > +--connection server_2 > +--let $rows_so_far=3 > +--eval INSERT INTO t1 VALUES ($rows_so_far, 'dummy3') > +--save_master_pos > +--echo # The gtid state on current master must be equal to ... > +SHOW VARIABLES LIKE 'gtid_binlog_pos'; > + > +--connection server_1 > +--sync_with_master > +--eval SELECT COUNT(*) = $rows_so_far as 'true' FROM t1 > +--echo # ... the gtid states on the slave: > +SHOW VARIABLES LIKE 'gtid_slave_pos'; > +SHOW VARIABLES LIKE 'gtid_binlog_pos'; > + > +--connection server_2 > +# > +# CRASH the new master and FAILOVER back to the original > +# > + > +# value 0 for the reverse server id 2 -> 1 failover > +--let $failover_to_slave=0 > +--let $query_to_crash = INSERT INTO t1 VALUES (4, REPEAT("x", 4100)) > +--let $query2_to_crash= INSERT INTO t1 VALUES (5, REPEAT("x", 4100)) > +--let $log_search_pattern=truncated binlog file:.*slave.*000001 > +--source rpl_semi_sync_crash.inc > + > +--connection server_1 > +--let $rows_so_far=6 > +--eval INSERT INTO t1 VALUES ($rows_so_far, 'Done') > +--save_master_pos > +--echo # The gtid state on current master must be equal to ... > +SHOW VARIABLES LIKE 'gtid_binlog_pos'; > + > +--connection server_2 > +--sync_with_master > +--eval SELECT COUNT(*) = $rows_so_far as 'true' FROM t1 > +--echo # ... the gtid states on the slave: > +SHOW VARIABLES LIKE 'gtid_slave_pos'; > +SHOW VARIABLES LIKE 'gtid_binlog_pos'; > + > + > +--let $diff_tables=server_1:t1, server_2:t1 > +--source include/diff_tables.inc > + > +# > +--echo # Cleanup > +# > +--connection server_1 > +DROP TABLE t1; > +--save_master_pos > + > +--connection server_2 > +--sync_with_master > +--source include/stop_slave.inc > + > +--connection server_1 > +set @@global.rpl_semi_sync_master_enabled = 0; > +set @@global.rpl_semi_sync_slave_enabled = 0; > +set @@global.rpl_semi_sync_master_wait_point=default; > +RESET SLAVE; > +RESET MASTER; > + > +--connection server_2 > +set @@global.rpl_semi_sync_master_enabled = 0; > +set @@global.rpl_semi_sync_slave_enabled = 0; > +set @@global.rpl_semi_sync_master_wait_point=default; > + > +evalp CHANGE MASTER TO master_host='127.0.0.1', master_port=$SERVER_MYPORT_1, master_user='root', master_use_gtid=no; > +--source include/start_slave.inc > + > +connection default; > +--enable_reconnect > +--source include/wait_until_connected_again.inc > + > +--source include/rpl_end.inc > diff --git a/sql/handler.h b/sql/handler.h > index fc69d9423b4..05a62ed0021 100644 > --- a/sql/handler.h > +++ b/sql/handler.h > @@ -873,6 +874,15 @@ typedef struct xid_t XID; > /* The 'buf' has to have space for at least SQL_XIDSIZE bytes. */ > uint get_sql_xid(XID *xid, char *buf); > > +/* struct for semisync slave binlog truncate recovery */ > +struct xid_recovery_member > +{ > + my_xid xid; > + uint in_engine_prepare; // number of engines that have xid prepared > + bool decided_to_commit; > + std::pair<uint, my_off_t> binlog_coord; // semisync recovery binlog offset wouldn't it be clearer to have a struct with named members? in fact, I'm somewhat surprised there's no such struct for binlog coords already. > +}; > + > /* for recover() handlerton call */ > #define MIN_XID_LIST_SIZE 128 > #define MAX_XID_LIST_SIZE (1024*128) > @@ -4820,7 +4830,8 @@ int ha_commit_one_phase(THD *thd, bool all); > int ha_commit_trans(THD *thd, bool all); > int ha_rollback_trans(THD *thd, bool all); > int ha_prepare(THD *thd); > -int ha_recover(HASH *commit_list); > +int ha_recover(HASH *commit_list, MEM_ROOT *mem_root= NULL); > +uint ha_recover_complete(HASH *commit_list, std::pair<uint, my_off_t> *coord= NULL); is coord a truncation position? > > /* transactions: these functions never call handlerton functions directly */ > int ha_enable_transaction(THD *thd, bool on); > diff --git a/sql/log_event.h b/sql/log_event.h > index 8a342cb5cd3..1036e9a44d4 100644 > --- a/sql/log_event.h > +++ b/sql/log_event.h > @@ -482,6 +482,16 @@ class String; > */ > #define LOG_EVENT_IGNORABLE_F 0x80 > > +/** > + @def LOG_EVENT_ACCEPT_OWN_F > + > + Flag sets by the semisync slave for accepting > + the same server_id ("own") events which the slave must not have > + in its state. Typically such events were never committed by > + their originator (this server) and discared at its semisync-slave recovery. > +*/ > +#define LOG_EVENT_ACCEPT_OWN_F 0x4000 may be, add an assert on all received events that such a flag is not set? it can only be set on events in relay log. also, consider the case when this event is read from a relay log, applied, and then sent to further slaves. In this case this flag must be removed before sending, otherwise they'll mistakenly might apply it if the server_id will match. > + > /** > @def LOG_EVENT_SKIP_REPLICATION_F > > @@ -3357,6 +3367,12 @@ class Gtid_log_event: public Log_event > uint64 commit_id; > uint32 domain_id; > uchar flags2; > + uint flags_extra; // more flags area placed after the regular flags2's one > + /* > + Extra to a "base" engine recoverable engines participating > + in the transaction. Zero, when the base engine only is present. what's a "base engine"? > + */ > + uint8 extra_engines; > > /* Flags2. */ > > diff --git a/sql/handler.cc b/sql/handler.cc > index c0a810a72bc..a46cef6b64c 100644 > --- a/sql/handler.cc > +++ b/sql/handler.cc > @@ -1637,9 +1672,17 @@ commit_one_phase_2(THD *thd, bool all, THD_TRANS *trans, bool is_real_trans) > DEBUG_SYNC(thd, "commit_one_phase_2"); > if (ha_info) > { > + int err; > + > + if (has_binlog_hton(ha_info) && can you replace has_binlog_hton() with, like, if trx cache is not empty or binlog enabled or something like that? > + (err= binlog_commit(thd, all, > + is_ro_1pc_trans(thd, ha_info, all, is_real_trans)))) > + { > + my_error(ER_ERROR_DURING_COMMIT, MYF(0), err); > + error= 1; > + } > for (; ha_info; ha_info= ha_info_next) > { > - int err; > handlerton *ht= ha_info->ht(); > if ((err= ht->commit(ht, thd, all))) > { > @@ -1962,8 +2008,177 @@ struct xarecover_st > XID *list; > HASH *commit_list; > bool dry_run; > + MEM_ROOT *mem_root; > + bool error; > }; > > +/** > + Inserts a new hash member. > + > + returns a successfully created and inserted @c xid_recovery_member > + into hash @c hash_arg, > + or NULL. > +*/ > +static xid_recovery_member* > +xid_member_insert(HASH *hash_arg, my_xid xid_arg, MEM_ROOT *ptr_mem_root) > +{ > + xid_recovery_member *member= (xid_recovery_member*) > + alloc_root(ptr_mem_root, sizeof(xid_recovery_member)); > + if (!member) > + return NULL; > + > + member->xid= xid_arg; > + member->in_engine_prepare= 1; > + member->decided_to_commit= false; > + > + return my_hash_insert(hash_arg, (uchar*) member) ? NULL : member; > +} > + > +/* > + Inserts a new or updates an existing hash member to increment > + the member's prepare counter. > + > + returns false on success, > + true otherwise. > +*/ > +static bool xid_member_replace(HASH *hash_arg, my_xid xid_arg, > + MEM_ROOT *ptr_mem_root) > +{ > + xid_recovery_member* member; > + if ((member= (xid_recovery_member *) > + my_hash_search(hash_arg, (uchar *)& xid_arg, sizeof(xid_arg)))) > + member->in_engine_prepare++; > + else > + member= xid_member_insert(hash_arg, xid_arg, ptr_mem_root); > + > + return member == NULL; > +} > + > +/* > + Decision to commit returns true, otherwise false for rollback. > + Flagged to commit member is destined to commit. If it is in doubt in case > + A. the caller does not specify coord_ptr (always so in the normal recovery), or > + B. coord_ptr is not NULL (can only be so in the semisync slave case) and its > + offset is greater than that of the member's the decision is rollback. > + If both A,B do not hold - which is the semisync slave recovery case - > + the decision is to rollback. > +*/ > +static bool xarecover_decide(xid_recovery_member* member, > + xid_t x, std::pair<uint, my_off_t> *coord_ptr) > +{ > + return > + member->decided_to_commit ? true : > + !coord_ptr ? false : > + (member->binlog_coord < *coord_ptr ? // semisync slave recovery > + true : false); > +} > + > +struct xarecover_iterate_arg > +{ > + handlerton *hton; > + std::pair<uint, my_off_t> *binlog_coord; > +}; > + > +/* > + Hash iterate function to complete with commit or rollback as either > + has been decided already or decide now (in the semisync recovery) > + via comparison against passed offset. > + Commit when the offset is greater than that of the member. > +*/ > +static my_bool xarecover_do_commit_or_rollback(void *member_arg, > + void *iter_arg) > +{ > + xid_recovery_member *member= (xid_recovery_member*) member_arg; > + handlerton *hton= ((xarecover_iterate_arg*) iter_arg)->hton; > + std::pair<uint, my_off_t> *max_coord_ptr= > + ((xarecover_iterate_arg*) iter_arg)->binlog_coord; > + xid_t x; > + my_bool rc; > + > + x.set(member->xid); > + > + rc= xarecover_decide(member, x, max_coord_ptr) ? > + hton->commit_by_xid(hton, &x) : hton->rollback_by_xid(hton, &x); > + > + DBUG_ASSERT(rc || member->in_engine_prepare > 0); > + > + if (!rc) > + { > + /* > + This block relies on Engine to report XAER_NOTA at > + "complete"_by_xid for unknown xid. > + */ > + member->in_engine_prepare--; > + if (global_system_variables.log_warnings > 2) > + sql_print_warning("%s transaction with xid %llu", may be not a sql_print_warning, but a sql_print_information? it's just an informational message > + member->decided_to_commit ? > + "Committed" : "Rolled back", (ulonglong) member->xid); > + } > + > + return false; > +} > + > +static my_bool xarecover_do_count_in_prepare(void *member_arg, > + void *ptr_count) > +{ > + xid_recovery_member *member= (xid_recovery_member*) member_arg; > + if (member->in_engine_prepare) > + { > + (*(uint*) ptr_count)++; > + if (global_system_variables.log_warnings > 2) > + sql_print_warning("Found prepared transaction with xid %llu", > + (ulonglong) member->xid); > + } > + > + return false; > +} > + > +struct xarecover_complete_arg > +{ > + HASH *commit_list; > + std::pair<uint, my_off_t> *binlog_coord; > +}; > + > +/* > + Completes binlog recovery to invoke a decider function for > + each transaction in doubt. > +*/ > +static my_bool xarecover_binlog_handlerton(THD *unused, > + plugin_ref plugin, > + void *arg) > +{ > + handlerton *hton= plugin_hton(plugin); > + > + if (hton->state == SHOW_OPTION_YES && hton->recover) > + { > + xarecover_iterate_arg iter_arg= > + { > + hton, > + ((xarecover_complete_arg*) arg)->binlog_coord > + }; > + my_hash_iterate(((xarecover_complete_arg*) arg)->commit_list, > + xarecover_do_commit_or_rollback, &iter_arg); > + } > + > + return FALSE; > +} > + > +/* > + Completes binlog recovery to invoke decider functions for > + each handerton. > + Returns the number of transactions remained doubtful. > +*/ > +uint ha_recover_complete(HASH *commit_list, std::pair<uint, my_off_t> *coord) > +{ > + uint count= 0; > + xarecover_complete_arg complete_arg= { commit_list, coord }; > + plugin_foreach(NULL, xarecover_binlog_handlerton, > + MYSQL_STORAGE_ENGINE_PLUGIN, &complete_arg); > + my_hash_iterate(commit_list, xarecover_do_count_in_prepare, &count); wouldn't it be cleaner to do everything in one commit_list scan? for every xid_recovery_member: run plugin_foreach, commit or rollback as needed increment a counter, if still in doubt > + > + return count; > +} > + > static my_bool xarecover_handlerton(THD *unused, plugin_ref plugin, > void *arg) > { > @@ -1973,6 +2188,9 @@ static my_bool xarecover_handlerton(THD *unused, plugin_ref plugin, > > if (hton->state == SHOW_OPTION_YES && hton->recover) > { > +#ifndef DBUG_OFF > + my_xid dbug_xid_list[128] __attribute__((unused)) = {0}; > +#endif What do you use it for? > while ((got= hton->recover(hton, info->list, info->len)) > 0 ) > { > sql_print_information("Found %d prepared transaction(s) in %s", > diff --git a/sql/log.cc b/sql/log.cc > index 8073f09ab88..a90d1e757e8 100644 > --- a/sql/log.cc > +++ b/sql/log.cc > @@ -2050,14 +2058,17 @@ static int binlog_commit(handlerton *hton, THD *thd, bool all) > Otherwise, we accumulate the changes. > */ > if (likely(!error) && ending_trans(thd, all)) > + { > + cache_mngr->ro_1pc= ro_1pc; > error= binlog_commit_flush_trx_cache(thd, all, cache_mngr); > + cache_mngr->ro_1pc= false; Why do you put it in cache_mngr, instead of passing it down, like you pass `all` ? > + } > > /* > This is part of the stmt rollback. > */ > if (!all) > cache_mngr->trx_cache.set_prev_position(MY_OFF_T_UNDEF); > - > THD_STAGE_INFO(thd, org_stage); > DBUG_RETURN(error); > } > @@ -9609,6 +9626,147 @@ int TC_LOG::using_heuristic_recover() > /****** transaction coordinator log for 2pc - binlog() based solution ******/ > #define TC_LOG_BINLOG MYSQL_BIN_LOG > > +/** > + Truncates the current binlog to specified position. Removes the rest of binlogs > + which are present after this binlog file. > + > + @param truncate_file Holds the binlog name to be truncated > + @param truncate_pos Position within binlog from where it needs to > + truncated. > + > + @retval true ok > + @retval false error > + > +*/ > +bool MYSQL_BIN_LOG::truncate_and_remove_binlogs(const char *file_name, > + my_off_t pos, > + rpl_gtid *ptr_gtid) > +{ > + int error= 0; > +#ifdef HAVE_REPLICATION > + LOG_INFO log_info; > + THD *thd= current_thd; > + my_off_t index_file_offset= 0; > + File file= -1; > + MY_STAT s; > + > + if ((error= find_log_pos(&log_info, file_name, 1))) > + { > + sql_print_error("Failed to locate binary log file:%s." > + "Error:%d", file_name, error); > + goto end; > + } > + > + while (!(error= find_next_log(&log_info, 1))) > + { > + if (!index_file_offset) > + { > + index_file_offset= log_info.index_file_start_offset; > + if ((error= open_purge_index_file(TRUE))) > + { > + sql_print_error("Failed to open purge index " > + "file:%s. Error:%d", purge_index_file_name, error); > + goto end; > + } > + } > + if ((error= register_purge_index_entry(log_info.log_file_name))) > + { > + sql_print_error("Failed to copy %s to purge index" > + " file. Error:%d", log_info.log_file_name, error); > + goto end; > + } > + } > + > + if (error != LOG_INFO_EOF) > + { > + sql_print_error("Failed to find the next binlog to " > + "add to purge index register. Error:%d", error); > + goto end; > + } > + > + if (is_inited_purge_index_file()) > + { > + if (!index_file_offset) > + index_file_offset= log_info.index_file_start_offset; > + > + if ((error= sync_purge_index_file())) > + { > + sql_print_error("Failed to flush purge index " > + "file. Error:%d", error); > + goto end; > + } > + > + // Trim index file > + if ((error= > + mysql_file_chsize(index_file.file, index_file_offset, '\n', > + MYF(MY_WME))) || > + (error= > + mysql_file_sync(index_file.file, MYF(MY_WME|MY_SYNC_FILESIZE)))) > + { > + sql_print_error("Failed to trim binlog index " > + "file:%s to offset:%llu. Error:%d", index_file_name, > + index_file_offset, error); > + goto end; > + } > + > + /* Reset data in old index cache */ > + if ((error= reinit_io_cache(&index_file, READ_CACHE, (my_off_t) 0, 0, 1))) > + { > + sql_print_error("Failed to reinit binlog index " > + "file. Error:%d", error); > + goto end; > + } > + > + /* Read each entry from purge_index_file and delete the file. */ > + if ((error= purge_index_entry(thd, NULL, TRUE))) > + { > + sql_print_error("Failed to process registered " > + "files that would be purged."); > + goto end; > + } > + } > + > + DBUG_ASSERT(pos); > + > + if ((file= mysql_file_open(key_file_binlog, file_name, > + O_RDWR | O_BINARY, MYF(MY_WME))) < 0) > + { > + error= 1; > + sql_print_error("Failed to open binlog file:%s for " > + "truncation.", file_name); > + goto end; > + } > + my_stat(file_name, &s, MYF(0)); > + > + /* Change binlog file size to truncate_pos */ > + if ((error= > + mysql_file_chsize(file, pos, 0, MYF(MY_WME))) || > + (error= mysql_file_sync(file, MYF(MY_WME|MY_SYNC_FILESIZE)))) > + { > + sql_print_error("Failed to trim the " > + "binlog file:%s to size:%llu. Error:%d", > + file_name, pos, error); > + goto end; > + } > + else > + { > + char buf[21]; > + > + longlong10_to_str(ptr_gtid->seq_no, buf, 10); > + sql_print_information("Successfully truncated binlog file:%s " > + "to pos:%llu to remove transactions starting from " > + "GTID %u-%u-%s", file_name, pos, > + ptr_gtid->domain_id, ptr_gtid->server_id, buf); > + } > + > +end: > + if (file >= 0) > + mysql_file_close(file, MYF(MY_WME)); Why you don't clean inuse flag here? You used to do it in the previous version of the patch. > + > + error= error || close_purge_index_file(); > +#endif > + return error > 0; > +} > int TC_LOG_BINLOG::open(const char *opt_name) > { > int error= 1; > @@ -10215,34 +10914,50 @@ int TC_LOG_BINLOG::recover(LOG_INFO *linfo, const char *last_log_name, > end_io_cache(&log); > mysql_file_close(file, MYF(MY_WME)); > file= -1; > + /* > + NOTE: reading other binlog's FD is necessary for finding out > + the checksum status of the respective binlog file. > + */ okay, but where do you read other binlog's FD? In the previous patch I reviewed you had case FORMAT_DESCRIPTION_EVENT: read FD and replace fdle. now you don't have it anymore. > + if (find_next_log(linfo, 1)) > + { > + sql_print_error("Error reading binlog files during recovery. " > + "Aborting."); > + goto err2; > + } > } > > +#ifdef HAVE_REPLICATION > + int rc= ctx.next_binlog_or_round(round, last_log_name, > + binlog_checkpoint_name, linfo, this); > + if (rc == -1) > + goto err2; > + else if (rc == 1) > + break; // all rounds done > +#else > if (!strcmp(linfo->log_file_name, last_log_name)) > break; // No more files to do > + round++; > +#endif > + > if ((file= open_binlog(&log, linfo->log_file_name, &errmsg)) < 0) > { > sql_print_error("%s", errmsg); > goto err2; > } > - /* > - We do not need to read the Format_description_log_event of other binlog > - files. It is not possible for a binlog checkpoint to span multiple > - binlog files written by different versions of the server. So we can use > - the first one read for reading from all binlog files. > - */ > - if (find_next_log(linfo, 1)) > - { > - sql_print_error("Error reading binlog files during recovery. Aborting."); > - goto err2; > - } > fdle->reset_crypto(); > - } > + } // end of for > > if (do_xa) > { > - if (ha_recover(&xids)) > - goto err2; > - > + if (binlog_checkpoint_found) > + { > +#ifndef HAVE_REPLICATION > + if (ha_recover_complete(&xids)) so, ha_recover_complete() is for no-semisync no-replication case? basically it should be the old behavior, exactly as before? why do you need ha_recover_complete() then if it didn't exist before? > +#else > + if (ctx.complete(this, xids)) > +#endif > + goto err2; > + } > free_root(&mem_root, MYF(0)); > my_hash_free(&xids); > } Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 57c19902326: MDEV-20017 Implement TO_CHAR() Oracle compatible function
by Sergei Golubchik 22 Apr '21

22 Apr '21

Hi, Monty! On Apr 13, Michael Widenius wrote: > commit 57c19902326 > Author: Michael Widenius <michael.widenius(a)gmail.com> > Date: Sun Jan 24 23:56:43 2021 +0200 > > diff --git a/sql/share/errmsg-utf8.txt b/sql/share/errmsg-utf8.txt > index aecb00563f7..b23522ac830 100644 > --- a/sql/share/errmsg-utf8.txt > +++ b/sql/share/errmsg-utf8.txt > @@ -7975,3 +7975,5 @@ ER_PK_INDEX_CANT_BE_IGNORED > eng "A primary key cannot be marked as IGNORE" > ER_FUNCTION_CANNOT_BE_USED_IN_CLAUSE > eng "Function '%s' cannot be used in the %s clause" > +ER_ORACLE_COMPAT_FUNCTION_ERROR > + eng "Oracle compatibility function error: %s" Why? We normally just say "invalid argument" or something, in no other case we say "oracle compatibility function" or "sybase compatibility function" or "odbc compatibility function". > diff --git a/sql/sql_string.cc b/sql/sql_string.cc > index 95a57017c53..9c57bb22085 100644 > --- a/sql/sql_string.cc > +++ b/sql/sql_string.cc > @@ -1275,3 +1275,15 @@ void Binary_string::shrink(size_t arg_length) > } > } > } > + > +bool Binary_string::strfill(char fill, size_t len) > +{ > + if (len) > + { > + if (alloc(length() + len)) > + return 1; > + memset(Ptr + str_length, fill, len); > + str_length+= (uint32) len; > + } > + return 0; > +} There's Binary_string::fill() already. better use it or, at the very least, declare bool strfill(char fill, size_t len) { return fill(str_length + len, fill); } in sql_string.h. And this swapped order of arguments is confusing, please fix it too. In fact, I think it's confusing to have both: fill(max_len, fill_char) strfill(fill_char, fill_length) if you want to keep both, it'd be better to rename them to something that somehow reflects the difference, for example: strfill -> append_many or append_repeated but really, I personally would just delete strfill. > diff --git a/sql/item_timefunc.h b/sql/item_timefunc.h > index af266956b05..9b78d6c159e 100644 > --- a/sql/item_timefunc.h > +++ b/sql/item_timefunc.h > @@ -978,6 +978,57 @@ class Item_func_time_format: public Item_func_date_format > }; > > > +/* the max length of datetime format models string in Oracle is 144 */ > +#define MAX_DATETIME_FORMAT_MODEL_LEN 144 > + > +class Item_func_tochar :public Item_str_func > +{ > + const MY_LOCALE *locale; > + THD *thd; > + String warning_message; > + bool fixed_length; > + > + /* > + When datetime format models is parsed, use uint16 integers to > + represent the format models and store in fmt_array. > + */ > + uint16 fmt_array[MAX_DATETIME_FORMAT_MODEL_LEN+1]; > + > + bool check_arguments() const override > + { > + return check_argument_types_can_return_text(1, arg_count); > + } > + > +public: > + Item_func_tochar(THD *thd, Item *a, Item *b): > + Item_str_func(thd, a, b), locale(0) > + { > + /* NOTE: max length of warning message is 64 */ > + warning_message.alloc(64); > + warning_message.length(0); > + } As far as I understand, this warning_message was introduced to issue the same error for every row, even if the format string is const_item and is parsed only once, in fix_fields. I don't think it's worth the trouble. Two simpler approaches are: * if the format string is invalid - parse it every time even if const, or * if the const format string is invalid - issue the error only once. so, please, remove warning_message and just use push_warning or my_error where the error is discovered. > + ~Item_func_tochar() { warning_message.free(); } > + String *val_str(String *str) override; > + LEX_CSTRING func_name_cstring() const override > + { > + static LEX_CSTRING name= {STRING_WITH_LEN("to_char") }; > + return name; > + } > + bool fix_length_and_dec() override; > + bool parse_format_string(const String *format, uint *fmt_len); > + > + bool check_vcol_func_processor(void *arg) override > + { > + if (arg_count > 2) > + return false; > + return mark_unsupported_function(func_name(), "()", arg, VCOL_SESSION_FUNC); > + } > + > + Item *get_copy(THD *thd) override > + { return get_item_copy<Item_func_tochar>(thd, this); } > +}; > + > + > class Item_func_from_unixtime :public Item_datetimefunc > { > bool check_arguments() const override > diff --git a/sql/item_timefunc.cc b/sql/item_timefunc.cc > index 04d913b0fca..44d2ec7912d 100644 > --- a/sql/item_timefunc.cc > +++ b/sql/item_timefunc.cc > @@ -1914,6 +1913,805 @@ String *Item_func_date_format::val_str(String *str) > return 0; > } > > +/* > + Oracle has many formatting models, we list all but only part of them > + are implemented, because some models depend on oracle functions > + which mariadb is not supported. > + > + Models for datetime, used by TO_CHAR/TO_DATE. Normal format characters are > + stored as short integer < 256, while format characters are stored as a > + integer > 256 > +*/ > + > +#define FMT_BASE 128 128? or 256? > +#define FMT_AD FMT_BASE+1 > +#define FMT_AD_DOT FMT_BASE+2 > +#define FMT_AM FMT_BASE+3 > +#define FMT_AM_DOT FMT_BASE+4 > +#define FMT_BC FMT_BASE+5 > +#define FMT_BC_DOT FMT_BASE+6 > +#define FMT_CC FMT_BASE+7 > +#define FMT_SCC FMT_BASE+8 > +#define FMT_D FMT_BASE+9 > +#define FMT_DAY FMT_BASE+10 > +#define FMT_DD FMT_BASE+11 > +#define FMT_DDD FMT_BASE+12 > +#define FMT_DL FMT_BASE+13 > +#define FMT_DS FMT_BASE+14 > +#define FMT_DY FMT_BASE+15 > +#define FMT_E FMT_BASE+16 > +#define FMT_EE FMT_BASE+17 > +#define FMT_FF FMT_BASE+18 > +#define FMT_FM FMT_BASE+19 > +#define FMT_FX FMT_BASE+20 > +#define FMT_HH FMT_BASE+21 > +#define FMT_HH12 FMT_BASE+22 > +#define FMT_HH24 FMT_BASE+23 > +#define FMT_IW FMT_BASE+24 > +#define FMT_I FMT_BASE+25 > +#define FMT_IY FMT_BASE+26 > +#define FMT_IYY FMT_BASE+27 > +#define FMT_IYYY FMT_BASE+28 > +#define FMT_J FMT_BASE+29 > +#define FMT_MI FMT_BASE+30 > +#define FMT_MM FMT_BASE+31 > +#define FMT_MON FMT_BASE+32 > +#define FMT_MONTH FMT_BASE+33 > +#define FMT_PM FMT_BASE+34 > +#define FMT_PM_DOT FMT_BASE+35 > +#define FMT_RM FMT_BASE+37 > +#define FMT_RR FMT_BASE+38 > +#define FMT_RRRR FMT_BASE+39 > +#define FMT_SS FMT_BASE+40 > +#define FMT_SSSSSS FMT_BASE+41 > +#define FMT_TS FMT_BASE+42 > +#define FMT_TZD FMT_BASE+43 > +#define FMT_TZH FMT_BASE+44 > +#define FMT_TZM FMT_BASE+45 > +#define FMT_TZR FMT_BASE+46 > +#define FMT_W FMT_BASE+47 > +#define FMT_WW FMT_BASE+48 > +#define FMT_X FMT_BASE+49 > +#define FMT_Y FMT_BASE+50 > +#define FMT_YY FMT_BASE+51 > +#define FMT_YYY FMT_BASE+52 > +#define FMT_YYYY FMT_BASE+53 > +#define FMT_YYYY_COMMA FMT_BASE+54 > +#define FMT_YEAR FMT_BASE+55 > +#define FMT_SYYYY FMT_BASE+56 > +#define FMT_SYEAR FMT_BASE+57 Not enum? Not even safe (with parentheses) #define? enum would be ideal here but at the very least make these defines safe. > + > + > +/** > + Modify the quotation flag and check whether the subsequent process is skipped Could you reword it please? > + > + @param cftm Character or FMT... format descriptor > + @param quotation_flag Points to 'true' if we are inside a quoted string > + > + @return true If we are inside a quoted string or if we found a '"' character > + @return false Otherwise > +*/ > + > +static inline bool check_quotation(uint16 cfmt, bool *quotation_flag) > +{ > + if (cfmt == '"') > + { > + *quotation_flag= !*quotation_flag; > + return true; > + } > + return *quotation_flag; > +} > + > +#define INVALID_CHARACTER(x) (((x) >= 'A' && (x) <= 'Z') ||((x) >= '0' && (x) <= '9') || (x) >= 127 || ((x) < 32)) why not to make this static inline too? side-effect safe, no risk of double evaluation of x. > + > + > +/** > + Special characters are directly output in the result > + > + @return 0 If found not acceptable character > + @return # Number of copied characters > +*/ > + > +static uint parse_special(char cfmt, const char *ptr, const char *end, > + uint16 *array) > +{ > + int offset= 0; > + char tmp1; > + > + /* Non-printable character and Multibyte encoded characters */ > + if (INVALID_CHARACTER(cfmt)) > + return 0; > + > + /* > + * '&' with text is used for variable input, but '&' with other > + * special charaters like '|'. '*' is used as separator > + */ > + if (cfmt == '&' && ptr + 1 < end) > + { > + tmp1= my_toupper(system_charset_info, *(ptr+1)); > + if (tmp1 >= 'A' && tmp1 <= 'Z') > + return 0; > + } > + > + do { > + /* > + Continuously store the special characters in fmt_array until non-special > + characters appear > + */ > + *array++= (uint16) (uchar) *ptr++; > + offset++; > + if (ptr == end) > + break; > + tmp1= my_toupper(system_charset_info, *ptr); > + } while (!INVALID_CHARACTER(tmp1) && tmp1 != '"'); > + return offset; > +} > + > + > +/** > + Parse the format string, convert it to an compact array and calculate the > + length of output string > + > + @param format Format string > + @param fmt_len Function will store max length of formated date string here > + > + @return 0 ok. fmt_len is updated > + @return 1 error. In this case 'warning_string' is set to error message > +*/ > + > +bool Item_func_tochar::parse_format_string(const String *format, uint *fmt_len) > +{ > + const char *ptr, *end; > + uint16 *tmp_fmt= fmt_array; > + uint tmp_len= 0; > + int offset= 0; > + bool quotation_flag= false; > + > + ptr= format->ptr(); > + end= ptr + format->length(); > + > + if (format->length() > MAX_DATETIME_FORMAT_MODEL_LEN) > + { > + warning_message.append(STRING_WITH_LEN("datetime format string is too " > + "long")); > + return 1; > + } > + > + for (; ptr < end; ptr++, tmp_fmt++) > + { > + uint ulen; > + char cfmt, next_char; > + > + cfmt= my_toupper(system_charset_info, *ptr); > + > + /* > + Oracle datetime format support text in double quotation marks like > + 'YYYY"abc"MM"xyz"DD', When this happens, store the text and quotation > + marks, and use the text as a separator in make_date_time_oracle. > + > + NOTE: the quotation mark is not print in return value. for example: > + select TO_CHAR(sysdate, 'YYYY"abc"MM"xyzDD"') will return 2021abc01xyz11 > + */ > + if (check_quotation(cfmt, &quotation_flag)) > + { > + *tmp_fmt= *ptr; > + tmp_len+= 1; > + continue; > + } > + > + switch (cfmt) { > + case 'A': // AD/A.D./AM/A.M. > + if (ptr+1 >= end) > + goto error; > + next_char= my_toupper(system_charset_info, *(ptr+1)); > + if (next_char == 'D') > + { > + *tmp_fmt= FMT_AD; > + ptr+= 1; > + tmp_len+= 2; > + } > + else if (next_char == 'M') > + { > + *tmp_fmt= FMT_AM; > + ptr+= 1; > + tmp_len+= 2; > + } > + else if (next_char == '.' && ptr+3 < end && *(ptr+3) == '.') > + { > + if (my_toupper(system_charset_info, *(ptr+2)) == 'D') > + { > + *tmp_fmt= FMT_AD_DOT; > + ptr+= 3; > + tmp_len+= 4; > + } > + else if (my_toupper(system_charset_info, *(ptr+2)) == 'M') > + { > + *tmp_fmt= FMT_AM_DOT; > + ptr+= 3; > + tmp_len+= 4; > + } > + else > + goto error; > + } > + else > + goto error; > + break; > + case 'B': // BC and B.C > + if (ptr+1 >= end) > + goto error; > + next_char= my_toupper(system_charset_info, *(ptr+1)); > + if (next_char == 'C') > + { > + *tmp_fmt= FMT_BC; > + ptr+= 1; > + tmp_len+= 2; > + } > + else if (next_char == '.' && ptr+3 < end && > + my_toupper(system_charset_info, *(ptr+2)) == 'C' && > + *(ptr+3) == '.') > + { > + *tmp_fmt= FMT_BC_DOT; > + ptr+= 3; > + tmp_len+= 4; > + } > + else > + goto error; > + break; > + case 'P': // PM or P.M. > + next_char= my_toupper(system_charset_info, *(ptr+1)); > + if (next_char == 'M') > + { > + *tmp_fmt= FMT_PM; > + ptr+= 1; > + tmp_len+= 2; > + } > + else if (next_char == '.' && > + my_toupper(system_charset_info, *(ptr+2)) == 'M' && > + my_toupper(system_charset_info, *(ptr+3)) == '.') > + { > + *tmp_fmt= FMT_PM_DOT; > + ptr+= 3; > + tmp_len+= 4; > + } > + else > + goto error; > + break; > + case 'Y': // Y, YY, YYY o YYYYY > + if (ptr + 1 == end || my_toupper(system_charset_info, *(ptr+1)) != 'Y') > + { > + *tmp_fmt= FMT_Y; > + tmp_len+= 1; > + break; > + } > + if (ptr + 2 == end || > + my_toupper(system_charset_info, *(ptr+2)) != 'Y') /* YY */ > + { > + *tmp_fmt= FMT_YY; > + ulen= 2; > + } > + else > + { > + if (ptr + 3 < end && my_toupper(system_charset_info, *(ptr+3)) == 'Y') > + { > + *tmp_fmt= FMT_YYYY; > + ulen= 4; > + } > + else > + { > + *tmp_fmt= FMT_YYY; > + ulen= 3; > + } > + } > + ptr+= ulen-1; > + tmp_len+= ulen; > + break; > + > + case 'R': // RR or RRRR > + if (ptr + 1 == end || my_toupper(system_charset_info, *(ptr+1)) != 'R') > + goto error; > + > + if (ptr + 2 == end || my_toupper(system_charset_info, *(ptr+2)) != 'R') > + { > + *tmp_fmt= FMT_RR; > + ulen= 2; > + } > + else > + { > + if (ptr + 3 >= end || my_toupper(system_charset_info, *(ptr+3)) != 'R') > + goto error; > + *tmp_fmt= FMT_RRRR; > + ulen= 4; > + } > + ptr+= ulen-1; > + tmp_len+= ulen; > + break; > + case 'M': > + { > + char tmp1; > + if (ptr + 1 >= end) > + goto error; > + > + tmp1= my_toupper(system_charset_info, *(ptr+1)); > + if (tmp1 == 'M') > + { > + *tmp_fmt= FMT_MM; > + tmp_len+= 2; > + ptr+= 1; > + } > + else if (tmp1 == 'I') > + { > + *tmp_fmt= FMT_MI; > + tmp_len+= 2; > + ptr+= 1; > + } > + else if (tmp1 == 'O') > + { > + if (ptr + 2 >= end) > + goto error; > + char tmp2= my_toupper(system_charset_info, *(ptr+2)); > + if (tmp2 != 'N') > + goto error; > + > + if (ptr + 4 >= end || > + my_toupper(system_charset_info, *(ptr+3)) != 'T' || > + my_toupper(system_charset_info, *(ptr+4)) != 'H') > + { > + *tmp_fmt= FMT_MON; > + tmp_len+= 3; > + ptr+= 2; > + } > + else > + { > + *tmp_fmt= FMT_MONTH; > + tmp_len+= (locale->max_month_name_length * > + my_charset_utf8mb3_bin.mbmaxlen); > + ptr+= 4; > + } > + } > + else > + goto error; > + } > + break; > + case 'D': // DD, DY, or DAY > + { > + if (ptr + 1 >= end) > + goto error; > + char tmp1= my_toupper(system_charset_info, *(ptr+1)); > + > + if (tmp1 == 'D') > + { > + *tmp_fmt= FMT_DD; > + tmp_len+= 2; > + } > + else if (tmp1 == 'Y') > + { > + *tmp_fmt= FMT_DY; > + tmp_len+= 3; > + } > + else if (tmp1 == 'A') // DAY > + { > + if (ptr + 2 == end || my_toupper(system_charset_info, *(ptr+2)) != 'Y') > + goto error; > + *tmp_fmt= FMT_DAY; > + tmp_len+= locale->max_day_name_length * my_charset_utf8mb3_bin.mbmaxlen; > + ptr+= 1; > + } > + else > + goto error; > + ptr+= 1; > + } > + break; > + case 'H': // HH, HH12 or HH23 > + { > + char tmp1, tmp2, tmp3; > + if (ptr + 1 >= end) > + goto error; > + tmp1= my_toupper(system_charset_info, *(ptr+1)); > + > + if (tmp1 != 'H') > + goto error; > + > + if (ptr+3 >= end) > + { > + *tmp_fmt= FMT_HH; > + ptr+= 1; > + } > + else > + { > + tmp2= *(ptr+2); > + tmp3= *(ptr+3); > + > + if (tmp2 == '1' && tmp3 == '2') > + { > + *tmp_fmt= FMT_HH12; > + ptr+= 3; > + } > + else if (tmp2 == '2' && tmp3 == '4') > + { > + *tmp_fmt= FMT_HH24; > + ptr+= 3; > + } > + else > + { > + *tmp_fmt= FMT_HH; > + ptr+= 1; > + } > + } > + tmp_len+= 2; > + break; > + } > + case 'S': // SS > + if (ptr + 1 == end || my_toupper(system_charset_info, *(ptr+1)) != 'S') > + goto error; > + > + *tmp_fmt= FMT_SS; > + tmp_len+= 2; > + ptr+= 1; > + break; > + case '|': > + /* > + If only one '|' just ignore it, else append others, for example: > + TO_CHAR('2000-11-05', 'YYYY|MM||||DD') --> 200011|||05 > + */ > + if (ptr + 1 == end || *(ptr+1) != '|') > + { > + tmp_fmt--; > + break; > + } > + ptr++; // Skip first '|' > + do > + { > + *tmp_fmt++= *ptr++; > + tmp_len++; > + } while ((ptr < end) && *ptr == '|'); > + ptr--; // Fix ptr for above for loop > + tmp_fmt--; > + break; > + > + default: > + offset= parse_special(cfmt, ptr, end, tmp_fmt); > + if (!offset) > + goto error; > + /* ptr++ is in the for loop, so we must move ptr to offset-1 */ > + ptr+= (offset-1); > + tmp_fmt+= (offset-1); > + tmp_len+= offset; > + break; > + } > + } > + *fmt_len= tmp_len; > + *tmp_fmt= 0; > + return 0; > + > +error: > + warning_message.append(STRING_WITH_LEN("date format not recognized at ")); > + warning_message.append(ptr, MY_MIN(8, end- ptr)); > + return 1; > +} > + > + > +static inline bool append_val(int val, int size, String *str) > +{ > + ulong len= 0; > + char intbuff[15]; > + > + len= (ulong) (int10_to_str(val, intbuff, 10) - intbuff); > + return str->append_with_prefill(intbuff, len, size, '0'); > +} > + > + > +static bool make_date_time_oracle(const uint16 *fmt_array, > + const MYSQL_TIME *l_time, > + const MY_LOCALE *locale, > + String *str) > +{ > + bool quotation_flag= false; > + const uint16 *ptr= fmt_array; > + uint hours_i; > + uint weekday; > + > + str->length(0); > + > + while (*ptr) > + { > + if (check_quotation(*ptr, &quotation_flag)) > + { > + /* don't display '"' in the result, so if it is '"', skip it */ > + if (*ptr != '"') > + { > + DBUG_ASSERT(*ptr <= 255); > + str->append((char) *ptr); > + } > + ptr++; > + continue; > + } > + > + switch (*ptr) { > + > + case FMT_AM: > + case FMT_PM: > + if (l_time->hour > 11) > + str->append("PM", 2); > + else > + str->append("AM", 2); > + break; > + > + case FMT_AM_DOT: > + case FMT_PM_DOT: > + if (l_time->hour > 11) > + str->append(STRING_WITH_LEN("P.M.")); > + else > + str->append(STRING_WITH_LEN("A.M.")); > + break; > + > + case FMT_AD: > + case FMT_BC: > + if (l_time->year > 0) > + str->append(STRING_WITH_LEN("AD")); > + else > + str->append(STRING_WITH_LEN("BC")); > + break; > + > + case FMT_AD_DOT: > + case FMT_BC_DOT: > + if (l_time->year > 0) > + str->append(STRING_WITH_LEN("A.D.")); > + else > + str->append(STRING_WITH_LEN("B.C.")); > + break; > + > + case FMT_Y: > + if (append_val(l_time->year%10, 1, str)) > + goto err_exit; > + break; > + > + case FMT_YY: > + case FMT_RR: > + if (append_val(l_time->year%100, 2, str)) > + goto err_exit; > + break; > + > + case FMT_YYY: > + if (append_val(l_time->year%1000, 3, str)) > + goto err_exit; > + break; > + > + case FMT_YYYY: > + case FMT_RRRR: > + if (append_val(l_time->year, 4, str)) > + goto err_exit; > + break; > + > + case FMT_MM: > + if (append_val(l_time->month, 2, str)) > + goto err_exit; > + break; > + > + case FMT_MON: > + { > + if (l_time->month == 0) > + { > + str->append("00", 2); > + } > + else > + { > + const char *month_name= (locale->ab_month_names-> > + type_names[l_time->month-1]); > + size_t m_len= strlen(month_name); > + str->append(month_name, m_len, system_charset_info); > + } > + } > + break; > + > + case FMT_MONTH: > + { > + if (l_time->month == 0) > + { > + str->append("00", 2); > + } > + else > + { > + const char *month_name= (locale->month_names-> > + type_names[l_time->month-1]); > + size_t month_byte_len= strlen(month_name); > + size_t month_char_len; > + str->append(month_name, month_byte_len, system_charset_info); > + month_char_len= my_numchars_mb(&my_charset_utf8mb3_general_ci, > + month_name, month_name + > + month_byte_len); > + if (str->strfill(' ', locale->max_month_name_length - month_char_len)) > + goto err_exit; > + } > + } > + break; > + > + case FMT_DD: > + if (append_val(l_time->day, 2, str)) > + goto err_exit; > + break; > + > + case FMT_DY: > + { > + if (l_time->day == 0) > + str->append("00", 2); > + else > + { > + weekday= calc_weekday(calc_daynr(l_time->year,l_time->month, > + l_time->day), 0); > + const char *day_name= locale->ab_day_names->type_names[weekday]; > + str->append(day_name, strlen(day_name), system_charset_info); > + } > + } > + break; > + > + case FMT_DAY: > + { > + if (l_time->day == 0) > + str->append("00", 2, system_charset_info); > + else > + { > + const char *day_name; > + size_t day_byte_len, day_char_len; > + weekday=calc_weekday(calc_daynr(l_time->year,l_time->month, > + l_time->day), 0); > + day_name= locale->day_names->type_names[weekday]; > + day_byte_len= strlen(day_name); > + str->append(day_name, day_byte_len, system_charset_info); > + day_char_len= my_numchars_mb(&my_charset_utf8mb3_general_ci, > + day_name, day_name + day_byte_len); > + if (str->strfill(' ', locale->max_day_name_length - day_char_len)) > + goto err_exit; > + } > + } > + break; > + > + case FMT_HH12: > + case FMT_HH: > + hours_i= (l_time->hour%24 + 11)%12+1; > + if (append_val(hours_i, 2, str)) > + goto err_exit; > + break; > + > + case FMT_HH24: > + if (append_val(l_time->hour, 2, str)) > + goto err_exit; > + break; > + > + case FMT_MI: > + if (append_val(l_time->minute, 2, str)) > + goto err_exit; > + break; > + > + case FMT_SS: > + if (append_val(l_time->second, 2, str)) > + goto err_exit; > + break; > + > + default: > + str->append((char) *ptr); > + } > + > + ptr++; > + }; > + return false; > + > +err_exit: > + return true; > +} > + > + > +bool Item_func_tochar::fix_length_and_dec() > +{ > + thd= current_thd; > + CHARSET_INFO *cs= thd->variables.collation_connection; > + Item *arg1= args[1]->this_item(); > + my_repertoire_t repertoire= arg1->collation.repertoire; > + StringBuffer<STRING_BUFFER_USUAL_SIZE> buffer; > + String *str; > + > + locale= thd->variables.lc_time_names; > + if (!thd->variables.lc_time_names->is_ascii) > + repertoire|= MY_REPERTOIRE_EXTENDED; > + collation.set(cs, arg1->collation.derivation, repertoire); > + > + /* first argument must be datetime or string */ > + enum_field_types arg0_mysql_type= args[0]->field_type(); > + > + max_length= 0; > + switch (arg0_mysql_type) { > + case MYSQL_TYPE_TIME: > + case MYSQL_TYPE_DATE: > + case MYSQL_TYPE_DATETIME: > + case MYSQL_TYPE_TIMESTAMP: > + case MYSQL_TYPE_VARCHAR: > + case MYSQL_TYPE_STRING: > + break; > + default: > + { > + my_printf_error(ER_ORACLE_COMPAT_FUNCTION_ERROR, > + ER(ER_ORACLE_COMPAT_FUNCTION_ERROR), > + MYF(0), > + "data type of first argument must be type " > + "date/datetime/time or string"); that's not how MariaDB works, it converts types. In particular, using an integer 20200101 in the date context is perfectly ok. > + return TRUE; > + } > + } > + if (args[1]->basic_const_item() && (str= args[1]->val_str(&buffer))) > + { > + uint ulen; > + fixed_length= 1; > + if (parse_format_string(str, &ulen)) > + { > + my_printf_error(ER_ORACLE_COMPAT_FUNCTION_ERROR, > + ER(ER_ORACLE_COMPAT_FUNCTION_ERROR), > + MYF(0), > + warning_message.c_ptr()); > + return TRUE; > + } > + max_length= (uint32) (ulen * collation.collation->mbmaxlen); > + } > + else > + { > + fixed_length= 0; > + max_length= (uint32) MY_MIN(arg1->max_length * 10 * > + collation.collation->mbmaxlen, > + MAX_BLOB_WIDTH); > + } > + set_maybe_null(); > + return FALSE; > +} > + > + > +String *Item_func_tochar::val_str(String* str) > + { > + StringBuffer<64> format_buffer; > + String *format; > + MYSQL_TIME l_time; > + const MY_LOCALE *lc= locale; > + date_conv_mode_t mode= TIME_CONV_NONE; > + size_t max_result_length= max_length; > + > + if (warning_message.length()) > + goto null_date; > + > + if ((null_value= args[0]->get_date(thd, &l_time, > + Temporal::Options(mode, thd)))) > + return 0; > + > + if (!fixed_length) > + { > + uint ulen; > + if (!(format= args[1]->val_str(&format_buffer)) || !format->length() || > + parse_format_string(format, &ulen)) > + goto null_date; > + max_result_length= ((size_t) ulen) * collation.collation->mbmaxlen; > + } > + > + if (str->alloc(max_result_length)) > + goto null_date; > + > + /* Create the result string */ > + str->set_charset(collation.collation); > + if (!make_date_time_oracle(fmt_array, &l_time, lc, str)) > + return str; > + > +null_date: > + > + if (warning_message.length()) > + { > + push_warning_printf(thd, > + Sql_condition::WARN_LEVEL_WARN, > + ER_ORACLE_COMPAT_FUNCTION_ERROR, > + ER_THD(thd, ER_ORACLE_COMPAT_FUNCTION_ERROR), > + warning_message.c_ptr()); > + if (!fixed_length) > + warning_message.length(0); > + } > + > + null_value= 1; > + return 0; > +} > + > > bool Item_func_from_unixtime::fix_length_and_dec() > { Regards, Sergei

2 1

Re: [Maria-developers] a1440737662: MDEV-20021 sql_mode="oracle" does not support MINUS set operator
by Sergei Golubchik 20 Apr '21

20 Apr '21

Hi, Monty! On Apr 13, Michael Widenius wrote: > revision-id: a1440737662 (mariadb-10.5.2-553-ga1440737662) > parent(s): 04a13e6ab8f > author: Michael Widenius <michael.widenius(a)gmail.com> > committer: Michael Widenius <michael.widenius(a)gmail.com> > timestamp: 2021-03-24 19:51:22 +0200 > message: > > MDEV-20021 sql_mode="oracle" does not support MINUS set operator > > MINUS is mapped to EXCEPT > > diff --git a/sql/sql_yacc.yy b/sql/sql_yacc.yy > index 57ba9df42c0..edd2f353dd0 100644 > --- a/sql/sql_yacc.yy > +++ b/sql/sql_yacc.yy > @@ -16037,6 +16038,7 @@ reserved_keyword_udt_not_param_type: > | MINUTE_MICROSECOND_SYM > | MINUTE_SECOND_SYM > | MIN_SYM > + | MINUS_ORACLE_SYM > | MODIFIES_SYM > | MOD_SYM > | NATURAL this is not good. MINUS should be in reserved_keyword_udt_not_param_type only in oracle mode, and otherwise it should be in keyword_sp_var_and_label (or not a keyword at all, but I don't think it's possible). Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] 04a13e6ab8f: MDEV-24285 support oracle build-in function: sys_guid
by Sergei Golubchik 20 Apr '21

20 Apr '21

Hi, Monty! On Apr 13, Michael Widenius wrote: > revision-id: 04a13e6ab8f (mariadb-10.5.2-552-g04a13e6ab8f) > parent(s): c76ac1e6de8 > author: Michael Widenius <michael.widenius(a)gmail.com> > committer: Michael Widenius <michael.widenius(a)gmail.com> > timestamp: 2021-03-24 19:42:30 +0200 > message: > > MDEV-24285 support oracle build-in function: sys_guid > > SYS_GUID() returns same as UUID(), but without any '-' I'd rather do it as a flag in Item_func_uuid. Less code, no need to copy-paste everything. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] c4de76aeff8: MDEV-17554 Auto-create new partition for system versioned tables with history partitioned by INTERVAL/LIMIT
by Sergei Golubchik 18 Apr '21

18 Apr '21

Hi, Aleksey! Note, it's a review of the `git diff 82e44d60d1e 8175ce8a5f1` so it's not only commit c4de76aeff8 On Apr 04, Aleksey Midenkov wrote: > revision-id: c4de76aeff8 (mariadb-10.5.2-540-gc4de76aeff8) > parent(s): 82e44d60d1e > author: Aleksey Midenkov <midenok(a)gmail.com> > committer: Aleksey Midenkov <midenok(a)gmail.com> > timestamp: 2021-03-31 21:17:55 +0300 > message: > > MDEV-17554 Auto-create new partition for system versioned tables with history partitioned by INTERVAL/LIMIT Overall it's very good! I have few minor questions/comments, see below. But it's almost done, I think. > diff --git a/mysql-test/suite/versioning/r/delete_history.result b/mysql-test/suite/versioning/r/delete_history.result > index cb865a835b3..90c9e4777bb 100644 > --- a/mysql-test/suite/versioning/r/delete_history.result > +++ b/mysql-test/suite/versioning/r/delete_history.result > @@ -154,3 +152,18 @@ select * from t1; > a > 1 > drop table t1; > +# > +# MDEV-17554 Auto-create new partition for system versioned tables with history partitioned by INTERVAL/LIMIT > +# > +# Don't auto-create new partition on DELETE HISTORY: > +create or replace table t (a int) with system versioning > +partition by system_time limit 1000 auto; > +delete history from t; > +show create table t; > +Table Create Table > +t CREATE TABLE `t` ( > + `a` int(11) DEFAULT NULL > +) ENGINE=DEFAULT_ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING > + PARTITION BY SYSTEM_TIME LIMIT 1000 AUTO > +PARTITIONS 2 Hmm, and if DELETE HISTORY would auto-create new partitions, what output would one expect here? I mean, how one can see whether the test result is correct or wrong? > +drop table t; > diff --git a/mysql-test/suite/versioning/t/partition.test b/mysql-test/suite/versioning/t/partition.test > index 445f5844630..57b80ca0b47 100644 > --- a/mysql-test/suite/versioning/t/partition.test > +++ b/mysql-test/suite/versioning/t/partition.test > @@ -1068,11 +1078,412 @@ alter table t1 add partition partitions 2; > --replace_result $default_engine DEFAULT_ENGINE > show create table t1; > alter table t1 add partition partitions 8; > +drop table t1; > + > +--echo # > +--echo # MDEV-17554 Auto-create new partition for system versioned tables with history partitioned by INTERVAL/LIMIT > +--echo # > +create or replace table t1 (x int) with system versioning > +partition by system_time limit 1 auto; > +--replace_result $default_engine DEFAULT_ENGINE > +show create table t1; > + > +--echo # INSERT, INSERT .. SELECT don't increment partitions it's not really "increment", better say "don't auto-create" > +insert into t1 values (1); > + > +create or replace table t2 (y int); > +insert into t2 values (2); > + > +insert into t1 select * from t2; > +insert into t2 select * from t1; why do you need a t2 table in this test? > +--replace_result $default_engine DEFAULT_ENGINE > +show create table t1; > + > +--echo # Too many partitions error > +set timestamp= unix_timestamp('2000-01-01 00:00:00'); > +create or replace table t1 (x int) with system versioning > +partition by system_time interval 1 hour auto; > +set timestamp= unix_timestamp('2001-01-01 00:01:00'); > +--error ER_TOO_MANY_PARTITIONS_ERROR > +update t1 set x= x + 1; > + > +--enable_info > +--echo # Increment from 3 to 5 > +create or replace table t1 (x int) with system versioning > +partition by system_time interval 3600 second > +starts '2000-01-01 00:00:00' auto partitions 3; > + > +insert into t1 values (1); > +--replace_result $default_engine DEFAULT_ENGINE > +show create table t1; > + > +set timestamp= unix_timestamp('2000-01-01 02:00:00'); > +update t1 set x= x + 1; > +--replace_result $default_engine DEFAULT_ENGINE > +show create table t1; > + > +set timestamp= unix_timestamp('2000-01-01 03:00:00'); > +update t1 set x= x + 2; > +--replace_result $default_engine DEFAULT_ENGINE > +show create table t1; > + > +--echo # Increment from 3 to 6, manual names, LOCK TABLES > +set timestamp= unix_timestamp('2000-01-01 00:00:00'); > +create or replace table t1 (x int) with system versioning > +partition by system_time interval 1 hour auto ( > + partition p1 history, > + partition p3 history, > + partition pn current); > + > +insert into t1 values (1); > --replace_result $default_engine DEFAULT_ENGINE > show create table t1; > > +set timestamp= unix_timestamp('2000-01-01 02:00:00'); > +update t1 set x= x + 3; > +--replace_result $default_engine DEFAULT_ENGINE > +show create table t1; > + > +set timestamp= unix_timestamp('2000-01-01 03:00:00'); > +update t1 set x= x + 4; > +--replace_result $default_engine DEFAULT_ENGINE > +show create table t1; > + > +set timestamp= unix_timestamp('2000-01-01 04:00:00'); > +lock tables t1 write; > +update t1 set x= x + 5; > +--replace_result $default_engine DEFAULT_ENGINE > +show create table t1; > +unlock tables; > +set timestamp= default; > + > +--echo # Couple of more LOCK TABLES cases > +create or replace table t1 (x int) with system versioning > +partition by system_time limit 1000 auto; > +lock tables t1 write; > +insert into t1 values (1); > +update t1 set x= x + 1; > +unlock tables; > +--replace_result $default_engine DEFAULT_ENGINE > +show create table t1; what does this test case demonstrate? > + > +--echo # Overflow prevention under LOCK TABLES > +create or replace table t1 (x int) > +with system versioning partition by system_time > +limit 10 auto; > + > +insert into t1 values (1), (2), (3), (4), (5), (6), (7), (8), (9); > +update t1 set x= x + 10; > + > +lock tables t1 write; > +update t1 set x= 1 where x = 11; > +update t1 set x= 2 where x = 12; > +update t1 set x= 3 where x = 13; > +unlock tables; > + > +select count(x) from t1 partition (p0); > +--replace_result $default_engine DEFAULT_ENGINE > +show create table t1; > +drop tables t1; > + > +--echo # Test VIEW, LOCK TABLES > +set timestamp= unix_timestamp('2000-01-01 00:00:00'); > +create or replace table t1 (x int) with system versioning > +partition by system_time interval 1 hour auto partitions 3; > +create or replace view v1 as select * from t1; > + > +insert into v1 values (1); why would a view matter in this test case? > +--replace_result $default_engine DEFAULT_ENGINE > +show create table t1; > + > +set timestamp= unix_timestamp('2000-01-01 01:00:00'); > +lock tables t1 write; > +update t1 set x= x + 3; > +--replace_result $default_engine DEFAULT_ENGINE > +show create table t1; > +unlock tables; > + > +drop view v1; > drop tables t1; > > +--echo # Multiple increments in single command > +set timestamp= unix_timestamp('2000-01-01 00:00:00'); > +create or replace table t1 (x int) with system versioning > +partition by system_time interval 1 hour auto partitions 3; > + > +create or replace table t2 (y int) with system versioning > +partition by system_time interval 1 hour auto; > + > +insert into t1 values (1); > +insert into t2 values (2); > + > +set timestamp= unix_timestamp('2000-01-01 01:00:00'); > +update t1, t2 set x= x + 1, y= y + 1; > +--replace_result $default_engine DEFAULT_ENGINE > +show create table t1; > +--replace_result $default_engine DEFAULT_ENGINE > +show create table t2; > + > +set timestamp= unix_timestamp('2000-01-01 02:00:00'); > +update t1, t2 set x= x + 1, y= y + 1; > +--replace_result $default_engine DEFAULT_ENGINE > +show create table t1; > +--replace_result $default_engine DEFAULT_ENGINE > +show create table t2; > + > +--echo # Here t2 is incremented too, but not updated > +set timestamp= unix_timestamp('2000-01-01 03:00:00'); > +update t1, t2 set t1.x= 0 where t1.x< t2.y; > +--replace_result $default_engine DEFAULT_ENGINE > +show create table t1; > +# Multiupdate_prelocking_strategy::handle_end() is processed after table open. > +# For PS it is possible to skip unneeded auto-creation because the above happens at > +# prepare stage and auto-creation is done at execute stage. > +--replace_result $default_engine DEFAULT_ENGINE 'PARTITIONS 4' 'PARTITIONS ok' 'PARTITIONS 5' 'PARTITIONS ok' eh... I don't think this is really "ok". As far as I remember, Multiupdate_prelocking_strategy knows what tables should be opened for writing and what for reading. Why would a new partition be created for t2? > +show create table t2; > + > +drop tables t1, t2; > + > +--echo # PS, SP, LOCK TABLES > +set timestamp= unix_timestamp('2000-01-01 00:00:00'); > +create or replace table t1 (x int) with system versioning > +partition by system_time interval 1 hour auto; > + > +insert into t1 values (1); > + > +set timestamp= unix_timestamp('2000-01-01 01:00:00'); > +execute immediate 'update t1 set x= x + 5'; > +--replace_result $default_engine DEFAULT_ENGINE > +show create table t1; > + > +prepare s from 'update t1 set x= x + 6'; > +set timestamp= unix_timestamp('2000-01-01 02:00:00'); > +execute s; execute s; > +--replace_result $default_engine DEFAULT_ENGINE > +show create table t1; > + > +set timestamp= unix_timestamp('2000-01-01 03:00:00'); > +lock tables t1 write; > +execute s; execute s; > +--replace_result $default_engine DEFAULT_ENGINE > +show create table t1; > +unlock tables; > +drop prepare s; > + > +create procedure sp() update t1 set x= x + 7; > +set timestamp= unix_timestamp('2000-01-01 04:00:00'); > +call sp; call sp; > +--replace_result $default_engine DEFAULT_ENGINE > +show create table t1; > +set timestamp= unix_timestamp('2000-01-01 05:00:00'); > +lock tables t1 write; > +call sp; call sp; > +--replace_result $default_engine DEFAULT_ENGINE > +show create table t1; > +unlock tables; > +drop procedure sp; > + > +set timestamp= unix_timestamp('2001-01-01 00:00:00'); > +create or replace table t1 (i int) with system versioning > +partition by system_time interval 1 day starts '2000-01-01 00:00:00'; > +insert into t1 values (0); > +set timestamp= unix_timestamp('2001-01-01 00:00:01'); > +prepare s from 'update t1 set i= i + 1'; > +execute s; > +set timestamp= unix_timestamp('2001-01-02 00:00:01'); > +execute s; > +drop prepare s; > + > +if (!$MTR_COMBINATION_HEAP) because of blobs? > +{ > +--echo # Complex table > +set timestamp= unix_timestamp('2000-01-01 00:00:00'); > +create or replace table t1 ( > + x int primary key auto_increment, ... > +--echo # Transaction > +create or replace table t1 (x int) with system versioning engine innodb > +partition by system_time interval 1 hour auto; > +start transaction; > +insert into t1 values (1); > +select * from t1; > + > +--connect con1, localhost, root > +set lock_wait_timeout= 1; > +set innodb_lock_wait_timeout= 1; > +--error ER_LOCK_WAIT_TIMEOUT > +update t1 set x= x + 111; > +select * from t1; what do you test here? (there is no show create table in the test) > + > +# cleanup > +--disconnect con1 > +--connection default > +drop table t1; > + > +set timestamp= unix_timestamp('2000-01-01 00:00:00'); > +create or replace table t1 (x int) with system versioning engine innodb > +partition by system_time interval 1 hour auto; > + > +insert into t1 values (1); > +set timestamp= unix_timestamp('2000-01-01 01:00:00'); > +start transaction; > +update t1 set x= 0; > +--connect con1, localhost, root > +select * from t1; if you add a show create table here, what would it show? > +--connection default > +commit; > +show create table t1; > + > +set timestamp= unix_timestamp('2000-01-01 02:00:00'); > +start transaction; > +update t1 set x= 1; > +--connection con1 > +select * from t1; > +--connection default > +rollback; > +show create table t1; > +--disconnect con1 > +--connection default > +drop table t1; > + > +--echo # > +--echo # MENT-724 Locking timeout caused by auto-creation affects original DML I'd better avoid MENT references here. Let's only mention Jira issues that users can actually see > +--echo # > +set timestamp= unix_timestamp('2000-01-01 00:00:00'); > +create or replace table t1 (x int primary key) with system versioning engine innodb > +partition by system_time interval 1 hour auto; > + > +insert into t1 values (1); > +start transaction; > +insert into t1 values (2); > + > +--connect con1, localhost, root > +set lock_wait_timeout= 1; > +set innodb_lock_wait_timeout= 1; > +set timestamp= unix_timestamp('2000-01-01 01:00:01'); > +update t1 set x= x + 122 where x = 1; isn't it a test that you already have above? with x = x + 111 > +--disconnect con1 > +--connection default > +select * from t1; > + > +# cleanup > +drop table t1; > +set timestamp= default; > + > --echo # > --echo # End of 10.5 tests > --echo # > diff --git a/mysql-test/suite/versioning/r/partition.result b/mysql-test/suite/versioning/r/partition.result > index 2a277b1c2ea..8369b40d98c 100644 > --- a/mysql-test/suite/versioning/r/partition.result > +++ b/mysql-test/suite/versioning/r/partition.result > @@ -1236,27 +1270,752 @@ t1 CREATE TABLE `t1` ( ... > +set timestamp= unix_timestamp('2000-01-01 03:00:00'); > +affected rows: 0 > +lock tables t1 write; > +affected rows: 0 > +execute s; > +affected rows: 1 > +info: Rows matched: 1 Changed: 1 Inserted: 1 Warnings: 0 > +execute s; > +affected rows: 1 > +info: Rows matched: 1 Changed: 1 Inserted: 1 Warnings: 0 > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `x` int(11) DEFAULT NULL > +) ENGINE=DEFAULT_ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING > + PARTITION BY SYSTEM_TIME INTERVAL 1 HOUR STARTS TIMESTAMP'2000-01-01 00:00:00' AUTO > +PARTITIONS 6 why did it add two partitions here? > +affected rows: 1 > +unlock tables; > +affected rows: 0 > +drop prepare s; > +affected rows: 0 > +create procedure sp() update t1 set x= x + 7; > +affected rows: 0 > +set timestamp= unix_timestamp('2000-01-01 04:00:00'); > +affected rows: 0 > +call sp; > +affected rows: 1 > +call sp; > +affected rows: 1 > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `x` int(11) DEFAULT NULL > +) ENGINE=DEFAULT_ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING > + PARTITION BY SYSTEM_TIME INTERVAL 1 HOUR STARTS TIMESTAMP'2000-01-01 00:00:00' AUTO > +PARTITIONS 6 > +affected rows: 1 > +set timestamp= unix_timestamp('2000-01-01 05:00:00'); > +affected rows: 0 > +lock tables t1 write; > +affected rows: 0 > +call sp; > +affected rows: 1 > +call sp; > +affected rows: 1 > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `x` int(11) DEFAULT NULL > +) ENGINE=DEFAULT_ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING > + PARTITION BY SYSTEM_TIME INTERVAL 1 HOUR STARTS TIMESTAMP'2000-01-01 00:00:00' AUTO > +PARTITIONS 8 and two here > +affected rows: 1 > +unlock tables; > +affected rows: 0 > +drop procedure sp; > +affected rows: 0 > +set timestamp= unix_timestamp('2001-01-01 00:00:00'); > +affected rows: 0 > +create or replace table t1 (i int) with system versioning > +partition by system_time interval 1 day starts '2000-01-01 00:00:00'; > +affected rows: 0 > +insert into t1 values (0); > +affected rows: 1 > +set timestamp= unix_timestamp('2001-01-01 00:00:01'); > +affected rows: 0 > +prepare s from 'update t1 set i= i + 1'; > +affected rows: 0 > +info: Statement prepared > +execute s; > +affected rows: 1 > +info: Rows matched: 1 Changed: 1 Inserted: 1 Warnings: 1 > +Warnings: > +Warning 4114 Versioned table `test`.`t1`: last HISTORY partition (`p0`) is out of INTERVAL, need more HISTORY partitions why? > +set timestamp= unix_timestamp('2001-01-02 00:00:01'); > +affected rows: 0 > +execute s; > +affected rows: 1 > +info: Rows matched: 1 Changed: 1 Inserted: 1 Warnings: 1 > +Warnings: > +Warning 4114 Versioned table `test`.`t1`: last HISTORY partition (`p0`) is out of INTERVAL, need more HISTORY partitions > +drop prepare s; > +affected rows: 0 > +# Complex table ... > diff --git a/mysql-test/suite/versioning/t/rpl.test b/mysql-test/suite/versioning/t/rpl.test > index b5be68feece..54258a8bdf1 100644 > --- a/mysql-test/suite/versioning/t/rpl.test > +++ b/mysql-test/suite/versioning/t/rpl.test > @@ -133,4 +133,44 @@ sync_slave_with_master; > connection master; > drop table t1; > > +--echo # > +--echo # MDEV-17554 Auto-create new partition for system versioned tables with history partitioned by INTERVAL/LIMIT > +--echo # > +set timestamp= unix_timestamp('2000-01-01 00:00:00'); > +create or replace table t1 (x int) with system versioning > +partition by system_time interval 1 hour auto; > +insert t1 values (); > +set timestamp= unix_timestamp('2000-01-01 01:00:00'); > +delete from t1; > +--replace_result InnoDB ENGINE MyISAM ENGINE MEMORY ENGINE > +show create table t1; > +--sync_slave_with_master > +--replace_result InnoDB ENGINE MyISAM ENGINE MEMORY ENGINE > +show create table t1; > +--connection master > +drop table t1; > +set timestamp= default; > + > +--echo # > +--echo # MENT-685 DML events for auto-partitioned tables are written into binary log twice same comment about MENT > +--echo # > +create table t1 (a int) with system versioning > +partition by system_time limit 1 auto; > + > +insert into t1 values (1); > +update t1 set a= a + 1; > +update t1 set a= a + 1; > +--replace_result InnoDB ENGINE MyISAM ENGINE MEMORY ENGINE > +show create table t1; > +select * from t1; > + > +--sync_slave_with_master > +--replace_result InnoDB ENGINE MyISAM ENGINE MEMORY ENGINE > +show create table t1; > + > +select * from t1; > +--connection master > +# cleanup > +drop table t1; may be show binlog events? you know, to verify that DML events aren't written into binary log twice > + > --source include/rpl_end.inc > diff --git a/sql/ha_partition.h b/sql/ha_partition.h > index 60a2d7f6762..7ade4419c22 100644 > --- a/sql/ha_partition.h > +++ b/sql/ha_partition.h > @@ -1610,7 +1610,7 @@ class ha_partition :public handler > for (; part_id < part_id_end; ++part_id) > { > handler *file= m_file[part_id]; > - DBUG_ASSERT(bitmap_is_set(&(m_part_info->read_partitions), part_id)); > + bitmap_set_bit(&(m_part_info->read_partitions), part_id); Where was it set before your patch? > file->info(HA_STATUS_VARIABLE | HA_STATUS_NO_LOCK | HA_STATUS_OPEN); > part_recs+= file->stats.records; > } > diff --git a/sql/handler.cc b/sql/handler.cc > index b312635c8ee..6bb6c279193 100644 > --- a/sql/handler.cc > +++ b/sql/handler.cc > @@ -1572,7 +1572,7 @@ int ha_commit_trans(THD *thd, bool all) > DBUG_ASSERT(thd->transaction->stmt.ha_list == NULL || > trans == &thd->transaction->stmt); > > - if (thd->in_sub_stmt) > + if (thd->in_sub_stmt & ~SUB_STMT_AUTO_HIST) > { where is this ha_commit_trans(thd, false) called from? mysql_alter_table that adds a new partition? > DBUG_ASSERT(0); > /* > diff --git a/sql/partition_info.cc b/sql/partition_info.cc > index 871411cf6c4..cf8dd140553 100644 > --- a/sql/partition_info.cc > +++ b/sql/partition_info.cc > @@ -816,10 +816,17 @@ bool partition_info::has_unique_name(partition_element *element) > vers_info->interval Limit by fixed time interval > vers_info->hist_part (out) Working history partition > */ > -void partition_info::vers_set_hist_part(THD *thd) > +uint partition_info::vers_set_hist_part(THD *thd, bool auto_hist) > { > + DBUG_ASSERT(!thd->lex->last_table() || > + !thd->lex->last_table()->vers_conditions.delete_history); is that right? Conceptually you need to test vers_conditions.delete_history for the table that owns this partition_info. Is it always last_table() ? I'd say it'd be more logical to do, like part_field_array[0]->table->pos_in_table_list > + > + uint create_count= 0; > + auto_hist= auto_hist && vers_info->auto_hist; > + > if (vers_info->limit) > { > + DBUG_ASSERT(!vers_info->interval.is_set()); > ha_partition *hp= (ha_partition*)(table->file); > partition_element *next= NULL; > List_iterator<partition_element> it(partitions); > @@ -862,9 +869,183 @@ void partition_info::vers_set_hist_part(THD *thd) > { > vers_info->hist_part= next; > if (next->range_value > thd->query_start()) > - return; > + { > + error= false; > + break; > + } > + } > + if (error) > + { > + if (auto_hist) > + { > + DBUG_ASSERT(thd->query_start() >= vers_info->hist_part->range_value); > + my_time_t diff= thd->query_start() - (my_time_t) vers_info->hist_part->range_value; > + if (diff > 0) > + { > + size_t delta= vers_info->interval.seconds(); > + create_count= (uint) (diff / delta + 1); > + if (diff % delta) > + create_count++; > + } > + else > + create_count= 1; > + } > + else > + { > + my_error(WARN_VERS_PART_FULL, MYF(ME_WARNING|ME_ERROR_LOG), after such an overflow a table becomes essentially corrupted, rows are in a wrong partition. So new history partitions cannot be added anymore without reorganizing the last partition. How is it handled now? (it's just a question, not part of the review, as it's not something you were changing in your patch) > + table->s->db.str, table->s->table_name.str, > + vers_info->hist_part->partition_name, "INTERVAL"); > + } > } > } > + > + /* > + When hist_part is almost full LOCK TABLES my overflow the partition as we s/my/may/ > + can't add new partitions under LOCK TABLES. Reserve one for this case. > + */ > + if (auto_hist && create_count == 0 && > + thd->lex->sql_command == SQLCOM_LOCK_TABLES && > + vers_info->hist_part->id + 1 == vers_info->now_part->id) > + create_count= 1; > + > + return create_count; > +} > + > + > +/** > + @brief Run fast_alter_partition_table() to add new history partitions > + for tables requiring them. > +*/ > +bool vers_add_auto_hist_parts(THD *thd, TABLE_LIST* tl, uint num_parts) > +{ > + bool result= true; > + HA_CREATE_INFO create_info; > + Alter_info alter_info; > + partition_info *save_part_info= thd->work_part_info; > + Query_tables_list save_query_tables; > + Reprepare_observer *save_reprepare_observer= thd->m_reprepare_observer; > + bool save_no_write_to_binlog= thd->lex->no_write_to_binlog; > + thd->m_reprepare_observer= NULL; > + thd->lex->reset_n_backup_query_tables_list(&save_query_tables); > + thd->in_sub_stmt|= SUB_STMT_AUTO_HIST; > + thd->lex->no_write_to_binlog= true; > + TABLE *table= tl->table; > + > + DBUG_ASSERT(!thd->is_error()); > + > + { > + DBUG_ASSERT(table->s->get_table_ref_type() == TABLE_REF_BASE_TABLE); > + DBUG_ASSERT(table->versioned()); > + DBUG_ASSERT(table->part_info); > + DBUG_ASSERT(table->part_info->vers_info); > + alter_info.reset(); > + alter_info.partition_flags= ALTER_PARTITION_ADD|ALTER_PARTITION_AUTO_HIST; > + create_info.init(); > + create_info.alter_info= &alter_info; > + Alter_table_ctx alter_ctx(thd, tl, 1, &table->s->db, &table->s->table_name); > + > + MDL_REQUEST_INIT(&tl->mdl_request, MDL_key::TABLE, tl->db.str, > + tl->table_name.str, MDL_SHARED_NO_WRITE, MDL_TRANSACTION); > + if (thd->mdl_context.acquire_lock(&tl->mdl_request, > + thd->variables.lock_wait_timeout)) > + goto exit; > + table->mdl_ticket= tl->mdl_request.ticket; > + > + create_info.db_type= table->s->db_type(); > + create_info.options|= HA_VERSIONED_TABLE; > + DBUG_ASSERT(create_info.db_type); > + > + create_info.vers_info.set_start(table->s->vers_start_field()->field_name); > + create_info.vers_info.set_end(table->s->vers_end_field()->field_name); > + > + partition_info *part_info= new partition_info(); > + if (unlikely(!part_info)) > + { > + my_error(ER_OUT_OF_RESOURCES, MYF(0)); > + goto exit; > + } > + part_info->use_default_num_partitions= false; > + part_info->use_default_num_subpartitions= false; > + part_info->num_parts= num_parts; > + part_info->num_subparts= table->part_info->num_subparts; > + part_info->subpart_type= table->part_info->subpart_type; > + if (unlikely(part_info->vers_init_info(thd))) > + { > + my_error(ER_OUT_OF_RESOURCES, MYF(0)); > + goto exit; > + } > + > + // NB: set_ok_status() requires DA_EMPTY > + thd->get_stmt_da()->reset_diagnostics_area(); would it not be DA_EMPTY without a reset? this is at the beginning of a statement, I'd expect it normally be DA_EMPTY here. What other DA states can you get here? > + > + thd->work_part_info= part_info; > + if (part_info->set_up_defaults_for_partitioning(thd, table->file, NULL, > + table->part_info->next_part_no(num_parts))) > + { > + push_warning(thd, Sql_condition::WARN_LEVEL_WARN, > + ER_VERS_HIST_PART_FAILED, > + "Auto-increment history partition: " > + "setting up defaults failed"); > + my_error(ER_VERS_HIST_PART_FAILED, MYF(ME_WARNING), > + tl->db.str, tl->table_name.str); > + goto exit; > + } > + bool partition_changed= false; > + bool fast_alter_partition= false; > + if (prep_alter_part_table(thd, table, &alter_info, &create_info, > + &partition_changed, &fast_alter_partition)) > + { > + push_warning(thd, Sql_condition::WARN_LEVEL_WARN, ER_VERS_HIST_PART_FAILED, > + "Auto-increment history partition: " > + "alter partitition prepare failed"); > + my_error(ER_VERS_HIST_PART_FAILED, MYF(ME_WARNING), > + tl->db.str, tl->table_name.str); > + goto exit; > + } > + if (!fast_alter_partition) > + { > + push_warning(thd, Sql_condition::WARN_LEVEL_WARN, ER_VERS_HIST_PART_FAILED, > + "Auto-increment history partition: " > + "fast alter partitition is not possible"); > + my_error(ER_VERS_HIST_PART_FAILED, MYF(ME_WARNING), > + tl->db.str, tl->table_name.str); > + goto exit; > + } > + DBUG_ASSERT(partition_changed); > + if (mysql_prepare_alter_table(thd, table, &create_info, &alter_info, > + &alter_ctx)) > + { > + push_warning(thd, Sql_condition::WARN_LEVEL_WARN, ER_VERS_HIST_PART_FAILED, > + "Auto-increment history partition: " > + "alter prepare failed"); > + my_error(ER_VERS_HIST_PART_FAILED, MYF(ME_WARNING), > + tl->db.str, tl->table_name.str); > + goto exit; > + } > + > + if (fast_alter_partition_table(thd, table, &alter_info, &create_info, > + tl, &table->s->db, &table->s->table_name)) > + { > + push_warning(thd, Sql_condition::WARN_LEVEL_WARN, ER_VERS_HIST_PART_FAILED, > + "Auto-increment history partition: " > + "alter partition table failed"); > + my_error(ER_VERS_HIST_PART_FAILED, MYF(ME_WARNING), > + tl->db.str, tl->table_name.str); > + goto exit; > + } > + } > + > + result= false; > + // NB: we have to return DA_EMPTY for new command may be DBUG_ASSERT(thd->get_stmt_da()->is_ok()); > + thd->get_stmt_da()->reset_diagnostics_area(); > + > +exit: > + thd->work_part_info= save_part_info; > + thd->m_reprepare_observer= save_reprepare_observer; > + thd->lex->restore_backup_query_tables_list(&save_query_tables); > + thd->in_sub_stmt&= ~SUB_STMT_AUTO_HIST; > + thd->lex->no_write_to_binlog= save_no_write_to_binlog; > + return result; > } > > > diff --git a/sql/partition_info.h b/sql/partition_info.h > index 0656238ec07..94093353d60 100644 > --- a/sql/partition_info.h > +++ b/sql/partition_info.h > @@ -72,9 +73,26 @@ struct Vers_part_info : public Sql_alloc > my_time_t start; > INTERVAL step; > enum interval_type type; > - bool is_set() { return type < INTERVAL_LAST; } > + bool is_set() const { return type < INTERVAL_LAST; } > + size_t seconds() const > + { > + if (step.second) > + return step.second; > + if (step.minute) > + return step.minute * 60; > + if (step.hour) > + return step.hour * 3600; > + if (step.day) > + return step.day * 3600 * 24; > + // comparison is used in rough estimates, it doesn't need to be calendar-correct > + if (step.month) > + return step.month * 3600 * 24 * 30; shouldn't it be `* 28` ? you need a most pessimistic estimate to make sure you never miss to create a partition when one is needed. You can sometimes create one when it's not needed yet, but it's less of a problem. > + DBUG_ASSERT(step.year); > + return step.year * 86400 * 30 * 365; that's one `* 30` too many :) > + } > } interval; > ulonglong limit; > + bool auto_hist; > partition_element *now_part; > partition_element *hist_part; > }; > diff --git a/sql/sql_base.cc b/sql/sql_base.cc > index 1a1186aca73..d0e255186da 100644 > --- a/sql/sql_base.cc > +++ b/sql/sql_base.cc > @@ -1625,6 +1625,52 @@ bool is_locked_view(THD *thd, TABLE_LIST *t) > } > > > +bool TABLE::vers_need_hist_part(const THD *thd, const TABLE_LIST *table_list) const > +{ > +#ifdef WITH_PARTITION_STORAGE_ENGINE > + if (part_info && part_info->part_type == VERSIONING_PARTITION && > + !table_list->vers_conditions.delete_history && > + !thd->stmt_arena->is_stmt_prepare() && > + table_list->lock_type >= TL_WRITE_ALLOW_WRITE && > + table_list->mdl_request.type >= MDL_SHARED_WRITE && > + table_list->mdl_request.type < MDL_EXCLUSIVE) > + { > + switch (thd->lex->sql_command) > + { SQLCOM_LOAD with DUP_REPLACE? > + case SQLCOM_INSERT: > + if (thd->lex->duplicates != DUP_UPDATE) > + break; > + /* fall-through: */ > + case SQLCOM_LOCK_TABLES: > + case SQLCOM_DELETE: > + case SQLCOM_UPDATE: > + case SQLCOM_REPLACE: > + case SQLCOM_REPLACE_SELECT: > + case SQLCOM_DELETE_MULTI: > + case SQLCOM_UPDATE_MULTI: > + return true; > + default:; > + break; > + } > + if (thd->rgi_slave && thd->rgi_slave->current_event && thd->lex->sql_command == SQLCOM_END) I wonder, would it be possible, instead of introducing rgi_slave->current_event, just make row events set thd->lex->sql_command appropriately? For example, currently row events increment thd->status_var.com_stat[] each event for its own SQLCOM_xxx, it won't be needed if they'll just set thd->lex->sql_command. > + { > + switch (thd->rgi_slave->current_event->get_type_code()) > + { > + case UPDATE_ROWS_EVENT: > + case UPDATE_ROWS_EVENT_V1: > + case DELETE_ROWS_EVENT: > + case DELETE_ROWS_EVENT_V1: > + return true; > + default:; > + break; > + } > + } > + } > +#endif > + return false; > +} > + > + > /** > Open a base table. > > @@ -1777,6 +1823,11 @@ bool open_table(THD *thd, TABLE_LIST *table_list, Open_table_context *ot_ctx) > DBUG_PRINT("info",("Using locked table")); > #ifdef WITH_PARTITION_STORAGE_ENGINE > part_names_error= set_partitions_as_used(table_list, table); > + if (table->vers_need_hist_part(thd, table_list)) > + { > + /* Rotation is still needed under LOCK TABLES */ a bit confusing comment, you left out stuff that are obvious to you but not to others. I'd suggest something like /* New partitions are not auto-created under LOCK TABLES (TODO: fix it) but rotation can still happen. */ > + table->part_info->vers_set_hist_part(thd, false); > + } > #endif > goto reset; > } > @@ -2032,6 +2083,20 @@ bool open_table(THD *thd, TABLE_LIST *table_list, Open_table_context *ot_ctx) > tc_add_table(thd, table); > } > > + if (!ot_ctx->vers_create_count && what does this condition mean? When is ot_ctx->vers_create_count > 0 ? > + table->vers_need_hist_part(thd, table_list)) > + { > + ot_ctx->vers_create_count= table->part_info->vers_set_hist_part(thd, true); > + if (ot_ctx->vers_create_count) > + { > + MYSQL_UNBIND_TABLE(table->file); > + tc_release_table(table); > + ot_ctx->request_backoff_action(Open_table_context::OT_ADD_HISTORY_PARTITION, > + table_list); > + DBUG_RETURN(TRUE); > + } > + } > + > if (!(flags & MYSQL_OPEN_HAS_MDL_LOCK) && > table->s->table_category < TABLE_CATEGORY_INFORMATION) > { > @@ -3172,10 +3239,18 @@ Open_table_context::recover_from_failed_open() > break; > case OT_DISCOVER: > case OT_REPAIR: > - if ((result= lock_table_names(m_thd, m_thd->lex->create_info, > - m_failed_table, NULL, > - get_timeout(), 0))) > + case OT_ADD_HISTORY_PARTITION: > + result= lock_table_names(m_thd, m_thd->lex->create_info, m_failed_table, > + NULL, get_timeout(), 0); > + if (result) > + { > + if (m_action == OT_ADD_HISTORY_PARTITION) > + { > + m_thd->clear_error(); why? > + result= false; > + } > break; > + } > > tdc_remove_table(m_thd, m_failed_table->db.str, > m_failed_table->table_name.str); > diff --git a/sql/sql_error.h b/sql/sql_error.h > index 318d5076534..92076616adb 100644 > --- a/sql/sql_error.h > +++ b/sql/sql_error.h > @@ -1195,7 +1195,6 @@ class Diagnostics_area: public Sql_state_errno, > > void copy_non_errors_from_wi(THD *thd, const Warning_info *src_wi); > > -private: It doesn't seem you're using get_warning_info() anywhere > Warning_info *get_warning_info() { return m_wi_stack.front(); } > > const Warning_info *get_warning_info() const { return m_wi_stack.front(); } > diff --git a/sql/sql_partition.cc b/sql/sql_partition.cc > index 86ba393fb7b..f4a960afe47 100644 > --- a/sql/sql_partition.cc > +++ b/sql/sql_partition.cc > @@ -7237,7 +7246,8 @@ uint fast_alter_partition_table(THD *thd, TABLE *table, > } > else if ((alter_info->partition_flags & ALTER_PARTITION_ADD) && > (part_info->part_type == RANGE_PARTITION || > - part_info->part_type == LIST_PARTITION)) > + part_info->part_type == LIST_PARTITION || > + alter_info->partition_flags & ALTER_PARTITION_AUTO_HIST)) it'd be good it adding new empty VERSIONING partitions would always go this way, auto or not. but it's a separate task. > { > /* > ADD RANGE/LIST PARTITIONS > @@ -7281,7 +7291,7 @@ uint fast_alter_partition_table(THD *thd, TABLE *table, > write_log_add_change_partition(lpt) || > ERROR_INJECT_CRASH("crash_add_partition_4") || > ERROR_INJECT_ERROR("fail_add_partition_4") || > - mysql_change_partitions(lpt) || > + mysql_change_partitions(lpt, false) || this way you skip trans_commit_stmt/trans_commit_implicit for ALTER TABLE ... ADD RANGE/LIST partition. is it always ok? A safer alternative would've been mysql_change_partitions(lpt, !(alter_info->partition_flags & ALTER_PARTITION_AUTO_HIST)) but it's more complex, so I'd prefer your variant, if we can be sure that it doesn't break anything. > ERROR_INJECT_CRASH("crash_add_partition_5") || > ERROR_INJECT_ERROR("fail_add_partition_5") || > alter_close_table(lpt) || Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

3 12

[Maria-developers] Initial prototype for MDEV-10825 Persist exact view definitions DDL
by Vicențiu Ciorbaru 17 Apr '21

17 Apr '21

Hi Sergei! This email is a followup to the brief discussion on Zulip here: https://mariadb.zulipchat.com/#narrow/stream/118759-general/topic/preserve.… You mentioned we store the view's definition (source) inside the FRM. I've used that information to extend the I_S.views table with a source column. The patch is very small, but I have 2 questions: 1. Is this how the feature should look like? I wonder if we should prepend *create view <view-name>* to the SOURCE column, to make it behave very similar to SHOW CREATE VIEW. Perhaps SOURCE as a column name is not the most well chosen name. 2. I don't know if I should use: table->field[11]->store(tables->source.str, tables->source.length, tables->view_creation_ctx->get_client_cs()); or table->field[11]->store(tables->source.str, tables->source.length, cs); when storing the source data. Here is the patch: https://github.com/MariaDB/server/compare/10.6-mdev-10825 As soon as we agree on the complete specs for the feature, I'll clean up test failures in other tests, etc. Vicențiu

2 1

Re: [Maria-developers] 7b20964dd240: MDEV-8334: Rename utf8 to utf8mb3
by Sergei Golubchik 15 Apr '21

15 Apr '21

Hi, Rucha! On Apr 15, Rucha Deodhar wrote: > revision-id: 7b20964dd240 > parent(s): e9a2c9e > author: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > timestamp: 2021-03-26 00:55:56 +0530 > message: > > MDEV-8334: Rename utf8 to utf8mb3 > > This patch is made as a part of MDEV-8334 to fix failing test in unit and > main test suite so that utf8mb3 characterset is recognized. Failing tests: > main.mysql_client_test > main.mysql_client_test_comp > unit.conc_basic-t > unit.conc_charset > unit.conc_connection > diff --git a/libmariadb/ma_charset.c b/libmariadb/ma_charset.c > index ee4b0f47..307cd522 100644 > --- a/libmariadb/ma_charset.c > +++ b/libmariadb/ma_charset.c > @@ -67,6 +67,10 @@ > #include <langinfo.h> > #endif > > +#define IS_UTF8(c)\ > +(!strcasecmp((c), "utf8") || !strcasecmp((c), "utf8mb3") ||\ > + !strcasecmp((c), "utf8mb4") || !strcasecmp((c), "utf-8")) > + > /* > +----------------------------------------------------------------------+ > | PHP Version 5 | > @@ -1269,7 +1275,7 @@ struct st_madb_os_charset MADB_OS_CHARSET[]= > {"57010", "ISCII Gujarati", NULL, NULL, MADB_CS_UNSUPPORTED}, > {"57011", "ISCII Punjabi", NULL, NULL, MADB_CS_UNSUPPORTED}, > {"65000", "utf-7 Unicode (UTF-7)", NULL, NULL, MADB_CS_UNSUPPORTED}, > - {"65001", "utf-8 Unicode (UTF-8)", "utf8", NULL, MADB_CS_EXACT}, > + {"65001", "utf-8 Unicode (UTF-8)", "utf8mb3", NULL, MADB_CS_EXACT}, No, keep this utf8, it's still a valid charset name, the server can figure it out what to map it to. > /* non Windows */ > #else > /* iconv encodings */ > @@ -1337,8 +1343,8 @@ struct st_madb_os_charset MADB_OS_CHARSET[]= > {"gb2312", "GB2312", "gb2312", "GB2312", MADB_CS_EXACT}, > {"gbk", "GBK", "gbk", "GBK", MADB_CS_EXACT}, > {"georgianps", "Georgian", "geostd8", "GEORGIAN-PS", MADB_CS_EXACT}, > - {"utf8", "UTF8", "utf8", "UTF-8", MADB_CS_EXACT}, > - {"utf-8", "UTF8", "utf8", "UTF-8", MADB_CS_EXACT}, > + {"utf8mb3", "UTF8MB3", "utf8mb3", "UTF-8", MADB_CS_EXACT}, > + {"utf-8", "UTF8MB3", "utf8mb3", "UTF-8", MADB_CS_EXACT}, same here > #endif > {NULL, NULL, NULL, NULL, 0} > }; > @@ -1361,8 +1367,8 @@ const char *madb_get_os_character_set() > return MADB_DEFAULT_CHARSET_NAME; > while (MADB_OS_CHARSET[i].identifier) > { > - if (MADB_OS_CHARSET[i].supported > MADB_CS_UNSUPPORTED && > - strcasecmp(MADB_OS_CHARSET[i].identifier, p) == 0) > + if ((MADB_OS_CHARSET[i].supported > MADB_CS_UNSUPPORTED && > + strcasecmp(MADB_OS_CHARSET[i].identifier, p) == 0) || IS_UTF8(p)) why? > return MADB_OS_CHARSET[i].charset; > i++; > } > diff --git a/unittest/libmariadb/basic-t.c b/unittest/libmariadb/basic-t.c > index c22e6c2b..e2943964 100644 > --- a/unittest/libmariadb/basic-t.c > +++ b/unittest/libmariadb/basic-t.c > @@ -310,7 +310,8 @@ static int use_utf8(MYSQL *my) > > while ((row= mysql_fetch_row(res)) != NULL) > { > - FAIL_IF(strcmp(row[0], "utf8"), "wrong character set"); > + FAIL_IF(strcmp(row[0], get_utf8_name(mysql_get_server_version(my),"utf8")), > + "wrong character set"); technically, C/C is a separate project, can run on any server with any config file. So it'd be safer to check that row[0] starts from utf8 and not assume that it depends on a server version in a specific way. > } > FAIL_IF(mysql_errno(my), mysql_error(my)); > mysql_free_result(res); > diff --git a/unittest/libmariadb/charset.c b/unittest/libmariadb/charset.c > index 898b6dad..ffa877bc 100644 > --- a/unittest/libmariadb/charset.c > +++ b/unittest/libmariadb/charset.c > @@ -71,14 +71,20 @@ int bug_8378(MYSQL *mysql) { > int test_client_character_set(MYSQL *mysql) > { > MY_CHARSET_INFO cs; > + char collation_name[19]; > char *csname= (char*) "utf8"; > char *csdefault= (char*)mysql_character_set_name(mysql); > > + strcpy(collation_name,(const char*)get_utf8_name(mysql_get_server_version(mysql), > + "utf8_general_ci")); > + This one is simpler. It only tests that mysql_set_character_set() works. Just don't use utf8, make it test on something else, e.g. on latin2. > FAIL_IF(mysql_set_character_set(mysql, csname), mysql_error(mysql)); > > mysql_get_character_set_info(mysql, &cs); > > - FAIL_IF(strcmp(cs.csname, "utf8") || strcmp(cs.name, "utf8_general_ci"), "Character set != UTF8"); > + FAIL_IF(strcmp(cs.csname, get_utf8_name(mysql_get_server_version(mysql),"utf8")) || > + strcmp(cs.name, collation_name), > + "Wrong UTF8 characterset"); > FAIL_IF(mysql_set_character_set(mysql, csdefault), mysql_error(mysql)); > > return OK; > @@ -537,6 +544,9 @@ static int test_bug30472(MYSQL *mysql) > > SKIP_MAXSCALE; > > + strcpy(collation_name,(const char*)get_utf8_name(mysql_get_server_version(mysql), > + "utf8_general_ci")); > + same here, the bug is https://bugs.mysql.com/bug.php?id=30472 "libmysql doesn't reset charset, insert_id after succ. mysql_change_user() call" so, does not need utf8 specifically. Change it to some easier to use charset. > if (mysql_get_server_version(mysql) < 50100 || !is_mariadb) > { > diag("Test requires MySQL Server version 5.1 or above"); > diff --git a/unittest/libmariadb/connection.c b/unittest/libmariadb/connection.c > index 70d347ce..eb9b39bb 100644 > --- a/unittest/libmariadb/connection.c > +++ b/unittest/libmariadb/connection.c > @@ -644,9 +644,8 @@ int test_conc26(MYSQL *unused __attribute__((unused))) > > FAIL_IF(my_test_connect(mysql, hostname, "notexistinguser", "password", schema, port, NULL, CLIENT_REMEMBER_OPTIONS), > "Error expected"); > - > - FAIL_IF(!mysql->options.charset_name || strcmp(mysql->options.charset_name, "utf8") != 0, > - "expected charsetname=utf8"); > + FAIL_IF(!mysql->options.charset_name || strcmp(mysql->options.charset_name, "utf8") != 0, > + "Wrong utf8 characterset for this version"); again, CONC-26 is "CLIENT_REMEMBER_OPTIONS flag missing" it doesn't apparently need utf8 specifically, so just use a different non-default charset there. > mysql_close(mysql); > > mysql= mysql_init(NULL); > @@ -981,7 +980,8 @@ static int test_sess_track_db(MYSQL *mysql) > printf("# SESSION_TRACK_VARIABLES: %*.*s\n", (int)len, (int)len, data); > } while (!mysql_session_track_get_next(mysql, SESSION_TRACK_SYSTEM_VARIABLES, &data, &len)); > diag("charset: %s", mysql->charset->csname); > - FAIL_IF(strcmp(mysql->charset->csname, "utf8"), "Expected charset 'utf8'"); > + FAIL_IF(strcmp(mysql->charset->csname, get_utf8_name(mysql_get_server_version(mysql),"utf8")), > + "Wrong utf8 characterset for this version"); same here > > rc= mysql_query(mysql, "SET NAMES latin1"); > check_mysql_rc(rc, mysql); > diff --git a/unittest/libmariadb/my_test.h b/unittest/libmariadb/my_test.h > index c30d1b6d..a040c3d9 100644 > --- a/unittest/libmariadb/my_test.h > +++ b/unittest/libmariadb/my_test.h > @@ -701,3 +701,23 @@ void run_tests(struct my_tests_st *test) { > } > } > > +static inline const char* get_utf8_name(unsigned long server_version, > + const char* name) > +{ > + const char *csname= server_version >= 100600 ? "utf8mb3" : "utf8"; > + char *corrected_name= malloc(19*sizeof(char)); > + corrected_name[18]='\0'; > + > + if (!strchr(name, '_')) > + { > + strcpy(corrected_name,csname); > + corrected_name[strlen(csname)]='\0'; > + } > + else > + { > + strcpy(corrected_name,csname); > + strcat(corrected_name,"_general_ci"); > + corrected_name[strlen(csname)+11]= '\0'; > + } > + return (const char*)corrected_name; > +} shouldn't be needed Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 21a84e23cf3: MDEV-16437: merge 5.7 P_S replication instrumentation and tables
by Sergei Golubchik 09 Apr '21

09 Apr '21

Hi, Sujatha! Note, it's a review of the combined `git diff e5fc78 21a84e` On Mar 31, Sujatha wrote: > revision-id: 21a84e23cf3 (mariadb-10.5.2-307-g21a84e23cf3) > parent(s): 4e6de585ff7 > author: Sujatha <sujatha.sivakumar(a)mariadb.com> > committer: Sujatha <sujatha.sivakumar(a)mariadb.com> > timestamp: 2020-11-30 13:22:32 +0530 > message: > > MDEV-16437: merge 5.7 P_S replication instrumentation and tables That was pretty good, thanks! Just few small comments below, no big changes will be needed. > diff --git a/mysql-test/suite/multi_source/simple.result b/mysql-test/suite/multi_source/simple.result > index a66d49e88cb..b57e146feb5 100644 > --- a/mysql-test/suite/multi_source/simple.result > +++ b/mysql-test/suite/multi_source/simple.result > @@ -21,7 +21,21 @@ show all slaves status; > Connection_name Slave_SQL_State Slave_IO_State Master_Host Master_User Master_Port Connect_Retry Master_Log_File Read_Master_Log_Pos Relay_Log_File Relay_Log_Pos Relay_Master_Log_File Slave_IO_Running Slave_SQL_Running Replicate_Do_DB Replicate_Ignore_DB Replicate_Do_Table Replicate_Ignore_Table Replicate_Wild_Do_Table Replicate_Wild_Ignore_Table Last_Errno Last_Error Skip_Counter Exec_Master_Log_Pos Relay_Log_Space Until_Condition Until_Log_File Until_Log_Pos Master_SSL_Allowed Master_SSL_CA_File Master_SSL_CA_Path Master_SSL_Cert Master_SSL_Cipher Master_SSL_Key Seconds_Behind_Master Master_SSL_Verify_Server_Cert Last_IO_Errno Last_IO_Error Last_SQL_Errno Last_SQL_Error Replicate_Ignore_Server_Ids Master_Server_Id Master_SSL_Crl Master_SSL_Crlpath Using_Gtid Gtid_IO_Pos Replicate_Do_Domain_Ids Replicate_Ignore_Domain_Ids Parallel_Mode SQL_Delay SQL_Remaining_Delay Slave_SQL_Running_State Slave_DDL_Groups Slave_Non_Transactional_Groups Slave_Transactional_Groups Retried_transactions Max_relay_log_size Executed_log_entries Slave_received_heartbeats Slave_heartbeat_period Gtid_Slave_Pos > slave1 Slave has read all relay log; waiting for more updates Waiting for master to send event 127.0.0.1 root MYPORT_1 60 master-bin.000001 <read_master_log_pos> mysqld-relay-bin-slave1.000002 <relay_log_pos> master-bin.000001 Yes Yes 0 0 <read_master_log_pos> <relay_log_space1> None 0 No 0 No 0 0 1 No optimistic 0 NULL Slave has read all relay log; waiting for more updates 0 0 0 0 1073741824 7 0 60.000 > slave2 Slave has read all relay log; waiting for more updates Waiting for master to send event 127.0.0.1 root MYPORT_2 60 master-bin.000001 <read_master_log_pos> mysqld-relay-bin-slave2.000002 <relay_log_pos> master-bin.000001 Yes Yes 0 0 <read_master_log_pos> <relay_log_space1> None 0 No 0 No 0 0 2 No optimistic 0 NULL Slave has read all relay log; waiting for more updates 0 0 0 0 1073741824 7 0 60.000 > +# > +# MDEV:16437: merge 5.7 P_S replication instrumentation and tables > +# > +select * from performance_schema.replication_connection_configuration; > +CONNECTION_NAME HOST PORT USER USING_GTID SSL_ALLOWED SSL_CA_FILE SSL_CA_PATH SSL_CERTIFICATE SSL_CIPHER SSL_KEY SSL_VERIFY_SERVER_CERTIFICATE SSL_CRL_FILE SSL_CRL_PATH CONNECTION_RETRY_INTERVAL CONNECTION_RETRY_COUNT HEARTBEAT_INTERVAL IGNORE_SERVER_IDS REPL_DO_DOMAIN_IDS REPL_IGNORE_DOMAIN_IDS > +slave2 # # root NO NO NO 60 86400 60.000 > +slave1 # # root NO NO NO 60 86400 60.000 Isn't the host always 127.0.0.1 ? Why to mask it? Also, may be it'd be easier to read the result, if you print it vertically? It looks that CONNECTION_RETRY_COUNT is 86400. And 86400 is clearly a timeout, not a retry count. > start all slaves; > +# > +# MDEV:16437: merge 5.7 P_S replication instrumentation and tables > +# > +select * from performance_schema.replication_applier_status_by_coordinator; > +CONNECTION_NAME THREAD_ID SERVICE_STATE LAST_SEEN_TRANSACTION LAST_ERROR_NUMBER LAST_ERROR_MESSAGE LAST_ERROR_TIMESTAMP LAST_TRANS_RETRY_COUNT > +slave2 # ON 0 0000-00-00 00:00:00 0 > +slave1 # ON 0 0000-00-00 00:00:00 0 > stop slave 'slave1'; > show slave 'slave1' status; > Slave_IO_State > diff --git a/mysql-test/suite/rpl/include/rpl_deadlock.test b/mysql-test/suite/rpl/include/rpl_deadlock.test > index e9191d5fcd8..bccbe044a36 100644 > --- a/mysql-test/suite/rpl/include/rpl_deadlock.test > +++ b/mysql-test/suite/rpl/include/rpl_deadlock.test > @@ -59,6 +59,16 @@ let $status_var_comparsion= >; > connection slave; > SELECT COUNT(*) FROM t2; > COMMIT; > + > +--echo > +--echo # Test that the performance schema coulumn shows > 0 values. > +--echo > + > +--let $assert_text= current number of retries should be more than the value saved before deadlock. > +--let $assert_cond= [SELECT COUNT_TRANSACTIONS_RETRIES FROM performance_schema.replication_applier_status, COUNT_TRANSACTIONS_RETRIES, 1] > "$slave_retried_transactions" > +--source include/assert.inc what's wrong with simple SELECT COUNT_TRANSACTIONS_RETRIES > $slave_retried_transactions FROM performance_schema.replication_applier_status ? > + > +source include/check_slave_is_running.inc; > sync_with_master; > > # Check the data > diff --git a/mysql-test/suite/rpl/r/rpl_perfschema_connect_config.result b/mysql-test/suite/rpl/r/rpl_perfschema_connect_config.result > new file mode 100644 > index 00000000000..4ace84ffac4 > --- /dev/null > +++ b/mysql-test/suite/rpl/r/rpl_perfschema_connect_config.result > @@ -0,0 +1,124 @@ > +include/master-slave.inc > +[connection master] > +# Asserted this: On master, the table should return an empty set. > +connection slave; > + > +# Verify that SELECT works for every field and produces an output > +# similar to the corresponding field in SHOW SLAVE STATUS(SSS). > + > +include/assert.inc [Value returned by SSS and PS table for Host should be same.] > +include/assert.inc [Value returned by SSS and PS table for Port should be same.] > +include/assert.inc [Value returned by SSS and PS table for User should be same.] > +include/assert.inc [Value returned by SSS and PS table for Using_Gtid should be same.] > +include/assert.inc [Value returned by SSS and PS table for SSL_Allowed should be same.] > +include/assert.inc [Value returned by SSS and PS table for SSL_CA_File should be same.] > +include/assert.inc [Value returned by SSS and PS table for SSL_CA_Path should be same.] > +include/assert.inc [Value returned by SSS and PS table for SSL_Certificate should be same.] > +include/assert.inc [Value returned by SSS and PS table for SSL_Cipher should be same.] > +include/assert.inc [Value returned by SSS and PS table for SSL_Key should be same.] > +include/assert.inc [Value returned by SSS and PS table for SSL_Verify_Server_Certificate should be same.] > +include/assert.inc [Value returned by SSS and PS table for SSL_Crl_File should be same.] > +include/assert.inc [Value returned by SSS and PS table for SSL_Crl_Path should be same.] > +include/assert.inc [Value returned by SSS and PS table for Connection_Retry_Interval should be same.] > +include/assert.inc [Value returned by PS table for Connection_Retry_Count should be 10.] this is just unreadable > + > +# Heartbeat_Interval is part of I_S and P_S. We will compare the > +# two to make sure both match. ... > diff --git a/mysql-test/suite/rpl/t/rpl_perfschema_applier_config.test b/mysql-test/suite/rpl/t/rpl_perfschema_applier_config.test > new file mode 100644 > index 00000000000..132d9912222 > --- /dev/null > +++ b/mysql-test/suite/rpl/t/rpl_perfschema_applier_config.test > @@ -0,0 +1,96 @@ > +# ==== Purpose ==== > +# > +# This test script serves as the functionality testing for the table > +# performance_schema.replication_applier_configuration. Test for ddl and dml > +# operations is a part of the perfschema suite. The ddl/dml tests are named: > +# 1) ddl_replication_applier_configuration.test and > +# 2) dml_replication_applier_configuration.test. > +# > +# The follwing scenarios are tested in this script: > +# > +# - Verify that output is same as SSS on a fresh slave. > +# - Verify that the value of this field is correct after STOP SLAVE. > +# - Verify that, when desired delay is set, the value is shown corectly. > +# - Verify that the value is preserved after STOP SLAVE. > +# - Verify that, when desired delay is reset, the value is shown corectly. > +# > +# ==== Related Worklog ==== > +# > +# MDEV-16437: merge 5.7 P_S replication instrumentation and tables > +# > + > +source include/master-slave.inc; > +source include/have_binlog_format_mixed.inc; master-slave should be included last > + > +let $assert_text= On master, the table should return an empty set.; > +let $assert_cond= count(*) = 0 from performance_schema.replication_applier_configuration; > +source include/assert.inc; again, what's wrong with just select * from performance_schema.replication_applier_configuration; or, may be, if you want to be very explicit select count(*) as 'must be 0' from performance_schema.replication_applier_configuration; > + > +--connection slave > + > +--echo > +--echo # Verify that SELECT works and produces an output similar to > +--echo # the corresponding field in SHOW SLAVE STATUS(SSS) in all scenarios. > +--echo > + > +--echo > +--echo # Verify that output is same as SSS on a fresh slave. > +--echo > + > +let $sss_value= query_get_value(SHOW SLAVE STATUS, SQL_Delay, 1); > +let $ps_value= query_get_value(select Desired_Delay from performance_schema.replication_applier_configuration, Desired_Delay, 1); > +let $assert_text= Value returned by SSS and PS table for Desired_Delay should be same.; > +let $assert_cond= "$sss_value" = "$ps_value"; > +source include/assert.inc; I'll stop commenting on every assert.inc, but please, please, stop overusing them. > + > +--echo > +--echo # Verify that the value of this field is correct after STOP SLAVE. > +--echo > + > +source include/stop_slave.inc; > + > +let $sss_value= query_get_value(SHOW SLAVE STATUS, SQL_Delay, 1); > +let $ps_value= query_get_value(select Desired_Delay from performance_schema.replication_applier_configuration, Desired_Delay, 1); > +let $assert_text= Value returned by SSS and PS table for Desired_Delay should be same.; > +let $assert_cond= "$sss_value" = "$ps_value"; > +source include/assert.inc; > + > +--echo > +--echo # Verify that, when desired delay is set, the value is shown corectly. > +--echo > + > +eval change master to master_delay= 2; > +source include/start_slave.inc; > + > +let $sss_value= query_get_value(SHOW SLAVE STATUS, SQL_Delay, 1); > +let $ps_value= query_get_value(select Desired_Delay from performance_schema.replication_applier_configuration, Desired_Delay, 1); > +let $assert_text= Value returned by SSS and PS table for Desired_Delay should be same.; > +let $assert_cond= "$sss_value" = "$ps_value"; > +source include/assert.inc; > + > +--echo > +--echo # Verify that the value is preserved after STOP SLAVE. > +--echo > + > +source include/stop_slave.inc; > + > +let $ss_value= query_get_value(SHOW SLAVE STATUS, SQL_Delay, 1); > +let $ps_value= query_get_value(select Desired_Delay from performance_schema.replication_applier_configuration, Desired_Delay, 1); > +let $assert_text= Value returned by SSS and PS table for Desired_Delay should be same.; > +let $assert_cond= "$sss_value" = "$ps_value"; > +source include/assert.inc; > + > +--echo > +--echo # Verify that, when desired delay is reset, the value is shown corectly. > +--echo > + > +eval change master to master_delay= 0; > +source include/start_slave.inc; > + > +let $sss_value= query_get_value(SHOW SLAVE STATUS, SQL_Delay, 1); > +let $ps_value= query_get_value(select Desired_Delay from performance_schema.replication_applier_configuration, Desired_Delay, 1); > +let $assert_text= Value returned by SSS and PS table for Desired_Delay should be same.; > +let $assert_cond= "$sss_value" = "$ps_value"; > +source include/assert.inc; > + > +source include/rpl_end.inc; > diff --git a/mysql-test/suite/rpl/t/rpl_perfschema_applier_status.test b/mysql-test/suite/rpl/t/rpl_perfschema_applier_status.test > new file mode 100644 > index 00000000000..52ee14cef2a > --- /dev/null > +++ b/mysql-test/suite/rpl/t/rpl_perfschema_applier_status.test > @@ -0,0 +1,72 @@ > +# ==== Purpose ==== > +# > +# This test script serves as the functionality testing for the table > +# performance_schema.replication_applier_status. Test for ddl and dml > +# operations is a part of the perfschema suite. The ddl/dml tests are named: > +# 1) ddl_replication_applier_status.test and > +# 2) dml_replication_applier_status.test. > +# > +# The follwing scenarios are tested in this script: > +# > +# - Verify that output is same as SSS on a fresh slave. > +# - Verify that the value of this field is correct after STOP SLAVE. > +# - Remaining delay is not tested. > +# - Count_trnsaction is partially tested here making sure it can be queried. > +# More testing in rpl/include/rpl_deadlock.test > +# > +# ==== Related Worklog ==== > +# > +# MDEV-16437: merge 5.7 P_S replication instrumentation and tables > +# > + > +source include/master-slave.inc; > +source include/have_binlog_format_mixed.inc; master-slave must always be last (I'll stop commenting on that too) > + > +let $assert_text= On master, the table should return an empty set.; > +let $assert_cond= count(*) = 0 from performance_schema.replication_applier_status; > +source include/assert.inc; > + > +--connection slave > + > +--echo > +--echo # Verify that SELECT works and produces an output similar to > +--echo # the corresponding field in SHOW SLAVE STATUS(SSS) in all scenarios. > +--echo > + > +--echo > +--echo # Verify that output is same as SSS on a fresh slave. > +--echo > + > +let $sss_value= query_get_value(SHOW SLAVE STATUS, Slave_SQL_Running, 1); > +let $ps_value= query_get_value(select Service_State from performance_schema.replication_applier_status, Service_State, 1); > +let $assert_text= SSS shows Slave_SQL_Running as "Yes". So, Service_State from this PS table should be "ON".; > +let $assert_cond= "$sss_value" = "Yes" AND "$ps_value"= "ON"; > +source include/assert.inc; > + > +let $ss_value= query_get_value(SHOW STATUS LIKE 'Slave_retried_transactions', Value, 1); > +let $ps_value= query_get_value(select count_transactions_retries from performance_schema.replication_applier_status, count_transactions_retries, 1); > +let $assert_text= COUNT_TRANSACTION_RETRIES should be equal to Slave_retried_transactions.; > +let $assert_cond= "$ps_value"= "$ss_value"; > +source include/assert.inc; > + > +--echo > +--echo # Verify that the fields show the correct values after STOP SLAVE. > +--echo > + > +source include/stop_slave.inc; > + > +let $sss_value= query_get_value(SHOW SLAVE STATUS, Slave_SQL_Running, 1); > +let $ps_value= query_get_value(select Service_State from performance_schema.replication_applier_status, Service_State, 1); > +let $assert_text= SSS shows Slave_SQL_Running as "No". So, Service_State from this PS table should be "OFF".; > +let $assert_cond= "$sss_value" = "No" AND "$ps_value"= "OFF"; > +source include/assert.inc; > + > +let $ss_value= query_get_value(SHOW STATUS LIKE 'Slave_retried_transactions', Value, 1); > +let $ps_value= query_get_value(select count_transactions_retries from performance_schema.replication_applier_status, count_transactions_retries, 1); > +let $assert_text= COUNT_TRANSACTION_RETRIES should be equal to Slave_retried_transactions.; > +let $assert_cond= "$ps_value"= "$ss_value"; > +source include/assert.inc; > + > +source include/start_slave.inc; > +source include/rpl_end.inc; > + > diff --git a/sql/rpl_mi.h b/sql/rpl_mi.h > index 4d47689ac18..946d138d618 100644 > --- a/sql/rpl_mi.h > +++ b/sql/rpl_mi.h > @@ -50,13 +57,6 @@ class Domain_id_filter > */ > DYNAMIC_ARRAY m_domain_ids[2]; > > -public: > - /* domain id list types */ > - enum enum_list_type { > - DO_DOMAIN_IDS= 0, > - IGNORE_DOMAIN_IDS > - }; > - Why did you move it up? Does it make any difference? > Domain_id_filter(); > > ~Domain_id_filter(); > diff --git a/storage/perfschema/table_replication_applier_status_by_coordinator.cc b/storage/perfschema/table_replication_applier_status_by_coordinator.cc > index beb8620b240..30cf56ce0c2 100644 > --- a/storage/perfschema/table_replication_applier_status_by_coordinator.cc > +++ b/storage/perfschema/table_replication_applier_status_by_coordinator.cc > @@ -55,12 +55,14 @@ table_replication_applier_status_by_coordinator::m_share= > sizeof(pos_t), /* ref length */ > &m_table_lock, > { C_STRING_WITH_LEN("CREATE TABLE replication_applier_status_by_coordinator(" > - "CHANNEL_NAME CHAR(64) collate utf8_general_ci not null," > + "CONNECTION_NAME VARCHAR(256) collate utf8_general_ci not null," please, don't rename existing columns, let's keep calling it CHANNEL_NAME everywhere. > "THREAD_ID BIGINT UNSIGNED," > "SERVICE_STATE ENUM('ON','OFF') not null," > + "LAST_SEEN_TRANSACTION CHAR(57) not null," I think it's better to put new columns at the end. > "LAST_ERROR_NUMBER INTEGER not null," > "LAST_ERROR_MESSAGE VARCHAR(1024) not null," > - "LAST_ERROR_TIMESTAMP TIMESTAMP(0) not null)") }, > + "LAST_ERROR_TIMESTAMP TIMESTAMP(0) not null," > + "LAST_TRANS_RETRY_COUNT INTEGER not null)") }, > false /* perpetual */ > }; > > @@ -104,15 +106,7 @@ int table_replication_applier_status_by_coordinator::rnd_next(void) > { > mi= (Master_info *)my_hash_element(&master_info_index->master_info_hash, m_pos.m_index); > > - /* > - Construct and display SQL Thread's (Coordinator) information in > - 'replication_applier_status_by_coordinator' table only in the case of > - multi threaded slave mode. Code should do nothing in the case of single > - threaded slave mode. In case of single threaded slave mode SQL Thread's > - status will be reported as part of > - 'replication_applier_status_by_worker' table. > - */ is this no longer true? > - if (mi && mi->host[0] && /*mi->rli.get_worker_count() > */ 0) > + if (mi && mi->host[0]) > { > make_row(mi); > m_next_pos.set_after(&m_pos); > @@ -147,13 +141,20 @@ int table_replication_applier_status_by_coordinator::rnd_pos(const void *pos) > void table_replication_applier_status_by_coordinator::make_row(Master_info *mi) > { > m_row_exists= false; > + rpl_gtid gtid; > + char buf[10+1+10+1+20+1]; > + String str(buf, sizeof(buf), system_charset_info); There's a special template for it: StringBuffer<10+1+10+1+20+1> str; > + bool first= true; > + > + str.length(0); not needed if you use StringBuffer<> > > DBUG_ASSERT(mi != NULL); > > mysql_mutex_lock(&mi->rli.data_lock); > > - m_row.channel_name_length= static_cast<uint>(mi->connection_name.length); > - memcpy(m_row.channel_name, mi->connection_name.str, m_row.channel_name_length); > + gtid= mi->rli.last_seen_gtid; > + m_row.connection_name_length= static_cast<uint>(mi->connection_name.length); > + memcpy(m_row.connection_name, mi->connection_name.str, m_row.connection_name_length); > > if (mi->rli.slave_running) > { > @@ -175,6 +176,18 @@ void table_replication_applier_status_by_coordinator::make_row(Master_info *mi) > else > m_row.service_state= PS_RPL_NO; > > + if ((gtid.seq_no > 0 && > + !rpl_slave_state_tostring_helper(&str, &gtid, &first))) > + { > + strmake(m_row.last_seen_transaction,str.ptr(), str.length()); > + m_row.last_seen_transaction_length= str.length(); > + } > + else > + { > + m_row.last_seen_transaction_length= 0; > + memcpy(m_row.last_seen_transaction, "", 1); a strange way to write m_row.last_seen_transaction[0]= 0, but ok :) > + } > + > mysql_mutex_lock(&mi->rli.err_lock); > > m_row.last_error_number= (long int) mi->rli.last_error().number; Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 4

[Maria-developers] JSON_TABLE: Why one can't just use Name_resolution_context
by Sergey Petrunia 09 Apr '21

09 Apr '21

Hello, There was this question raised: Why can't JSON_TABLE code just use the current Name_resolution_context objects? == Context == Name_resolution_context object has these two members that specify a subset of tables to use: TABLE_LIST *first_name_resolution_table; TABLE_LIST *last_name_resolution_table; JSON_TABLE code also introduces this one: table_map ignored_tables; With the meaning that name resolution process will not search for the field of interest in the tables listed in the bitmap. == Rationale == For a JSON_TABLE invocation select ... from json_table(ARG, ...) as JT .... the ARG may refer to any table that precedes JT in the from clause. Let's call the set of the tables ALLOWED_DEPS. Can one take a JSON_TABLE invocation and pick first_name_resolution_table=FIRST, last_name_resolution_table=LAST such that ALLOWED_DEPS = { FIRST, SECOND= FIRST->next_name_resolution_table, THIRD= SECOND->next_name_resolution_table, ... LAST } ? I claim that there are cases where this is not possible. == An example == select * from t1, (t2 natural join (json_table('{}', '$' COLUMNS (d for ordinality)) as JT natural join t3 ) ); Here, JT's ALLOWED_DEPS={t1, t2}. Looking at the data structures (each Table name denotes a TABLE_LIST object referring to the table) select_lex.name_resolution_context.first_name_resolution_table= t1 t1->next_name_resolution_table= $join_nest1 $join_nest1->next_name_resolution_table= NULL $join_nest1->nested_join.join_list= { $join_nest2, t2 } t2->next_name_resolution_table= $join_nest2 $join_nest2->next_name_resolution_table= NULL $join_nest2->nested_join.join_list= {t3, JT} JT->next_name_resolution_table= t3 t3->next_name_resolution_table= NULL Apparently there is no single chain that includes t1 and t2. == Possible objections == "But nested (outer) joins do it" Nested outer joins never need to pick a subset that includes only - one of the tables in a NATURAL JOIN - one of the tables outside that NATURAL JOIN. so there's no comparison. BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://petrunia.net

1 0

Re: [Maria-developers] ee538938345: MDEV-21117: refine the server binlog-based recovery for semisync
by Sergei Golubchik 08 Apr '21

08 Apr '21

Hi, Andrei! On Apr 07, Andrei Elkin wrote: > Sergei, salve. > > I should've asked for clarification to > > >>> + } > >>> + } else > >> > >> add into the else branch something like > >> > >> /* this branch is only for the legacy TC_LOG_MMAP */ > >> compile_time_assert(sizeof(TC_LOG_MMAP) > 1 ); > > > > [x] > > Actually the branch, which is > > } else > if (IF_WSREP((wsrep_emulate_bin_log && > wsrep_is_wsrep_xid(info->list + i) && > x <= wsrep_limit), false) || > (info->commit_list ? > my_hash_search(info->commit_list, (uchar *)&x, sizeof(x)) != 0 : > tc_heuristic_recover == TC_HEURISTIC_RECOVER_COMMIT)) > { > #ifndef DBUG_OFF > int rc= > #endif > hton->commit_by_xid(hton, info->list+i); > > deals with tc_heuristic_recover. > > I am not sure what you meant. I wanted to make sure all TC_LOG_MMAP code is removed when TC_LOG_MMAP itself goes away. But apparenlty I overlooked the heuristic recover path. So this code deals with both heuristic recover and TC_LOG_MMAP. Okay than, forget that comment. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

[Maria-developers] JSON_TABLE: on the question of temp table "cache" in MySQL-8
by Sergey Petrunia 07 Apr '21

07 Apr '21

1 0

Re: [Maria-developers] d221068ae8b: Change CHARSET_INFO character set and collaction names to LEX_CSTRING
by Sergei Golubchik 06 Apr '21

06 Apr '21

Hi, Michael! On Apr 01, Michael Widenius wrote: > revision-id: d221068ae8b (mariadb-10.5.2-540-gd221068ae8b) > parent(s): 896c3b0a00a > author: Michael Widenius <michael.widenius(a)gmail.com> > committer: Michael Widenius <michael.widenius(a)gmail.com> > timestamp: 2021-03-24 19:13:42 +0200 > message: > > Change CHARSET_INFO character set and collaction names to LEX_CSTRING > > This change removed 68 strlen() calls from the code. > > The following renames was to ensure we don't use the old names > when merging code from earlier releases, as using the new variables > for print function could result in crashes: > - charset->csname renamed to charset->cs_name > - charset->name renamed to charset->col_name Two comments: 1. please, use coll_name. in our codebase "cs" conventionally means "charset", "col" is usually used for "column", "coll" - for "collation" 2. You changed the client api, as can be seen from > diff --git a/client/mysql.cc b/client/mysql.cc > index e6c7d0f091c..8c64d01539d 100644 > --- a/client/mysql.cc > +++ b/client/mysql.cc > @@ -4719,9 +4719,8 @@ sql_real_connect(char *host,char *database,char *user,char *password, > return -1; // Retryable > } > > - charset_info= get_charset_by_name(mysql.charset->name, MYF(0)); > - > - > + charset_info= get_charset_by_name(IF_EMBEDDED(mysql.charset->col_name.str, > + mysql.charset->name), MYF(0)); > connected=1; > #ifndef EMBEDDED_LIBRARY > mysql_options(&mysql, MYSQL_OPT_RECONNECT, &debug_info_flag); Luckily, C/C is a separate project now, so most clients won't be affected. But anything linking with embedded might be. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] e3971006cca: Give a readable error in mtr if resolve_at_variable fails
by Sergei Golubchik 06 Apr '21

06 Apr '21

Hi, Monty! On Apr 05, Michael Widenius wrote: > revision-id: e3971006cca (mariadb-10.5.2-547-ge3971006cca) > parent(s): ae009435e72 > author: Michael Widenius <michael.widenius(a)gmail.com> > committer: Michael Widenius <michael.widenius(a)gmail.com> > timestamp: 2021-03-24 19:25:09 +0200 > message: > > Give a readable error in mtr if resolve_at_variable fails This change is wrong. Clearly, the || '' part was intentional. Check the history, it was added in the commit commit 897b51db434 Author: Sergei Golubchik <serg(a)mariadb.com> Date: Thu Sep 10 12:12:47 2020 +0200 make S3 tests to run when S3 is statically linked * use the environment variable HA_S3_SO, not a literal ha_s3 in cnf files * make ConfigFactory to support empty option values * update no_s3.result after MDEV-11412 so, apparently, the idea was that you should be able to write plugin-load-add=$HA_S3_SO and it will *not* fail when $HA_S3_SO is not defined, because S3 engine is statically linked in. > diff --git a/mysql-test/lib/My/ConfigFactory.pm b/mysql-test/lib/My/ConfigFactory.pm > index 3249a06256c..5a5b691998c 100644 > --- a/mysql-test/lib/My/ConfigFactory.pm > +++ b/mysql-test/lib/My/ConfigFactory.pm > @@ -347,8 +347,15 @@ sub resolve_at_variable { > or croak "There is no group named '$group_name' that ", > "can be used to resolve '$option_name' for test '$self->{testname}'"; > > - my $value= $from_group->value($option_name) || ''; > - $res .= $before.$value; > + my $value= $from_group->value($option_name); > + if (!defined($value)) > + { > + ::mtr_verbose("group: $group_name option_name: $option_name is undefined"); > + } > + else > + { > + $res .= $before.$value; > + } > } > $res .= $after; > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] 405a89353b2: Don't reset StringBuffers in loops when not needed
by Sergei Golubchik 06 Apr '21

06 Apr '21

Hi, Monty! On Apr 02, Michael Widenius wrote: > revision-id: 405a89353b2 (mariadb-10.5.2-542-g405a89353b2) > parent(s): 2dd8d472fcc > author: Michael Widenius <michael.widenius(a)gmail.com> > committer: Michael Widenius <michael.widenius(a)gmail.com> > timestamp: 2021-03-24 19:25:08 +0200 > message: > > Don't reset StringBuffers in loops when not needed > > - Moved out creating StringBuffers in loops and instead create them > outside and just reset the buffer if it was not allocated (to avoid > a possible malloc/free for every entry) > > diff --git a/sql/sql_string.h b/sql/sql_string.h > index 29d01779b71..8df50ac5e66 100644 > --- a/sql/sql_string.h > +++ b/sql/sql_string.h > @@ -1045,6 +1045,12 @@ class StringBuffer : public String > { > length(0); > } > + void set_buffer_if_not_allocated(CHARSET_INFO *cs) > + { > + if (is_alloced()) > + Static_binary_string::set(buff, buff_sz); it seems to be doing exactly the opposite of what the name suggests. It sets the buffer *if allocated*. Also, it looks logical to do length(0) here, so that this method would reset the StringBuffer to its initial state. > + set_charset(cs); > + } > }; Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

[Maria-developers] MDEV-9504
by Jason Alpers 02 Apr '21

02 Apr '21

Hello, We're hitting the problem in MDEV-9504 ( https://jira.mariadb.org/browse/MDEV-9504) I see from the commit at https://github.com/MariaDB/server/commit/07b8aefe90ca830d2de068f2966cd2288b… that this change was backed out due to test failures. I ran through all the mysql-tests with and without the change and haven't been able to pinpoint a difference though. The reverting commit indicates the correct value for select_limit is only computed when the query plan changes. Any chance someone could provide more detail on what needs to be changed to avoid regression with this fix? If there are sample queries that regress with only the original MDEV-9504 changes, that would be helpful. Thanks in advance! Jase

1 0

Re: [Maria-developers] GSOC21: MDEV-16375 & MDEV-23143
by Hollow Man 02 Apr '21

02 Apr '21

Hi community! I've had my proposal shared with https://drive.google.com/file/d/1sv0qbqt9W-ob3GqxygWwRGurpRS1lCiv/view , hope to get some feedback from the community.   Songlin   ------------------ Original ------------------ From:  "Hollow Man"<hollowman(a)hollowman.ml>; Date:  Thu, Mar 11, 2021 00:17 AM To:  "maria-developers"<maria-developers(a)lists.launchpad.net>; Subject:  GSOC21: MDEV-16375 & MDEV-23143   Hi MariaDB community!    Glad to be here! My github account is @HollowMan6. Though I'm new to MariaDB community, I'm interested in MDEV-16375 & MDEV-23143: Function to normalize a json value & missing a JSON_EQUALS function for this year's GSOC project. Here are my first thoughts on these issues:    I have checked part of the codebase and I think the two issues can be merged into one. First we can create a function named JSON_NORMALIZE to normalize the json, which automatically parses the inputed json document, recursively sorts the keys (for objects) / sorts the numbers (for arrays), removes the spaces, and then return the json document string.    Then we create a function named JSON_EQUALS, which can be used to compare 2 json documents for equality realized by first seperately normalize the two json documents using JSON_NORMALIZE, then the 2 can be compared exactly as binary strings.    I have taken some inspirations from the Item_func_json_keys and json_scan_start for parsing json documents, and I think it's possible to sort the keys using std::map in STL for objects.    That's all for my ideas so far. Please correct me if I made some mistakes, and I'm going to work on my ideas later. Cheers! Hollow Man

3 4

Re: [Maria-developers] JSON_TABLE: On name resolution question
by Sergey Petrunia 02 Apr '21

02 Apr '21

On Thu, Apr 01, 2021 at 10:14:04PM -0700, Igor Babaev wrote: > Sergey, > > Can you explain this: > > MariaDB [test]> explain format=json select * from t1 as t2 where item_name > in (select item_name from t2 as s, json_table(t2.item_props,'$' columns( > color varchar(100) path '$.color')) as t) and t2.item_name='Jeans'; > +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ > | EXPLAIN > > > > > > > | > +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ > | { > "query_block": { > "select_id": 1, > "table": { > "table_name": "t2", > "access_type": "ALL", > "rows": 2, > "filtered": 100, > "attached_condition": "t2.item_name = 'Jeans'" > }, > "table": { > "table_name": "s", > "access_type": "ALL", > "rows": 2, > "filtered": 100, > "attached_condition": "s.item_name = 'Jeans'" > }, > "table": { > "table_name": "t", > "access_type": "ALL", > "rows": 40, > "filtered": 100, > "table_function": "json_table", > "first_match": "t2" > } > } > } | > +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ > How do get get s.item_name = 'Jeans' ? > Check out the optimizer trace: First, the subquery is converted into a semi-join: "transformation": { "select_id": 2, "from": "IN (SELECT)", "to": "semijoin", "converted_to_semi_join": true } then, there's a multiple equality: "transformation": "equality_propagation", "resulting_condition": "1 and multiple equal('Jeans', t2.item_name, s.item_name)" and then, after substituion: "attaching_conditions_to_tables": { "original_condition": "t2.item_name = 'Jeans' and s.item_name = 'Jeans'", BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://petrunia.net

1 0

Re: [Maria-developers] 9abbb589e6d: MDEV-20220: Merge 5.7 P_S replication table 'replication_applier_status_by_worker
by Sergei Golubchik 01 Apr '21

01 Apr '21

Hi, Sujatha! thanks! ok to push. one question below On Apr 01, Sujatha wrote: > revision-id: 9abbb589e6d (mariadb-10.5.2-531-g9abbb589e6d) > parent(s): 04c7d5f1518 > author: Sujatha <sujatha.sivakumar(a)mariadb.com> > committer: Sujatha <sujatha.sivakumar(a)mariadb.com> > timestamp: 2021-03-29 22:57:26 +0530 > message: > > MDEV-20220: Merge 5.7 P_S replication table 'replication_applier_status_by_worker > > Step 3: > ====== > > Preserve worker pool information on either STOP SLAVE/Error. In case STOP > SLAVE is executed worker threads will be gone, hence worker threads will be > unavailable. Querying the table at this stage will give empty rows. To > address this case when worker threads are about to stop, due to an error or > forced stop, create a backup pool and preserve the data which is relevant to > populate performance schema table. Clear the backup pool upon slave start. > > > diff --git a/mysql-test/suite/perfschema/r/dml_handler.result b/mysql-test/suite/perfschema/r/dml_handler.result > index ce4ae6bf9a3..61bbba3189e 100644 > --- a/mysql-test/suite/perfschema/r/dml_handler.result > +++ b/mysql-test/suite/perfschema/r/dml_handler.result > @@ -12,78 +12,81 @@ Warning 1287 '<select expression> INTO <destination>;' is deprecated and will be > # For each table in the performance schema, attempt HANDLER...OPEN, > # which should fail with an error 1031, ER_ILLEGAL_HA. > # ... > ERROR HY000: Storage engine PERFORMANCE_SCHEMA of the table `performance_schema`.`replication_connection_configuration` doesn't have this option > +SELECT TABLE_NAME INTO @table_name FROM table_list WHERE id=57; > +HANDLER performance_schema.replication_applier_status_by_worker OPEN; > +ERROR HY000: Storage engine PERFORMANCE_SCHEMA of the table `performance_schema`.`replication_applier_status_by_worker` doesn't have this option > SELECT TABLE_NAME INTO @table_name FROM table_list WHERE id=56; > diff --git a/mysql-test/suite/perfschema/r/information_schema.result b/mysql-test/suite/perfschema/r/information_schema.result > index 3f4a8cf6336..c0aff9bb956 100644 > --- a/mysql-test/suite/perfschema/r/information_schema.result > +++ b/mysql-test/suite/perfschema/r/information_schema.result > @@ -59,6 +59,7 @@ performance_schema prepared_statements_instances def > performance_schema replication_applier_configuration def > performance_schema replication_applier_status def > performance_schema replication_applier_status_by_coordinator def > +performance_schema replication_applier_status_by_worker def > performance_schema replication_connection_configuration def > performance_schema rwlock_instances def > performance_schema session_account_connect_attrs def etc, etc. many tests showing that this commit adds a new performance_schema.replication_applier_status_by_worker table. but looking at the commit, I don't think it does. incorrectly split commits, perhaps? Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 04c7d5f1518: MDEV-20220: Merge 5.7 P_S replication table 'replication_applier_status_by_worker
by Sergei Golubchik 01 Apr '21

01 Apr '21

Hi, Sujatha! thanks! ok to push On Apr 01, Sujatha wrote: > revision-id: 04c7d5f1518 (mariadb-10.5.2-530-g04c7d5f1518) > parent(s): a73681e1f03 > author: Sujatha <sujatha.sivakumar(a)mariadb.com> > committer: Sujatha <sujatha.sivakumar(a)mariadb.com> > timestamp: 2021-03-29 17:36:16 +0530 > message: > > MDEV-20220: Merge 5.7 P_S replication table 'replication_applier_status_by_worker > > Step2: > ===== > Add two extra columns mentioned below. > --------------------------------------------------------------------------- > |Column Name: | Description: | > |-------------------------------------------------------------------------| > | | | > |WORKER_IDLE_TIME | Total idle time in seconds that the worker | > | | thread has spent waiting for work from | > | | co-ordinator thread | > | | | > |LAST_TRANS_RETRY_COUNT | Total number of retries attempted by last | > | | transaction | > --------------------------------------------------------------------------- > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] a73681e1f03: MDEV-20220: Merge 5.7 P_S replication table 'replication_applier_status_by_worker
by Sergei Golubchik 01 Apr '21

01 Apr '21

Hi, Sujatha! thanks! ok to push On Apr 01, Sujatha wrote: > revision-id: a73681e1f03 (mariadb-10.5.2-529-ga73681e1f03) > parent(s): bd43f39bd5a > author: Sujatha <sujatha.sivakumar(a)mariadb.com> > committer: Sujatha <sujatha.sivakumar(a)mariadb.com> > timestamp: 2021-03-29 17:27:54 +0530 > message: > > MDEV-20220: Merge 5.7 P_S replication table 'replication_applier_status_by_worker > > Step1: > ===== > Backport 'replication_applier_status_by_worker' from upstream. > > Iterate through rpl_parallel_thread_pool and display slave worker thread > specific information as part of 'replication_applier_status_by_worker' > table. > > --------------------------------------------------------------------------- > |Column Name: | Description: | > |-------------------------------------------------------------------------| > | | | > |CHANNEL_NAME | Name of replication channel through which the | > | | transaction is received. | > | | | > |THREAD_ID | Thread_Id as displayed in 'performance_schema. | > | | threads' table for thread with name | > | | 'thread/sql/rpl_parallel_thread' | > | | | > | | THREAD_ID will be NULL when worker threads are | > | | stopped due to an error/force stop | > | | | > |SERVICE_STATE | Thread is running or not | > | | | > |LAST_SEEN_TRANSACTION | Last GTID executed by worker | > | | | > |LAST_ERROR_NUMBER | Last Error that occured on a particular worker | > | | | > |LAST_ERROR_MESSAGE | Last error specific message | > | | | > |LAST_ERROR_TIMESTAMP | Time stamp of last error | > | | | > --------------------------------------------------------------------------- Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] b530cc73ddb: MDEV-22010: use executables MariaDB named in scripts
by Sergei Golubchik 01 Apr '21

01 Apr '21

Hi, Rucha! Looks almost perfect! a couple of comments about mysqld_multi.sh below On Apr 01, Rucha Deodhar wrote: > revision-id: b530cc73ddb (mariadb-10.5.2-386-gb530cc73ddb) > parent(s): de407e7cb4d > author: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > committer: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > timestamp: 2021-03-23 01:42:03 +0530 > message: > > MDEV-22010: use executables MariaDB named in scripts > > As a part of this MDEV following changes were made: > 1) Mariadb named executables used instead of mysql named executables in scripts > 2) renamed mysql-test-run and mysql-stress-test to mariadb-test-run and > mariadb-stress-test and created a symlink. > > diff --git a/scripts/mysqld_multi.sh b/scripts/mysqld_multi.sh > index 0b33f61710e..191caf22454 100644 > --- a/scripts/mysqld_multi.sh > +++ b/scripts/mysqld_multi.sh > @@ -353,7 +353,7 @@ sub start_mysqlds() > $tmp.= " $options[$j]"; > } > } > - if ($opt_verbose && $com =~ m/\/(safe_mysqld|mysqld_safe)$/ && !$info_sent) > + if ($opt_verbose && $com =~ m/\/(mariadb_safe|mariadbd-safe)$/ && !$info_sent) this doesn't make much sense. there was never a mariadb_safe tool, was it? if there was not, then you don't need to search for it, just write m/\/mariadbd-safe$/ although I'd rather written m[/mariadbd-safe$] - if / is not used as a delimiter, there's no need to escape it in a regex. > { > print "WARNING: $1 is being used to start mysqld. In this case you "; here it should, probably, say "to start mariadbd" ? > print "may need to pass\n\"ledir=...\" under groups [mysqldN] to "; > @@ -667,14 +667,14 @@ sub example > # > # 2.PID-FILE > # > -# If you are using mysqld_safe to start mysqld, make sure that every > -# MariaDB server has a separate pid-file. In order to use mysqld_safe > +# If you are using mariadbd-safe to start mariadbd, make sure that every > +# MariaDB server has a separate pid-file. In order to use mariadbd-safe > # via $my_progname, you need to use two options: > # > -# mysqld=/path/to/mysqld_safe > +# mysqld=/path/to/mariadbd-safe > # ledir=/path/to/mysqld-binary/ and here /path/to/mariadbd-binary/ ? > # > -# ledir (library executable directory), is an option that only mysqld_safe > +# ledir (library executable directory), is an option that only mariadbd-safe > # accepts, so you will get an error if you try to pass it to mysqld directly. > # For this reason you might want to use the above options within [mysqld#] > # group directly. > @@ -741,9 +741,9 @@ language = @datadir@/mysql/english > user = unix_user1 > > [mysqld3] > -mysqld = /path/to/mysqld_safe > +mysqld = /path/to/mariadbd-safe > ledir = /path/to/mysqld-binary/ and here > -mysqladmin = /path/to/mysqladmin > +mysqladmin = /path/to/mariadb-admin > socket = /tmp/mysql.sock3 > port = 3308 > pid-file = @localstatedir@3/hostname.pid3 Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] JSON_TABLE: On CTE question
by Sergey Petrunia 01 Apr '21

01 Apr '21

Hello Igor, with recursive cte1 as ( select * from t1, json_table(t1.item_props,'$' columns( color varchar(100) path '$.color')) as t ), cte2 as ( select * from (select 1 as n from dual) x, cte1 union all select cte2.n+1, cte2.item_name, cte2.item_props, cte2.color from cte2, json_table(cte2.item_props,'$' columns( color varchar(100) path '$.color')) as t where t.item_name=cte2.item_name and cte2.n < 3 ) select * from cte2; ERROR 1054 (42S22): Unknown column 't.item_name' in 'where clause' with recursive cte1 as ( select * from t1, json_table(t1.item_props,'$' columns( color varchar(100) path '$.color')) as t ), cte2 as ( select * from (select 1 as n from dual) x, cte1 union all select cte2.n+1, cte2.item_name, cte2.item_props, cte2.color from cte2, cte1 as t where t.item_name=cte2.item_name and cte2.n < 3 ) select * from cte2; +------+-----------+-----------------------------------+-------+ | n | item_name | item_props | color | +------+-----------+-----------------------------------+-------+ | 1 | Laptop | {"color": "black", "price": 1000} | black | | 1 | Jeans | {"color": "blue", "price": 50} | blue | | 2 | Laptop | {"color": "black", "price": 1000} | black | | 2 | Jeans | {"color": "blue", "price": 50} | blue | | 3 | Laptop | {"color": "black", "price": 1000} | black | | 3 | Jeans | {"color": "blue", "price": 50} | blue | +------+-----------+-----------------------------------+-------+ Igor> It means that json table t does not inherit the column name item_name from cte2. I'm not sure what do you mean by "inherit"? json_table t has one column named "color". Could you elaborate? BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://petrunia.net

1 0

[Maria-developers] JSON_TABLE: On name resolution question
by Sergey Petrunia 01 Apr '21

01 Apr '21

Hello Igor, I was looking at the example you've posted earlier this week on Slack: Igor> @holyfoot, @spetrunia: here' serious problem with identification for JSON_TABLEs: create table t1 (item_name varchar(32), item_props varchar(1024)); insert into t1 values ('Laptop', '{"color": "black", "price": 1000}'); insert into t1 values ('Jeans', '{"color": "blue", "price": 50}'); create table t2 (item_name varchar(32), item_props varchar(1024)); insert into t2 values ('Laptop', '{"color": "black", "price": 1200}'); insert into t2 values ('Shirt', '{"color": "blue", "price": 20}'); MariaDB [test]> select * from t1 as t2 where item_name not in (select item_name from t2 as s, json_table(t2.item_props,'$' columns( color varchar(100) path '$.color')) as t); +-----------+--------------------------------+ | item_name | item_props | +-----------+--------------------------------+ | Jeans | {"color": "blue", "price": 50} | +-----------+--------------------------------+ Igor> t2.item_props should not be valid, because it refers to t2 as s. Let me format the query: select * from t1 as t2 where item_name not in (select item_name from t2 as s, json_table(t2.item_props, '$' columns( color varchar(100) path '$.color') ) as t ); My reply is: "t2.item_props" does NOT refer to (t2_as_s).item_props. It refers to the top level select's (t1_as_t2).item_props. Proof: let's rename the t1-as-t2 in the top-level select: select * from t1 as t2qqqqqqqqq where item_name not in (select item_name from t2 as s, json_table(t2.item_props, '$' columns( color varchar(100) path '$.color') ) as t ); ERROR 1054 (42S22): Unknown column 't2.item_props' in 'JSON_TABLE argument' Igor> This easily can be seen after adding MariaDB [test]> insert into t2 values ('Jeans', '{"color": "black", "price": 20}'); Query OK, 1 row affected (0.004 sec) MariaDB [test]> select * from t1 as t2 where item_name not in (select item_name from t2 as s, json_table(t2.item_props,'$' columns( color varchar(100) path '$.color')) as t); I get an empty set for this query. Is there any issue? BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://petrunia.net

1 0

Re: [Maria-developers] f46382ba93c: Optimize usage of c_ptr(), c_ptr_quick() and String::alloc()
by Sergei Golubchik 01 Apr '21

01 Apr '21

Hi, Michael! On Mar 27, Michael Widenius wrote: > revision-id: f46382ba93c (mariadb-10.5.2-515-gf46382ba93c) > parent(s): c157c285db9 > author: Michael Widenius <michael.widenius(a)gmail.com> > committer: Michael Widenius <michael.widenius(a)gmail.com> > timestamp: 2021-03-24 14:31:53 +0200 > message: > > Optimize usage of c_ptr(), c_ptr_quick() and String::alloc() > > The proble was that hen using String::alloc() to allocate a string, "hen" ? after a minute of staring I realized that you probably meant "when". (also, "problem") > the String ensures that there is space for an extra NULL byte in the > buffer and if not, reallocates the string. This is a problem with the > String::set_int() that calls alloc(21), which forces > an extra malloc/free to happen. > > - We do not anymore re-allocate String if alloc() is called with the > Allocated_length. This reduces number of malloc() allocations, > especially one big re-allocation in Protocol::send_result_Set_metadata() > for almost every query that produced a result to the connnected client. > - Avoid extra mallocs when using LONGLONG_BUFFER_SIZE > This can now be done as alloc() doesn't increase buffers if new length is > not bigger than old one. > - c_ptr() is redesigned to be safer (but a bit longer) than before. > - Remove wrong usage of c_ptr_quick() > c_ptr_quick() where used in many cases to get the pointer to the used s/where/was/ > buffer, even when it didn't need to be \0 terminated. In this case > ptr() is a better substitute. > Another problem with c_ptr_quick() is that it didn't guarantee that > the string would be \0 terminated. > - item_val_str(), an API function not used currently by the server, > now always returns a null terminated string (before it didn't always > do that). > - Ensure that all String allocations uses STRING_PSI_MEMORY_KEY. The old > mixed usage of performance keys caused assert's when String buffers > where shrunk. > - Binary_string::shrink() is simplifed > > diff --git a/sql/item.cc b/sql/item.cc > index f6f3e2720fa..a8a43c6266a 100644 > --- a/sql/item.cc > +++ b/sql/item.cc > @@ -4703,7 +4703,7 @@ Item *Item_param::value_clone_item(THD *thd) > return 0; // Should create Item_decimal. See MDEV-11361. > case STRING_RESULT: > return new (mem_root) Item_string(thd, name, > - Lex_cstring(value.m_string.c_ptr_quick(), > + Lex_cstring(value.m_string.ptr(), Hmm, you said that LEX_CSTRING::str should always be \0-terminated. If that's the case, then c_ptr() would be correct here, not ptr() > value.m_string.length()), > value.m_string.charset(), > collation.derivation, > diff --git a/sql/partition_info.cc b/sql/partition_info.cc > index 871411cf6c4..b3e1bceec31 100644 > --- a/sql/partition_info.cc > +++ b/sql/partition_info.cc > @@ -126,7 +126,7 @@ partition_info *partition_info::get_clone(THD *thd) > /** > Mark named [sub]partition to be used/locked. > > - @param part_name Partition name to match. > + @param part_name Partition name to match. Must be \0 terminated! > @param length Partition name length. > > @return Success if partition found > @@ -172,9 +172,9 @@ bool partition_info::add_named_partition(const char *part_name, size_t length) > else > bitmap_set_bit(&read_partitions, part_def->part_id); > } > - DBUG_PRINT("info", ("Found partition %u is_subpart %d for name %s", > + DBUG_PRINT("info", ("Found partition %u is_subpart %d for name %.*s", > part_def->part_id, part_def->is_subpart, > - part_name)); > + length, part_name)); These two changes contradict each other. Either part_name must be \0-terminated, and then you don't need to specify a length in printf. Or it is not \0-terminated and you must specify a length. > DBUG_RETURN(false); > } > > diff --git a/sql/sql_string.cc b/sql/sql_string.cc > index f2a0f55aec8..95a57017c53 100644 > --- a/sql/sql_string.cc > +++ b/sql/sql_string.cc > @@ -118,9 +121,14 @@ bool Binary_string::realloc_raw(size_t alloc_length) > return FALSE; > } > > + > bool String::set_int(longlong num, bool unsigned_flag, CHARSET_INFO *cs) > { > - uint l=20*cs->mbmaxlen+1; > + /* > + This allocates a few bytes extra in the unlikely case that cs->mb_maxlen > + > 1, but we can live with that dunno, it seems that utf8 is the *likely* case and it's getting more and more likely with the time, > + */ > + uint l= LONGLONG_BUFFER_SIZE * cs->mbmaxlen; > int base= unsigned_flag ? 10 : -10; > > if (alloc(l)) > @@ -1254,21 +1262,16 @@ bool String::append_semi_hex(const char *s, uint len, CHARSET_INFO *cs) > // Shrink the buffer, but only if it is allocated on the heap. > void Binary_string::shrink(size_t arg_length) > { > - if (!is_alloced()) > - return; > - if (ALIGN_SIZE(arg_length + 1) < Alloced_length) > + if (is_alloced() && ALIGN_SIZE(arg_length + 1) < Alloced_length) > + { > + char *new_ptr; > + /* my_realloc() can't fail as new buffer is less than the original one */ > + if ((new_ptr= (char*) my_realloc(STRING_PSI_MEMORY_KEY, Ptr, arg_length, you don't need an if() because, as you wrote yourself "my_realloc() can't fail" here. > + MYF(thread_specific ? > + MY_THREAD_SPECIFIC : 0)))) > { > diff --git a/sql/sql_string.h b/sql/sql_string.h > index b3eca118b63..ba0cff68fb4 100644 > --- a/sql/sql_string.h > +++ b/sql/sql_string.h > @@ -600,25 +600,34 @@ class Binary_string: public Static_binary_string > > inline char *c_ptr() > { > - DBUG_ASSERT(!alloced || !Ptr || !Alloced_length || > - (Alloced_length >= (str_length + 1))); > - > - if (!Ptr || Ptr[str_length]) // Should be safe > - (void) realloc(str_length); > + if (unlikely(!Ptr)) > + return (char*) ""; > + /* > + Here we assume that any buffer used to initalize String has > + an end \0 or have at least an accessable character at end. > + This is to handle the case of String("Hello",5) efficently. > + */ > + if (unlikely(!alloced && !Ptr[str_length])) > + return Ptr; No, this is not good. Note the difference between String a("Hello", 5) and char hello[5]; String a(buf, 5); Your assumption should only apply to the first case, not to the second. In the first case alloced=Alloced_length=0, in the second case only alloced=0 and Alloced_length=5. So in the if() above you need to look at Alloced_length: if (!Alloced_length && !Ptr[str_length]) return Ptr; > + if (str_length < Alloced_length) > + { > + Ptr[str_length]=0; > + return Ptr; > + } > + (void) realloc(str_length+1); /* This will add end \0 */ > return Ptr; > } > + /* Use c_ptr() instead. This will be deleted soon, kept for compatiblity */ > inline char *c_ptr_quick() > { > - if (Ptr && str_length < Alloced_length) > - Ptr[str_length]=0; > - return Ptr; > + return c_ptr_safe(); 1. why not to remove it now? 2. it's strange that you write "use c_ptr() instead" but you actually use c_ptr_safe() instead. > } > inline char *c_ptr_safe() > { > if (Ptr && str_length < Alloced_length) > Ptr[str_length]=0; > else > - (void) realloc(str_length); > + (void) realloc(str_length + 1); > return Ptr; > } could you write a comment, explaining when one should use c_ptr() and when - c_ptr_safe() ? Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 7

Re: [Maria-developers] 9bf4b92cbc5: MDEV-17554 Auto-create new partition for system versioned tables with history partitioned by INTERVAL/LIMIT
by Sergei Golubchik 31 Mar '21

31 Mar '21

Hi, Aleksey! On Jun 29, Aleksey Midenkov wrote: > revision-id: 9bf4b92cbc5 (mariadb-10.5.2-168-g9bf4b92cbc5) > parent(s): 478301d9b9a > author: Aleksey Midenkov <midenok(a)gmail.com> > committer: Aleksey Midenkov <midenok(a)gmail.com> > timestamp: 2020-04-17 17:04:24 +0300 > message: > > MDEV-17554 Auto-create new partition for system versioned tables with history partitioned by INTERVAL/LIMIT > > == Syntax change == > > Keyword AUTO_INCREMENT (or AUTO) enables partition auto-creation. I still think that AUTO_INCREMENT here looks rather out-of-place, I don't know how to explain to users why AUTO_INCREMENT is allowed here. But if we'll have it, we'll have to support it here for a long time for compatibility reasons. "Because somebody might be already using it" I'd suggest to support only AUTO here. Or, if the parser can do it easily, a much more readable syntax would be create table t1 (x int) with system versioning auto partition by system_time interval 1 hour; this would be the best, easy to read, matches the documentation (where it could be called "auto partitioning"). If it won't cause any difficult parser problems, I'd prefer us to use that. > create or replace table t1 (x int) with system versioning > partition by system_time interval 1 hour auto_increment; > > create or replace table t1 (x int) with system versioning > partition by system_time limit 1000 auto; > > Or with explicit partitions: > > create or replace table t1 (x int) with system versioning > partition by system_time interval 1 hour auto > (partition p0 history, partition pn current); > > == Description == > > Before executing history-generating DML command add N history > partitions, so that N would be sufficient for potentially generated > history. N > 1 may be required when history is rotated by INTERVAL and > timestamp was jumped to future further than interval value. > > It is assumed that one DML command will not generate history rows more > than specified LIMIT. Otherwise history would overflow generated > partition, a warning would be printed, and user action would be > required to rebuild partitions. > > Auto-creation is implemented by synchronous > fast_alter_partition_table() call from the thread of the executed DML > command before the command itself (by the fallback and retry mechanism > similar to Discovery feature, see Open_table_context). > > Creating history partitions was made by the principle of minimal > disruption of the main business process. I.e. when procedure of > creation fails it will not (if possible) fail the original DML > command. Warning info will display what happened and the last history > partition may overflow. In such case partition rebuild is required to > correctly display history info. User application may detect warning > code ER_VERS_HIST_PART_FAILED and stop execution if that is preferred. > > The name for newly added partitions are generated like default > partition names with extension of MDEV-22155 (which avoids name > clashes by extending assignment counter to next free-enough gap). > > These DML commands trigger auto-creation: > > * DELETE (including multi-delete, excluding DELETE HISTORY) > * UPDATE (including multi-update) > * REPLACE (including REPLACE .. SELECT) INSERT ... ON DUPLICATE KEY UPDATE ? > diff --git a/mysql-test/suite/versioning/common.inc b/mysql-test/suite/versioning/common.inc > index 355b571e5a0..b35a5138015 100644 > --- a/mysql-test/suite/versioning/common.inc > +++ b/mysql-test/suite/versioning/common.inc > @@ -6,6 +6,7 @@ if (!$TEST_VERSIONING_SO) > source include/have_innodb.inc; > > set @@session.time_zone='+00:00'; > +set @@global.time_zone='+00:00'; Why is that? I'd expect your auto-adding of partitions to happen in the context of a session, using session time zone. Is it for replication? > select ifnull(max(transaction_id), 0) into @start_trx_id from mysql.transaction_registry; > set @test_start=now(6); > > diff --git a/mysql-test/suite/versioning/r/delete_history.result b/mysql-test/suite/versioning/r/delete_history.result > index cb865a835b3..90c9e4777bb 100644 > --- a/mysql-test/suite/versioning/r/delete_history.result > +++ b/mysql-test/suite/versioning/r/delete_history.result > @@ -63,8 +63,6 @@ insert into t values (1); > update t set a= 2; > update t set a= 3; > delete history from t; > -Warnings: > -Warning 4114 Versioned table `test`.`t`: last HISTORY partition (`p1`) is out of LIMIT, need more HISTORY partitions good riddance, it didn't make much sense anyway > # The above warning is one command late (MDEV-20345) ^^^ > select * from t for system_time all; > a > diff --git a/sql/ha_partition.h b/sql/ha_partition.h > index 85cb736b5bd..5fb893b4f0e 100644 > --- a/sql/ha_partition.h > +++ b/sql/ha_partition.h > @@ -1623,7 +1623,7 @@ class ha_partition :public handler could you also, please, fix (in a separate commit) ha_partition::part_records() to take a proper partition_element* argument, not void* ? > for (; part_id < part_id_end; ++part_id) > { > handler *file= m_file[part_id]; > - DBUG_ASSERT(bitmap_is_set(&(m_part_info->read_partitions), part_id)); > + bitmap_set_bit(&(m_part_info->read_partitions), part_id); > file->info(HA_STATUS_VARIABLE | HA_STATUS_NO_LOCK | HA_STATUS_OPEN); > part_recs+= file->stats.records; > } > diff --git a/sql/partition_info.cc b/sql/partition_info.cc > index e9dbf2b49c3..72312ee7ac4 100644 > --- a/sql/partition_info.cc > +++ b/sql/partition_info.cc > @@ -814,10 +814,16 @@ bool partition_info::has_unique_name(partition_element *element) > vers_info->interval Limit by fixed time interval > vers_info->hist_part (out) Working history partition > */ > -void partition_info::vers_set_hist_part(THD *thd) > +uint partition_info::vers_set_hist_part(THD *thd, bool auto_inc) let's not all it auto_inc. it'll add noise to every grep for auto_inc issues. What about auto_add ? > { > + DBUG_ASSERT(!thd->lex->last_table()->vers_conditions.delete_history); > + > + uint create_count= 0; > + auto_inc= auto_inc && vers_info->auto_inc; > + > if (vers_info->limit) > { > + DBUG_ASSERT(!vers_info->interval.is_set()); > ha_partition *hp= (ha_partition*)(table->file); > partition_element *next= NULL; > List_iterator<partition_element> it(partitions); > @@ -836,22 +842,26 @@ void partition_info::vers_set_hist_part(THD *thd) > { > if (next == vers_info->now_part) > { > - my_error(WARN_VERS_PART_FULL, MYF(ME_WARNING|ME_ERROR_LOG), > - table->s->db.str, table->s->table_name.str, > - vers_info->hist_part->partition_name, "LIMIT"); > + if (auto_inc) > + create_count= 1; > + else > + my_error(WARN_VERS_PART_FULL, MYF(ME_WARNING|ME_ERROR_LOG), > + table->s->db.str, table->s->table_name.str, > + vers_info->hist_part->partition_name, "LIMIT"); > } > else > vers_info->hist_part= next; > } > - return; > + // reserve at least one history partition > + if (auto_inc && create_count == 0 && > + vers_info->hist_part->id + 1 == vers_info->now_part->id) > + create_count= 1; Questionable. What does it solve? > } > - > - if (vers_info->interval.is_set()) > + else if (vers_info->interval.is_set() && > + vers_info->hist_part->range_value <= thd->query_start()) > { > - if (vers_info->hist_part->range_value > thd->query_start()) > - return; > - > partition_element *next= NULL; > + bool error= true; > List_iterator<partition_element> it(partitions); > while (next != vers_info->hist_part) > next= it++; > @@ -860,12 +870,200 @@ void partition_info::vers_set_hist_part(THD *thd) > { > vers_info->hist_part= next; > if (next->range_value > thd->query_start()) > - return; > + { > + error= false; > + break; but here you don't "reserve at least one history partition" Why INTERVAL is different? > + } > + } > + if (error) > + { > + if (auto_inc) > + { > + DBUG_ASSERT(thd->query_start() >= vers_info->hist_part->range_value); > + my_time_t diff= thd->query_start() - vers_info->hist_part->range_value; > + if (diff > 0) > + { > + size_t delta= vers_info->interval.seconds(); > + create_count= diff / delta + 1; > + if (diff % delta) > + create_count++; > + } > + else > + create_count= 1; > + } > + else > + { > + my_error(WARN_VERS_PART_FULL, MYF(ME_WARNING|ME_ERROR_LOG), > + table->s->db.str, table->s->table_name.str, > + vers_info->hist_part->partition_name, "INTERVAL"); > + } > } > - my_error(WARN_VERS_PART_FULL, MYF(ME_WARNING|ME_ERROR_LOG), > - table->s->db.str, table->s->table_name.str, > - vers_info->hist_part->partition_name, "INTERVAL"); > } > + > + return create_count; > +} > + > + > +/** > + @brief Run fast_alter_partition_table() to add new history partitions > + for tables requiring them. > +*/ > +void vers_add_auto_parts(THD *thd, TABLE_LIST* tl, uint num_parts) > +{ > + HA_CREATE_INFO create_info; > + Alter_info alter_info; > + String query; > + partition_info *save_part_info= thd->work_part_info; > + Query_tables_list save_query_tables; > + Reprepare_observer *save_reprepare_observer= thd->m_reprepare_observer; > + Diagnostics_area new_stmt_da(thd->query_id, false, true); > + Diagnostics_area *save_stmt_da= thd->get_stmt_da(); > + bool save_no_write_to_binlog= thd->lex->no_write_to_binlog; > + const CSET_STRING save_query= thd->query_string; > + thd->m_reprepare_observer= NULL; > + thd->lex->reset_n_backup_query_tables_list(&save_query_tables); > + thd->in_sub_stmt|= SUB_STMT_AUTO_HIST; > + thd->lex->no_write_to_binlog= !thd->is_current_stmt_binlog_format_row(); > + TABLE *table= tl->table; > + > + DBUG_ASSERT(!thd->is_error()); > + /* NB: we have to preserve m_affected_rows, m_row_count_func, m_last_insert_id, etc */ > + thd->set_stmt_da(&new_stmt_da); > + new_stmt_da.set_overwrite_status(true); Is it needed? You've just started using a new Diagnostics_area, the status should be DA_EMPTY here, nothing to overwrite. > + > + { > + DBUG_ASSERT(table->s->get_table_ref_type() == TABLE_REF_BASE_TABLE); > + DBUG_ASSERT(table->versioned()); > + DBUG_ASSERT(table->part_info); > + DBUG_ASSERT(table->part_info->vers_info); > + alter_info.reset(); > + alter_info.partition_flags= ALTER_PARTITION_ADD|ALTER_PARTITION_AUTO_HIST; > + create_info.init(); > + create_info.alter_info= &alter_info; > + Alter_table_ctx alter_ctx(thd, tl, 1, &table->s->db, &table->s->table_name); > + > + MDL_REQUEST_INIT(&tl->mdl_request, MDL_key::TABLE, tl->db.str, > + tl->table_name.str, MDL_SHARED_NO_WRITE, MDL_TRANSACTION); > + if (thd->mdl_context.acquire_lock(&tl->mdl_request, > + thd->variables.lock_wait_timeout)) > + goto exit; > + table->mdl_ticket= tl->mdl_request.ticket; > + > + create_info.db_type= table->s->db_type(); > + create_info.options|= HA_VERSIONED_TABLE; > + DBUG_ASSERT(create_info.db_type); > + > + create_info.vers_info.set_start(table->s->vers_start_field()->field_name); > + create_info.vers_info.set_end(table->s->vers_end_field()->field_name); > + > + partition_info *part_info= new partition_info(); > + if (unlikely(!part_info)) > + { > + my_error(ER_OUT_OF_RESOURCES, MYF(0)); > + goto exit; > + } > + part_info->use_default_num_partitions= false; > + part_info->use_default_num_subpartitions= false; > + part_info->num_parts= num_parts; > + part_info->num_subparts= table->part_info->num_subparts; > + part_info->subpart_type= table->part_info->subpart_type; > + if (unlikely(part_info->vers_init_info(thd))) > + { > + my_error(ER_OUT_OF_RESOURCES, MYF(0)); > + goto exit; > + } > + > + // NB: set_ok_status() requires DA_EMPTY > + thd->get_stmt_da()->reset_diagnostics_area(); Is it needed? You've just started using a new Diagnostics_area, the status should be DA_EMPTY here. > + > + thd->work_part_info= part_info; > + if (part_info->set_up_defaults_for_partitioning(thd, table->file, NULL, > + table->part_info->next_part_no(num_parts))) > + { > + push_warning(thd, Sql_condition::WARN_LEVEL_WARN, > + ER_VERS_HIST_PART_FAILED, > + "Auto-increment history partition: " > + "setting up defaults failed"); > + my_error(ER_VERS_HIST_PART_FAILED, MYF(ME_WARNING), > + tl->db.str, tl->table_name.str); > + goto exit; > + } > + bool partition_changed= false; > + bool fast_alter_partition= false; > + if (prep_alter_part_table(thd, table, &alter_info, &create_info, > + &partition_changed, &fast_alter_partition)) > + { > + push_warning(thd, Sql_condition::WARN_LEVEL_WARN, ER_VERS_HIST_PART_FAILED, > + "Auto-increment history partition: " > + "alter partitition prepare failed"); > + my_error(ER_VERS_HIST_PART_FAILED, MYF(ME_WARNING), > + tl->db.str, tl->table_name.str); > + goto exit; > + } > + if (!fast_alter_partition) > + { > + push_warning(thd, Sql_condition::WARN_LEVEL_WARN, ER_VERS_HIST_PART_FAILED, > + "Auto-increment history partition: " > + "fast alter partitition is not possible"); > + my_error(ER_VERS_HIST_PART_FAILED, MYF(ME_WARNING), > + tl->db.str, tl->table_name.str); > + goto exit; > + } > + DBUG_ASSERT(partition_changed); > + if (mysql_prepare_alter_table(thd, table, &create_info, &alter_info, > + &alter_ctx)) > + { > + push_warning(thd, Sql_condition::WARN_LEVEL_WARN, ER_VERS_HIST_PART_FAILED, > + "Auto-increment history partition: " > + "alter prepare failed"); > + my_error(ER_VERS_HIST_PART_FAILED, MYF(ME_WARNING), > + tl->db.str, tl->table_name.str); > + goto exit; > + } > + > + // Forge query string for rpl logging > + if (!thd->lex->no_write_to_binlog) > + { > + query.set(STRING_WITH_LEN("ALTER TABLE `"), &my_charset_latin1); better use StringBuffer<...> query; because you're doing lots of reallocs now. > + > + if (query.append(table->s->db) || > + query.append(STRING_WITH_LEN("`.`")) || > + query.append(table->s->table_name) || this is wrong, the table and db names can have backticks inside. use append_identifier() instead. > + query.append("` ADD PARTITION PARTITIONS ") || > + query.append_ulonglong(part_info->num_parts) || > + query.append(" AUTO")) > + { > + my_error(ER_OUT_OF_RESOURCES, MYF(ME_ERROR_LOG)); > + goto exit; > + } > + CSET_STRING qs(query.c_ptr(), query.length(), &my_charset_latin1); > + thd->set_query(qs); > + } Should it be binlogged at all? May be just leave it to the slave to auto-add the partition as needed? Looks a bit suspicious now, you log an ALTER TABLE possibly in the middle of a transaction. It will be replayed differently, not as auto-adding, in particular, it will commit. So, possibly different results on the slave, perhaps different gtid. Do clear it out with Andrei please. Or don't binlog auto-adding at all. > + > + if (fast_alter_partition_table(thd, table, &alter_info, &create_info, > + tl, &table->s->db, &table->s->table_name)) > + { > + push_warning(thd, Sql_condition::WARN_LEVEL_WARN, ER_VERS_HIST_PART_FAILED, > + "Auto-increment history partition: " > + "alter partition table failed"); > + my_error(ER_VERS_HIST_PART_FAILED, MYF(ME_WARNING), > + tl->db.str, tl->table_name.str); > + } > + } > + > + if (!thd->transaction.stmt.is_empty()) > + trans_commit_stmt(thd); 1. why would this operation register a transaction? 2. you amended a couple of checks like - if (thd->in_sub_stmt) + if (thd->in_sub_stmt & ~SUB_STMT_AUTO_HIST) but what will happen if it's, for example, an insert inside a trigger? So a real sub_stmt and SUB_STMT_AUTO_HIST? > + > +exit: > + thd->work_part_info= save_part_info; > + thd->m_reprepare_observer= save_reprepare_observer; > + thd->lex->restore_backup_query_tables_list(&save_query_tables); > + thd->in_sub_stmt&= ~SUB_STMT_AUTO_HIST; > + if (!new_stmt_da.is_warning_info_empty()) > + save_stmt_da->copy_sql_conditions_from_wi(thd, new_stmt_da.get_warning_info()); > + thd->set_stmt_da(save_stmt_da); > + thd->lex->no_write_to_binlog= save_no_write_to_binlog; > + thd->set_query(save_query); > } > > > diff --git a/sql/partition_info.h b/sql/partition_info.h > index 7ae2d168068..ca68e61cbe2 100644 > --- a/sql/partition_info.h > +++ b/sql/partition_info.h > @@ -72,9 +73,34 @@ struct Vers_part_info : public Sql_alloc > my_time_t start; > INTERVAL step; > enum interval_type type; > - bool is_set() { return type < INTERVAL_LAST; } > + bool is_set() const { return type < INTERVAL_LAST; } > + size_t seconds() const > + { > + if (step.second) > + return step.second; > + if (step.minute) > + return step.minute * 60; > + if (step.hour) > + return step.hour * 3600; > + if (step.day) > + return step.day * 3600 * 24; > + // comparison is used in rough estimates, it doesn't need to be calendar-correct Are you sure the approximate value is enough here? You use it to estimate how many partitions to create. It's not a big deal if you create more than necessary (although it's not nice and we'll definitely get bug reports if that will happen). But you surely don't want to create less. > + if (step.month) > + return step.month * 3600 * 24 * 30; > + DBUG_ASSERT(step.year); > + return step.year * 86400 * 30 * 365; > + } > + bool lt(size_t secs) const > + { > + return seconds() < secs; > + } > + bool ge(size_t seconds) const > + { > + return !(this->lt(seconds)); > + } lt() and ge() don't seem to be used anywhere. > } interval; > ulonglong limit; > + bool auto_inc; > partition_element *now_part; > partition_element *hist_part; > }; > diff --git a/sql/sql_base.cc b/sql/sql_base.cc > index b89be77f282..5583d70f8eb 100644 > --- a/sql/sql_base.cc > +++ b/sql/sql_base.cc > @@ -1862,6 +1862,25 @@ bool open_table(THD *thd, TABLE_LIST *table_list, Open_table_context *ot_ctx) > DBUG_PRINT("info",("Using locked table")); > #ifdef WITH_PARTITION_STORAGE_ENGINE > part_names_error= set_partitions_as_used(table_list, table); > + if (table->part_info && > + table->part_info->part_type == VERSIONING_PARTITION && > + !table_list->vers_conditions.delete_history && > + table_list->lock_type >= TL_WRITE_ALLOW_WRITE && > + table_list->mdl_request.type == MDL_SHARED_WRITE) > + { > + switch (thd->lex->sql_command) > + { > + case SQLCOM_DELETE: > + case SQLCOM_UPDATE: > + case SQLCOM_REPLACE: > + case SQLCOM_REPLACE_SELECT: > + case SQLCOM_DELETE_MULTI: > + case SQLCOM_UPDATE_MULTI: > + /* Rotation is still needed under LOCK TABLES */ > + table->part_info->vers_set_hist_part(thd, false); ALTER TABLE works under LOCK TABLES, so this should be possible too, but let's have it as a separate task. > + default:; > + } > + } > #endif > goto reset; > } > @@ -2104,6 +2123,37 @@ bool open_table(THD *thd, TABLE_LIST *table_list, Open_table_context *ot_ctx) > tc_add_table(thd, table); > } > > +#ifdef WITH_PARTITION_STORAGE_ENGINE > + if (table->part_info && > + table->part_info->part_type == VERSIONING_PARTITION && > + !table_list->vers_conditions.delete_history && > + table_list->lock_type >= TL_WRITE_ALLOW_WRITE && > + table_list->mdl_request.type >= MDL_SHARED_WRITE && > + table_list->mdl_request.type < MDL_EXCLUSIVE) > + { > + switch (thd->lex->sql_command) > + { > + case SQLCOM_LOCK_TABLES: > + case SQLCOM_DELETE: > + case SQLCOM_UPDATE: > + case SQLCOM_REPLACE: > + case SQLCOM_REPLACE_SELECT: > + case SQLCOM_DELETE_MULTI: > + case SQLCOM_UPDATE_MULTI: this is quite complex condition, better not to duplicate it. May be something like bool need_set_hist_part(TABLE_LIST *table_list, enum_sql_command sql_command) { if (table_list->table->part_info && table_list->table->part_info->part_type == VERSIONING_PARTITION && !table_list->vers_conditions.delete_history && table_list->lock_type >= TL_WRITE_ALLOW_WRITE && table_list->mdl_request.type >= MDL_SHARED_WRITE && table_list->mdl_request.type < MDL_EXCLUSIVE) { switch(sql_command) { case SQLCOM_LOCK_TABLES: case SQLCOM_DELETE: case SQLCOM_UPDATE: case SQLCOM_REPLACE: case SQLCOM_REPLACE_SELECT: case SQLCOM_DELETE_MULTI: case SQLCOM_UPDATE_MULTI: return true; } default:; } return false; } > + ot_ctx->vers_create_count= table->part_info->vers_set_hist_part(thd, true); > + if (ot_ctx->vers_create_count) > + { > + ot_ctx->request_backoff_action(Open_table_context::OT_ADD_HISTORY_PARTITION, > + table_list); > + MYSQL_UNBIND_TABLE(table->file); > + tc_release_table(table); There's no MYSQL_UNBIND_TABLE/tc_release_table after other request_backoff_action invocations. Why this one is special? > + DBUG_RETURN(TRUE); > + } > + default:; > + } > + } > +#endif > + > if (!(flags & MYSQL_OPEN_HAS_MDL_LOCK) && > table->s->table_category < TABLE_CATEGORY_INFORMATION) > { > diff --git a/sql/sql_class.h b/sql/sql_class.h > index 14516811262..145ac5c5f64 100644 > --- a/sql/sql_class.h > +++ b/sql/sql_class.h > @@ -3346,6 +3347,10 @@ class THD: public THD_count, /* this must be first */ > > #ifdef WITH_PARTITION_STORAGE_ENGINE > partition_info *work_part_info; > + /** > + List of tables requiring new history partition. > + */ > + List<TABLE_SHARE> vers_auto_part_tables; this doesn't seem to be used anywhere > #endif > > #ifndef EMBEDDED_LIBRARY > diff --git a/sql/sql_yacc.yy b/sql/sql_yacc.yy > index 7cc1faea79b..3f6c1793432 100644 > --- a/sql/sql_yacc.yy > +++ b/sql/sql_yacc.yy > @@ -7521,14 +7529,17 @@ add_partition_rule: > > add_part_extra: > /* empty */ > - | '(' part_def_list ')' > + | '(' part_def_list ')' opt_vers_auto_inc > { > - LEX *lex= Lex; > - lex->part_info->num_parts= lex->part_info->partitions.elements; > + Lex->part_info->num_parts= Lex->part_info->partitions.elements; > + if ($4) > + Lex->alter_info.partition_flags|= ALTER_PARTITION_AUTO_HIST; > } > - | PARTITIONS_SYM real_ulong_num > + | PARTITIONS_SYM real_ulong_num opt_vers_auto_inc > { > Lex->part_info->num_parts= $2; > + if ($3) > + Lex->alter_info.partition_flags|= ALTER_PARTITION_AUTO_HIST; I'm confused. I thought that ALTER_PARTITION_AUTO_HIST is what you set in vers_add_auto_parts() to mark auto-adding of a new partition. But here you set it in the parser, when a user specifies AUTO in partition specifications. So it seems you're using this flag for two very different purposes. How do you distinguish between them? And why did you decide to do it this way? > } > ; > > diff --git a/sql/sql_partition.cc b/sql/sql_partition.cc > index fbed614489d..25017ee5425 100644 > --- a/sql/sql_partition.cc > +++ b/sql/sql_partition.cc > @@ -4825,6 +4829,13 @@ uint prep_alter_part_table(THD *thd, TABLE *table, Alter_info *alter_info, > DBUG_RETURN(TRUE); > } > > + if (alter_info->partition_flags & ALTER_PARTITION_AUTO_HIST && > + (!table->part_info || !table->part_info->vers_info)) > + { > + my_error(ER_SYNTAX_ERROR, MYF(0)); can it even happen? Or should it be DBUG_ASSERT(0) here? > + DBUG_RETURN(TRUE); > + } > + > partition_info *alt_part_info= thd->lex->part_info; > /* > This variable is TRUE in very special case when we add only DEFAULT > @@ -5312,7 +5323,9 @@ that are reorganised. > now_part= el; > } > } > - if (*fast_alter_table && tab_part_info->vers_info->interval.is_set()) > + if (*fast_alter_table && > + !(alter_info->partition_flags & ALTER_PARTITION_AUTO_HIST) && > + tab_part_info->vers_info->interval.is_set()) this !ALTER_PARTITION_AUTO_HIST - do you mean not vers_add_auto_parts() or not sql statement that uses AUTO ? > { > partition_element *hist_part= tab_part_info->vers_info->hist_part; > if (hist_part->range_value <= thd->query_start()) > @@ -5347,7 +5360,8 @@ that are reorganised. > */ > if (!(alter_info->partition_flags & ALTER_PARTITION_TABLE_REORG)) > { > - if (!alt_part_info->use_default_partitions) > + if (!alt_part_info->use_default_partitions && > + !(alter_info->partition_flags & ALTER_PARTITION_AUTO_HIST)) Sorry, I don't understand that if() at all. What does it do? > { > DBUG_PRINT("info", ("part_info: %p", tab_part_info)); > tab_part_info->use_default_partitions= FALSE; Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 4

Re: [Maria-developers] f8901333a82: Add support for minimum field width for strings to my_vsnprintf()
by Sergei Golubchik 31 Mar '21

31 Mar '21

Hi, Michael! On Mar 29, Michael Widenius wrote: > revision-id: f8901333a82 (mariadb-10.5.2-534-gf8901333a82) > parent(s): a194c5cfd41 > author: Michael Widenius <michael.widenius(a)gmail.com> > committer: Michael Widenius <michael.widenius(a)gmail.com> > timestamp: 2021-03-24 15:53:35 +0200 > message: > > Add support for minimum field width for strings to my_vsnprintf() > > diff --git a/mysql-test/main/almost_full.result b/mysql-test/main/almost_full.result > index b2d7092aa51..4b5c8417412 100644 > --- a/mysql-test/main/almost_full.result > +++ b/mysql-test/main/almost_full.result > @@ -7,24 +7,24 @@ INSERT INTO t1 SET b=repeat('a',600); > ERROR HY000: The table 't1' is full > CHECK TABLE t1 EXTENDED; > Table Op Msg_type Msg_text > -test.t1 check warning Datafile is almost full, 65448 of 65534 used > +test.t1 check warning Datafile is almost full, 65448 of 65534 used This happens because mi_check.c has mi_check_print_warning(param, "Datafile is almost full, %10s of %10s used", ... could you please remove explicit width specification from these messages? this looks rather ugly in warnings. This applies both to MyISAM and Aria. > test.t1 check status OK > UPDATE t1 SET b=repeat('a', 800) where a=10; > ERROR HY000: The table 't1' is full > diff --git a/mysql-test/main/grant4.result b/mysql-test/main/grant4.result > index 981bbdeddf5..ba64010f1bb 100644 > --- a/mysql-test/main/grant4.result > +++ b/mysql-test/main/grant4.result > @@ -187,8 +187,8 @@ connection con1; > check table mysqltest_db1.t1; > Table Op Msg_type Msg_text > mysqltest_db1.t1 check warning 1 client is using or hasn't closed the table properly > -mysqltest_db1.t1 check error Size of indexfile is: 1024 Should be: 2048 > -mysqltest_db1.t1 check warning Size of datafile is: 14 Should be: 7 > +mysqltest_db1.t1 check error Size of indexfile is: 1024 Should be: 2048 > +mysqltest_db1.t1 check warning Size of datafile is: 14 Should be: 7 Same here. > mysqltest_db1.t1 check error Corrupt > # The below statement should fail before repairing t1. > # Otherwise info about such repair will be missing from its result-set. > diff --git a/mysql-test/main/myisam_recover.result b/mysql-test/main/myisam_recover.result > index da96682186c..d3c385bc49b 100644 > --- a/mysql-test/main/myisam_recover.result > +++ b/mysql-test/main/myisam_recover.result > @@ -69,9 +69,9 @@ flush table t1; > # check table is needed to mark the table as crashed. > check table t1; > Table Op Msg_type Msg_text > -test.t1 check warning Size of datafile is: 42 Should be: 21 > -test.t1 check error Record-count is not ok; is 6 Should be: 3 > -test.t1 check warning Found 6 key parts. Should be: 3 > +test.t1 check warning Size of datafile is: 42 Should be: 21 > +test.t1 check error Record-count is not ok; is 6 Should be: 3 > +test.t1 check warning Found 6 key parts. Should be: 3 Same. Must be in many messages in mi_check.c and ma_check.c > test.t1 check error Corrupt > # > diff --git a/strings/my_vsnprintf.c b/strings/my_vsnprintf.c > index a2e3f9b738d..1621db2e412 100644 > --- a/strings/my_vsnprintf.c > +++ b/strings/my_vsnprintf.c > @@ -224,12 +224,27 @@ static char *backtick_string(CHARSET_INFO *cs, char *to, const char *end, > */ > > static char *process_str_arg(CHARSET_INFO *cs, char *to, const char *end, > - size_t width, char *par, uint print_type, > - my_bool nice_cut) > + longlong length_arg, size_t width, char *par, > + uint print_type, my_bool nice_cut) > { > int well_formed_error; > uint dots= 0; > size_t plen, left_len= (size_t) (end - to) + 1, slen=0; > + my_bool left_fill= 1; > + size_t length; > + > + /* > + The sign of the length argument specific the string should be right > + or left adjusted > + */ > + if (length_arg < 0) You're trying to support left alignment, but you have yourself added a test that shows that it doesn't work. So this code is dead, never used. See below. > + { > + length= -length_arg; > + left_fill= 0; > + } > + else > + length= (size_t) length_arg; > + > if (!par) > par = (char*) "(null)"; > > @@ -693,7 +720,8 @@ size_t my_vsnprintf_ex(CHARSET_INFO *cs, char *to, size_t n, > if (*fmt == 's' || *fmt == 'T') /* String parameter */ > { > reg2 char *par= va_arg(ap, char *); > - to= process_str_arg(cs, to, end, width, par, print_type, (*fmt == 'T')); > + to= process_str_arg(cs, to, end, (longlong) length, width, par, see? length is unsigned type size_t, the sign before length is ignored by the parser. Whether you cast length to a signed type or not and whether you checks for length < 0 or not, left alignment still won't work. That's why you have added this test: > + test1("Negative width is ignored for strings < x> < y>", > + "Negative width is ignored for strings <%-4s> <%-5s>", "x", "y"); Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 2

Re: [Maria-developers] 1bbc852ef71: Changed field_index to use field_index_t instead of uint16
by Sergei Golubchik 31 Mar '21

31 Mar '21

Hi, Michael! On Mar 29, Michael Widenius wrote: > revision-id: 1bbc852ef71 (mariadb-10.5.2-525-g1bbc852ef71) > parent(s): 039f4a054bb > author: Michael Widenius <michael.widenius(a)gmail.com> > committer: Michael Widenius <michael.widenius(a)gmail.com> > timestamp: 2021-03-24 14:31:54 +0200 > message: > > Changed field_index to use field_index_t instead of uint16 Just curious. Why? > diff --git a/sql/unireg.cc b/sql/unireg.cc > index 51dd1fb6e8c..f1a0c5c5269 100644 > --- a/sql/unireg.cc > +++ b/sql/unireg.cc > @@ -146,8 +146,8 @@ get_fieldno_by_name(HA_CREATE_INFO *create_info, List<Create_field> &create_fiel > { > if (field_name.streq(sql_field->field_name)) > { > - DBUG_ASSERT(field_no <= uint16(~0U)); > + DBUG_ASSERT(field_no < NO_CACHED_FIELD_INDEX); this look suspicios. NO_CACHED_FIELD_INDEX is ((uint)(-1)) which is always greater than anything you can put into field_index_t. You want to ensure that field_no is valid when casted into field_index_t. I'd suggest to redefine NO_CACHED_FIELD_INDEX as ((field_index_t)~0U) > - return uint16(field_no); > + return (field_index_t) field_no; why not to declare field_no to be field_index_t? then you won't need a cast. > } > } > > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] 1d577194654: Revert MDEV-16592 "Change Item::with_sum_func to a virtual method"
by Sergei Golubchik 31 Mar '21

31 Mar '21

Hi, Michael! On Mar 29, Michael Widenius wrote: > revision-id: 1d577194654 (mariadb-10.5.2-526-g1d577194654) > parent(s): 1bbc852ef71 > author: Michael Widenius <michael.widenius(a)gmail.com> > committer: Michael Widenius <michael.widenius(a)gmail.com> > timestamp: 2021-03-24 15:03:04 +0200 > message: > > Revert MDEV-16592 "Change Item::with_sum_func to a virtual method" > > diff --git a/sql/item.cc b/sql/item.cc > index e5be8699d80..e0ba4ff2c89 100644 > --- a/sql/item.cc > +++ b/sql/item.cc > @@ -975,9 +975,7 @@ class Item :public Value_source, > void share_name_with(const Item *item) > { > name= item->name; > - common_flags= static_cast<uint8> > - ((common_flags & ~IS_AUTO_GENERATED_NAME) | > - (item->common_flags & IS_AUTO_GENERATED_NAME)); > + is_autogenerated_name= item->is_autogenerated_name; this, apparently, should be in the commit 039f4a054bb "Removed Item::common_flags and replaced it with bit fields" > } Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] 25fe7dbad5d: Reduce usage of strlen()
by Sergei Golubchik 31 Mar '21

31 Mar '21

Hi, Michael! On Mar 31, Michael Widenius wrote: > commit 25fe7dbad5d > Author: Michael Widenius <michael.widenius(a)gmail.com> > Date: Wed Aug 12 20:29:55 2020 +0300 > > Reduce usage of strlen() I'll only mention actual mistakes, no style comments or suggestions or question, just want to get it over with. > diff --git a/sql/item.h b/sql/item.h > index 9a6d0b4abfe..d6ff10bf3ba 100644 > --- a/sql/item.h > +++ b/sql/item.h > @@ -1093,7 +1093,7 @@ class Item :public Value_source, > void share_name_with(const Item *item) > { > name= item->name; > - copy_flags(item, item_base_t::IS_AUTOGENERATED_NAME); > + copy_flags(item, item_base_t::IS_EXPLICIT_NAME); belongs to a previous commit > } > virtual void cleanup(); > virtual void make_send_field(THD *thd, Send_field *field); > diff --git a/sql/item_windowfunc.cc b/sql/item_windowfunc.cc > index 262481d019b..a9e008b0b92 100644 > --- a/sql/item_windowfunc.cc > +++ b/sql/item_windowfunc.cc > @@ -99,13 +99,15 @@ Item_window_func::fix_fields(THD *thd, Item **ref) > > if (window_spec->window_frame && is_frame_prohibited()) > { > - my_error(ER_NOT_ALLOWED_WINDOW_FRAME, MYF(0), window_func()->func_name()); > + my_error(ER_NOT_ALLOWED_WINDOW_FRAME, MYF(0), > + window_func()->func_name()); This should be func_name().str, otherwise it'll crash or print garbage on Windows. Same below, in a couple of places. > return true; > } Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] dbb62d20f36: Fixes that enables my_new.cc (new wrapper using my_malloc)
by Sergei Golubchik 29 Mar '21

29 Mar '21

Hi, Michael! On Mar 28, Michael Widenius wrote: > revision-id: dbb62d20f36 (mariadb-10.5.2-518-gdbb62d20f36) > parent(s): 0c37211c2a2 > author: Michael Widenius <michael.widenius(a)gmail.com> > committer: Michael Widenius <michael.widenius(a)gmail.com> > timestamp: 2021-03-24 14:31:53 +0200 > message: > > Fixes that enables my_new.cc (new wrapper using my_malloc) > > This is not enabled by default as there are leaks in the > server that needs to be fixed first. One can compile > with -DREALLY_USE_MYSYS_NEW -DSF_REMEMBER_FRAMES=14 to find the > memory leaks from 'new' The change in my_new.cc is ok. The change in my_global.h and CMakeLists.txt is confusing and redundant. You unconditionaly define USE_MYSYS_NEW then conditionally undefine it again? Everything already worked just fine. By default cxx new is used, if one wants to use mysys version, it's done with -DHAVE_CXX_NEW=0 this works for me in 10.5 and fails to compile my_new.cc, as it should. Please, only keed your my_new.cc changes and say in the commit message "One can compile with -DHAVE_CXX_NEW=0 to find memory leaks from 'new'" As for -DSF_REMEMBER_FRAMES=14, perhaps it's better to do it automatically? Like #ifdef USE_MYSYS_NEW #define SF_REMEMBER_FRAMES 14 #else #define SF_REMEMBER_FRAMES 8 #endif or may be even just always make it 14. It should be easy to enable this feature, right? > diff --git a/CMakeLists.txt b/CMakeLists.txt > index a0b540f12d2..c51fb98abdc 100644 > --- a/CMakeLists.txt > +++ b/CMakeLists.txt > @@ -366,10 +366,8 @@ ENDIF() > # Run platform tests > INCLUDE(configure.cmake) > > -# force -DUSE_MYSYS_NEW unless already done by HAVE_CXX_NEW > -IF(NOT HAVE_CXX_NEW) > - ADD_DEFINITIONS(-DUSE_MYSYS_NEW) > -ENDIF() > +# Always use USE_MYSYS_NEW > +ADD_DEFINITIONS(-DUSE_MYSYS_NEW) > > # Find header files from the bundled libraries > # (wolfssl, readline, pcre2, etc) > diff --git a/include/my_global.h b/include/my_global.h > index e999e555bf7..6e3b48a00fd 100644 > --- a/include/my_global.h > +++ b/include/my_global.h > @@ -487,6 +487,9 @@ typedef unsigned short ushort; > duplicate declaration of __cxa_pure_virtual, solved by declaring it a > weak symbol. > */ > +#ifndef REALLY_USE_MYSYS_NEW > +#undef USE_MYSYS_NEW > +#endif > #if defined(USE_MYSYS_NEW) && ! defined(DONT_DECLARE_CXA_PURE_VIRTUAL) > C_MODE_START > int __cxa_pure_virtual () __attribute__ ((weak)); > diff --git a/mysys/my_static.c b/mysys/my_static.c > index 3c4b9efc1f8..96a23cdb283 100644 > --- a/mysys/my_static.c > +++ b/mysys/my_static.c > @@ -48,6 +48,7 @@ PSI_memory_key key_memory_my_err_head; > PSI_memory_key key_memory_my_file_info; > PSI_memory_key key_memory_pack_frm; > PSI_memory_key key_memory_charsets; > +PSI_memory_key key_memory_new; where do you initialize it? I'd suggest statically, like PSI_memory_key key_memory_new= PSI_INSTRUMENT_MEM; > > #ifdef _WIN32 > PSI_memory_key key_memory_win_SECURITY_ATTRIBUTES; Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 4

Re: [Maria-developers] 0c37211c2a2: Fixed my_addr_resolve
by Sergei Golubchik 28 Mar '21

28 Mar '21

Hi, Michael! On Mar 28, Michael Widenius wrote: > revision-id: 0c37211c2a2 (mariadb-10.5.2-517-g0c37211c2a2) > parent(s): 555c4470130 > author: Michael Widenius <michael.widenius(a)gmail.com> > committer: Michael Widenius <michael.widenius(a)gmail.com> > timestamp: 2021-03-24 14:31:53 +0200 > message: > > Fixed my_addr_resolve > > At least on openSUSE Leap 15.1 one should not use the dli_fbase offset > provided by dladdr() for addresses in the main program. > Without this patch the stack traces from sf_report_leaked_memory() > are not usable. This is very strange. First, the whole purpose of this code is to use dli_fbase offset for the main program. That's why it was added in the first place. PIE binaries can be loaded at any address in memory and the address you see with addr2line or nm or readelf is not the same address you'll see in gdb or in a strack trace. We have to subtract the base address first. But may be openSUSE is different? So, I built on openSUSE Leap and the stack trace was perfectly usable without your patch. I tried both `kill -ABRT` and an artificially introduced memory leak. So I suspect there was some misunderstanding and the problem was somewhere else (I cannot know what, of course, but just as an example - the build was bad and recompilation fixed it). Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 2

Re: [Maria-developers] ee538938345: MDEV-21117: refine the server binlog-based recovery for semisync
by Sergei Golubchik 24 Mar '21

24 Mar '21

Hi, Andrei! On Mar 24, Andrei Elkin wrote: > >> Here it goes. > >> > >> As after master crash and failover to slave, the demoted-to-slave > >> ex-master must be ready to face and accept its own (generated by) > >> events, without generally necessary --replicate-same-server-id. > >> > >> So the acceptance conditions are refined/relaxed for the semisync > >> slave connected to master in the GTID mode which ensures (under > >> gtid_strict_mode ON) that there can't be any duplicate events for > >> execution on such an ex-master slave. Non-GTID "binlog-truncated" > >> ex-master semisync slave has to be configured with > >> --replicate-same-server-id. > > > > Why would you generally care about same server id in the gtid mode? > > Maybe you did not care in above to the 'connection' word in 'the gtid > connection mode' which is about > > Change-Master to ... master_use_gtid= slave_pos > > So the "legacy" pre-gtid one corresponds to master_use_gtid= no. > [Thinking into future, it may stay for some more time actually to > serve for manual recovery of the slave.] As discussed on slack, let's now only support fully automatic recovery for the case of master_use_gtid=slave_pos. If one uses master_use_gtid=none the failover is non-trivial anyway. We can always add flipping of replicate-same-server-id later, if needed. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] ee538938345: MDEV-21117: refine the server binlog-based recovery for semisync
by Sergei Golubchik 23 Mar '21

23 Mar '21

Hi, Andrei! On Mar 22, Andrei Elkin wrote: > > Here it goes. > > As after master crash and failover to slave, the demoted-to-slave > ex-master must be ready to face and accept its own (generated by) > events, without generally necessary --replicate-same-server-id. > > So the acceptance conditions are refined/relaxed for the semisync slave > connected to master in the GTID mode which ensures (under gtid_strict_mode ON) > that there can't be any duplicate events for execution > on such an ex-master slave. > Non-GTID "binlog-truncated" ex-master semisync slave has to be > configured with --replicate-same-server-id. Why would you generally care about same server id in the gtid mode? If gtid is not present locally, then it should be applied, right? Why to look at server id at all? It's a thing of the past, from pre-gtid times. > >> >> +--let $query1 = INSERT INTO t VALUES (20) > >> >> +--let $query2 = DELETE FROM t2 WHERE f = 666 /* no such record */ > >> >> +--source binlog_truncate_active_log.inc > >> >> + > >> >> +--echo # Case B. > >> >> +# The inverted sequence ends up to truncate only $query2 > >> >> +--let $this_search_pattern = Successfully truncated.*to remove transactions starting from GTID 0-1-10 > >> >> +--let $query1 = DELETE FROM t2 WHERE f = 0 > >> > > >> > why not `= 666 /* no such record */` ? > >> > is `= 0` important here? > >> > >> Yes, there's a fine detail here. > >> > >> 'The inverted sequence ends up' to leave in binlog > >> > >> $query1 = DELETE FROM t2 WHERE f = 0 > >> > >> which is as ineffective as 666. The patch recognizes the ineffectiveness > >> through a special value of engines involved counter in Gtid event. > >> In principle the truncation could > >> start from it rather than the following $query2. > >> But that would mean more complicated coding. > >> So the semantics of the zero is to be good, as we're so used to :-)! > > > > Sorry, I didn't understand anything. What does the value in the WHERE > > clause has to do with engines and where the binlog is truncated? ok. after discussing the issue on slack, the answer appears to be "the value in WHERE has nothing to do with any of that, it doesn't matter whether it's 0 or 666 as long as it doesn't match any rows" then, I'd say, better change 0 to 667 in the test. We wouldn't be having this discussion at all if you'd have some number that appeared arbitrary (and "667" is fine). My confusion was exactly caused by the fact that it looked like =0 matters, that it was important to hav 0 there. > >> >> +--let $query2 = INSERT INTO t VALUES (20) > >> >> +--source binlog_truncate_active_log.inc > > ... > >> >> diff --git a/sql/sql_class.h b/sql/sql_class.h > >> >> index 140394fefc1..87fa88d4c89 100644 > >> >> --- a/sql/sql_class.h > >> >> +++ b/sql/sql_class.h > >> >> @@ -4632,7 +4632,7 @@ class THD :public Statement, > >> >> LF_PINS *tdc_hash_pins; > >> >> LF_PINS *xid_hash_pins; > >> >> bool fix_xid_hash_pins(); > >> >> - > >> >> + bool is_1pc_ro_trans; > >> > > >> > I don't like how it looks, it's in THD (for every single connection > >> > and more) and it's only used in some exotic cases. It's set in two > >> > places around ha_commit_one_phase(), when it would be better to set > >> > it inside ha_commit_one_phase(), in one place only. > >> > > >> > But I cannot suggest how to get rid of it yet as I don't understand > >> > what it's for, I've asked a question about it below, in > >> > Gtid_log_event::Gtid_log_event > >> > >> The flag is to help sorting out ineffective (no engine data change) > >> STATEMENT binlog-format logged transaction. I thought it's practical > >> enough to face and important to handle. > >> Through an example, > >> let the table `t' be initially empty, and > >> let binlog contain > >> > >> trx1: INSERT INTO t SET a = 1; /* slave acked, trx got committed in engine */ > >> trx2: INSERT INTO t SET a = 2; /* not acked, trx remains prepared in engine */ > >> trx3: DELETE FROM t WHERE a = 0; /* ineffective, does not have Xid event */ > >> > >> prior a crash with their states as commented. > >> The truncate trx should be trx2 as the trx3's status > >> suggests it's harmless for involving into > >> truncation. > >> However we have yet to distinguish it from unsafe (to truncate) > >> event group ("transaction") that also ends with COMMIT query not Xid event. > > > > Hmm. I see. But thd isn't a universal dumpster for passing > > arguments to the function, don't use it like that. > > Well, I had to resort to that. To explain, > > > Please, pass this "read-only transaction with no xid" flag to the > > Gtid_log_event constructor as an argument. Or recalculate it inside > > the constructor, it's not such an expensive thing to do. > > To pass a flag ensues adding an extra arg to > struct handlerton::commit > which I don't feel easy about considering the reason. yes, one more reason to remove binlog_hton. But for now, like with binlog_commit_by_xid, I'd say, call binlog_commit directly and make binlog_hton->commit a no-op. Like: --- a/sql/handler.cc +++ b/sql/handler.cc @@ -1935,6 +1935,7 @@ commit_one_phase_2(THD *thd, bool all, THD_TRANS *trans, if (ha_info) { + binlog_commit(ht, thd, all); for (; ha_info; ha_info= ha_info_next) { int err; > >> >> + if (!truncate_validated) > >> >> + { > >> >> + DBUG_ASSERT(round <= 2); > >> >> + /* > >> >> + Either truncate was not set or was reset, else > >> >> + it gets incremented, otherwise it may only set to an earlier > >> >> + offset only at the turn out of the 1st round. > >> > > >> > sorry, I cannot parse that :( > >> > >> The comments are for the assert in a branch that acts when the truncate position > >> (gtid) is not yet finalized. > >> Let me annotate under each assert's line. > >> > >> > > >> >> + */ > >> >> + DBUG_ASSERT(truncate_gtid.seq_no == 0 || > >> > >> 'truncate was not set or was reset' ^, that is truncate_gtid. > >> The 'was reset' is actually not present in the assert - that'd be > >> reflected with > >> truncate_reset_done == true > >> > >> > >> >> + last_gtid_coord >= binlog_truncate_coord || > >> > >> 'it gets incremented' ^ > > > > why does that mean "it gets incremented" ? > > I think my English failed me. > > It is getting incremented I understand that "it gets incremented" means "it = it + 1", no problem. How this assert verifies that "it gets incremented" ? > meant to say (in my last reply as well). > The use case is that at this point there's a truncate candidate /* 1 */ > and it has not been settled yet /* 2 */, > > !truncate_validated == true /* 2 */ && truncate_gtid.seq_no > 0 /* 1 */ > > and the current gtid is still "better" to take over as the candidate in > this block. The candidate's offset must not decrease (unless... > > >> >> + (round == 2 && truncate_set_in_1st)); > >> > >> And if it's neither of the above there must've been truncate candidate > >> set in the 1st round. > > > > ... > ... the above ). > > >> >> switch (typ) > >> >> { > >> >> + case FORMAT_DESCRIPTION_EVENT: > >> >> + if (round > 1) > >> >> + { > >> >> + if (fdle != fdle_arg) > >> >> + delete fdle; > >> >> + fdle= (Format_description_log_event*) ev; > >> > > >> > why is it needed? all binlog files should have the same > >> > Format_description_log_event anyway. > >> > >> Actually contrary to what was stated (the patch removed that claim) in > >> the parent revision of this commit, the FD:s of the > >> recovery binlog files may have different checksums. I.e > >> > >> set @@global.binlog_checksum = value > >> > >> can be issued to rotate binlog to a new hot file having different checksum value > >> while the binlog *CheckPoint* file is behind. So this is possible to > >> have at recovery: > >> > >> B_1(CP, checksum = NONE), B_2(hot, checksum = CRC32) > >> > >> which also manifests a normal use case bug I have not reported any bug (though wanted to). > > > > okay, checksum can differ. But how does it affect > > Format_description_log_event ? > > FD holds knowledge of whether events in the file are with the checksum > info. In my example above B_1 events would be (are) considered as checksummed > by the base. Okay. Please, 1. add a comment about it 2. add a test case for it ideally, also, move it into a separate commit Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] f1db957c5da: MDEV-21469: Implement crash-safe binary logging of the user XA
by Sergei Golubchik 22 Mar '21

22 Mar '21

Hi, Andrei! On Mar 16, Andrei Elkin wrote: > > On Mar 14, Sujatha wrote: > >> revision-id: f1db957c5da (mariadb-10.5.2-247-gf1db957c5da) > >> parent(s): dc92235f21e > >> author: Sujatha <sujatha.sivakumar(a)mariadb.com> > >> committer: Andrei Elkin <andrei.elkin(a)mariadb.com> > >> timestamp: 2020-12-21 16:10:46 +0200 > >> message: > >> > >> MDEV-21469: Implement crash-safe binary logging of the user XA ... > >> diff --git a/sql/log.cc b/sql/log.cc > >> index 731bb3e98f0..c69b8518cf4 100644 > >> --- a/sql/log.cc > >> +++ b/sql/log.cc > >> @@ -2019,28 +2027,49 @@ static int binlog_xa_recover_dummy(handlerton *hton __attribute__((unused)), > >> return 0; > >> } > >> > >> - > >> +/* > >> + The function invokes binlog_commit() and returns its result > >> + when it has not yet called it already. > >> + binlog_cache_mngr::completed_by_xid remembers the fact of > >> + the 1st of maximum two subsequent calls. > >> +*/ > >> static int binlog_commit_by_xid(handlerton *hton, XID *xid) > >> { > >> + int rc= 0; > >> THD *thd= current_thd; > >> - > >> - (void) thd->binlog_setup_trx_data(); > >> + binlog_cache_mngr *cache_mngr= thd->binlog_setup_trx_data(); > >> > >> DBUG_ASSERT(thd->lex->sql_command == SQLCOM_XA_COMMIT); > >> > >> - return binlog_commit(hton, thd, TRUE); > >> -} > >> + if (!cache_mngr->completed_by_xid) > > > > On what code path can you have completed_by_xid == false here? > > Let me answer broadly, the purpose of `completed_by_xid' is to > make sure binlog_hton->commit_by_xid() does not invoke lower level > binlog_commit() for the 2nd time at the following function. I understand that. And I mean that if commit_by_xid is always true, then binlog_hton->commit_by_xid can be a noop, because you always invoke binlog_commit_by_xid (or binlog_rollback_by_xid) explicitly. > I leave only two essential statements prepared with explanatory comments: > > ha_commit_or_rollback_by_xid() > { > // binlog_commit() has to be called as first hton ... > > binlog_hton->commit_by_xid(binlog_hton, xid); > > // ... So the above raises the flag (sets up a fence) to block binlog_hton > // run for the 2nd time in the loop here: > > plugin_foreach(NULL, commit ? xacommit_handlerton : xarollback_handlerton, > MYSQL_STORAGE_ENGINE_PLUGIN, &xaop); > > Now I am thinking a (much) better option exists to block that 2nd time invocation > of binlog_commit() with > merely testing the type of hton in > > static my_bool xacommit_handlerton(THD *unused1, plugin_ref plugin, void *arg) > { > handlerton *hton= plugin_hton(plugin); > - if (hton->recover) > + if (hton->recover && hton != binlog_hton) > { > hton->commit_by_xid(); > > and the same to xarollback_handlerton. > I am now committed to turn to it. No, see above. Ideally, binlog won't have a hton and it won't be in the list of engines at all. But if that's a too big step to make now, then just make binlog_hton->commit_by_xid noop (or NULL, if that works) and invoke binlog_commit_by_xid() explicitly by name. > >> + { > >> + rc= binlog_commit(hton, thd, TRUE); > >> + cache_mngr->completed_by_xid= true; > >> + } > >> > >> + return rc; > >> +} > >> > >> @@ -10333,14 +10386,108 @@ start_binlog_background_thread() ... > >> + > >> +/** > >> + Performs recovery based on transaction coordinator log for 2pc. At the > >> + time of crash, if the binary log was in active state, then recovery for > >> + "implicit" 'xid's and explicit 'XA' transactions is initiated, > >> + otherwise merely the gtid binlog state is updated. > >> + For 'xid' and 'XA' based recovery the following steps are performed. > >> + > >> + Identify the active binlog checkpoint file. > >> + Scan the binary log from the beginning. > >> + From GTID_LIST and GTID_EVENTs reconstruct the gtid binlog state. > >> + Prepare a list of 'xid's for recovery. > >> + Prepare a list of explicit 'XA' transactions for recovery. > >> + Recover the 'xid' transactions. > >> + The explicit 'XA' transaction recovery is initiated once all the server > >> + components are initialized. Please check 'execute_xa_for_recovery()'. > > > > why explicit XA recovery is delayed? > > It has to wait for at least until after > init_update_queries(); > in init_server_components() without which Better move init_update_queries() to happen before recovery. As far as I can see, init_update_queries() only sets flags in the array of commands, it does not depend on anything and can be done anytime. Ideally all the recovery could happen in one place, not split over two separate functions that are invoked at separate points in time during the startup. > >> @@ -10561,6 +10734,219 @@ int TC_LOG_BINLOG::recover(LOG_INFO *linfo, const char *last_log_name, > >> + { > >> + enable_apply_event= true; > >> + // partially engine-prepared XA is first cleaned out prior replay > >> + thd->lex->sql_command= SQLCOM_XA_ROLLBACK; > >> + ha_commit_or_rollback_by_xid(&gev->xid, 0); > >> + } > >> + else > >> + --recover_xa_count; > >> + } > >> + } > >> + else if (gev->flags2 & Gtid_log_event::FL_COMPLETED_XA) > >> + { > >> + if (member->state == XA_COMPLETE && > >> + member->in_engine_prepare > 0) > >> + enable_apply_event= true; > > > > why? you cannot replay a fully prepared and partially committed > > transaction > > We might make use of slave_exec_mode = IDEMPOTENT but only ROW format > events are subject to the mode. Yes, so why you're trying to replay it instead of just completing the commit? Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 269328884e5: MDEV-21365: Check $MARIADB_HOME/my.cnf in addition to $MYSQL_HOME/my.cnf
by Sergei Golubchik 22 Mar '21

22 Mar '21

Hi, Rucha! On Mar 22, Rucha Deodhar wrote: > revision-id: 269328884e5 (mariadb-10.5.2-269-g269328884e5) > parent(s): 46fab5b32a8 > author: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > committer: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > timestamp: 2021-03-22 16:18:18 +0530 > message: > > MDEV-21365: Check $MARIADB_HOME/my.cnf in addition to $MYSQL_HOME/my.cnf > > Currently for the server, we only check if $MYSQL_HOME is set. Added a check > if $MARIADB_HOME is set and try to read the configuration file from this > directory. > To keep behavior consistent with MariaDB Connector/C, checking $MARIADB_HOME > after $MYSQL_HOME. The patch is fine. But the commit comment says exactly the opposite of what you've implemented. You check MARIADB_HOME before MYSQL_HOME (which is correct), not after. Please fix the comment and ok to push. > diff --git a/mysys/my_default.c b/mysys/my_default.c > index c76ab8c6eb1..ec94aee290c 100644 > --- a/mysys/my_default.c > +++ b/mysys/my_default.c > @@ -1051,8 +1051,17 @@ static const char **init_default_directories(MEM_ROOT *alloc) > > #endif > > - if ((env= getenv("MYSQL_HOME"))) > + /* > + If value of $MARIADB_HOME environment variable name is NULL, check > + for $MYSQL_HOME > + */ > + if ((env= getenv("MARIADB_HOME"))) > errors += add_directory(alloc, env, dirs); > + else > + { > + if ((env= getenv("MYSQL_HOME"))) > + errors += add_directory(alloc, env, dirs); > + } > > /* Placeholder for --defaults-extra-file=<path> */ > errors += add_directory(alloc, "", dirs); > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] e5fc78f84e3: MDEV-20220: Merge 5.7 P_S replication table 'replication_applier_status_by_worker
by Sergei Golubchik 22 Mar '21

22 Mar '21

Hi, Sujatha! Could you split this patch, please? 1. Just add replication_applier_status_by_worker table. With CHANNEL_NAME column and (perhaps, not sure if it applies) with WORKER_ID column. Without extra columns and backup. 2. Add extra columns. 3. Add backup. With these three commits you'll have exactly the same diff as now, just split (and with CHANNEL_NAME column). But really, I wonder whether this backup functionality is needed at all? In MySQL there is persistent information about these workers, it doesn't go away when they're stopped, it's stored persistently in a table, indexed by worker_id. If we don't have anything persistent like that and all workers completely disappear into oblivion, then, may be, replication_applier_status_by_worker should not show anything when they aren't running? If you agree, then you'll only need two commits. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org On Mar 21, Sujatha wrote: > revision-id: e5fc78f84e3 (mariadb-10.5.2-303-ge5fc78f84e3) > parent(s): 8b8969929d7 > author: Sujatha <sujatha.sivakumar(a)mariadb.com> > committer: Sujatha <sujatha.sivakumar(a)mariadb.com> > timestamp: 2020-11-27 12:59:42 +0530 > message: > > MDEV-20220: Merge 5.7 P_S replication table 'replication_applier_status_by_worker > > Fix: > === > Iterate through rpl_parallel_thread_pool and display slave worker thread > specific information as part of 'replication_applier_status_by_worker' table. > > --------------------------------------------------------------------------------- > |Column Name: | Description: | > |-------------------------------------------------------------------------------| > | | | > |THREAD_ID | Thread_Id as displayed in 'performance_schema.threads'| > | | table for thread with name | > | | 'thread/sql/rpl_parallel_thread' | > | | | > | | THREAD_ID will be NULL when worker threads are stopped| > | | due to error/force stop | > | | | > |SERVICE_STATE | Thread is running or not | > | | | > |LAST_SEEN_TRANSACTION | Last GTID executed by worker | > | | | > |LAST_ERROR_NUMBER | Last Error that occurred on a particular worker | > | | | > |LAST_ERROR_MESSAGE | Last error specific message | > | | | > |LAST_ERROR_TIMESTAMP | Time stamp of last error | > | | | > |WORKER_IDLE_TIME | Total idle time in seconds that the worker thread has | > | | spent waiting for work from SQL thread | > | | | > |LAST_TRANS_RETRY_COUNT | Total number of retries attempted by last transaction | > --------------------------------------------------------------------------------- > > > In case STOP SLAVE is executed worker threads will be gone, hence worker > threads will be unavailable. Querying the table at this stage will give empty > rows. To address this case when worker threads are about to stop, due to an > error or forced stop, create a backup pool and preserve the data which is > relevant to populate performance schema table. Clear the backup pool upon > slave start.

2 4

Re: [Maria-developers] e5fc78f84e3: MDEV-20220: Merge 5.7 P_S replication table 'replication_applier_status_by_worker
by sujatha 22 Mar '21

22 Mar '21

Hello Andrei, Good Morning. On 22/03/21 3:50 pm, Andrei Elkin wrote: > Sujatha, howdy. > > Just one note from me at this point, > >> ==> https://dev.mysql.com/doc/refman/5.7/en/replication-channels.html >> >> A multi-source replica can also be set up as a multi-threaded >> replica, by setting the slave_parallel_workers system variable to a >> value greater than 0. When you do this on a multi-source replica, each >> channel on the replica has the specified number of applier >> threads, plus a coordinator thread to manage them. You cannot >> configure the number of applier threads for individual channels. >> >> In case of MySQL, worker threads are dedicated to particular channel. >> >> In MariaDB "The pool of replication worker threads is shared among all >> multi-source master connections, and among all replication domains >> that can replicate in parallel using out-of-order". >> >> Because of this I didn't include CHANNEL_NAME. In MariaDB Slave_worker >> thread's don't have WORKER_ID they only have 'thread_id'. > Would it make sense to report > > LAST_CHANNEL (or LAST_SOURCE) similar to other 'LAST_' like > > |LAST_SEEN_TRANSACTION | Last GTID executed by worker | > > ? Yes, as 'LAST_SOURCE', we can display 'mi->connection_name'. In MariaDB documentation 'CONNECTION_NAME' is used to identify a source, so I used 'CONNECTION_NAME' instead of 'CHANNEL_NAME'(MySQL'). Thank you. S.Sujatha > >>>> Fix: >>>> === >>>> Iterate through rpl_parallel_thread_pool and display slave worker thread >>>> specific information as part of 'replication_applier_status_by_worker' table. >>>> >>>> --------------------------------------------------------------------------------- >>>> |Column Name: | Description: | >>>> |-------------------------------------------------------------------------------| >>>> | | | >>>> |THREAD_ID | Thread_Id as displayed in 'performance_schema.threads'| >>>> | | table for thread with name | >>>> | | 'thread/sql/rpl_parallel_thread' | >>>> | | | >>>> | | THREAD_ID will be NULL when worker threads are stopped| >>>> | | due to error/force stop | >>>> | | | >>>> |SERVICE_STATE | Thread is running or not | >>>> | | | >>>> |LAST_SEEN_TRANSACTION | Last GTID executed by worker | >>>> | | | >>>> |LAST_ERROR_NUMBER | Last Error that occurred on a particular worker | >>>> | | | >>>> |LAST_ERROR_MESSAGE | Last error specific message | >>>> | | | >>>> |LAST_ERROR_TIMESTAMP | Time stamp of last error | >>>> | | | >>>> |WORKER_IDLE_TIME | Total idle time in seconds that the worker thread has | >>>> | | spent waiting for work from SQL thread | >>>> | | | >>>> |LAST_TRANS_RETRY_COUNT | Total number of retries attempted by last transaction | >>>> --------------------------------------------------------------------------------- > Cheers, > > Andrei

1 0

Re: [Maria-developers] d86fe72c027: MDEV-22010: use executables MariaDB named in scripts
by Sergei Golubchik 21 Mar '21

21 Mar '21

Hi, Rucha! On Mar 21, Rucha Deodhar wrote: > revision-id: d86fe72c027 (mariadb-10.5.2-386-gd86fe72c027) > parent(s): de407e7cb4d > author: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > committer: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > timestamp: 2021-03-18 00:58:43 +0530 > message: > > MDEV-22010: use executables MariaDB named in scripts > > As a part of this MDEV following changes were made: > 1) Mariadb named executables used instead of mysql named executables in scripts > 2) renamed mysql-test-run and mysql-stress-test to mariadb-test-run and > mariadb-stress-test and created a symlink. > > diff --git a/.gitignore b/.gitignore > index d58dd2aa2b8..1a840d1a51d 100644 > --- a/.gitignore > +++ b/.gitignore > @@ -600,3 +600,7 @@ tests/mariadb-client-test > versioninfo_dll.rc > versioninfo_exe.rc > win/packaging/ca/symlinks.cc > +mysql-test/mysql-stress-test.pl > +mysql-test/mysql-test-run > +mysql-test/mysql-test-run.pl > +mysql-test/mtr Please, add these lines together with other mysql-test/ lines in the file. Then you'll see that mysql-test-run and mtr are already ignored. > diff --git a/scripts/mysql_install_db.sh b/scripts/mysql_install_db.sh > index 5f183afe8fc..147bd7351bc 100644 > --- a/scripts/mysql_install_db.sh > +++ b/scripts/mysql_install_db.sh > @@ -338,11 +338,11 @@ then > cannot_find_file resolveip @resolveip_locations@ > exit 1 > fi > - mysqld=`find_in_dirs mysqld @mysqld_locations@` > + mysqld=`find_in_dirs mariadbd @mysqld_locations@` > if test -z "$mysqld" > then > - cannot_find_file mysqld @mysqld_locations@ > - exit 1 > + cannot_find_file mysqld @mysqld_locations@ you wanted to say "cannot_find_file mariadbd @mysqld_locations@" > + exit 1 > fi > langdir=`find_in_dirs --dir errmsg.sys @errmsg_locations@` > if test -z "$langdir" > @@ -579,7 +579,7 @@ else > echo > echo " shell> $mysqld --skip-grant-tables --general-log &" > echo > - echo "and use the command line tool $bindir/mysql" > + echo "and use the command line tool $bindir/mariadb or $bindir/mysql" I don't think you need any "or mysql" here. > echo "to connect to the mysql database and look at the grant tables:" > echo > echo " shell> $bindir/mysql -u root mysql" > @@ -613,7 +613,8 @@ then > echo "PLEASE REMEMBER TO SET A PASSWORD FOR THE MariaDB root USER !" > echo "To do so, start the server, then issue the following commands:" > echo > - echo "'$bindir/mysqladmin' -u root password 'new-password'" > + echo "'$bindir/mariadb-admin' -u root password 'new-password' or" > + echo "'$bindir/mariadb-admin' -u root -h $hostname password 'new-password' or" and not here. > echo "'$bindir/mysqladmin' -u root -h $hostname password 'new-password'" > echo > echo "Alternatively you can run:" > diff --git a/scripts/mysql_secure_installation.sh b/scripts/mysql_secure_installation.sh > index b2a9edf4953..6098e0148b5 100644 > --- a/scripts/mysql_secure_installation.sh > +++ b/scripts/mysql_secure_installation.sh > @@ -159,15 +159,15 @@ then > cannot_find_file my_print_defaults $basedir/bin $basedir/extra > exit 1 > fi > - mysql_command=`find_in_basedir mysql bin` > + mysql_command=`find_in_basedir mariadb bin` > if test -z "$mysql_command" > then > - cannot_find_file mysql $basedir/bin > - exit 1 > + cannot_find_file mysql $basedir/bin should be cannot_find_file mariadb $basedir/bin" > + exit 1 > fi > else > print_defaults="@bindir@/my_print_defaults" > - mysql_command="@bindir@/mysql" > + mysql_command="@bindir@/mariadb" > fi > > if test ! -x "$print_defaults" > diff --git a/scripts/mysqld_multi.sh b/scripts/mysqld_multi.sh > index 0b33f61710e..d83663e03fc 100644 > --- a/scripts/mysqld_multi.sh > +++ b/scripts/mysqld_multi.sh > @@ -728,7 +728,7 @@ sub example > > [mysqld_multi] > mysqld = @bindir@/mysqld_safe mariadbd-safe ? (here and everywhere in this file) Also, you've missed mysqld_safe in FIND_PROC set in scripts/CMakeLists.txt > -mysqladmin = @bindir@/mysqladmin > +mysqladmin = @bindir@/mariadb-admin > user = multi_admin > password = my_password > > diff --git a/scripts/mysqld_safe.sh b/scripts/mysqld_safe.sh > index 126eb924814..7763fb9db91 100644 > --- a/scripts/mysqld_safe.sh > +++ b/scripts/mysqld_safe.sh > @@ -531,7 +531,11 @@ else > ledir='@libexecdir@' > fi > > -helper=`find_in_bin mysqld_safe_helper` > +helper=`find_in_bin mariadbd-safe-helper` > +if test -x helper > +then > + helper=`find_in_bin mysqld_safe_helper` > +fi no, remove the if, no need to fallback (and you've got the condition wrong anyway) > print_defaults=`find_in_bin my_print_defaults` > # Check if helper exists > command -v $helper --help >/dev/null 2>&1 > diff --git a/scripts/wsrep_sst_common.sh b/scripts/wsrep_sst_common.sh > index 5e134570881..b8621a76d6f 100644 > --- a/scripts/wsrep_sst_common.sh > +++ b/scripts/wsrep_sst_common.sh > @@ -273,16 +273,16 @@ SCRIPTS_DIR="$(cd $(dirname "$0"); pwd -P)" > EXTRA_DIR="$SCRIPTS_DIR/../extra" > CLIENT_DIR="$SCRIPTS_DIR/../client" > > -if [ -x "$CLIENT_DIR/mysql" ]; then > - MYSQL_CLIENT="$CLIENT_DIR/mysql" > +if [-x "$CLIENT_DIR/mariadb"]; then "[" is a name of an executable, always leave a space after it. there's no [-x executable. > + MYSQL_CLIENT="$CLIENT_DIR/mariadb" > else > - MYSQL_CLIENT=mysql > + MYSQL_CLIENT=mariadb > fi Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 2858f9fb7e0: MDEV-22189: Change error messages inside code to have mariadb instead of
by Sergei Golubchik 21 Mar '21

21 Mar '21

Hi, Rucha! First, please check the buildbot and make sure all tests pass. You've changed MySQL->MariaDB in result files, but without corresponding changes in the code your result files now differ from the actual test output. One more comment below On Mar 21, Rucha Deodhar wrote: > revision-id: 2858f9fb7e0 (mariadb-10.5.2-274-g2858f9fb7e0) > parent(s): 41485551c58 > author: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > committer: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > timestamp: 2021-03-18 13:58:34 +0530 > message: > > MDEV-22189: Change error messages inside code to have mariadb instead of > mysql > > Fix: Changed error messages, rerecorded results and changed other relevant > files. > > diff --git a/debian/additions/innotop/innotop b/debian/additions/innotop/innotop > index 19229a57a82..a045a479be9 100644 > --- a/debian/additions/innotop/innotop > +++ b/debian/additions/innotop/innotop > @@ -987,12 +987,12 @@ sub parse_tx_text { > # Parsing the line that begins 'MySQL thread id' is complicated. The only > # thing always in the line is the thread and query id. See function > # innobase_mysql_print_thd in InnoDB source file sql/ha_innodb.cc. > - my ( $thread_line ) = $txn =~ m/^(MySQL thread id .*)$/m; > + my ( $thread_line ) = $txn =~ m/^(MariaDB thread id .*)$/m; > my ( $mysql_thread_id, $query_id, $hostname, $ip, $user, $query_status ); > > if ( $thread_line ) { > # These parts can always be gotten. > - ( $mysql_thread_id, $query_id ) = $thread_line =~ m/^MySQL thread id $d, .*?query id $d/m; > + ( $mysql_thread_id, $query_id ) = $thread_line =~ m/^MariaDB thread id $d, .*?query id $d/m; > > # If it's a master/slave thread, "Has (read|sent) all" may be the thread's > # proc_info. In these cases, there won't be any host/ip/user info > @@ -4144,15 +4144,15 @@ my $innodb_parser = InnoDBParser->new; > > my $nonfatal_errs = join('|', > 'Access denied for user', > - 'Unknown MySQL server host', > + 'Unknown MariaDB server host', > 'Unknown database', > - 'Can\'t connect to local MySQL server through socket', > - 'Can\'t connect to MySQL server on', > + 'Can\'t connect to local MariaDB server through socket', > + 'Can\'t connect to MariaDB server on', > 'MySQL server has gone away', > 'Cannot call SHOW INNODB STATUS', > 'Access denied', > 'AutoCommit', > - 'Lost connection to MySQL server', > + 'Lost connection to MariaDB server', > 'Too many connections', > ); innotop is generally a separate tool that technically could work both with MariaDB and MySQL. So, better use (?:MariaDB|MySQL) in all regexp patterns above. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] MDEV-13467 review.
by Alexey Botchkov 20 Mar '21

20 Mar '21

One thing i'd still like you to do differently. + if (g->get_class_info()->m_type_id == Geometry::wkb_multipoint) + { + const char point_size= 4 + WKB_HEADER_SIZE + POINT_DATA_SIZE+1; //1 for the type + char point_temp[point_size]; + memset(point_temp+4, Geometry::wkb_point, 1); + memcpy(point_temp+5, static_cast<const Gis_multi_point *>(g)->get_data_ptr()+5, 4); + memcpy(point_temp+4+WKB_HEADER_SIZE, g->get_data_ptr()+4+WKB_ HEADER_SIZE, + POINT_DATA_SIZE); + point_temp[point_size-1]= '\0'; + Geometry_buffer gbuff; + Geometry *gg= Geometry::construct(&gbuff, point_temp, point_size-1); > Gis_multi_point::geometry_n(uint32 num, String *result) stores the point in String *, > but I need Geometry* to use further, so still would need some recalculation. You could call the ::construct with the string->prr() instead of point_temp. Like str->q_append(0); //SRID g->geometry_n(str, 1); gg= Geometry::construct(&gbuff, str->ptr(), str->length()); But I propose an even simpler method. { Gis_pont p; p.set_data_ptr(g->get_data_ptr()+4+WKB_HEADER_SIZE, POINT_DATA_SIZE); calculate_haversine(&p, radius); } > Regarding radius, Daniel has suggestion: when r=NULL, result= NULL, but r=0, > how should that be treated? > Here is MySQL example (no error, but warning) There i see that MySQL returns the error when the radius is 0 or below. Right? I think it's ok for us to do same. > In a single function we are calculating radian for x,y, I think it is faster than to call function 2 times for x and y. I doubt so. Really would like to compare the speed here. Still i'm ok with that single function if you like it better. Best regards. HF On Tue, Mar 2, 2021 at 1:16 PM Anel Husakovic <anel(a)mariadb.org> wrote: > Hi Alexey, > thanks for your review, hope everything is ok now. > Here is the new patch 1f618e36577a2a322807 > <https://github.com/MariaDB/server/commit/1f618e36577a2a32280705c640d6f9a748…> according > to your review and comments below. > > On Sat, Jan 9, 2021 at 2:09 PM Alexey Botchkov <holyfoot(a)mariadb.com> > wrote: > >> Hi, Anel! >> >> Sorry for that long delay with this review. Had real difficult weeks of >> my life :( >> Now it seems to be over, so getting back to normal. >> Below is my opinions of the code i see in the branch >> bb-10.1-anel-MDEV-13467-gis-feature-st_sphere-v2 >> Hope it's the recent version. >> >> + Adapter for functions that takes two or three arguments. >> +*/ >> + >> +class Create_func_arg2_or_3 : public Create_func >> +{ >> Do you think we need the separate Create_fund_arg2_or_3 class? >> If you don't see any other class to inherit from it, >> i'd recommend to leave the Create_func_distance_sphere alone. >> No need to have two constructors in the Item_func_sphere_distance >> too. Just use the list argument as it's done for the >> Item_func_json_contains (in 10.2). >> > > Ack ^ droped the Create_func_arg2_or_3 and inherited from > Create_native_function which can use a variable number of parameters. > Thanks > > >> Anyway i belive we should get rid of duplicating code >> about the 'param1' and 'param2'. They can be popped/checked >> with no if (arg_count == 2) condition. >> >> + double distance= 0.0; >> + null_value= (args[0]->null_value || args[1]->null_value); >> + if (null_value) >> + { >> + // Returned item must be set to Null >> + DBUG_ASSERT(maybe_null); >> don't think it's needed. >> + return 0; >> Should be return 0.0 i belive. Or goto handle_errors. >> Thet goes to all but one 'return' statements in this function. >> >> + } >> >> > ... >> > if (!arg1 || !arg2) >> >> This last condition should never happen since we have not-null_value. >> So i'd remove that 'if' section completely. >> And anyway it should do the 'return' in that branch. >> >> + null_value= args[2]->null_value; >> + if (null_value) >> + { >> + return 0; >> + } >> >> seems nicer to me this way >> if (args[2]->null_value) >> { >> null_value= true; >> goto handle_errors; >> } >> >> > Ack. > Regarding radius, Daniel has suggestion > <https://github.com/MariaDB/server/commit/e42b69136a31cb596e940d1ceafffa5c11…>: > when r=NULL, result= NULL, but r=0, how should that be treated? > Here is MySQL > <https://dbfiddle.uk/?rdbms=mysql_8.0&fiddle=d467903dce23464df0ebf396220da886>example > (no error, but warning) > > > >> + if (!(g1= Geometry::construct(&buffer1, arg1->ptr(), arg1->length())) >> || >> + !(g2= Geometry::construct(&buffer2, arg2->ptr(), arg2->length()))) >> + { >> + goto handle_errors; >> >> we should launch the ER_GIS_INVALID_DATA here. >> >> > ER_GIS_INVALID_DATA is introduced in 10.2, I will change this (and test > case) when pushing to 10.2, but I would also push to 10.1 since there may > be users using it in 10.1. > Also when inspecting this error in 10.2 noted that lot tests have > commented this error, so will remove that comment in the same commit. > > mysql-test/suite/innodb_gis/t/gis.test > > mysql-test/suite/innodb_gis/t/1.test > > mysql-test/suite/innodb_gis/t/bug16236208.test > > mysql-test/suite/innodb_gis/t/precise.test > > mysql-test/suite/innodb_gis/t/rtree.test > > > + // Generate error message in case different geometry is used? >> Yes, i think the explainig message here is good to have. >> >> +double Item_func_sphere_distance::spherical_distance(Geometry *g1, >> + Geometry *g2, >> + const double >> sphere_radius) >> +double Item_func_sphere_distance::spherical_distance_points(Geometry *g1, >> + Geometry *g2, >> + const double >> r) >> Why these two are member static functions, not simply static? >> If you plan any 'external' use for them, please add comments about >> what exacly they do. >> >> > Changed function to be member function only. > > > >> +double Item_func_sphere_distance::spherical_distance(Geometry *g1, >> + Geometry *g2, >> + const double >> sphere_radius) >> +{ >> + double res= 0.0; >> + switch (g1->get_class_info()->m_type_id) >> + { >> + case Geometry::wkb_point: >> + res= spherical_distance_points(g1, g2, sphere_radius); >> + break; >> + >> + case Geometry::wkb_multipoint: >> + res= spherical_distance_points(g1, g2, sphere_radius); >> + break; >> >> Why do we need this function? >> We confirmed already that the type is either wkb_point or >> wkb_multipolygon, >> so just can call spherical_distance_points. No? >> >> > Indeed, left from earlier implementation (v1). > > >> >>>>> >> >> >> +class Item_func_sphere_distance: public Item_real_func >> +{ >> + double sphere_radius= 6370986.0; // Default radius equals Earth radius >> >> Some compilers don't like this syntax, and we didn't use it yet, so >> i'd recomment to move this to the constructor or even better to the >> ::val_real(). >> >> +double Gis_point::calculate_haversine(const Geometry *g, >> + const double sphere_radius) >> +{ >> + DBUG_ASSERT(sphere_radius > 0); >> + double x1r= 0.0; >> + double x2r= 0.0; >> + double y1r= 0.0; >> + double y2r= 0.0; >> That lot of unnecessary initializations seems excessive. >> >> Ack > > >> + if (g->get_class_info()->m_type_id == Geometry::wkb_multipoint) >> + { >> + const char point_size= 4 + WKB_HEADER_SIZE + POINT_DATA_SIZE+1; //1 >> for the type >> + char point_temp[point_size]; >> + memset(point_temp+4, Geometry::wkb_point, 1); >> + memcpy(point_temp+5, static_cast<const Gis_multi_point >> *>(g)->get_data_ptr()+5, 4); >> + memcpy(point_temp+4+WKB_HEADER_SIZE, >> g->get_data_ptr()+4+WKB_HEADER_SIZE, >> + POINT_DATA_SIZE); >> + point_temp[point_size-1]= '\0'; >> + Geometry_buffer gbuff; >> + Geometry *gg= Geometry::construct(&gbuff, point_temp, point_size-1); >> >> There is the Gis_multi_point::geometry_n() function, should be used here >> to get the point. >> >> Gis_multi_point::geometry_n(uint32 num, String *result) stores the point > in String *, but I need Geometry* to use further, so still would need some > recalculation. > > I didn’t find a multipoint API to convert String * to Geometry* so I did > my own calculation after debugging from row data. > > > I don't like that lot of static_cast-s. >> Isn't it nicer to have just the function of >> double calculate_harvesine_points(Geometry *p1, Geometry *p2, double >> radius) >> double calculate_harvesine_multipoint_point(Geometry *mp, Geometry *p, >> double radius) >> double calculate_harvesine_multipoint_multipoint(Geometry *mp1, Geometry >> *mp2, double radius) >> which would call each other? >> >> > I would prefer to have 2 static member functions and use static cast where > needed, if you don't mind. > The same could be done with 3 functions but with the code repeated. > > >> The ::get_xy_radian() method looks unneeded to me. >> It's enough to have the to_radian(double d) function. >> >> > In a single function we are calculating radian for x,y, I think it is > faster than to call function 2 times for x and y. > > >> BTW it's possible to do the multipoint/multipoint calculations faster >> than n^2. >> Not necessary to do it right now, just not to remember. >> >> > Regarding this there are some implementations that don't use Multipoints > <https://postgis.net/docs/manual-1.4/ST_Distance_Sphere.html> as both > arguments, MySQL > <https://dev.mysql.com/doc/refman/8.0/en/spatial-convenience-functions.html>also > use only Multipoints,Points combination, not MultiPoints,MultiPoints, so we > need to decide what to support. > However, there are algorithms to calculate max > <http://citeseerx.ist.psu.edu/viewdoc/download;jsessionid=CA2AE9B932194B2970…> > ,min <https://core.ac.uk/download/pdf/82507217.pdf> distance and > O(nlong) using convex hull <https://en.wikipedia.org/wiki/Convex_hull> > (which we have implemented <https://mariadb.com/kb/en/st_convexhull/>) > and using rotating calipers > <https://en.wikipedia.org/wiki/Rotating_calipers> from N points from > convex hull it could be O(n). > > Also there is suggested usage of iteration > <https://github.com/MariaDB/server/commit/e42b69136a31cb596e940d1ceafffa5c11…>from > Daniel. > > > <https://github.com/MariaDB/server/commit/e42b69136a31cb596e940d1ceafffa5c11…> > > > >> Best regards. >> HF >> >> > > -- > *Anel Husaković, * > *Software developer* > *MariaDB Foundation* >

1 0

Re: [Maria-developers] 0fc221a012d: MDEV-8334: Rename utf8 to utf8mb3
by Sergei Golubchik 19 Mar '21

19 Mar '21

Hi, Rucha! Thanks for combining all commits in one and a cleanup. See comments below. On Mar 18, Rucha Deodhar wrote: > revision-id: 0fc221a012d (mariadb-10.5.2-402-g0fc221a012d) > parent(s): a1542f8a573 > author: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > committer: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > timestamp: 2021-03-16 23:31:12 +0530 > message: > > MDEV-8334: Rename utf8 to utf8mb3 > > This patch changes the main name of 3 byte character set from utf8 to > utf8mb3. New old_mode UTF8_IS_UTF8MB3 is added and set TRUE by default, > so that utf8 would mean utf8mb3. If not set, utf8 would mean utf8mb4. > diff --git a/client/mysqlcheck.c b/client/mysqlcheck.c > index fb3103a318d..a8990d8cb6b 100644 > --- a/client/mysqlcheck.c > +++ b/client/mysqlcheck.c > @@ -437,7 +437,7 @@ static int get_options(int *argc, char ***argv) > if (!default_charset) > { > if (opt_fix_db_names || opt_fix_table_names) > - default_charset= (char*) "utf8"; > + default_charset= (char*) "utf8mb3"; why not to keep it utf8? > else > default_charset= (char*) MYSQL_AUTODETECT_CHARSET_NAME; > } > diff --git a/client/mysqldump.c b/client/mysqldump.c > index 7c363973da2..900456b31b2 100644 > --- a/client/mysqldump.c > +++ b/client/mysqldump.c > @@ -3235,7 +3235,7 @@ static uint get_table_structure(const char *table, const char *db, char *table_t > { > fprintf(sql_file, > "/*!40101 SET @saved_cs_client = @@character_set_client */;\n" > - "/*!40101 SET character_set_client = utf8 */;\n" > + "/*!40101 SET character_set_client = utf8mb3 */;\n" why not to keep it utf8? > "%s%s;\n" > "/*!40101 SET character_set_client = @saved_cs_client */;\n", > is_log_table ? "CREATE TABLE IF NOT EXISTS " : "", > diff --git a/extra/mariabackup/backup_mysql.cc b/extra/mariabackup/backup_mysql.cc > index 3083326a7e0..c62252257b9 100644 > --- a/extra/mariabackup/backup_mysql.cc > +++ b/extra/mariabackup/backup_mysql.cc > @@ -117,7 +117,7 @@ xb_mysql_connect() > mysql_options(connection, MYSQL_PLUGIN_DIR, xb_plugin_dir); > } > mysql_options(connection, MYSQL_OPT_PROTOCOL, &opt_protocol); > - mysql_options(connection,MYSQL_SET_CHARSET_NAME, "utf8"); > + mysql_options(connection,MYSQL_SET_CHARSET_NAME, "utf8mb3"); why not to keep it utf8? > > msg("Connecting to MySQL server host: %s, user: %s, password: %s, " > "port: %s, socket: %s", opt_host ? opt_host : "localhost", > @@ -1506,7 +1506,7 @@ write_xtrabackup_info(MYSQL *connection, const char * filename, bool history, > "incremental ENUM('Y', 'N') DEFAULT NULL," > "format ENUM('file', 'tar', 'xbstream') DEFAULT NULL," > "compressed ENUM('Y', 'N') DEFAULT NULL" > - ") CHARACTER SET utf8 ENGINE=innodb", false); > + ") CHARACTER SET utf8mb3 ENGINE=innodb", false); why not to keep it utf8? > > > #define ESCAPE_BOOL(expr) ((expr)?"'Y'":"'N'") > diff --git a/extra/mariabackup/xtrabackup.cc b/extra/mariabackup/xtrabackup.cc > index 62fdace654d..0f500c35ff6 100644 > --- a/extra/mariabackup/xtrabackup.cc > +++ b/extra/mariabackup/xtrabackup.cc > @@ -1716,7 +1716,7 @@ static int create_bootstrap_file() > if(!f) > return -1; > > - fputs("SET NAMES UTF8;\n",f); > + fputs("SET NAMES UTF8MB3;\n",f); why not to keep it utf8? > enumerate_ibd_files(append_export_table); > for (std::set<std::string>::iterator it = tables_for_export.begin(); > it != tables_for_export.end(); it++) > diff --git a/mysql-test/main/create-uca.test b/mysql-test/main/create-uca.test > index 0acb51f7286..f73f6114962 100644 > --- a/mysql-test/main/create-uca.test > +++ b/mysql-test/main/create-uca.test > @@ -1,5 +1,5 @@ > # Prerequisites > -let collation=utf8_unicode_ci; > +let collation=utf8mb3_unicode_ci; that's fine, as we generally don't want tests to depend on the old_mode. > --source include/have_collation.inc > > # Initial cleanup > diff --git a/mysql-test/main/ctype_ldml.result b/mysql-test/main/ctype_ldml.result > index 22b7a316111..7c284520733 100644 > --- a/mysql-test/main/ctype_ldml.result > +++ b/mysql-test/main/ctype_ldml.result > @@ -8,7 +8,6 @@ Variable_name Value > character_sets_dir MYSQL_TEST_DIR/std_data/ldml/ > show collation like 'utf8_phone_ci'; > Collation Charset Id Default Compiled Sortlen > -utf8_phone_ci utf8 352 8 I suppose the test should show there is a _phone_ci collation. As you've renamed it, you need to adjust the test to do show collation like 'utf8mb3_phone_ci'; > CREATE TABLE t1 ( > name VARCHAR(64), > phone VARCHAR(64) CHARACTER SET utf8 COLLATE utf8_phone_ci > @@ -37,7 +36,6 @@ Bar +7-912-800-80-01 > DROP TABLE t1; > show collation like 'utf8_test_ci'; > Collation Charset Id Default Compiled Sortlen > -utf8_test_ci utf8 353 8 same > create table t1 (c1 char(1) character set utf8 collate utf8_test_ci); > insert into t1 values ('a'); > select * from t1 where c1='b'; > @@ -526,7 +524,6 @@ DROP TABLE t1; > SET NAMES utf8 COLLATE utf8_phone_ci; > SHOW COLLATION LIKE 'utf8_phone_ci'; > Collation Charset Id Default Compiled Sortlen > -utf8_phone_ci utf8 352 8 and here > SET NAMES utf8; > SELECT hex(weight_string(_utf8mb4'a' collate utf8mb4_test_400_ci)); > hex(weight_string(_utf8mb4'a' collate utf8mb4_test_400_ci)) > diff --git a/mysql-test/main/show_check.result b/mysql-test/main/show_check.result > index d031c792922..1c85ef596c9 100644 > --- a/mysql-test/main/show_check.result > +++ b/mysql-test/main/show_check.result > @@ -874,12 +874,11 @@ set names utf8; > ---------------------------------------------------------------- > SHOW CHARACTER SET LIKE 'utf8'; > Catalog Database Table Table_alias Column Column_alias Type Length Max length Is_null Flags Decimals Charsetnr > -def information_schema CHARACTER_SETS CHARACTER_SETS CHARACTER_SET_NAME Charset 253 96 4 N 1 0 33 > -def information_schema CHARACTER_SETS CHARACTER_SETS DESCRIPTION Description 253 180 13 N 1 0 33 > -def information_schema CHARACTER_SETS CHARACTER_SETS DEFAULT_COLLATE_NAME Default collation 253 96 15 N 1 0 33 > -def information_schema CHARACTER_SETS CHARACTER_SETS MAXLEN Maxlen 8 3 1 N 32769 0 63 > +def information_schema CHARACTER_SETS CHARACTER_SETS CHARACTER_SET_NAME Charset 253 96 0 N 1 0 33 > +def information_schema CHARACTER_SETS CHARACTER_SETS DESCRIPTION Description 253 180 0 N 1 0 33 > +def information_schema CHARACTER_SETS CHARACTER_SETS DEFAULT_COLLATE_NAME Default collation 253 96 0 N 1 0 33 > +def information_schema CHARACTER_SETS CHARACTER_SETS MAXLEN Maxlen 8 3 0 N 32769 0 63 > Charset Description Default collation Maxlen > -utf8 UTF-8 Unicode utf8_general_ci 3 again, I suspect this test should now do `SHOW CHARACTER SET LIKE 'utf8mb3'; > ---------------------------------------------------------------- > SHOW COLLATION LIKE 'latin1_bin'; > Catalog Database Table Table_alias Column Column_alias Type Length Max length Is_null Flags Decimals Charsetnr > diff --git a/mysql-test/suite/funcs_1/r/charset_collation.result b/mysql-test/suite/funcs_1/r/charset_collation.result > index 31bd30c5acf..6b52e80d6ba 100644 > --- a/mysql-test/suite/funcs_1/r/charset_collation.result > +++ b/mysql-test/suite/funcs_1/r/charset_collation.result > @@ -9,7 +9,6 @@ ORDER BY character_set_name; > CHARACTER_SET_NAME DEFAULT_COLLATE_NAME DESCRIPTION MAXLEN > binary binary Binary pseudo charset 1 > latin1 latin1_swedish_ci cp1252 West European 1 > -utf8 utf8_general_ci UTF-8 Unicode 3 and here too, changing the test is in order > > SELECT * > FROM information_schema.collations > diff --git a/mysql-test/suite/funcs_1/r/is_column_privileges.result b/mysql-test/suite/funcs_1/r/is_column_privileges.result > index b6be9118048..46b2d515041 100644 > --- a/mysql-test/suite/funcs_1/r/is_column_privileges.result > +++ b/mysql-test/suite/funcs_1/r/is_column_privileges.result > @@ -45,7 +45,7 @@ COLUMN_PRIVILEGES CREATE TEMPORARY TABLE `COLUMN_PRIVILEGES` ( > `COLUMN_NAME` varchar(64) NOT NULL DEFAULT '', > `PRIVILEGE_TYPE` varchar(64) NOT NULL DEFAULT '', > `IS_GRANTABLE` varchar(3) NOT NULL DEFAULT '' > -) ENGINE=MEMORY DEFAULT CHARSET=utf8 > +) ENGINE=MEMORY DEFAULT CHARSET=utf8mb3 Just a thought. Did you also fix all --embedded, all --ps, and all --big --big tests to pass? > SHOW COLUMNS FROM information_schema.COLUMN_PRIVILEGES; > Field Type Null Key Default Extra > GRANTEE varchar(190) NO > diff --git a/mysql-test/suite/innodb/r/innodb_ctype_ldml.result b/mysql-test/suite/innodb/r/innodb_ctype_ldml.result > index 502f57156c3..8a96c023134 100644 > --- a/mysql-test/suite/innodb/r/innodb_ctype_ldml.result > +++ b/mysql-test/suite/innodb/r/innodb_ctype_ldml.result > @@ -8,7 +8,6 @@ Variable_name Value > character_sets_dir MYSQL_TEST_DIR/std_data/ldml/ > show collation like 'utf8_phone_ci'; > Collation Charset Id Default Compiled Sortlen > -utf8_phone_ci utf8 352 8 deja vu. I rememeber I've commented on this very thing already :) please, update this test too. > CREATE TABLE t1 ( > name VARCHAR(64), > phone VARCHAR(64) CHARACTER SET utf8 COLLATE utf8_phone_ci > Binary files a/mysql-test/suite/sys_vars/r/character_set_results_basic.result and b/mysql-test/suite/sys_vars/r/character_set_results_basic.result differ > diff --git a/mysql-test/suite/sys_vars/r/old_mode_basic.result b/mysql-test/suite/sys_vars/r/old_mode_basic.result > index 39c8e554be2..a6b95f1c60c 100644 > --- a/mysql-test/suite/sys_vars/r/old_mode_basic.result > +++ b/mysql-test/suite/sys_vars/r/old_mode_basic.result > @@ -167,8 +167,100 @@ NO_PROGRESS_INFO > SET @@global.old_mode = @global_start_value; > SELECT @@global.old_mode; > @@global.old_mode > - > +UTF8_IS_UTF8MB3 > SET @@session.old_mode = @session_start_value; > SELECT @@session.old_mode; > @@session.old_mode > - > +UTF8_IS_UTF8MB3 > +# > +# Beginning of 10.6 test > +# > +# MDEV-8334: Rename utf8 to utf8mb3 > +# Ah! Great. > +# Save and display old values > +SET @save_old_mode = @@OLD_MODE; > +SET @save_character_set_server = @@character_set_server; > +SET @save_character_set_client = @@character_set_client; > +SET @save_character_set_results = @@character_set_results; > +SET @save_character_set_connection = @@character_set_connection; > +SET @save_character_set_filesystem = @@character_set_filesystem; > +SET @save_character_set_database = @@character_set_database; > +SET @save_collation_connection = @@collation_connection; > +SET @save_collation_server = @@collation_server; > +SET @save_collation_database = @@collation_database; > +SELECT @@OLD_MODE; > +@@OLD_MODE > +UTF8_IS_UTF8MB3 > +SELECT @@character_set_server,@@character_set_client,@@character_set_results, > +@@character_set_connection, @@character_set_filesystem, @@character_set_database, > +@@collation_connection, @@collation_server, @@collation_database; > +@@character_set_server @@character_set_client @@character_set_results @@character_set_connection @@character_set_filesystem @@character_set_database @@collation_connection @@collation_server @@collation_database > +latin1 latin1 latin1 latin1 binary latin1 latin1_swedish_ci latin1_swedish_ci latin1_swedish_ci > +# > +# UTF8MB3 alias for UTF8 > +# > +SET @@character_set_server = utf8; > +SET @@character_set_client = utf8; > +SET @@character_set_results = utf8; > +SET @@character_set_connection = utf8; > +SET @@character_set_filesystem = utf8; > +SET @@character_set_database = utf8; > +SET @@collation_connection = utf8_general_ci; > +SET @@collation_server = utf8_unicode_ci; > +SET @@collation_database = utf8_bin; > +SELECT @@character_set_server, @@character_set_client, @@character_set_results, > +@@character_set_connection, @@character_set_filesystem, @@character_set_database, > +@@collation_connection, @@collation_server, @@collation_database; > +@@character_set_server @@character_set_client @@character_set_results @@character_set_connection @@character_set_filesystem @@character_set_database @@collation_connection @@collation_server @@collation_database > +utf8mb3 utf8mb3 utf8mb3 utf8mb3 utf8mb3 utf8mb3 utf8mb3_general_ci utf8mb3_unicode_ci utf8mb3_bin > +CREATE DATABASE db1 CHARACTER SET = 'utf8' COLLATE = 'utf8_general_ci'; > +ALTER DATABASE db1 CHARACTER SET = 'utf8' COLLATE = 'utf8_unicode_ci'; > +CREATE TABLE tb1 (id1 INT) CHARACTER SET 'utf8' COLLATE 'utf8_bin'; > +SHOW CREATE TABLE tb1; > +Table Create Table > +tb1 CREATE TABLE `tb1` ( > + `id1` int(11) DEFAULT NULL > +) ENGINE=MyISAM DEFAULT CHARSET=utf8mb3 COLLATE=utf8mb3_bin > +DROP TABLE tb1; > +DROP DATABASE db1; > +# > +# UTF8MB4 is alias for UTF8 > +# > +SET @@OLD_MODE=0; > +SET @@character_set_server = utf8; > +SET @@character_set_client = utf8; > +SET @@character_set_results = utf8; > +SET @@character_set_connection = utf8; > +SET @@character_set_filesystem = utf8; > +SET @@character_set_database = utf8; > +SET @@collation_connection = utf8_general_ci; > +SET @@collation_server = utf8_unicode_ci; > +SET @@collation_database = utf8_bin; > +SELECT @@character_set_server, @@character_set_client, @@character_set_results, > +@@character_set_connection, @@character_set_filesystem, @@character_set_database, > +@@collation_connection, @@collation_server, @@collation_database; > +@@character_set_server @@character_set_client @@character_set_results @@character_set_connection @@character_set_filesystem @@character_set_database @@collation_connection @@collation_server @@collation_database > +utf8mb4 utf8mb4 utf8mb4 utf8mb4 utf8mb4 utf8mb4 utf8mb4_general_ci utf8mb4_unicode_ci utf8mb4_bin > +CREATE DATABASE db1 CHARACTER SET = 'utf8' COLLATE = 'utf8_general_ci'; > +ALTER DATABASE db1 CHARACTER SET = 'utf8' COLLATE = 'utf8_unicode_ci'; > +CREATE TABLE tb1 (id1 INT) CHARACTER SET 'utf8' COLLATE 'utf8_bin'; > +SHOW CREATE TABLE tb1; > +Table Create Table > +tb1 CREATE TABLE `tb1` ( > + `id1` int(11) DEFAULT NULL > +) ENGINE=MyISAM DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin > +DROP TABLE tb1; > +DROP DATABASE db1; > +SET @@OLD_MODE = @save_old_mode; > +SET @@character_set_server = @save_character_set_server; > +SET @@character_set_client = @save_character_set_client; > +SET @@character_set_results = @save_character_set_results; > +SET @@character_set_connection = @save_character_set_connection; > +SET @@character_set_filesystem = @save_character_set_filesystem; > +SET @@character_set_database = @save_character_set_database; > +SET @@collation_connection = @save_collation_connection; > +SET @@collation_server = @save_collation_server; > +SET @@collation_database = @save_collation_database; > +# > +# End of 10.6 test > +# > diff --git a/mysys/charset.c b/mysys/charset.c > index 32cfeb56e2d..dbb2749d217 100644 > --- a/mysys/charset.c > +++ b/mysys/charset.c > @@ -763,22 +764,13 @@ get_charset_number_internal(const char *charset_name, uint cs_flags) > } > > > -static const char* > -get_charset_name_alias(const char *name) > -{ > - if (!my_strcasecmp(&my_charset_latin1, name, "utf8mb3")) > - return "utf8"; > - return NULL; > -} > - > - > uint get_charset_number(const char *charset_name, uint cs_flags) > { > uint id; > my_pthread_once(&charsets_initialized, init_available_charsets); > if ((id= get_charset_number_internal(charset_name, cs_flags))) > return id; > - if ((charset_name= get_charset_name_alias(charset_name))) > + if ((charset_name= !my_strcasecmp(&my_charset_latin1, charset_name, "utf8") ? "utf8mb3" : NULL)) Huh? Why do you not check MY_UTF8_IS_UTF8MB3 here? > return get_charset_number_internal(charset_name, cs_flags); > return 0; > } > @@ -820,7 +812,7 @@ static CHARSET_INFO *find_collation_data_inheritance_source(CHARSET_INFO *cs) > char name[MY_CS_NAME_SIZE + 1]; > memcpy(name, beg, end - beg); > name[end - beg]= '\0'; > - return inheritance_source_by_id(cs, get_collation_number(name)); > + return inheritance_source_by_id(cs, get_collation_number(name,MYF(0))); and not here? > } > return NULL; > } > @@ -961,7 +953,28 @@ my_collation_get_by_name(MY_CHARSET_LOADER *loader, > CHARSET_INFO *get_charset_by_name(const char *cs_name, myf flags) > { > MY_CHARSET_LOADER loader; > + my_bool utf8_is_utf8mb3= flags & MY_UTF8_IS_UTF8MB3 ? 1 : 0; > + char *copy_of_name= (char*)cs_name; > + char start[6], result[64]; > + char *temp_cs_name; > + > my_charset_loader_init_mysys(&loader); > + > + if (!strcasecmp("utf8",copy_of_name)) > + cs_name = (const char*)(utf8_is_utf8mb3 ? "utf8mb3" : "utf8mb4"); > + > + strncpy(start, cs_name, 5); > + temp_cs_name= (char *)(utf8_is_utf8mb3 ? "utf8mb3_":"utf8mb4_"); > + > + if (!strncasecmp("utf8_", start,5)) > + { > + copy_of_name+= 5; > + result[63]='\0'; > + strcpy(result, temp_cs_name); > + strcat(result, copy_of_name); > + result[strlen(copy_of_name)+strlen(temp_cs_name)]='\0'; > + cs_name= (const char *) result; > + } And why do you do all that ^^^ ? Old code didn't try to change utf8mb3 to utf8 here, because, I suppose, my_collation_get_by_name() below did all that. Why did you add alias resolution where none was? > return my_collation_get_by_name(&loader, cs_name, flags); > } > > @@ -1005,12 +1018,16 @@ get_charset_by_csname(const char *cs_name, uint cs_flags, myf flags) > { > MY_CHARSET_LOADER loader; > my_charset_loader_init_mysys(&loader); > + > + if (!strcasecmp("utf8",cs_name)) > + cs_name= (const char*)(flags & MY_UTF8_IS_UTF8MB3 ? "utf8mb3" : "utf8mb4"); same here > + > return my_charset_get_by_name(&loader, cs_name, cs_flags, flags); > } > > > /** > - Resolve character set by the character set name (utf8, latin1, ...). > + Resolve character set by the character set name (utf8mb3, latin1, ...). > > The function tries to resolve character set by the specified name. If > there is character set with the given name, it is assigned to the "cs" > @@ -1453,8 +1472,8 @@ static const MY_CSET_OS_NAME charsets[] = > > {"US-ASCII", "latin1", my_cs_approx}, > > - {"utf8", "utf8", my_cs_exact}, > - {"utf-8", "utf8", my_cs_exact}, > + {"utf8mb3", "utf8mb3", my_cs_exact}, eh, no. Try to understand what this array is for. > + {"utf-8", "utf8mb3", my_cs_exact}, > #endif > {NULL, NULL, 0} > }; > diff --git a/plugin/handler_socket/client/hslongrun.cpp b/plugin/handler_socket/client/hslongrun.cpp > index b7c02951340..7f88d48fff2 100644 > --- a/plugin/handler_socket/client/hslongrun.cpp > +++ b/plugin/handler_socket/client/hslongrun.cpp > @@ -897,7 +897,7 @@ hs_longrun_init_table(const config& conf, int num_prepare, > "v1 varchar(32) not null," > "v2 varchar(32) not null," > "v3 varchar(32) not null" > - ") character set utf8 collate utf8_bin engine = innodb"); > + ") character set utf8mb3 collate utf8_bin engine = innodb"); just keep it utf8 > for (int i = 0; i < num_prepare; ++i) { > const std::string i_str = to_stdstring(i); > const std::string v1 = "pv1_" + i_str; > diff --git a/plugin/handler_socket/client/hstest.pl b/plugin/handler_socket/client/hstest.pl > index 1363e153c44..5924d8a0ce5 100755 > --- a/plugin/handler_socket/client/hstest.pl > +++ b/plugin/handler_socket/client/hstest.pl > @@ -52,7 +52,7 @@ for my $action (@actions) { > "k $keytype primary key" . > ",v varchar(32) not null" . > $moreflds . > - ") character set utf8 collate utf8_bin " . > + ") character set utf8mb3 collate utf8_bin " . and here. forget about handlersocket > "engine = $engine"); > } elsif ($action eq "insert") { > print("INSERT $db.$table tablesize=$tablesize\n"); > diff --git a/plugin/win_auth_client/common.cc b/plugin/win_auth_client/common.cc > index 8b7319252ac..ddd34aec7da 100644 > --- a/plugin/win_auth_client/common.cc > +++ b/plugin/win_auth_client/common.cc > @@ -384,7 +384,7 @@ char* wchar_to_utf8(const wchar_t *string, size_t *len) > buf= (char*)malloc(buf_len + 1); > if (!buf) > { > - DBUG_PRINT("error",("Out of memory when converting string '%S' to utf8", > + DBUG_PRINT("error",("Out of memory when converting string '%S' to utf8mb3", Nope, see what this function is doing. > string)); > return NULL; > } > @@ -408,7 +408,7 @@ char* wchar_to_utf8(const wchar_t *string, size_t *len) > > #ifndef DBUG_OFF > Error_message_buf error_buf; > - DBUG_PRINT("error", ("Could not convert string '%S' to utf8" > + DBUG_PRINT("error", ("Could not convert string '%S' to utf8mb3" same > ", WideCharToMultiByte() failed with error %X (%s)", > string, GetLastError(), > get_last_error_message(error_buf))); > @@ -451,7 +451,7 @@ wchar_t* utf8_to_wchar(const char *string, size_t *len) > > if (!buf) > { > - DBUG_PRINT("error",("Out of memory when converting utf8 string '%s'" > + DBUG_PRINT("error",("Out of memory when converting utf8mb3 string '%s'" same > " to wide-char representation", string)); > return NULL; > } > diff --git a/scripts/fill_help_tables.sql b/scripts/fill_help_tables.sql > index d0efb750330..ad7c4fce9a4 100644 > --- a/scripts/fill_help_tables.sql > +++ b/scripts/fill_help_tables.sql > @@ -22,7 +22,7 @@ don't change help tables, please. they'll be regenerated from the documentation > > -- mysql -u root -p mysql < file_name > > -set names 'utf8'; > +set names 'utf8mb3'; > > set sql_log_bin = 0; > > diff --git a/scripts/mysql_system_tables.sql b/scripts/mysql_system_tables.sql > index e390f36a98b..7c8532577a1 100644 > --- a/scripts/mysql_system_tables.sql > +++ b/scripts/mysql_system_tables.sql > @@ -209,7 +209,7 @@ SET @create_transaction_registry="CREATE TABLE IF NOT EXISTS transaction_registr > UNIQUE KEY (commit_id), > INDEX (begin_timestamp), > INDEX (commit_timestamp, transaction_id) > -) ENGINE=INNODB DEFAULT CHARSET=utf8 COLLATE=utf8_bin STATS_PERSISTENT=0"; > +) ENGINE=INNODB DEFAULT CHARSET=utf8mb3 COLLATE=utf8_bin STATS_PERSISTENT=0"; here and in all other places in this file and other .sql files: you need to change the collation too, not just the charset. > > SET @str=IF(@have_innodb <> 0, @create_innodb_table_stats, "SET @dummy = 0"); > PREPARE stmt FROM @str; > diff --git a/sql/mysqld.cc b/sql/mysqld.cc > index 0bf21e02002..cc7568990b4 100644 > --- a/sql/mysqld.cc > +++ b/sql/mysqld.cc > @@ -4039,7 +4039,10 @@ static int init_common_variables() > *next_character_set_name++= '\0'; > if (!(default_charset_info= > get_charset_by_csname(default_character_set_name, > - MY_CS_PRIMARY, MYF(MY_WME)))) > + MY_CS_PRIMARY, > + global_system_variables.old_behavior & > + OLD_MODE_UTF8_IS_UTF8MB3 ? > + MYF(MY_UTF8_IS_UTF8MB3 | MY_WME) : MYF(MY_WME)))) may be you'd better get this multi-line assignment out of if() ? e.g. myf utf8_flag= global_system_variables.old_behavior & OLD_MODE_UTF8_IS_UTF8MB3 : 0; default_charset_info= get_charset_by_csname(default_character_set_name, MY_CS_PRIMARY, MYF(utf8_flag | MY_WME)); if (!default_charset_info) ... or add a helper thd->utf8_alias() as I suggested below > { > if (next_character_set_name) > { > @@ -4056,7 +4059,10 @@ static int init_common_variables() > if (default_collation_name) > { > CHARSET_INFO *default_collation; > - default_collation= get_charset_by_name(default_collation_name, MYF(0)); > + default_collation= get_charset_by_name(default_collation_name, > + global_system_variables.old_behavior & > + OLD_MODE_UTF8_IS_UTF8MB3 ? > + MYF(MY_UTF8_IS_UTF8MB3) : MYF(0)); and here you'll be able to use utf8_flag without the conditional operator > if (!default_collation) > { > #ifdef WITH_PERFSCHEMA_STORAGE_ENGINE > @@ -4097,7 +4103,10 @@ static int init_common_variables() > > if (!(character_set_filesystem= > get_charset_by_csname(character_set_filesystem_name, > - MY_CS_PRIMARY, MYF(MY_WME)))) > + MY_CS_PRIMARY, > + global_system_variables.old_behavior & > + OLD_MODE_UTF8_IS_UTF8MB3 ? > + MYF(MY_UTF8_IS_UTF8MB3 | MY_WME) : MYF(MY_WME)))) and here > return 1; > global_system_variables.character_set_filesystem= character_set_filesystem; > > @@ -7415,7 +7424,9 @@ static void usage(void) > DBUG_ENTER("usage"); > if (!(default_charset_info= get_charset_by_csname(default_character_set_name, > MY_CS_PRIMARY, > - MYF(MY_WME)))) > + global_system_variables.old_behavior & > + OLD_MODE_UTF8_IS_UTF8MB3 ? > + MYF(MY_UTF8_IS_UTF8MB3 | MY_WME) : MYF(MY_WME)))) split this one too? > exit(1); > if (!default_collation_name) > default_collation_name= (char*) default_charset_info->name; > diff --git a/sql/sql_class.h b/sql/sql_class.h > index 50b746fe514..15aa9ca8199 100644 > --- a/sql/sql_class.h > +++ b/sql/sql_class.h > @@ -1007,6 +1008,36 @@ inline void update_global_memory_status(int64 size) > my_atomic_add64_explicit(ptr, size, MY_MEMORY_ORDER_RELAXED); > } > > +inline const char* get_alias_collation_or_charset_name(const char* name, > + bool utf8_is_utf8mb3) > +{ > + char *copy_of_name= (char*)name; > + char start[6], result[64]; > + char *temp_cs_name; > + > + if (!strchr(name,'_')) > + { > + if (!strcasecmp("utf8",name)) > + name = utf8_is_utf8mb3 ? "utf8mb3" : "utf8mb4"; > + return name; > + } > + else > + { > + strncpy(start, name, 5); > + temp_cs_name= (char *)(utf8_is_utf8mb3 ? "utf8mb3_":"utf8mb4_"); > + if (!strncasecmp("utf8_", start,5)) > + { > + copy_of_name+= 5; > + result[63]='\0'; > + strcpy(result, temp_cs_name); > + strcat(result, copy_of_name); > + result[strlen(copy_of_name)+strlen(temp_cs_name)]='\0'; > + strcpy((char*)name,result); > + } > + } > + return name; > +} Please, no. First, you cannot just copy `result` into `name`, because `result` is longer, you'll overwrite whatever was in memory after name's value. Second, you don't need to resolve aliases here, charset code already does it, don't duplicate that. Just pass MY_UTF8_IS_UTF8MB3 down to my_collation_get_by_name() below. > + > /** > Get collation by name, send error to client on failure. > @param name Collation name > diff --git a/sql/sql_db.cc b/sql/sql_db.cc > index 9bf16220535..f471d8edc66 100644 > --- a/sql/sql_db.cc > +++ b/sql/sql_db.cc > @@ -583,9 +583,14 @@ bool load_db_opt(THD *thd, const char *path, Schema_specification_st *create) > default-collation commands. > */ > if (!(create->default_table_charset= > - get_charset_by_csname(pos+1, MY_CS_PRIMARY, MYF(0))) && > + get_charset_by_csname(pos+1, MY_CS_PRIMARY, > + thd->variables.old_behavior & > + OLD_MODE_UTF8_IS_UTF8MB3 ? > + MYF(MY_UTF8_IS_UTF8MB3) : MYF(0))) && > !(create->default_table_charset= > - get_charset_by_name(pos+1, MYF(0)))) > + get_charset_by_name(pos+1, thd->variables.old_behavior & > + OLD_MODE_UTF8_IS_UTF8MB3 ? > + MYF(MY_UTF8_IS_UTF8MB3) : MYF(0)))) again, please move myf flags manipulations out of if() > { > sql_print_error("Error while loading database options: '%s':",path); > sql_print_error(ER_THD(thd, ER_UNKNOWN_CHARACTER_SET),pos+1); > @@ -595,7 +600,9 @@ bool load_db_opt(THD *thd, const char *path, Schema_specification_st *create) > else if (!strncmp(buf,"default-collation", (pos-buf))) > { > if (!(create->default_table_charset= get_charset_by_name(pos+1, > - MYF(0)))) > + thd->variables.old_behavior & > + OLD_MODE_UTF8_IS_UTF8MB3 ? > + MYF(MY_UTF8_IS_UTF8MB3) : MYF(0)))) and here > { > sql_print_error("Error while loading database options: '%s':",path); > sql_print_error(ER_THD(thd, ER_UNKNOWN_COLLATION),pos+1); > diff --git a/sql/sql_lex.cc b/sql/sql_lex.cc > index 6871699dc5b..ad322eda097 100644 > --- a/sql/sql_lex.cc > +++ b/sql/sql_lex.cc > @@ -2789,7 +2789,10 @@ int Lex_input_stream::scan_ident_middle(THD *thd, Lex_ident_cli_st *str, > body_utf8_append(m_cpp_text_start, m_cpp_tok_start + length); > ErrConvString csname(str->str + 1, str->length - 1, &my_charset_bin); > CHARSET_INFO *cs= get_charset_by_csname(csname.ptr(), > - MY_CS_PRIMARY, MYF(0)); > + MY_CS_PRIMARY, > + thd->variables.old_behavior & > + OLD_MODE_UTF8_IS_UTF8MB3 ? > + MYF(MY_UTF8_IS_UTF8MB3) : MYF(0)); may be it could be a helper in thd, like THD::utf8_alias() { return variables.old_behavior & OLD_MODE_UTF8_IS_UTF8MB3 ? MY_UTF8_IS_UTF8MB3 : 0; } > if (cs) > { > *introducer= cs; > diff --git a/storage/connect/mysql-test/connect/my.cnf b/storage/connect/mysql-test/connect/my.cnf > index 6310772d01f..83f0aa8ab30 100644 > --- a/storage/connect/mysql-test/connect/my.cnf > +++ b/storage/connect/mysql-test/connect/my.cnf > @@ -14,4 +14,4 @@ MASTER_MYSOCK= @mysqld.1.socket > SLAVE_MYPORT= @mysqld.2.port > SLAVE_MYSOCK= @mysqld.2.socket > > -PGCLIENTENCODING= UTF8 > +PGCLIENTENCODING= UTF8MB3 eh... really? Are you sure you've tested it and it worked? by the name of it I suspect it's a postgresql client encoding. > diff --git a/storage/connect/mysql-test/connect/t/odbc_postgresql.sql b/storage/connect/mysql-test/connect/t/odbc_postgresql.sql > index 1c302294393..3c78120a7a2 100644 > --- a/storage/connect/mysql-test/connect/t/odbc_postgresql.sql > +++ b/storage/connect/mysql-test/connect/t/odbc_postgresql.sql > @@ -4,13 +4,13 @@ > -- Run this script as a admin user: > -- psql -U postgres < odbc_postgresql.sql > > -SET NAMES 'UTF8'; > +SET NAMES 'UTF8MB3'; same, postgresql > > DROP DATABASE IF EXISTS mtr; > DROP USER IF EXISTS mtr; > > CREATE USER mtr WITH PASSWORD 'mtr'; > -CREATE DATABASE mtr OWNER=mtr ENCODING='UTF8'; > +CREATE DATABASE mtr OWNER=mtr ENCODING='UTF8MB3'; same > GRANT ALL ON DATABASE mtr TO mtr; > \c mtr > SET role mtr; > diff --git a/storage/innobase/fts/fts0opt.cc b/storage/innobase/fts/fts0opt.cc > index e5164fcc4fa..a26a05c2cf9 100644 > --- a/storage/innobase/fts/fts0opt.cc > +++ b/storage/innobase/fts/fts0opt.cc > @@ -330,7 +330,7 @@ fts_word_t* > fts_word_init( > /*==========*/ > fts_word_t* word, /*!< in: word to initialize */ > - byte* utf8, /*!< in: UTF-8 string */ > + byte* utf8mb3, /*!< in: UTF-8 string */ don't rename variables, please > ulint len) /*!< in: length of string in bytes */ > { > mem_heap_t* heap = mem_heap_create(sizeof(fts_node_t)); > diff --git a/storage/mroonga/mysql-test/mroonga/storage/r/collation_utf8_unicode_ci_french.result b/storage/mroonga/mysql-test/mroonga/storage/r/collation_utf8_unicode_ci_french.result > index 3f24de87035..3659aa5aee8 100644 > --- a/storage/mroonga/mysql-test/mroonga/storage/r/collation_utf8_unicode_ci_french.result > +++ b/storage/mroonga/mysql-test/mroonga/storage/r/collation_utf8_unicode_ci_french.result > @@ -7,5 +7,4 @@ FULLTEXT INDEX (content) > INSERT INTO diaries VALUES ("Je suis un garçon."); > SELECT * FROM diaries WHERE MATCH (content) AGAINST ("garcon"); > content > -Je suis un garçon. looks like a bug > DROP TABLE diaries; > diff --git a/storage/mroonga/mysql-test/mroonga/storage/r/collation_utf8_unicode_ci_japanese.result b/storage/mroonga/mysql-test/mroonga/storage/r/collation_utf8_unicode_ci_japanese.result > index 94ef2608b81..79dac1e63a7 100644 > --- a/storage/mroonga/mysql-test/mroonga/storage/r/collation_utf8_unicode_ci_japanese.result > +++ b/storage/mroonga/mysql-test/mroonga/storage/r/collation_utf8_unicode_ci_japanese.result > @@ -7,5 +7,4 @@ FULLTEXT INDEX (content) > INSERT INTO diaries VALUES ("ひらがなとカタカナを覚えました。"); > SELECT * FROM diaries WHERE MATCH (content) AGAINST ("かたかな"); > content > -ひらがなとカタカナを覚えました。 that too > DROP TABLE diaries; > diff --git a/storage/mroonga/vendor/groonga/CMakeLists.txt b/storage/mroonga/vendor/groonga/CMakeLists.txt > index d271d4c4eb9..fc134b81cde 100644 > --- a/storage/mroonga/vendor/groonga/CMakeLists.txt > +++ b/storage/mroonga/vendor/groonga/CMakeLists.txt > @@ -268,7 +268,7 @@ if(UNIX) > ac_check_funcs(pthread_condattr_setpshared) > endif() > > -option(GRN_WITH_NFKC "use NFKC based UTF8 normalization." ON) > +option(GRN_WITH_NFKC "use NFKC based UTF8MB3 normalization." ON) not here, please > > if(WIN32) > ac_check_headers(winsock2.h) > diff --git a/storage/mroonga/vendor/groonga/benchmark/bench-nfkc.c b/storage/mroonga/vendor/groonga/benchmark/bench-nfkc.c > index ebae95b273b..3997b933e87 100644 > --- a/storage/mroonga/vendor/groonga/benchmark/bench-nfkc.c > +++ b/storage/mroonga/vendor/groonga/benchmark/bench-nfkc.c > @@ -83,11 +83,11 @@ static void > bench_char_type(gpointer user_data) > { > uint64_t code_point; > - char utf8[7]; > + char utf8mb3[7]; don't rename variables > > for (code_point = 1; code_point < MAX_UNICODE; code_point++) { > - ucs2utf8(code_point, (unsigned char *)utf8); > - grn_nfkc50_char_type(utf8); > + ucs2utf8(code_point, (unsigned char *)utf8mb3); > + grn_nfkc50_char_type(utf8mb3); > } > } > > diff --git a/storage/mroonga/vendor/groonga/vendor/plugins/groonga-normalizer-mysql/tool/dump_difference_utf8.rb b/storage/mroonga/vendor/groonga/vendor/plugins/groonga-normalizer-mysql/tool/dump_difference_utf8.rb > index 4b6fde8c7b0..ce20a2c5b40 100644 > --- a/storage/mroonga/vendor/groonga/vendor/plugins/groonga-normalizer-mysql/tool/dump_difference_utf8.rb > +++ b/storage/mroonga/vendor/groonga/vendor/plugins/groonga-normalizer-mysql/tool/dump_difference_utf8.rb > @@ -36,8 +36,8 @@ parser.sorted_pages.each do |page, characters| > next if base == sort > n_differences += 1 > utf8s = [base, upper, lower, sort] > - formatted_code_points = utf8s.collect do |utf8| > - "%#07x" % Unicode.from_utf8(utf8) > + formatted_code_points = utf8s.collect do |utf8mb3| > + "%#07x" % Unicode.from_utf8(utf8mb3) just revert all changes under storage/mroonga/vendor/* > end > if sort.bytesize > base.bytesize > n_expanded_sort_characters += 1 > diff --git a/storage/sphinx/mysql-test/sphinx/my.cnf b/storage/sphinx/mysql-test/sphinx/my.cnf > index f60380b7171..22cc06914f4 100644 > --- a/storage/sphinx/mysql-test/sphinx/my.cnf > +++ b/storage/sphinx/mysql-test/sphinx/my.cnf > @@ -7,7 +7,7 @@ xmlpipe_command = cat @ENV.MTR_SUITE_DIR/testdata.xml > [index test1] > source = src1 > docinfo = extern > -charset_type = utf-8 > +charset_type = utf-8mb3 revert > path = @ENV.MYSQLTEST_VARDIR/searchd/test1 > > [indexer] > diff --git a/storage/spider/mysql-test/spider/bg/my.cnf b/storage/spider/mysql-test/spider/bg/my.cnf > index 246099c623e..39f5bd01c67 100644 > --- a/storage/spider/mysql-test/spider/bg/my.cnf > +++ b/storage/spider/mysql-test/spider/bg/my.cnf > @@ -75,15 +75,15 @@ MASTER_1_MYSOCK= @mysqld.1.1.socket > MASTER_1_ENGINE_TYPE= Spider > #MASTER_1_ENGINE_TYPE= MyISAM > MASTER_1_ENGINE= ENGINE=Spider > -MASTER_1_CHARSET= DEFAULT CHARSET=utf8 > +MASTER_1_CHARSET= DEFAULT CHARSET=utf8mb3 Make sure to update both charset and collation here and in all other .cnf files in the spider suite > MASTER_1_ENGINE2= ENGINE=MyISAM > -MASTER_1_CHARSET2= DEFAULT CHARSET=utf8 > -MASTER_1_CHARSET3= DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci > +MASTER_1_CHARSET2= DEFAULT CHARSET=utf8mb3 > +MASTER_1_CHARSET3= DEFAULT CHARSET=utf8mb3 COLLATE=utf8_unicode_ci > SLAVE1_1_MYPORT= @mysqld.4.1.port > SLAVE1_1_MYSOCK= @mysqld.4.1.socket > SLAVE1_1_ENGINE_TYPE= MyISAM > SLAVE1_1_ENGINE= ENGINE=MyISAM > -SLAVE1_1_CHARSET= DEFAULT CHARSET=utf8 > +SLAVE1_1_CHARSET= DEFAULT CHARSET=utf8mb3 > USE_CHILD_GROUP2= 1 > OUTPUT_CHILD_GROUP2= 0 > CHILD2_1_MYPORT= @mysqld.2.1.port > diff --git a/storage/spider/spd_init_query.h b/storage/spider/spd_init_query.h > index 19b04d50b82..f12cef377e2 100644 > --- a/storage/spider/spd_init_query.h > +++ b/storage/spider/spd_init_query.h > @@ -559,7 +559,7 @@ static LEX_STRING spider_init_queries[] = { > " table_name char(64) not null default ''," > " primary key (table_id)," > " unique uk1(db_name, table_name)" > - " ) engine=Aria transactional=1 default charset=utf8 collate=utf8_bin;" > + " ) engine=Aria transactional=1 default charset=utf8mb3 collate=utf8_bin;" always change both charset and collation (everywhere in this file) > " create table if not exists mysql.spider_rewrite_table_tables(" > " table_id bigint unsigned not null," > " partition_id bigint unsigned not null auto_increment," > diff --git a/tests/mysql_client_test.c b/tests/mysql_client_test.c > index 0043786d477..5d9213591fb 100644 > --- a/tests/mysql_client_test.c > +++ b/tests/mysql_client_test.c > @@ -19236,7 +19236,7 @@ static void test_bug12337762() > rc= mysql_query(mysql, "create table charset_tab("\ > "txt1 varchar(32) character set Latin1,"\ > "txt2 varchar(32) character set Latin1 collate latin1_bin,"\ > - "txt3 varchar(32) character set utf8 collate utf8_bin"\ > + "txt3 varchar(32) character set utf8mb3 collate utf8_bin"\ both charset and collation > ")"); > > DIE_UNLESS(rc == 0); Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

[Maria-developers] Old buildbot 10.5 branch failure review
by Otto Kekäläinen 19 Mar '21

19 Mar '21

Hello! I noticed there are quite a lot of upgrade test failures on the 10.5 branch, which is a stable release and should definitely be all green all the time. Here is quick review of those issues with suggested fixes. Maybe Daniel or Elena can do them? They are all easy ones and would make 7 currently red CI steps turn into green. ## Overview: http://buildbot.askmonty.org/buildbot/grid?branch=10.5&category=main&catego… ## kvm-deb-buster-amd6 minor-upgrade-all (http://buildbot.askmonty.org/buildbot/builders/kvm-deb-buster-amd64/builds/…) ++ diff -U1000 /home/buildbot/mariadb.org-tools-master/buildbot/baselines/ldd.10.5.buster.amd64 /home/buildbot/ldd.new --- /home/buildbot/mariadb.org-tools-master/buildbot/baselines/ldd.10.5.buster.amd64 2021-03-16 21:03:04.000000000 -0400 +++ /home/buildbot/ldd.new 2021-03-17 17:23:42.066634818 -0400 - liblz4.so.1 => /usr/lib/x86_64-linux-gnu/liblz4.so.1 Seems some dependency stopped including liblz4. This didn't affect MariaDB. This is a valid change and the fix is to "reset" the ldd.10.5.buster.amd64 file on buildbot.askmonty.org so that there would be no diff no more. ## kvm-deb-buster-amd64 minor-upgrade-all - liblz4.so.1 => /usr/lib/x86_64-linux-gnu/liblz4.so.1 Same as above. ## kvm-deb-sid-x86 - minor-upgrade-all and ## kvm-deb-sid-amd64 - minor-upgrade-all: - libhogweed.so.5 => /lib/x86_64-linux-gnu/libhogweed.so.5 + libhogweed.so.6 => /lib/x86_64-linux-gnu/libhogweed.so.6 - libnettle.so.7 => /lib/x86_64-linux-gnu/libnettle.so.7 + libnettle.so.8 => /lib/x86_64-linux-gnu/libnettle.so.8 Similar issue to buster minor-upgrade-all failure. This is a valid dependency change and the fix is to "reset" the ldd.10.5.sid.amd64 and ldd.10.5.sid.x86 filse on buildbot.askmonty.org so that there would be no diff no more. ## kvm-zyp-opensuse150-amd64 minor-upgrade-all: --- /tmp/ldd.baseline 2021-03-17 19:42:17.468328791 +0000 +++ /home/buildbot/ldd.new 2021-03-17 19:42:16.928328910 +0000 === /usr/sbin/mysqld === /usr/bin/mysql + libedit.so.0 => /usr/lib64/libedit.so.0 === /usr/bin/mysqlaccess === /usr/bin/mysqladmin === /usr/bin/mysqlbinlog === /usr/bin/mysqlcheck === /usr/bin/mysql_client_test === /usr/bin/mysql_client_test_embedded === /usr/bin/mysql_config === /usr/bin/mysql_convert_table_format === /usr/bin/mysqld_multi === /usr/bin/mysqld_safe === /usr/bin/mysqld_safe_helper === /usr/bin/mysqldump === /usr/bin/mysqldumpslow === /usr/bin/mysql_embedded + libedit.so.0 => /usr/lib64/libedit.so.0 Seems legit, library has correct path now. Fix by "resetting" the ldd.baseline. # kvm-zyp-sles150-amd64 minor-upgrade-all: --- /tmp/ldd.baseline 2021-03-17 20:59:56.412361298 +0000 +++ /home/buildbot/ldd.new 2021-03-17 20:59:56.024361282 +0000 === /usr/sbin/mysqld - libpcre2-8.so.0 => /usr/lib64/libpcre2-8.so.0 === /usr/bin/mysql === /usr/bin/mysqlaccess === /usr/bin/mysqladmin === /usr/bin/mysqlbinlog === /usr/bin/mysqlcheck === /usr/bin/mysql_client_test === /usr/bin/mysql_client_test_embedded - libpcre2-8.so.0 => /usr/lib64/libpcre2-8.so.0 === /usr/bin/mysql_config === /usr/bin/mysql_convert_table_format === /usr/bin/mysqld_multi === /usr/bin/mysqld_safe === /usr/bin/mysqld_safe_helper === /usr/bin/mysqldump === /usr/bin/mysqldumpslow === /usr/bin/mysql_embedded - libpcre2-8.so.0 => /usr/lib64/libpcre2-8.so.0 === /usr/bin/mysql_find_rows === /usr/bin/mysql_fix_extensions === /usr/bin/mysqlhotcopy === /usr/bin/mysqlimport === /usr/bin/mysql_install_db === /usr/bin/mysql_ldb === /usr/bin/mysql_plugin === /usr/bin/mysql_secure_installation === /usr/bin/mysql_setpermission === /usr/bin/mysqlshow === /usr/bin/mysqlslap === /usr/bin/mysqltest - libpcre2-8.so.0 => /usr/lib64/libpcre2-8.so.0 - libpcre2-posix.so.2 => /usr/lib64/libpcre2-posix.so.2 === /usr/bin/mysqltest_embedded - libpcre2-8.so.0 => /usr/lib64/libpcre2-8.so.0 - libpcre2-posix.so.2 => /usr/lib64/libpcre2-posix.so.2 Seems legit, library has correct path now. Fix by "resetting" the ldd.baseline. ## kvm-deb-sid-amd64 minor-upgrade-columnstore: The following packages have unmet dependencies: mariadb-plugin-columnstore : Depends: mariadb-server-10.5 (= 1:10.5.8+maria~sid) but 1:10.5.9+maria~sid is to be installed E: Unable to correct problems, you have held broken packages. The mariadb-plugin-columnstore binary packages don't exist for kvm-deb-sid-amd64, as they were skipped when making the 10.5.9 release. Either build them or disable the test until 10.5.10 is out and they are again built. By disabling the test there would be less "red" and less time wasted by developers checking the red ones on why their commits (potentially) fail on CI. Thanks!

4 3

[Maria-developers] 0038c24f2d3: MDEV-16146: MariaDB slave stops with following errors.
by sujatha 18 Mar '21

18 Mar '21

revision-id: 0038c24f2d388b9b0f3bff9219a53c8900153d02 (mariadb-10.5.4-539-g0038c24f2d3) parent(s): 190a8312f598fc4892331225104297f6288f23ac author: Sujatha committer: Sujatha timestamp: 2021-03-18 18:41:30 +0530 message: MDEV-16146: MariaDB slave stops with following errors. Problem: ======== 180511 11:07:58 [ERROR] Slave I/O: Unexpected master's heartbeat data: heartbeat is not compatible with local info;the event's data: log_file_name mysql-bin.000009 log_pos 1054262041, Error_code: 1623 Analysis: ========= In replication setup when master server doesn't have any events to send to slave server it sends an 'Heartbeat_log_event'. This event carries the current binary log filename and offset details. The offset values is stored within 4 bytes of event header. When the size of binary log is higher than UINT32_MAX the log_pos values will not fit in 4 bytes memory. It overflows and hence slave stops with an error. Fix: === Since we cannot extend the fixed header of Log_event class an additional header named 'extra_header' is introduced. 'extra_header' contains HB_FLAGS - 2 bytes HB_LONG_LOG_POS_OFFSET - 8 bytes This 'extra_header' is added only in a case where log_pos > UINT32_MAX and 'HB_LONG_LOG_POS_OFFSET_F' will be enabled. On slave while reading the 'Heartbeat_log_event' if 'HB_LONG_LOG_POS_OFFSET_F' is found to be set then value of 'log_pos' is extracted from 'extra_header'. --- .../suite/rpl/r/rpl_incompatible_heartbeat.result | 20 +++++++++ .../suite/rpl/t/rpl_incompatible_heartbeat.test | 47 ++++++++++++++++++++++ sql/log_event.h | 19 ++++++++- sql/log_event_server.cc | 8 +++- sql/slave.cc | 14 +++++++ sql/sql_repl.cc | 26 ++++++++++-- 6 files changed, 128 insertions(+), 6 deletions(-) diff --git a/mysql-test/suite/rpl/r/rpl_incompatible_heartbeat.result b/mysql-test/suite/rpl/r/rpl_incompatible_heartbeat.result new file mode 100644 index 00000000000..fe2b0436f2d --- /dev/null +++ b/mysql-test/suite/rpl/r/rpl_incompatible_heartbeat.result @@ -0,0 +1,20 @@ +include/master-slave.inc +[connection master] +connection master; +SET @saved_dbug = @@GLOBAL.debug_dbug; +SET @@global.debug_dbug= '+d,simulate_pos_4G'; +connection slave; +include/stop_slave.inc +SET @saved_dbug = @@GLOBAL.debug_dbug; +SET @@global.debug_dbug= '+d,simulate_pos_4G'; +CHANGE MASTER TO MASTER_HEARTBEAT_PERIOD=0.001; +include/start_slave.inc +connection master; +SET @@GLOBAL.debug_dbug = @saved_dbug; +connection slave; +SET @@GLOBAL.debug_dbug = @saved_dbug; +connection master; +CREATE TABLE t (f INT) ENGINE=INNODB; +INSERT INTO t VALUES (10); +DROP TABLE t; +include/rpl_end.inc diff --git a/mysql-test/suite/rpl/t/rpl_incompatible_heartbeat.test b/mysql-test/suite/rpl/t/rpl_incompatible_heartbeat.test new file mode 100644 index 00000000000..b1aa5435ff4 --- /dev/null +++ b/mysql-test/suite/rpl/t/rpl_incompatible_heartbeat.test @@ -0,0 +1,47 @@ +# ==== Purpose ==== +# +# Test verifies that slave IO thread can process heartbeat events with log_pos +# values higher than UINT32_MAX. +# +# ==== Implementation ==== +# +# Steps: +# 0 - Stop slave threads. Configure a small master_heartbeat_period. +# 1 - Using debug points, simulate a huge binlog offset higher than +# UINT32_MAX on master. +# 2 - Start the slave and observe that slave IO thread is able to process +# the offset received through heartbeat event. +# +# ==== References ==== +# +# MDEV-16146: MariaDB slave stops with incompatible heartbeat +# +--source include/have_debug.inc +--source include/have_innodb.inc +--source include/have_binlog_format_mixed.inc +# Test simulates binarylog offsets higher than UINT32_MAX +--source include/have_64bit.inc +--source include/master-slave.inc + +--connection master +SET @saved_dbug = @@GLOBAL.debug_dbug; +SET @@global.debug_dbug= '+d,simulate_pos_4G'; + +--connection slave +--source include/stop_slave.inc +SET @saved_dbug = @@GLOBAL.debug_dbug; +SET @@global.debug_dbug= '+d,simulate_pos_4G'; +CHANGE MASTER TO MASTER_HEARTBEAT_PERIOD=0.001; +--source include/start_slave.inc + +--connection master +sleep 1; +SET @@GLOBAL.debug_dbug = @saved_dbug; +--sync_slave_with_master +SET @@GLOBAL.debug_dbug = @saved_dbug; + +--connection master +CREATE TABLE t (f INT) ENGINE=INNODB; +INSERT INTO t VALUES (10); +DROP TABLE t; +--source include/rpl_end.inc diff --git a/sql/log_event.h b/sql/log_event.h index 4e193232f4b..72d8766cea5 100644 --- a/sql/log_event.h +++ b/sql/log_event.h @@ -576,6 +576,22 @@ class String; #define MARIA_SLAVE_CAPABILITY_MINE MARIA_SLAVE_CAPABILITY_GTID +#define HB_EXTRA_HEADER_LEN 10 +/* + Extra header contains flags specific to Heartbeat_log_event and + respective data. The length needs to be updated when new memebers are + added to this header +*/ +#define HB_FLAGS_OFFSET 0 +/* + When the size of 'log_pos' within Heartbeat_log_event exceeds UINT_MAX32 it + cannot be stored in standard header as 'log_pos' is of 4 bytes size. Hence + extra_header is introduced. First 2 bytes represent flags. In case of long + 'log_pos' value 'HB_LONG_LOG_POS_OFFSET_F' bit within the flag will be set. + The log_pos is stored witin 'long_log_pos' variables. +*/ +#define HB_LONG_LOG_POS_OFFSET_F 0x1 +#define HB_LONG_LOG_POS_OFFSET 2 /** @enum Log_event_type @@ -5718,7 +5734,8 @@ bool copy_cache_to_file_wrapped(IO_CACHE *body, class Heartbeat_log_event: public Log_event { public: - Heartbeat_log_event(const char* buf, uint event_len, + uint16 hb_flags; + Heartbeat_log_event(const char* buf, ulong event_len, const Format_description_log_event* description_event); Log_event_type get_type_code() { return HEARTBEAT_LOG_EVENT; } bool is_valid() const diff --git a/sql/log_event_server.cc b/sql/log_event_server.cc index 6dee9b9adf6..2701f4f2f9a 100644 --- a/sql/log_event_server.cc +++ b/sql/log_event_server.cc @@ -8493,14 +8493,18 @@ void Ignorable_log_event::pack_info(Protocol *protocol) #if defined(HAVE_REPLICATION) -Heartbeat_log_event::Heartbeat_log_event(const char* buf, uint event_len, +Heartbeat_log_event::Heartbeat_log_event(const char* buf, ulong event_len, const Format_description_log_event* description_event) :Log_event(buf, description_event) { + uint16 hb_flags; uint8 header_size= description_event->common_header_len; - ident_len = event_len - header_size; + ident_len = event_len - (header_size + HB_EXTRA_HEADER_LEN); set_if_smaller(ident_len,FN_REFLEN-1); log_ident= buf + header_size; + hb_flags= uint2korr(buf + header_size + ident_len); + if (hb_flags & HB_LONG_LOG_POS_OFFSET_F) + log_pos= uint8korr(buf + header_size + ident_len + HB_LONG_LOG_POS_OFFSET); } #endif diff --git a/sql/slave.cc b/sql/slave.cc index 1da030084ef..f365a1e7b5c 100644 --- a/sql/slave.cc +++ b/sql/slave.cc @@ -6493,6 +6493,12 @@ static int queue_event(Master_info* mi,const char* buf, ulong event_len) TODO: handling `when' for SHOW SLAVE STATUS' snds behind */ + DBUG_EXECUTE_IF("simulate_pos_4G", + { + inc_pos= mi->master_log_pos; // temp_save + mi->master_log_pos= ((ulong)2394967295); + };); + if (memcmp(mi->master_log_name, hb.get_log_ident(), hb.get_ident_len()) || mi->master_log_pos > hb.log_pos) { /* missed events of heartbeat from the past */ @@ -6504,6 +6510,14 @@ static int queue_event(Master_info* mi,const char* buf, ulong event_len) error_msg.append_ulonglong(hb.log_pos); goto err; } + DBUG_EXECUTE_IF("simulate_pos_4G", + { + if (hb.log_pos > UINT32_MAX) + { + mi->master_log_pos= inc_pos; // restore + DBUG_SET("-d, simulate_pos_4G"); + } + };); /* Heartbeat events doesn't count in the binlog size, so we don't have to diff --git a/sql/sql_repl.cc b/sql/sql_repl.cc index 010ea794254..d68ae05b60e 100644 --- a/sql/sql_repl.cc +++ b/sql/sql_repl.cc @@ -824,12 +824,14 @@ get_slave_until_gtid(THD *thd, String *out_str) */ static int send_heartbeat_event(binlog_send_info *info, NET* net, String* packet, - const struct event_coordinates *coord, + struct event_coordinates *coord, enum enum_binlog_checksum_alg checksum_alg_arg) { DBUG_ENTER("send_heartbeat_event"); ulong ev_offset; + char extra_buf[HB_EXTRA_HEADER_LEN]; + uint16 hb_flags= 0; if (reset_transmit_packet(info, info->flags, &ev_offset, &info->errmsg)) DBUG_RETURN(1); @@ -850,19 +852,30 @@ static int send_heartbeat_event(binlog_send_info *info, size_t event_len = ident_len + LOG_EVENT_HEADER_LEN + (do_checksum ? BINLOG_CHECKSUM_LEN : 0); int4store(header + SERVER_ID_OFFSET, global_system_variables.server_id); + if (coord->pos <= UINT32_MAX) + { + int4store(header + LOG_POS_OFFSET, coord->pos); // log_pos + } + else + { + hb_flags|= HB_LONG_LOG_POS_OFFSET_F; + int2store(extra_buf + HB_FLAGS_OFFSET, hb_flags); + int8store(extra_buf + HB_LONG_LOG_POS_OFFSET, coord->pos); + event_len+= HB_EXTRA_HEADER_LEN; + } int4store(header + EVENT_LEN_OFFSET, event_len); int2store(header + FLAGS_OFFSET, 0); - int4store(header + LOG_POS_OFFSET, coord->pos); // log_pos - packet->append(header, sizeof(header)); packet->append(p, ident_len); // log_file_name + packet->append(extra_buf, sizeof(extra_buf)); if (do_checksum) { char b[BINLOG_CHECKSUM_LEN]; ha_checksum crc= my_checksum(0, (uchar*) header, sizeof(header)); crc= my_checksum(crc, (uchar*) p, ident_len); + crc= my_checksum(crc, (uchar*) extra_buf, sizeof(extra_buf)); int4store(b, crc); packet->append(b, sizeof(b)); } @@ -2525,6 +2538,13 @@ static int wait_new_events(binlog_send_info *info, /* in */ } #endif mysql_bin_log.unlock_binlog_end_pos(); + + DBUG_EXECUTE_IF("simulate_pos_4G", + { + coord.pos= ((ulong)5394967295); + DBUG_SET("-d, simulate_pos_4G"); + };); + ret= send_heartbeat_event(info, info->net, info->packet, &coord, info->current_checksum_alg);

1 0

Re: [Maria-developers] 9acabefd276: MDEV-22010: use executables MariaDB named in scripts
by Sergei Golubchik 17 Mar '21

17 Mar '21

Hi, Rucha! On Mar 17, Rucha Deodhar wrote: > > mysql-test-run and mysql-stress-test was also supposed to be renamed as > part of this task. > I need to check again if debs and rpms package symlink. > > > > > > diff --git a/mysql-test/.mtr b/mysql-test/.mtr > > > new file mode 120000 > > > index 00000000000..68986c8cd7e > > > --- /dev/null > > > +++ b/mysql-test/.mtr > > > @@ -0,0 +1 @@ > > > +./mariadb-test-run > > > > What is that? What is .mtr file? > > I think after i built it got committed along with other files. please remove it from the commit > > > diff --git a/mysql-test/mysql-stress-test.pl b/mysql-test/ > > mysql-stress-test.pl > > > new file mode 120000 > > > index 00000000000..e704a5d70bf > > > --- /dev/null > > > +++ b/mysql-test/mysql-stress-test.pl > > > @@ -0,0 +1 @@ > > > +./mariadb-stress-test.pl > > > > looks like you've checked in files you shouldn't have. Like symlinks. > > As mysql-test-run and mysql-stress-test was supposed to be renamed too. So, > I created a symlink > so that ./mysql-test-run and ./mysql-stress-test would still work if > anybody uses it. symlinks - yes. But they are created by cmake during the build, they should not be added to a commit. please, remove. You might want to add them to the .gitignore file though. > > > \ No newline at end of file > > > diff --git a/scripts/msql2mysql.sh b/scripts/msql2mysql.sh > > > index 72a609fa6e7..11923fa15c7 100644 > > > --- a/scripts/msql2mysql.sh > > > +++ b/scripts/msql2mysql.sh > > > @@ -16,4 +16,4 @@ > > > # Free Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston > > > # MA 02110-1335 USA. > > > > > > -@bindir@/replace msqlConnect mysql_connect msqlListDBs mysql_list_dbs > > msqlNumRows mysql_num_rows msqlFetchRow mysql_fetch_row msqlFetchField > > mysql_fetch_field msqlFreeResult mysql_free_result msqlListFields > > mysql_list_fields msqlListTables mysql_list_tables msqlErrMsg > > 'mysql_error(mysql)' msqlStoreResult mysql_store_result msqlQuery > > mysql_query msqlField mysql_field msqlSelect mysql_select msqlSelectDB > > mysql_select_db msqlNumFields mysql_num_fields msqlClose mysql_close > > msqlDataSeek mysql_data_seek m_field MYSQL_FIELD m_result MYSQL_RES m_row > > MYSQL_ROW msql mysql mSQL mySQL MSQL MYSQL msqlCreateDB mysql_create_db > > msqlDropDB mysql_drop_db msqlFieldSeek mysql_field_seek -- $* > > > +@bindir@/replace msqlConnect mysql_connect msqlListDBs mysql_list_dbs > > msqlNumRows mysql_num_rows msqlFetchRow mysql_fetch_row msqlFetchField > > mysql_fetch_field msqlFreeResult mysql_free_result msqlListFields > > mysql_list_fields msqlListTables mysql_list_tables msqlErrMsg > > 'mysql_error(mysql)' msqlStoreResult mysql_store_result msqlQuery > > mysql_query msqlField mysql_field msqlSelect mysql_select msqlSelectDB > > mysql_select_db msqlNumFields mysql_num_fields msqlClose mysql_close > > msqlDataSeek mysql_data_seek m_field MYSQL_FIELD m_result MYSQL_RES m_row > > MYSQL_ROW msql mysql mysql mariadb mSQL mySQL MSQL MYSQL msqlCreateDB > > mysql_create_db msqlDropDB mysql_drop_db msqlFieldSeek mysql_field_seek -- > > $* > > > > Why? > > So that mysql in bin directory would get replaced with mariadb no, that's not it. See other renames, msqlListTables -> mysql_list_tables, msqlFetchRow -> mysql_fetch_row etc this (prehistoric) script was supposed to help migrating programs that work with mSQL (https://en.wikipedia.org/wiki/MSQL) to MySQL by renaming C API calls in the source code. In short, revert this change, it doesn't belong to this script. > > > diff --git a/scripts/mysql_install_db.sh b/scripts/mysql_install_db.sh > > > index 5f183afe8fc..084af42d01c 100644 > > > --- a/scripts/mysql_install_db.sh > > > +++ b/scripts/mysql_install_db.sh > > > @@ -338,11 +338,15 @@ then > > > cannot_find_file resolveip @resolveip_locations@ > > > exit 1 > > > fi > > > - mysqld=`find_in_dirs mysqld @mysqld_locations@` > > > + mysqld=`find_in_dirs mariadbd @mysqld_locations@` > > > if test -z "$mysqld" > > > then > > > - cannot_find_file mysqld @mysqld_locations@ > > > - exit 1 > > > + mysqld=`find_in_dirs mysqld @mysqld_locations@` > > > + if test -z "$mysqld" > > > + then > > > + cannot_find_file mysqld @mysqld_locations@ > > > > Hmm. At least, it should be mariadbd in the cannot_find_file line. > > > > Why do you search for mysqld after mariadbd wasn't found? > > > I added it for additional check, just in case there is mysqld and not > mariadbd or if it could affect compatibility. I don't think it could. These scripts you've edited are installed together with MariaDB, there is no supported configuration where these scripts exist and mariadb* binaries don't. That is, I believe this extra check is not needed. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 9acabefd276: MDEV-22010: use executables MariaDB named in scripts
by Sergei Golubchik 17 Mar '21

17 Mar '21

Hi, Rucha! On Mar 17, Rucha Deodhar wrote: > revision-id: 9acabefd276 (mariadb-10.5.2-386-g9acabefd276) > parent(s): de407e7cb4d > author: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > committer: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > timestamp: 2021-02-11 15:19:29 +0530 > message: > > MDEV-22010: use executables MariaDB named in scripts > > As a part of this MDEV following changes were made: > 1) Mariadb named executables used instead of mysql named executables in scripts > 2) renamed mysql-test-run and mysql-stress-test to mariadb-test-run and > mariadb-stress-test and created a symlink. > > diff --git a/debian/mariadb-test.links b/debian/mariadb-test.links > index 718a5355474..b3577667d01 100644 > --- a/debian/mariadb-test.links > +++ b/debian/mariadb-test.links > @@ -2,5 +2,7 @@ usr/bin/mariadb-client-test usr/bin/mysql_client_test > usr/bin/mariadb-client-test-embedded usr/bin/mysql_client_test_embedded > usr/bin/mariadb-test usr/bin/mysqltest > usr/bin/mariadb-test-embedded usr/bin/mysqltest_embedded > -usr/share/mysql/mysql-test/mysql-test-run.pl usr/share/mysql/mysql-test/mtr > -usr/share/mysql/mysql-test/mysql-test-run.pl usr/share/mysql/mysql-test/mysql-test-run > +usr/share/mysql/mysql-test/mariadb-test-run.pl usr/share/mysql/mysql-test/mysql-test-run.pl > +usr/share/mysql/mysql-test/mariadb-stress-test.pl usr/share/mysql/mysql-test/mysql-stress-test.pl > +usr/share/mysql/mysql-test/mariadb-test-run.pl usr/share/mysql/mysql-test/mysql-test-run > +usr/share/mysql/mysql-test/mariadb-test-run.pl usr/share/mysql/mysql-test/mtr Not sure that was part of the task, but ok. Did you check that both debs and rpms package all new symlinks correctly? > diff --git a/mysql-test/.mtr b/mysql-test/.mtr > new file mode 120000 > index 00000000000..68986c8cd7e > --- /dev/null > +++ b/mysql-test/.mtr > @@ -0,0 +1 @@ > +./mariadb-test-run What is that? What is .mtr file? > \ No newline at end of file > diff --git a/mysql-test/mysql-stress-test.pl b/mysql-test/mysql-stress-test.pl > new file mode 120000 > index 00000000000..e704a5d70bf > --- /dev/null > +++ b/mysql-test/mysql-stress-test.pl > @@ -0,0 +1 @@ > +./mariadb-stress-test.pl looks like you've checked in files you shouldn't have. Like symlinks. > \ No newline at end of file > diff --git a/mysql-test/mysql-test-run.pl b/mysql-test/mysql-test-run.pl > new file mode 120000 > index 00000000000..56150a0ecad > --- /dev/null > +++ b/mysql-test/mysql-test-run.pl > @@ -0,0 +1 @@ > +./mariadb-test-run.pl another one > \ No newline at end of file > diff --git a/scripts/msql2mysql.sh b/scripts/msql2mysql.sh > index 72a609fa6e7..11923fa15c7 100644 > --- a/scripts/msql2mysql.sh > +++ b/scripts/msql2mysql.sh > @@ -16,4 +16,4 @@ > # Free Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston > # MA 02110-1335 USA. > > -@bindir@/replace msqlConnect mysql_connect msqlListDBs mysql_list_dbs msqlNumRows mysql_num_rows msqlFetchRow mysql_fetch_row msqlFetchField mysql_fetch_field msqlFreeResult mysql_free_result msqlListFields mysql_list_fields msqlListTables mysql_list_tables msqlErrMsg 'mysql_error(mysql)' msqlStoreResult mysql_store_result msqlQuery mysql_query msqlField mysql_field msqlSelect mysql_select msqlSelectDB mysql_select_db msqlNumFields mysql_num_fields msqlClose mysql_close msqlDataSeek mysql_data_seek m_field MYSQL_FIELD m_result MYSQL_RES m_row MYSQL_ROW msql mysql mSQL mySQL MSQL MYSQL msqlCreateDB mysql_create_db msqlDropDB mysql_drop_db msqlFieldSeek mysql_field_seek -- $* > +@bindir@/replace msqlConnect mysql_connect msqlListDBs mysql_list_dbs msqlNumRows mysql_num_rows msqlFetchRow mysql_fetch_row msqlFetchField mysql_fetch_field msqlFreeResult mysql_free_result msqlListFields mysql_list_fields msqlListTables mysql_list_tables msqlErrMsg 'mysql_error(mysql)' msqlStoreResult mysql_store_result msqlQuery mysql_query msqlField mysql_field msqlSelect mysql_select msqlSelectDB mysql_select_db msqlNumFields mysql_num_fields msqlClose mysql_close msqlDataSeek mysql_data_seek m_field MYSQL_FIELD m_result MYSQL_RES m_row MYSQL_ROW msql mysql mysql mariadb mSQL mySQL MSQL MYSQL msqlCreateDB mysql_create_db msqlDropDB mysql_drop_db msqlFieldSeek mysql_field_seek -- $* Why? > diff --git a/scripts/mysql_install_db.sh b/scripts/mysql_install_db.sh > index 5f183afe8fc..084af42d01c 100644 > --- a/scripts/mysql_install_db.sh > +++ b/scripts/mysql_install_db.sh > @@ -338,11 +338,15 @@ then > cannot_find_file resolveip @resolveip_locations@ > exit 1 > fi > - mysqld=`find_in_dirs mysqld @mysqld_locations@` > + mysqld=`find_in_dirs mariadbd @mysqld_locations@` > if test -z "$mysqld" > then > - cannot_find_file mysqld @mysqld_locations@ > - exit 1 > + mysqld=`find_in_dirs mysqld @mysqld_locations@` > + if test -z "$mysqld" > + then > + cannot_find_file mysqld @mysqld_locations@ Hmm. At least, it should be mariadbd in the cannot_find_file line. Why do you search for mysqld after mariadbd wasn't found? > + exit 1 > + fi > fi > langdir=`find_in_dirs --dir errmsg.sys @errmsg_locations@` > if test -z "$langdir" > @@ -579,7 +583,7 @@ else > echo > echo " shell> $mysqld --skip-grant-tables --general-log &" > echo > - echo "and use the command line tool $bindir/mysql" > + echo "and use the command line tool $bindir/mariadb or $bindir/mysql" again, why both? > echo "to connect to the mysql database and look at the grant tables:" > echo > echo " shell> $bindir/mysql -u root mysql" > @@ -613,7 +617,9 @@ then > echo "PLEASE REMEMBER TO SET A PASSWORD FOR THE MariaDB root USER !" > echo "To do so, start the server, then issue the following commands:" > echo > + echo "'$bindir/mariadb-admin' -u root password 'new-password' or" > echo "'$bindir/mysqladmin' -u root password 'new-password'" > + echo "'$bindir/mariadb-admin' -u root -h $hostname password 'new-password' or" > echo "'$bindir/mysqladmin' -u root -h $hostname password 'new-password'" and here > echo > echo "Alternatively you can run:" > diff --git a/scripts/mysql_secure_installation.sh b/scripts/mysql_secure_installation.sh > index b2a9edf4953..12364f1be43 100644 > --- a/scripts/mysql_secure_installation.sh > +++ b/scripts/mysql_secure_installation.sh > @@ -159,15 +159,19 @@ then > cannot_find_file my_print_defaults $basedir/bin $basedir/extra > exit 1 > fi > - mysql_command=`find_in_basedir mysql bin` > + mysql_command=`find_in_basedir mariadb bin` > if test -z "$mysql_command" > then > - cannot_find_file mysql $basedir/bin > - exit 1 > + mysql_command=`find_in_basedir mysql bin` > + if test -z "$mysql_command" > + then > + cannot_find_file mysql $basedir/bin > + exit 1 > + fi same as above > fi > else > print_defaults="@bindir@/my_print_defaults" > - mysql_command="@bindir@/mysql" > + mysql_command="@bindir@/mariadb" > fi > > if test ! -x "$print_defaults" > diff --git a/scripts/mysqld_safe.sh b/scripts/mysqld_safe.sh > index 126eb924814..7763fb9db91 100644 > --- a/scripts/mysqld_safe.sh > +++ b/scripts/mysqld_safe.sh > @@ -531,7 +531,11 @@ else > ledir='@libexecdir@' > fi > > -helper=`find_in_bin mysqld_safe_helper` > +helper=`find_in_bin mariadbd-safe-helper` > +if test -x helper > +then > + helper=`find_in_bin mysqld_safe_helper` > +fi and here > print_defaults=`find_in_bin my_print_defaults` > # Check if helper exists > command -v $helper --help >/dev/null 2>&1 > diff --git a/scripts/wsrep_sst_common.sh b/scripts/wsrep_sst_common.sh > index 5e134570881..cf38bda8f70 100644 > --- a/scripts/wsrep_sst_common.sh > +++ b/scripts/wsrep_sst_common.sh > @@ -273,13 +273,21 @@ SCRIPTS_DIR="$(cd $(dirname "$0"); pwd -P)" > EXTRA_DIR="$SCRIPTS_DIR/../extra" > CLIENT_DIR="$SCRIPTS_DIR/../client" > > -if [ -x "$CLIENT_DIR/mysql" ]; then > +if [-x "$CLIENT_DIR/mariadb"]; then > + MYSQL_CLIENT="$CLIENT_DIR/mariadb" > +elif [-x mariadb] then; > + MYSQL_CLIENT=mariadb > +elif [ -x "$CLIENT_DIR/mysql" ]; then > MYSQL_CLIENT="$CLIENT_DIR/mysql" > else > MYSQL_CLIENT=mysql > fi > > -if [ -x "$CLIENT_DIR/mysqldump" ]; then > +if [ -x "$CLIENT_DIR/mariadb-dump" ]; then > + MYSQLDUMP="$CLIENT_DIR/mariadb-dump" > +elif [-x mariadb-dump]; then > + MYSQLDUMP=mariadb-dump > +elif [ -x "$CLIENT_DIR/mysqldump" ]; then > MYSQLDUMP="$CLIENT_DIR/mysqldump" > else > MYSQLDUMP=mysqldump and here Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] ee538938345: MDEV-21117: refine the server binlog-based recovery for semisync
by Sergei Golubchik 16 Mar '21

16 Mar '21

Hi, Andrei! Few comments/questions below. I still have troubles understanding the logic of your solution, so I did not ask any questions about those your replies - as I'd be just repeating the same thing over and over. After I'll start getting it finally, I'll go over your reply again. Meanwhile - minor comments/questions, not related to the main algorithm. On Mar 04, Andrei Elkin wrote: > >> To facilitate the failover on the slave side conditions to accept > >> own events (having been discarded by the above recovery) are > >> relaxed to let so for the semisync slave that connects to master in > >> gtid mode. gtid_strict_mode is further recommended to secure from > >> inadvertent re-applying out of order gtids in general. Non-gtid > >> mode connected semisync slave would require > >> --replicate-same-server-id (mind --log-slave-updates must be OFF > >> then). > > > > Sorry, I failed to understand this paragraph :( > > 'gtid_strict_mode is further recommended to secure > from inadvertent re-applying out of order gtids in general.' > > attempted to promote GTID connection with master, > which of course should not be of this patch's > business, but in a way it is 'cos the non-gtid's mode > (CHANGE MASTER TO ... master_use_gtid = NO) is covered by the patch > with a limitation of '--log-slave-updates must be OFF'. > > The GTID-ON connection is covered, and without any `--replicate-same-server-id'. This doesn't seem to clarify anything. I mean the whole paragraph, starting from "To facilitate" and to "then)." Could you try to rephrase it to make it clearer, please? > >> +--let $query1 = INSERT INTO t VALUES (20) > >> +--let $query2 = DELETE FROM t2 WHERE f = 666 /* no such record */ > >> +--source binlog_truncate_active_log.inc > >> + > >> +--echo # Case B. > >> +# The inverted sequence ends up to truncate only $query2 > >> +--let $this_search_pattern = Successfully truncated.*to remove transactions starting from GTID 0-1-10 > >> +--let $query1 = DELETE FROM t2 WHERE f = 0 > > > > why not `= 666 /* no such record */` ? > > is `= 0` important here? > > Yes, there's a fine detail here. > > 'The inverted sequence ends up' to leave in binlog > > $query1 = DELETE FROM t2 WHERE f = 0 > > which is as ineffective as 666. The patch recognizes the ineffectiveness > through a special value of engines involved counter in Gtid event. > In principle the truncation could > start from it rather than the following $query2. > But that would mean more complicated coding. > So the semantics of the zero is to be good, as we're so used to :-)! Sorry, I didn't understand anything. What does the value in the WHERE clause has to do with engines and where the binlog is truncated? > > > >> +--let $query2 = INSERT INTO t VALUES (20) > >> +--source binlog_truncate_active_log.inc ... > >> +--connection master2 > >> +SET DEBUG_SYNC= "now WAIT_FOR master1_ready"; > >> +if ($pre_q2) > > > > is $pre_q2 a flag or a query string? > > it's used as a flag, it's set as a query string. > > It's actually both with the idea execute it when defined. > Legit, right? legit, but it's not both. Were it both, you'd have if ($pre_q2) { eval $pre_q2; } but you have, if ($pre_q2) { CALL sp_blank_xa; } > >> +{ > >> + CALL sp_blank_xa; > >> +} ... > >> diff --git a/mysql-test/suite/binlog/t/binlog_truncate_multi_engine.test b/mysql-test/suite/binlog/t/binlog_truncate_multi_engine.test > >> new file mode 100644 > >> index 00000000000..fe153e5703c > >> --- /dev/null > >> +++ b/mysql-test/suite/binlog/t/binlog_truncate_multi_engine.test > >> +# > >> +# ==== Implementation ==== > >> +# > >> +# Steps: > >> +# 0 - Create two tables in innodb and rocksdb engines, > >> +# > >> +# In loop for A,B,C cases (below) do 1-5: > > > > there's no loop now (which is good, don't add it please :) [x] ? > >> +# The 1st trx binlogs, rotate binlog and hold on before committing at engine > >> +SET DEBUG_SYNC= "commit_after_release_LOCK_after_binlog_sync SIGNAL master1_ready WAIT_FOR master1_go"; > > > > you don't wait for master1_ready anywhere. intentional? > > No, actually > > SET DEBUG_SYNC= "now WAIT_FOR master1_ready"; > > must be run by below master2 (sim to how master3 waits for master2). meaning [x] ? > >> +--source include/have_binlog_format_row.inc > >> +--let $rpl_topology=1->2 > >> +--source include/rpl_init.inc > > > > isn't this a normal master-slave.inc topology? > > Initially yes. > > I am guessing you mean why not to > --source it. > The test heavily uses 'server_`index`' format to automate > replication direction. as far as I remember, master-slave.inc also defines server_1 and server_2 connections. > >> diff --git a/sql/sql_class.h b/sql/sql_class.h > >> index 140394fefc1..87fa88d4c89 100644 > >> --- a/sql/sql_class.h > >> +++ b/sql/sql_class.h > >> @@ -4632,7 +4632,7 @@ class THD :public Statement, > >> LF_PINS *tdc_hash_pins; > >> LF_PINS *xid_hash_pins; > >> bool fix_xid_hash_pins(); > >> - > >> + bool is_1pc_ro_trans; > > > > I don't like how it looks, it's in THD (for every single connection > > and more) and it's only used in some exotic cases. It's set in two > > places around ha_commit_one_phase(), when it would be better to set > > it inside ha_commit_one_phase(), in one place only. > > > > But I cannot suggest how to get rid of it yet as I don't understand > > what it's for, I've asked a question about it below, in > > Gtid_log_event::Gtid_log_event > > The flag is to help sorting out ineffective (no engine data change) > STATEMENT binlog-format logged transaction. I thought it's practical > enough to face and important to handle. > Through an example, > let the table `t' be initially empty, and > let binlog contain > > trx1: INSERT INTO t SET a = 1; /* slave acked, trx got committed in engine */ > trx2: INSERT INTO t SET a = 2; /* not acked, trx remains prepared in engine */ > trx3: DELETE FROM t WHERE a = 0; /* ineffective, does not have Xid event */ > > prior a crash with their states as commented. > The truncate trx should be trx2 as the trx3's status > suggests it's harmless for involving into > truncation. > However we have yet to distinguish it from unsafe (to truncate) > event group ("transaction") that also ends with COMMIT query not Xid event. Hmm. I see. But thd isn't a universal dumpster for passing arguments to the function, don't use it like that. Please, pass this "read-only transaction with no xid" flag to the Gtid_log_event constructor as an argument. Or recalculate it inside the constructor, it's not such an expensive thing to do. > One rather drastic solution could be to not binlog DELETE altogether. > I did not look into this with much of interest earlier, but how about it? > Perhaps it'd more like a bug fixes than breaking legacy. No, I suspect it's too drastic > >> /* Members related to temporary tables. */ > >> public: > >> /* Opened table states. */ > >> diff --git a/sql/slave.cc b/sql/slave.cc > >> index 28e08e32346..9b3f73e5341 100644 > >> --- a/sql/slave.cc > >> +++ b/sql/slave.cc > >> @@ -6944,7 +6945,9 @@ static int queue_event(Master_info* mi,const char* buf, ulong event_len) > >> } > >> else > >> if ((s_id == global_system_variables.server_id && > >> - !mi->rli.replicate_same_server_id) || > >> + (!mi->rli.replicate_same_server_id && > >> + !(semisync_recovery= (rpl_semi_sync_slave_enabled && > >> + mi->using_gtid != Master_info::USE_GTID_NO)))) || > > > > How can "semisync recovery" code path reach queue_event()? > > This is a failover now, after recovery. > The slave which is ex-master tries to adopt "own" (generated by it) events. 1. don't call it "semisync recovery" then 2. add a comment explaining this case (it's a rare and not obvious one) 3. why does it have to be specially recognized anyway? > > > >> event_that_should_be_ignored(buf) || > >> /* > >> the following conjunction deals with IGNORE_SERVER_IDS, if set > >> diff --git a/sql/log_event.h b/sql/log_event.h > >> index 8a342cb5cd3..1588ab85104 100644 > >> --- a/sql/log_event.h > >> +++ b/sql/log_event.h > >> @@ -482,6 +482,16 @@ class String; > >> */ > >> #define LOG_EVENT_IGNORABLE_F 0x80 > >> > >> +/** > >> + @def LOG_EVENT_ACCEPT_OWN_F > >> + > >> + Flag sets by the gtid-mode connected semisync slave for > >> + the same server_id ("own") events which the slave must not have > >> + in its state. Typically such events were never committed by > >> + their originator (this server) and discared at its crash recovery > > > > sorry, I failed to understand that > > This should've been explained above. > The slave which is ex-master truncated at recovery some of its ("own") transactions. > The flag serves to help recognize that at execution of the replication events. 1. yes, this is clearer, thanks. may be you should rewrite your comment putting this explanation in it? 2. why does it have to be specially recognized anyway? ... > > 3. also it's somewhat an unfounded assumption that only r/w engines > > will have the transaction prepared. But we can make it a fact, if we > > won't prepare r/o engines at all (in ha_prepare). > > [?] > > Sorry, I don't get it. > Do you mean that an r/o engine (branch) might need to be counted by > `ha_count_rw()`? But then it would be contributing with binlogged > events of its branch... I can't imagine that. I mean the following. Say, you have two engines participating in a transaction, like in INSERT innodb_table SELECT * FROM rocksdb_table; One is r/w, the other is r/o. It's a 2pc transaction, the server does prepare for everyone, then commit for everyone. I don't think there is any guarantee that prepare for an r/o engine will be a no-op. It is completely up to the engine what to do in this case, it's possible that the engine will optimize it to a no-op, but it is not a certainty. What I think we should do is not to call prepare() for r/o engines at all. Just skip straight to commit/rollback on the second phase. > >> diff --git a/sql/handler.cc b/sql/handler.cc > >> index 6792e80b8fe..247ab7267bf 100644 > >> --- a/sql/handler.cc > >> +++ b/sql/handler.cc > >> @@ -1245,6 +1245,24 @@ int ha_prepare(THD *thd) > >> DBUG_RETURN(error); > >> } > >> > >> +/* > >> + Returns counted number of > >> + read-write recoverable transaction participants. > >> +*/ > >> +uint ha_count_rw(THD *thd, bool all) > > > > the name doesn't match the implementation, please, rename > > [?] > > Please help here. > I could not find what specifically you mean. > It > returns rw_ha_count > > which I think as 'read-write recoverable transaction participants.' I read the name is "a number of rw engines in the ha_info", it does not mention "recoverable", and I from the name I woulnd't have guessed. May be ha_count_rw_2pc ? > >> +{ > >> + unsigned rw_ha_count= 0; > >> + THD_TRANS *trans=all ? &thd->transaction.all : &thd->transaction.stmt; > >> + > >> + for (Ha_trx_info * ha_info= trans->ha_list; ha_info; > >> + ha_info= ha_info->next()) > >> + { > >> + if (ha_info->is_trx_read_write() && ha_info->ht()->recover) > >> + ++rw_ha_count; > >> + } > >> + return rw_ha_count; > >> +} > >> + > >> /** > >> Check if we can skip the two-phase commit. > >> > >> @@ -1960,8 +1982,122 @@ struct xarecover_st > >> XID *list; > >> HASH *commit_list; > >> bool dry_run; > >> + MEM_ROOT *mem_root; > >> + bool error; > >> }; > >> > >> +/** > >> + Inserts a new hash member. > >> + > >> + returns a successfully created and inserted @c xid_recovery_member > >> + into hash @c hash_arg, > >> + or NULL. > >> +*/ > >> +static xid_recovery_member* > >> +xid_member_insert(HASH *hash_arg, my_xid xid_arg, MEM_ROOT *ptr_mem_root) > >> +{ > >> + xid_recovery_member *member= (xid_recovery_member*) > >> + alloc_root(ptr_mem_root, sizeof(xid_recovery_member)); > >> + if (!member) > >> + return NULL; > >> + > >> + member->xid= xid_arg; > >> + member->in_engine_prepare= 1; > >> + member->decided_to_commit= false; > >> + > >> + return my_hash_insert(hash_arg, (uchar*) member) ? NULL : member; > >> +} > >> + > >> +/* > >> + Inserts a new or updates an existing hash member to increment > >> + the member's prepare counter. > >> + > >> + returns false on success, > >> + true otherwise. > >> +*/ > >> +static bool xid_member_replace(HASH *hash_arg, my_xid xid_arg, > >> + MEM_ROOT *ptr_mem_root) > >> +{ > >> + xid_recovery_member* member; > >> + if ((member= (xid_recovery_member *) > >> + my_hash_search(hash_arg, (uchar *)& xid_arg, sizeof(xid_arg)))) > >> + member->in_engine_prepare++; > >> + else > >> + member= xid_member_insert(hash_arg, xid_arg, ptr_mem_root); > >> + > >> + return member == NULL; > >> +} > >> + > >> +/* > >> + Hash iterate function to complete with commit or rollback as decided > >> + (typically at binlog recovery processing) in member->in_engine_prepare. > >> +*/ > >> +static my_bool xarecover_do_commit_or_rollback(void *member_arg, > >> + void *hton_arg) > >> +{ > >> + xid_recovery_member *member= (xid_recovery_member*) member_arg; > >> + handlerton *hton= (handlerton*) hton_arg; > >> + xid_t x; > >> + my_bool rc; > >> + > >> + x.set(member->xid); > >> + rc= member->decided_to_commit ? hton->commit_by_xid(hton, &x) : > >> + hton->rollback_by_xid(hton, &x); > >> + > >> + DBUG_ASSERT(rc || member->in_engine_prepare > 0); > >> + > >> + if (!rc) > >> + { > > > > I don't think you can trust rc on that. > > if it's non-zero, it's an error all right. > > but it's a bit of a stretch to presume that > > nonexisting xid is always an error. > > > > also commit_by_xid returns int, not my_bool. > > [?] I mean, if an engine returns success when you ask it to commit or rollback an unknown xid - this is not a bug, right? Commit means "make all changes by this xid persistent", if there are no changes, there is nothing to do, so doing nothing and reporting success isn't exactly wrong. Same for rollback. > I'd rather to stick with the patch and require from any engine to > return \cite{xa spec} > > [XAER_NOTA] The specified XID is not known by the resource manager > > What'd you say? Okay. I'd say I was wrong, if the standard requires a resource manager to return an error for an unknown XID, then you surely can rely on that :) Sorry for confusion. > >> + > >> + return false; > >> +} > >> + > >> +static my_bool xarecover_do_count_in_prepare(void *member_arg, > >> + void *ptr_count) > >> +{ > >> + xid_recovery_member *member= (xid_recovery_member*) member_arg; > >> + if (member->in_engine_prepare) > >> + { > >> + *(uint*) ptr_count += member->in_engine_prepare; > > > > This is a rather weird semantics. > > it's kind of a number of transactions times number of engines. > > if one transaction wasn't committed in two engines and another > > transaction wasn't rolled back in one engine, the counter will be 3. > > how are you going to explain this to users? > > So `ptr_count` actually is for transaction branches, not transactions. > But the ultimate warning speaks erronously the latter > > + sql_print_warning("Could not complete %u number of transactions in " > "engines.", count_in_prepare); > > So I'd convert the counter to count the transactions: > > (*(uint*) ptr_count) ++ > > [?] yes, that'd be easier to understand, thanks > >> diff --git a/sql/log.cc b/sql/log.cc > >> index 8073f09ab88..a9808ed8823 100644 > >> --- a/sql/log.cc > >> +++ b/sql/log.cc ... > >> + char buf[21]; > >> + > >> + longlong10_to_str(ptr_gtid->seq_no, buf, 10); > >> + sql_print_information("Successfully truncated binlog file:%s " > >> + "to pos:%llu to remove transactions starting from " > >> + "GTID %u-%u-%s", file_name, pos, > >> + ptr_gtid->domain_id, ptr_gtid->server_id, buf); > >> + } > >> + if (!(error= init_io_cache(&cache, file, IO_SIZE, WRITE_CACHE, > >> + (my_off_t) pos, 0, MYF(MY_WME|MY_NABP)))) > >> + { > >> + /* > >> + Write Stop_log_event to ensure clean end point for the binary log being > >> + truncated. > >> + */ > > > > I'm not sure about it. The less you touch the corrupted binlog the better. > > simply truncating it would be enough, wouldn't it? > > Yes, it would. > But I somewhat preferred a tidy-up style. > > [?] > > If you feel strong to remove it will be done. I feel it'd be safer not to write anything into the corrupted binlog. > >> + Stop_log_event se; > >> + se.checksum_alg= cs_alg; > >> + if ((error= write_event(&se, NULL, &cache))) > >> + { > >> + sql_print_error("Failed to write stop event to " > >> + "binary log. Errno:%d", > >> + (cache.error == -1) ? my_errno : error); > >> + goto end; > >> + } > >> + clear_inuse_flag_when_closing(cache.file); > >> + if ((error= flush_io_cache(&cache)) || > >> + (error= mysql_file_sync(file, MYF(MY_WME|MY_SYNC_FILESIZE)))) > >> + { > >> + sql_print_error("Faild to write stop event to " > >> + "binary log. Errno:%d", > >> + (cache.error == -1) ? my_errno : error); > >> + } > >> + } > >> + else > >> + sql_print_error("Failed to initialize binary log " > >> + "cache for writing stop event. Errno:%d", > >> + (cache.error == -1) ? my_errno : error); > > > > 1. I don't see why you need to sync after every operation. > > Yep, there's one fsync before Stop event writing, redundant if Stop will > stay in the final patch. As far as I understand, if Stop is removed, you shouldn't need any syncs at all, just truncate and close. > >> + int next_binlog_or_round(int& round, > >> + const char *last_log_name, > >> + const char *binlog_checkpoint_name, > >> + LOG_INFO *linfo, MYSQL_BIN_LOG *log); > >> + bool is_safe() > > > > what do you mean "safe" ? > > Safe to truncate condition is explained elsewhere to mean there are no > non-transactional group of events within the truncated tail. > Earlier mentioned ineffective COMMIT-query ended "transactions" are safe. may be is_safe_to_truncate() then? ... > >> + if (!truncate_validated) > >> + { > >> + DBUG_ASSERT(round <= 2); > >> + /* > >> + Either truncate was not set or was reset, else > >> + it gets incremented, otherwise it may only set to an earlier > >> + offset only at the turn out of the 1st round. > > > > sorry, I cannot parse that :( > > The comments are for the assert in a branch that acts when the truncate position > (gtid) is not yet finalized. > Let me annotate under each assert's line. > > > > >> + */ > >> + DBUG_ASSERT(truncate_gtid.seq_no == 0 || > > 'truncate was not set or was reset' ^, that is truncate_gtid. > The 'was reset' is actually not present in the assert - that'd be > reflected with > truncate_reset_done == true > > > >> + last_gtid_coord >= binlog_truncate_coord || > > 'it gets incremented' ^ why does that mean "it gets incremented" ? > > >> + (round == 2 && truncate_set_in_1st)); > > And if it's neither of the above there must've been truncate candidate > set in the 1st round. ... > >> switch (typ) > >> { > >> + case FORMAT_DESCRIPTION_EVENT: > >> + if (round > 1) > >> + { > >> + if (fdle != fdle_arg) > >> + delete fdle; > >> + fdle= (Format_description_log_event*) ev; > > > > why is it needed? all binlog files should have the same > > Format_description_log_event anyway. > > Actually contrary to what was stated (the patch removed that claim) in > the parent revision of this commit, the FD:s of the > recovery binlog files may have different checksums. I.e > > set @@global.binlog_checksum = value > > can be issued to rotate binlog to a new hot file having different checksum value > while the binlog *CheckPoint* file is behind. So this is possible to > have at recovery: > > B_1(CP, checksum = NONE), B_2(hot, checksum = CRC32) > > which also manifests a normal use case bug I have not reported any bug (though wanted to). okay, checksum can differ. But how does it affect Format_description_log_event ? Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] ee538938345: MDEV-21117: refine the server binlog-based recovery for semisync
by Sergei Golubchik 15 Mar '21

15 Mar '21

Hi, Andrei! On Mar 01, Andrei Elkin wrote: > > I've reviewed almost everything, see comments below. But not the > > Recovery_context methods. Please explain how it works and how all > > these truncate_validated, truncate_reset_done, truncate_set_in_1st, > > etc all work together. > > ...specifically to this point. Just in case I hope you did not miss to > read recovery_design.txt from the MDEV, which does not go into coding > details that you're effectively about in above. I've read it now. Still I don't understand why you need an extra round of binlog scanning (two rounds if only one file, three rounds if many). Also, this recovery_design.txt is not part of the code, so whoever will look at it later will be just as puzzled as I was. ... > [ G1, G2, ..., G_k, g_{k+1}, ... g_n ] > > here the uppercase `G' stands for committed trx, the smallcase `g' for > prepared,`_k' - sub-scripts in the recovery sequence. As the capital > letter first rules in the single-engine case the first occurrence of a > pattern `G_k,g_k+1' identifies the truncate index. The patch reflects > such fact with raising `truncate_validated' flag. > > But it's more complicated in the multiple binlog files / engines case. > The first `Gg' letter-case drop is not guaranteed to be the only drop > so `truncate_reset_done' and `truncate_set_in_1st' are introduced to > help with truncate index identification. Why is it not guaranteed? > When `truncate_validated' is set that indicates the truncate index is > determined and may not change in the current (1st of 2nd) nor future > rounds. `truncate_reset_done' says that an "inverse" `g_k,G_k+1' pair > is found so that any earlier truncation candidate gets reset (to > "zero"). If there will be later any candidate found in *this* (1st or > 2nd) round in the sequence its index will be obviously greater. > > `truncate_set_in_1st' function is to remember that the truncate > candidate was found in the 1st round (in the "hot" binlog file), but > if the candidate has not been validated `!truncate_validated' it may > be exacted in the 2nd round and then to an earlier transaction. So the > flag helps to handle exception from truncate candidate monotony rule: > e.g the hot binlog B2 contains `[g5,g6,...g_n]' and a ref to binlog > checkpoint file B1 that contains `[G1,g2,G3,g4]'. The first round > truncate candidate of g5 would be first exacted to `g2' before finally > ascertained to `g4' in the 2nd round. (Notice `g2 -> g4' preserves the > truncate index monotony). > > Notice that due to exacting like `g2 -> g4' in the 2nd round of the > above example `g2' got "to be up-cased" into `G2' for committing > (feasible with two trx:s on two different engine scenario - g2 with > Innodb only got prepared, G4 - on Rocksdb, and got committed prior the > crash). That's what the 3rd round is for. > > I hope this will be helpful. Unfortunately, not very much. It describes how you juggle with variables and scanning rounds. But not *what* you're trying to find. And it's kind of difficult to reverse engineer your "how" back into "what" :( Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

[Maria-developers] MDEV-17399: JSON_TABLE: Incorrect code with table elimination
by Sergey Petrunia 15 Mar '21

15 Mar '21

Hi Alexey, > diff --git a/sql/opt_table_elimination.cc b/sql/opt_table_elimination.cc > index 3958797ec44..f2497d524ec 100644 > --- a/sql/opt_table_elimination.cc > +++ b/sql/opt_table_elimination.cc > @@ -637,6 +637,16 @@ void eliminate_tables(JOIN *join) > List_iterator<Item> it(join->fields_list); > while ((item= it++)) > used_tables |= item->used_tables(); > + > + { > + List_iterator<TABLE_LIST> it(*join->join_list); > + TABLE_LIST *tbl; > + while ((tbl= it++)) > + { > + if (tbl->table_function) > + used_tables|= tbl->table_function->used_tables(); > + } > + } > This only walks the tables that are "at the top level" of the join. if a JSON_TABLE(...) is inside a nested outer join, it will not be found. Please walk select_lex->leaf_tables instead. Please add the testcase (I provide one below). Please also add a comment, clarifying what is being done, something like: Table function JSON_TABLE() can have references to other tables. Do not eliminate the tables that JSON_TABLE() refers to. Note: the JSON_TABLE itself cannot be eliminated as it doesn't have unique keys. == Testcase == create table t20 (a int not null); create table t21 (a int not null primary key, js varchar(100)); insert into t20 select seq from seq_1_to_100; insert into t21 select a, '{"a":100}' from t20; create table t31(a int); create table t32(b int); insert into t31 values (1); insert into t32 values (1); explain select t20.a, jt1.ab from t20 left join t21 on t20.a=t21.a join (t31 left join (t32 join JSON_TABLE(t21.js,'$' COLUMNS (ab INT PATH '$.a')) AS jt1) on t31.a<3); BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://petrunia.net

2 3

Re: [Maria-developers] f1db957c5da: MDEV-21469: Implement crash-safe binary logging of the user XA
by Sergei Golubchik 14 Mar '21

14 Mar '21

Hi, Sujatha! Note, this is a review of a combined diff dc92235f21e + f1db957c5da On Mar 14, Sujatha wrote: > revision-id: f1db957c5da (mariadb-10.5.2-247-gf1db957c5da) > parent(s): dc92235f21e > author: Sujatha <sujatha.sivakumar(a)mariadb.com> > committer: Andrei Elkin <andrei.elkin(a)mariadb.com> > timestamp: 2020-12-21 16:10:46 +0200 > message: > > MDEV-21469: Implement crash-safe binary logging of the user XA > > Make XA PREPARE, XA COMMIT and XA ROLLBACK statements > crash-safe when --log-bin is specified. > > At execution of XA PREPARE, XA COMMIT and XA ROLLBACK their replication > events are made written into the binary log prior to execution > of the commands in storage engines. > In case the server crashes, after writing to binary log but before all > involved engines have processed the command, the following recovery > will execute the command's replication events to equalize the states > of involved engines with that of binlog. > That applies to all XA PREPARE *group* and XA COMMIT or ROLLBACK. > > On the implementation level the recovery time binary log parsing > is augmented to pay attention to > the user XA xids to identify the XA transactions' state:s in binary log, and > eventually match them against their states in engines, see > MYSQL_BIN_LOG::recover_explicit_xa_prepare(). > In discrepancy cases the outdated state in the engines is corrected with > resubmitting the transaction prepare group of events, or completion > ones. The multi-engine partly prepared XA PREPARE case > the XA is rolled back first. > The fact of multiple-engine involved is registered into > Gtid_log_event::flags2 as one bit. The boolean value is > sufficient and precise to deal with two engines in XA transaction. > With more than 2 recoverable engines the flag method is still correct though > may be pessimistic, as it treats all recoverable engines as XA > participants. So when the number of such Engines exceeds the number > of prepared engines of the XA that XA is treated > as partially completed, with all that ensued. > As an optimization no new bit is allocated in flags2, instead a > pre-existing ones (of MDEV-742) are reused, observing that > A. XA "COMPLETE" does not require multi-engine hint for its recovery and that > B. the MDEV-742 XA-completion bit is not anyhow used by > XA-PREPARE and its GTID log event. > > Notice the multi-engine flagging is superceded by MDEV-21117 extension > in Gtid log event so this part should be taken from there. > diff --git a/mysql-test/suite/binlog/t/binlog_xa_multi_binlog_crash_recovery.test b/mysql-test/suite/binlog/t/binlog_xa_multi_binlog_crash_recovery.test > new file mode 100644 > index 00000000000..d0852de2fcc > --- /dev/null > +++ b/mysql-test/suite/binlog/t/binlog_xa_multi_binlog_crash_recovery.test > @@ -0,0 +1,86 @@ > +# ==== Purpose ==== > +# > +# Test verifies that XA crash recovery works fine across multiple binary logs. > +# > +# ==== Implementation ==== > +# > +# Steps: > +# 0 - Generate an explicit XA transaction. Using debug simulation hold the "debug sync" != "debug sim" :) > +# execution of XA PREPARE statement after the XA PREPARE is written to > +# the binary log. With this the prepare will not be done in engine. > +# 1 - By executing FLUSH LOGS generate multiple binary logs. > +# 2 - Now make the server to disappear at this point. > +# 3 - Restart the server. During recovery the XA PREPARE from the binary > +# log will be read. It is cross checked with engine. Since it is not > +# present in engine it will be executed once again. > +# 4 - When server is up execute XA RECOVER to check that the XA is > +# prepared in engine as well. > +# 5 - XA COMMIT the transaction and check the validity of the data. > +# > +# ==== References ==== > +# > +# MDEV-21469: Implement crash-safe logging of the user XA > +# > + > +--source include/have_innodb.inc > +--source include/have_debug.inc > +--source include/have_debug_sync.inc > +--source include/have_log_bin.inc > + > +RESET MASTER; > + > +CREATE TABLE t1 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; > +CALL mtr.add_suppression("Found 1 prepared XA transactions"); > + > +connect(con1,localhost,root,,); > +XA START 'xa1'; > +INSERT INTO t1 SET a=1; > +SET DEBUG_SYNC= "simulate_hang_after_binlog_prepare SIGNAL con1_ready WAIT_FOR con1_go"; > +SET GLOBAL DEBUG_DBUG="d,simulate_crash_after_binlog_prepare"; > +XA END 'xa1'; > +--send XA PREPARE 'xa1'; > + > +connection default; > +SET DEBUG_SYNC= "now WAIT_FOR con1_ready"; > +FLUSH LOGS; > +FLUSH LOGS; > +FLUSH LOGS; > + > +--source include/show_binary_logs.inc > +--let $binlog_file= master-bin.000004 > +--let $binlog_start= 4 > +--source include/show_binlog_events.inc > + > +--write_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect > +wait I don't see why you'd need to wait first, it'd be simpler to restart right away. > +EOF > + > +--error 0,2013 > +SET DEBUG_SYNC= "now SIGNAL con1_go"; > +--source include/wait_until_disconnected.inc > + > +--connection con1 > +--error 2013 > +--reap > +--source include/wait_until_disconnected.inc > + > +# > +# Server restart > +# > +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect > +restart > +EOF > + > +connection default; > +--enable_reconnect > +--source include/wait_until_connected_again.inc > + > +XA RECOVER; > +XA COMMIT 'xa1'; > + > +SELECT * FROM t1; > + > +# Clean up. > +connection default; > +DROP TABLE t1; > + > diff --git a/mysql-test/suite/rpl/t/rpl_xa_commit_crash_safe.test b/mysql-test/suite/rpl/t/rpl_xa_commit_crash_safe.test > new file mode 100644 > index 00000000000..e972e3f09de > --- /dev/null > +++ b/mysql-test/suite/rpl/t/rpl_xa_commit_crash_safe.test > @@ -0,0 +1,98 @@ > +# ==== Purpose ==== > +# > +# Test verifies that XA COMMIT statements are crash safe. > +# > +# ==== Implementation ==== > +# > +# Steps: > +# 0 - Generate 2 explicit XA transactions. 'xa1' and 'xa2'. > +# 'xa1' will be prepared and committed. > +# 1 - For 'xa2' let the XA COMMIT be done in binary log and crash the > +# server so that it is not committed in engine. > +# 2 - Restart the server. The recovery code should successfully recover > +# 'xa2'. The COMMIT should be executed during recovery. > +# 3 - Check the data in table. Both rows should be present in table. > +# 4 - Trying to commit 'xa2' should report unknown 'XA' error as COMMIT is > +# already complete during recovery. > +# > +# ==== References ==== > +# > +# MDEV-21469: Implement crash-safe logging of the user XA > + > + > +--source include/have_innodb.inc > +--source include/master-slave.inc master-slave must be always included last > +--source include/have_debug.inc > + > +connect (master2,localhost,root,,); why do you need master2 connection? > +--connection master > +CALL mtr.add_suppression("Found 1 prepared XA transactions"); > + > +CREATE TABLE t ( f INT ) ENGINE=INNODB; > +XA START 'xa1'; > +INSERT INTO t VALUES (20); > +XA END 'xa1'; > +XA PREPARE 'xa1'; > +XA COMMIT 'xa1'; > +--sync_slave_with_master > +--source include/stop_slave.inc > + > +--connection master1 > +XA START 'xa2'; > +INSERT INTO t VALUES (40); > +XA END 'xa2'; > +XA PREPARE 'xa2'; > + > +--write_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect > +wait > +EOF Same. Why do you wait? > + > +SET GLOBAL DEBUG_DBUG="d,simulate_crash_after_binlog_commit_or_rollback"; > +--error 2013 # CR_SERVER_LOST > +XA COMMIT 'xa2'; > +--source include/wait_until_disconnected.inc > + > +--connection master1 > +--source include/wait_until_disconnected.inc > + > +--connection master > +--source include/wait_until_disconnected.inc > + > +# > +# Server restart > +# > +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect > +restart > +EOF > + > +connection default; > +--enable_reconnect > +--source include/wait_until_connected_again.inc > + > +# rpl_end.inc needs to use the connection server_1 > +connection server_1; > +--enable_reconnect > +--source include/wait_until_connected_again.inc > + > +--connection master > +--enable_reconnect > +--source include/wait_until_connected_again.inc > + > +--connection slave > +--source include/start_slave.inc > +--sync_with_master > + > +--connection master > +SELECT * FROM t; > +XA RECOVER; > +--error 1397 # ER_XAER_NOTA > +XA COMMIT 'xa2'; > +SELECT * FROM t; > +--sync_slave_with_master > + > +SELECT * FROM t; > + > +--connection master > +DROP TABLE t; > +--sync_slave_with_master > +--source include/rpl_end.inc > diff --git a/mysql-test/suite/rpl/t/rpl_xa_event_apply_failure.test b/mysql-test/suite/rpl/t/rpl_xa_event_apply_failure.test > new file mode 100644 > index 00000000000..14cebbd9b13 > --- /dev/null > +++ b/mysql-test/suite/rpl/t/rpl_xa_event_apply_failure.test > @@ -0,0 +1,119 @@ > +# ==== Purpose ==== > +# > +# Test verifies that if for some reason an event cannot be applied during > +# recovery, appropriate error is reported. > +# > +# ==== Implementation ==== > +# > +# Steps: > +# 0 - Generate 2 explicit XA transactions. 'xa1' and 'xa2'. > +# 'xa1' will be prepared and committed. > +# 1 - For 'xa2' let the XA COMMIT be done in binary log and crash the > +# server so that it is not committed in engine. > +# 2 - Restart the server. Using debug simulation point make XA COMMIT 'xa2' > +# execution to fail. The server will resume anyway > +# to leave the error in the errlog (see "Recovery: Error.."). > +# 3 - Work around the simulated failure with Commit once again > +# from a connection that turns OFF binlogging. > +# Slave must catch up with the master. > +# > +# ==== References ==== > +# > +# MDEV-21469: Implement crash-safe logging of the user XA > + > + > +--source include/have_innodb.inc > +--source include/master-slave.inc master-slave must be always included last > +--source include/have_debug.inc > + > +connect (master2,localhost,root,,); why do you need master2 connection? > +--connection master > +CALL mtr.add_suppression("Found 1 prepared XA transactions"); > +CALL mtr.add_suppression("Failed to execute binlog query event"); > +CALL mtr.add_suppression("Recovery: Error .Out of memory.."); > +CALL mtr.add_suppression("Crash recovery failed."); > +CALL mtr.add_suppression("Can.t init tc log"); > +CALL mtr.add_suppression("Aborting"); > + > +CREATE TABLE t ( f INT ) ENGINE=INNODB; > +XA START 'xa1'; > +INSERT INTO t VALUES (20); > +XA END 'xa1'; > +XA PREPARE 'xa1'; > +XA COMMIT 'xa1'; > +--sync_slave_with_master > +--source include/stop_slave.inc > + > +--connection master1 > +XA START 'xa2'; > +INSERT INTO t VALUES (40); > +XA END 'xa2'; > +XA PREPARE 'xa2'; > + > +--write_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect > +wait ditto > +EOF > + > +SET GLOBAL DEBUG_DBUG="d,simulate_crash_after_binlog_commit_or_rollback"; > +--error 2013 # CR_SERVER_LOST > +XA COMMIT 'xa2'; > +--source include/wait_until_disconnected.inc > + > +--connection master1 > +--source include/wait_until_disconnected.inc > + > +--connection master > +--source include/wait_until_disconnected.inc > + > +# > +# Server restart > +# > +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect > +restart: --debug-dbug=d,trans_xa_commit_fail > +EOF > + > +connection default; > +--source include/wait_until_disconnected.inc > + > +connection default; > +--enable_reconnect > +--source include/wait_until_connected_again.inc > + > +--connection master > +--enable_reconnect > +--echo *** must be no 'xa2' commit seen, as it's still prepared: > +SELECT * FROM t; > +XA RECOVER; > + > +# Commit it manually now to work around the extra binlog record > +# by turning binlogging OFF by the connection. > + > +SET GLOBAL DEBUG_DBUG=""; > +SET SQL_LOG_BIN=0; > +--error 0 why error 0? What will happen if you don't disable binlog? > +XA COMMIT 'xa2'; > +SET SQL_LOG_BIN=1; > + > + > +# rpl_end.inc needs to use the connection server_1 > +connection server_1; > +--enable_reconnect > +--source include/wait_until_connected_again.inc > + > +--connection master > +--source include/wait_until_connected_again.inc > + > +--connection slave > +--source include/start_slave.inc > +--sync_with_master > + > +--connection master > +SELECT * FROM t; > + > +--sync_slave_with_master > +SELECT * FROM t; > + > +--connection master > +DROP TABLE t; > +--sync_slave_with_master > +--source include/rpl_end.inc > diff --git a/mysql-test/suite/rpl/t/rpl_xa_prepare_commit_prepare.test b/mysql-test/suite/rpl/t/rpl_xa_prepare_commit_prepare.test > new file mode 100644 > index 00000000000..7b987c7f29b > --- /dev/null > +++ b/mysql-test/suite/rpl/t/rpl_xa_prepare_commit_prepare.test > @@ -0,0 +1,95 @@ > +# ==== Purpose ==== > +# > +# Test verifies that XA PREPARE transactions are crash safe. > +# > +# ==== Implementation ==== > +# > +# Steps: > +# 0 - Generate 2 explicit XA transactions. 'xa1' and 'xa2'. > +# 'xa1' will be prepared and committed. > +# 1 - For 'xa2' let the XA PREPARE be done in binary log and crash the > +# server so that it is not prepared in engine. > +# 2 - Restart the server. The recovery code should successfully recover > +# 'xa2'. > +# 3 - When server is up, execute XA RECOVER and verify that 'xa2' is > +# present. > +# 4 - Commit the XA transaction and verify its correctness. > +# > +# ==== References ==== > +# > +# MDEV-21469: Implement crash-safe logging of the user XA > + > +--source include/have_innodb.inc > +--source include/master-slave.inc ditto > +--source include/have_debug.inc > + > +connect (master2,localhost,root,,); ditto > +--connection master > +CALL mtr.add_suppression("Found 1 prepared XA transactions"); > + > +CREATE TABLE t ( f INT ) ENGINE=INNODB; > +XA START 'xa1'; > +INSERT INTO t VALUES (20); > +XA END 'xa1'; > +XA PREPARE 'xa1'; > +XA COMMIT 'xa1'; > +--sync_slave_with_master > +--source include/stop_slave.inc > + > +--connection master1 > +XA START 'xa2'; > +INSERT INTO t VALUES (40); > +XA END 'xa2'; > + > +--write_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect > +wait ditto > +EOF > + > +SET GLOBAL DEBUG_DBUG="d,simulate_crash_after_binlog_prepare"; > +--error 2013 # CR_SERVER_LOST > +XA PREPARE 'xa2'; > +--source include/wait_until_disconnected.inc > + > +--connection master1 > +--source include/wait_until_disconnected.inc > + > +--connection master > +--source include/wait_until_disconnected.inc > + > +# > +# Server restart > +# > +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect > +restart > +EOF > + > +connection default; > +--enable_reconnect > +--source include/wait_until_connected_again.inc > + > +# rpl_end.inc needs to use the connection server_1 > +connection server_1; > +--enable_reconnect > +--source include/wait_until_connected_again.inc > + > +--connection master > +--enable_reconnect > +--source include/wait_until_connected_again.inc > + > +--connection slave > +--source include/start_slave.inc > +--sync_with_master > + > +--connection master > +SELECT * FROM t; > +XA RECOVER; > +XA COMMIT 'xa2'; > +SELECT * FROM t; > +--sync_slave_with_master > + > +SELECT * FROM t; > + > +--connection master > +DROP TABLE t; > +--sync_slave_with_master > +--source include/rpl_end.inc > diff --git a/mysql-test/suite/rpl/t/rpl_xa_prepare_crash_safe.test b/mysql-test/suite/rpl/t/rpl_xa_prepare_crash_safe.test > new file mode 100644 > index 00000000000..9d2c5cce528 > --- /dev/null > +++ b/mysql-test/suite/rpl/t/rpl_xa_prepare_crash_safe.test > @@ -0,0 +1,117 @@ > +# ==== Purpose ==== > +# > +# Test verifies that XA PREPARE transactions are crash safe. > +# > +# ==== Implementation ==== > +# > +# Steps: > +# 0 - Generate 3 explicit XA transactions. 'xa1', 'xa2' and 'xa3'. > +# Using debug simulation hold the execution of second XA PREPARE > +# statement after the XA PREPARE is written to the binary log. > +# With this the prepare will not be done in engine. > +# 1 - For 'xa3' allow the PREPARE statement to be written to binary log and > +# simulate server crash. > +# 2 - Restart the server. The recovery code should successfully recover > +# 'xa2' and 'xa3'. > +# 3 - When server is up, execute XA RECOVER and verify that 'xa2' and 'xa3' > +# are present along with 'xa1'. > +# 4 - Commit all the XA transactions and verify their correctness. > +# > +# ==== References ==== > +# > +# MDEV-21469: Implement crash-safe logging of the user XA > + > + > +--source include/have_innodb.inc > +--source include/master-slave.inc ditto > +--source include/have_debug.inc > + > +connect (master2,localhost,root,,); ok, in this test you actually use master2 :) > +--connection master > +CALL mtr.add_suppression("Found 1 prepared XA transactions"); > +CALL mtr.add_suppression("Found 2 prepared XA transactions"); > +CALL mtr.add_suppression("Found 3 prepared XA transactions"); > + > +CREATE TABLE t ( f INT ) ENGINE=INNODB; > +XA START 'xa1'; > +INSERT INTO t VALUES (20); > +XA END 'xa1'; > +XA PREPARE 'xa1'; > +--sync_slave_with_master > +--source include/stop_slave.inc > + > +--connection master1 > +use test; > +xa start 'xa2'; > +insert into t values (30); > +xa end 'xa2'; > +SET DEBUG_SYNC="simulate_hang_after_binlog_prepare SIGNAL reached WAIT_FOR go"; > +send xa prepare 'xa2'; > + > +--connection master2 > +let $wait_condition= > + SELECT COUNT(*) = 1 FROM INFORMATION_SCHEMA.PROCESSLIST > + WHERE STATE like "debug sync point: simulate_hang_after_binlog_prepare%"; > +--source include/wait_condition.inc eh? you can just 'WAIT_FOR signal', couldn't you? > + > +XA START 'xa3'; > +INSERT INTO t VALUES (40); > +XA END 'xa3'; > + > +--write_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect > +wait ditto > +EOF > + > +SET GLOBAL DEBUG_DBUG="d,simulate_crash_after_binlog_prepare"; > +--error 2013 # CR_SERVER_LOST > +XA PREPARE 'xa3'; > +--source include/wait_until_disconnected.inc > + > +--connection master1 > +--error 2013 > +--reap > +--source include/wait_until_disconnected.inc > + > +--connection master > +--source include/wait_until_disconnected.inc > + > +# > +# Server restart > +# > +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect > +restart > +EOF > + > +connection default; > +--enable_reconnect > +--source include/wait_until_connected_again.inc > + > +# rpl_end.inc needs to use the connection server_1 > +connection server_1; > +--enable_reconnect > +--source include/wait_until_connected_again.inc > + > +--connection master > +--enable_reconnect > +--source include/wait_until_connected_again.inc > + > + > +--connection slave > +--source include/start_slave.inc > +--sync_with_master > + > +--connection master > +SELECT * FROM t; > +XA RECOVER; > +XA COMMIT 'xa1'; > +XA COMMIT 'xa2'; > +XA COMMIT 'xa3'; > +SELECT * FROM t; > +--sync_slave_with_master > + > +SELECT * FROM t; > + > +--connection master > +DROP TABLE t; > +--sync_slave_with_master > +--source include/rpl_end.inc > diff --git a/mysql-test/suite/rpl/t/rpl_xa_rollback_commit_crash_safe.test b/mysql-test/suite/rpl/t/rpl_xa_rollback_commit_crash_safe.test > new file mode 100644 > index 00000000000..1d19f96116e > --- /dev/null > +++ b/mysql-test/suite/rpl/t/rpl_xa_rollback_commit_crash_safe.test > @@ -0,0 +1,97 @@ > +# ==== Purpose ==== > +# > +# Test verifies that XA COMMIT statements are crash safe. XA ROLLBACK > +# > +# ==== Implementation ==== > +# > +# Steps: > +# 0 - Generate 2 explicit XA transactions. 'xa1' and 'xa2'. > +# 'xa1' will be prepared and committed. > +# 1 - For 'xa2' let the XA ROLLBACK be done in binary log and crash the > +# server so that it is not committed in engine. not rolled back > +# 2 - Restart the server. The recovery code should successfully recover > +# 'xa2'. The ROLLBACK should be executed during recovery. > +# 3 - Check the data in table. Only one row should be present in table. > +# 4 - Trying to rollback 'xa2' should report unknown 'XA' error as rollback > +# is already complete during recovery. > +# > +# ==== References ==== > +# > +# MDEV-21469: Implement crash-safe logging of the user XA > + > +--source include/have_innodb.inc > +--source include/master-slave.inc ditto > +--source include/have_debug.inc > + > +connect (master2,localhost,root,,); unused again > +--connection master > +CALL mtr.add_suppression("Found 1 prepared XA transactions"); > + > +CREATE TABLE t ( f INT ) ENGINE=INNODB; > +XA START 'xa1'; > +INSERT INTO t VALUES (20); > +XA END 'xa1'; > +XA PREPARE 'xa1'; > +XA COMMIT 'xa1'; > +--sync_slave_with_master > +--source include/stop_slave.inc > + > +--connection master1 > +XA START 'xa2'; > +INSERT INTO t VALUES (40); > +XA END 'xa2'; > +XA PREPARE 'xa2'; > + > +--write_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect > +wait ditto > +EOF > + > +SET GLOBAL DEBUG_DBUG="d,simulate_crash_after_binlog_commit_or_rollback"; > +--error 2013 # CR_SERVER_LOST > +XA ROLLBACK 'xa2'; > +--source include/wait_until_disconnected.inc > + > +--connection master1 > +--source include/wait_until_disconnected.inc > + > +--connection master > +--source include/wait_until_disconnected.inc > + > +# > +# Server restart > +# > +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect > +restart > +EOF > + > +connection default; > +--enable_reconnect > +--source include/wait_until_connected_again.inc > + > +# rpl_end.inc needs to use the connection server_1 > +connection server_1; > +--enable_reconnect > +--source include/wait_until_connected_again.inc > + > +--connection master > +--enable_reconnect > +--source include/wait_until_connected_again.inc > + > +--connection slave > +--source include/start_slave.inc > +--sync_with_master > + > +--connection master > +SELECT * FROM t; > +XA RECOVER; > +--error 1397 # ER_XAER_NOTA > +XA ROLLBACK 'xa2'; > +SELECT * FROM t; > +--sync_slave_with_master > + > +SELECT * FROM t; > + > +--connection master > +DROP TABLE t; > +--sync_slave_with_master > +--source include/rpl_end.inc > diff --git a/storage/rocksdb/mysql-test/rocksdb_rpl/t/rpl_rocksdb_xa_recover.test b/storage/rocksdb/mysql-test/rocksdb_rpl/t/rpl_rocksdb_xa_recover.test > new file mode 100644 > index 00000000000..c7cefbda4bf > --- /dev/null > +++ b/storage/rocksdb/mysql-test/rocksdb_rpl/t/rpl_rocksdb_xa_recover.test > @@ -0,0 +1,46 @@ > +# MDEV-742, MDEV-21469 XA replication, and xa crash-safe. > +# Tests prove xa state is recovered to a prepared or completed state > +# upon post-crash recovery, incl a multi-engine case. > + > +--source include/have_rocksdb.inc > +--source include/have_innodb.inc > +--source include/have_debug.inc > +--source include/master-slave.inc ditto > +--source include/have_binlog_format_row.inc > + > +--connection slave > +--source include/stop_slave.inc > + > +--connection master > +CALL mtr.add_suppression("Found . prepared XA transactions"); > +CALL mtr.add_suppression("Failed to execute binlog query event"); > +CREATE TABLE t1 (a INT PRIMARY KEY) ENGINE=rocksdb; > +CREATE TABLE t2 (a INT PRIMARY KEY) ENGINE=innodb; > +INSERT INTO t1 SET a = 1; > +INSERT INTO t2 SET a = 1; > + > +--let $xa=xa1 > +--let $when=after_binlog > +--source rpl_rocksdb_xa_recover.inc > + > +--let $xa=xa2 > +--let $when=after_first_engine > +--let $finally_expected=$xa in the list > +--source rpl_rocksdb_xa_recover.inc > + > +--connection slave > +--source include/start_slave.inc > + > +--connection master > +--sync_slave_with_master > +let $diff_tables= master:t1, slave:t1; > +source include/diff_tables.inc; > +let $diff_tables= master:t2, slave:t2; > +source include/diff_tables.inc; > + > +--connection master > +SET SESSION DEBUG_DBUG=""; > +drop table t1,t2; > +--sync_slave_with_master > + > +--source include/rpl_end.inc > diff --git a/storage/rocksdb/mysql-test/rocksdb_rpl/t/rpl_rocksdb_xa_recover.inc b/storage/rocksdb/mysql-test/rocksdb_rpl/t/rpl_rocksdb_xa_recover.inc > new file mode 100644 > index 00000000000..312b8b6a19e > --- /dev/null > +++ b/storage/rocksdb/mysql-test/rocksdb_rpl/t/rpl_rocksdb_xa_recover.inc > @@ -0,0 +1,67 @@ > +# callee of rpl_rocksdb_xa_recover.test > +# requires t1,t2 as defined in the caller. > + > +--let $at=_prepare > +--let $finally_expected=$xa in the list > +--let $init_val=`SELECT MAX(a) from t1 as t_1` why as t_1 ? > +--eval XA START '$xa' > +--eval INSERT INTO t1 SET a=$init_val + 1 > +--eval INSERT INTO t2 SET a=$init_val + 1 > +--eval XA END '$xa' > +--eval SET SESSION DEBUG_DBUG="d,simulate_crash_$when$at" > + > +--exec echo "restart" > $MYSQLTEST_VARDIR/tmp/mysqld.1.expect > +--error 2013 # CR_SERVER_LOST > +--eval XA PREPARE '$xa' > +--source include/wait_until_disconnected.inc > +--let $rpl_server_number = 1 > +--source include/rpl_reconnect.inc > +--echo "*** $finally_expected" > +XA RECOVER; > +--eval XA ROLLBACK '$xa' > +--eval SELECT MAX(a) - $init_val as zero FROM t1 > +--eval SELECT MAX(a) - $init_val as zero FROM t2 > + > + > +--let $at=_commit_or_rollback > +--let $finally_expected=empty list (rolled back) > +--eval XA START '$xa' > +--eval INSERT INTO t1 SET a=$init_val + 2 > +--eval INSERT INTO t2 SET a=$init_val + 2 > +--eval XA END '$xa' > +--eval XA PREPARE '$xa' > + > +--eval SET SESSION DEBUG_DBUG="d,simulate_crash_$when$at" > +--exec echo "restart" > $MYSQLTEST_VARDIR/tmp/mysqld.1.expect > +--error 2013 # CR_SERVER_LOST > +--eval XA ROLLBACK '$xa' > +--source include/wait_until_disconnected.inc > +--let $rpl_server_number = 1 > +--source include/rpl_reconnect.inc > + > +--echo "*** $finally_expected" > +XA RECOVER; > +--eval SELECT MAX(a) - $init_val as zero FROM t1 > +--eval SELECT MAX(a) - $init_val as zero FROM t2 > + > + > +--let $at=_commit_or_rollback > +--let $finally_expected=empty list (committed away) > +--eval XA START '$xa' > +--eval INSERT INTO t1 SET a=$init_val + 3 > +--eval INSERT INTO t2 SET a=$init_val + 3 > +--eval XA END '$xa' > +--eval XA PREPARE '$xa' > + > +--eval SET SESSION DEBUG_DBUG="d,simulate_crash_$when$at" > +--exec echo "restart" > $MYSQLTEST_VARDIR/tmp/mysqld.1.expect > +--error 2013 # CR_SERVER_LOST > +--eval XA COMMIT '$xa' > +--source include/wait_until_disconnected.inc > +--let $rpl_server_number = 1 > +--source include/rpl_reconnect.inc > + > +--echo "*** $finally_expected" > +XA RECOVER; > +--eval SELECT MAX(a) - $init_val as three FROM t1 > +--eval SELECT MAX(a) - $init_val as three FROM t2 > diff --git a/sql/log_event.h b/sql/log_event.h > index 639cbfbe7aa..e1880339e85 100644 > --- a/sql/log_event.h > +++ b/sql/log_event.h > @@ -3614,6 +3616,12 @@ class Gtid_log_event: public Log_event > static const uchar FL_PREPARED_XA= 64; > /* FL_"COMMITTED or ROLLED-BACK"_XA is set for XA transaction. */ > static const uchar FL_COMPLETED_XA= 128; > + /* > + To mark the fact of multiple transactional engine participants > + in the prepared XA. The FL_COMPLETED_XA bit is reused by XA_PREPARE_LOG_EVENT, What do you mean by that? XA_prepare_log_event does not inherit from Gtid_log_event, it cannot use Gtid_log_event flags. And, please, explain in a comment why you can reuse a bit for two different flags. Like "the ambiguity between FL_COMPLETED_XA and FL_MULTI_ENGINE_XA is resolved by the flag FL_PREPARED_XA. if it's set, then 128 means FL_MULTI_ENGINE_XA, if it's not set, then 128 means FL_COMPLETED_XA" > + oth the XA completion events do not need such marking. other > + */ > + static const uchar FL_MULTI_ENGINE_XA= 128; > > #ifdef MYSQL_SERVER > Gtid_log_event(THD *thd_arg, uint64 seq_no, uint32 domain_id, bool standalone, > diff --git a/sql/handler.h b/sql/handler.h > index 0eff7bd930d..9acdd47bfc2 100644 > --- a/sql/handler.h > +++ b/sql/handler.h > @@ -1880,8 +1891,18 @@ class Ha_trx_info > m_ht= ht_arg; > m_flags= (int) TRX_READ_ONLY; /* Assume read-only at start. */ > > - m_next= trans->ha_list; > - trans->ha_list= this; > + if (likely(!past_head)) > + { > + m_next= trans->ha_list; > + trans->ha_list= this; > + } > + else > + { > + DBUG_ASSERT(trans->ha_list); > + > + m_next= trans->ha_list->m_next; > + trans->ha_list->m_next= this; > + } I don't like this fake generalization. There's no point for an arbitrary engine to be put at a specific place in the list. Only binlog is special. At least, remove this new argument and check for binlog_hton here. But really, think of this - binlog is so special, may be it'd be easier not to pretend it's a storage engine at all? No need for a fake binlog_hton, no need to register it, etc. It kind of made sense in 2005, but it doesn't necessarily make it now. > } > > /** Clear, prepare for reuse. */ > diff --git a/sql/handler.cc b/sql/handler.cc > index ffb89b30d92..b981294f0d6 100644 > --- a/sql/handler.cc > +++ b/sql/handler.cc > @@ -1338,6 +1345,9 @@ int ha_prepare(THD *thd) > handlerton *ht= ha_info->ht(); > if (ht->prepare) > { > + DBUG_EXECUTE_IF("simulate_crash_after_first_engine_prepare", > + if (!ha_info->next()) DBUG_SUICIDE();); looks like not simulate_crash_after_first_engine_prepare, but more like simulate_crash_before_last_engine_prepare. it's the same point in time if you have only two engines, but the code still implements "before last", not "after first" > + > if (unlikely(prepare_or_error(ht, thd, all))) > { > ha_rollback_trans(thd, all); > @@ -1368,22 +1378,22 @@ int ha_prepare(THD *thd) > } > > /* > - Like ha_check_and_coalesce_trx_read_only to return counted number of > - read-write transaction participants limited to two, but works in the 'all' > - context. > - Also returns the last found rw ha_info through the 2nd argument. > + Returns counted number of > + read-write recoverable transaction participants optionally limited to two. > + Also optionally returns the last found rw ha_info through the 2nd argument. > */ > -uint ha_count_rw_all(THD *thd, Ha_trx_info **ptr_ha_info) > +uint ha_count_rw_all(THD *thd, Ha_trx_info **ptr_ha_info, bool count_through) I don't understand why you need a new argument. ptr_ha_info==NULL means don't return a value there, this is a standard convention used everywhere. also, it's not an "count rw all" it's are there more rw-all htons than N where N can be 1 or 2. So: * either take N as an argument and return as soon as rw_ha_count > N * or don't overoptimize here and just return full count every time > { > unsigned rw_ha_count= 0; > > for (auto ha_info= thd->transaction.all.ha_list; ha_info; > ha_info= ha_info->next()) > { > - if (ha_info->is_trx_read_write()) > + if (ha_info->is_trx_read_write() && ha_info->ht()->recover) > { > - *ptr_ha_info= ha_info; > - if (++rw_ha_count > 1) > + if (ptr_ha_info) > + *ptr_ha_info= ha_info; > + if (++rw_ha_count > 1 && !count_through) > break; > } > } > @@ -1876,6 +1886,10 @@ commit_one_phase_2(THD *thd, bool all, THD_TRANS *trans, bool is_real_trans) > { > int err; > handlerton *ht= ha_info->ht(); > + > + DBUG_EXECUTE_IF("simulate_crash_after_first_engine_commit_or_rollback", > + if (!ha_info->next()) DBUG_SUICIDE();); 1. same as above 2. why not to use *different* names for these two crashes? in case, you know, you'll want to crash at a specific point in the code, not just somewhere? And in a case you'll actually want to crash just somewhere you can always use "+d,simulate_crash_after_first_engine_commit,simulate_crash_after_first_engine_rollback" (and it should be "before_last", of course, not "after_first") > + > if ((err= ht->commit(ht, thd, all))) > { > my_error(ER_ERROR_DURING_COMMIT, MYF(0), err); > @@ -1987,6 +2001,10 @@ int ha_rollback_trans(THD *thd, bool all) > { > int err; > handlerton *ht= ha_info->ht(); > + > + DBUG_EXECUTE_IF("simulate_crash_after_first_engine_commit_or_rollback", > + if (!ha_info->next()) DBUG_SUICIDE();); ditto > + > if ((err= ht->rollback(ht, thd, all))) > { // cannot happen > my_error(ER_ERROR_DURING_ROLLBACK, MYF(0), err); > @@ -2098,8 +2116,24 @@ int ha_commit_or_rollback_by_xid(XID *xid, bool commit) > xaop.xid= xid; > xaop.result= 1; > > + if (binlog_hton->recover) > + { > + /* > + When the binlogging service is enabled complete the transaction > + by it first. > + */ > + if (commit) > + binlog_hton->commit_by_xid(binlog_hton, xid); > + else > + binlog_hton->rollback_by_xid(binlog_hton, xid); > + } > plugin_foreach(NULL, commit ? xacommit_handlerton : xarollback_handlerton, > MYSQL_STORAGE_ENGINE_PLUGIN, &xaop); > + if (binlog_hton->recover) > + { > + THD *thd= current_thd; > + thd->reset_binlog_completed_by_xid(); > + } 1. this, precisely, shows that binlog does not need a hton anymoreo 2. how can binlog_hton->recover be NULL here? > > return xaop.result; > } > @@ -2222,6 +2258,7 @@ static my_bool xarecover_handlerton(THD *unused, plugin_ref plugin, > > if (hton->recover) > { > + info->recover_htons++; don't forget to subtract 1 for binlog_hton->recover, otherwise it'll make you to rollback everything with a FL_MULTI_ENGINE_XA flag > while ((got= hton->recover(hton, info->list, info->len)) > 0 ) > { > sql_print_information("Found %d prepared transaction(s) in %s", > @@ -2263,7 +2300,21 @@ static my_bool xarecover_handlerton(THD *unused, plugin_ref plugin, > _db_doprnt_("ignore xid %s", xid_to_str(buf, info->list[i])); > }); > xid_cache_insert(info->list + i); > + XID *foreign_xid= info->list + i; swap those two lines ^^^ then you can write xid_cache_insert(foreign_xid); > info->found_foreign_xids++; > + > + /* > + For each foreign xid prepraed in engine, check if it is present in > + xa_prepared_list of binlog. > + */ > + if (info->xa_prepared_list) for simplicity I'd rather always initialize xa_prepared_list and remove this if() > + { > + struct xa_recovery_member *member= NULL; > + if ((member= (xa_recovery_member *) > + my_hash_search(info->xa_prepared_list, foreign_xid->key(), > + foreign_xid->key_length()))) > + member->in_engine_prepare++; > + } > continue; > } > if (IF_WSREP(!(wsrep_emulate_bin_log && > @@ -2362,7 +2424,7 @@ int ha_recover(HASH *commit_list) > info.found_my_xids, opt_tc_log_file); > DBUG_RETURN(1); > } > - if (info.commit_list) > + if (info.commit_list && !info.found_foreign_xids) Why? after the finished crash recovery one can still have transactions in doubt. > sql_print_information("Crash recovery finished."); > DBUG_RETURN(0); > } > diff --git a/sql/log.cc b/sql/log.cc > index 731bb3e98f0..c69b8518cf4 100644 > --- a/sql/log.cc > +++ b/sql/log.cc > @@ -2019,28 +2027,49 @@ static int binlog_xa_recover_dummy(handlerton *hton __attribute__((unused)), > return 0; > } > > - > +/* > + The function invokes binlog_commit() and returns its result > + when it has not yet called it already. > + binlog_cache_mngr::completed_by_xid remembers the fact of > + the 1st of maximum two subsequent calls. > +*/ > static int binlog_commit_by_xid(handlerton *hton, XID *xid) > { > + int rc= 0; > THD *thd= current_thd; > - > - (void) thd->binlog_setup_trx_data(); > + binlog_cache_mngr *cache_mngr= thd->binlog_setup_trx_data(); > > DBUG_ASSERT(thd->lex->sql_command == SQLCOM_XA_COMMIT); > > - return binlog_commit(hton, thd, TRUE); > -} > + if (!cache_mngr->completed_by_xid) On what code path can you have completed_by_xid == false here? > + { > + rc= binlog_commit(hton, thd, TRUE); > + cache_mngr->completed_by_xid= true; > + } > > + return rc; > +} > > +/* > + The function invokes binlog_rollback() and returns its results similarly > + to a commit branch. > +*/ > static int binlog_rollback_by_xid(handlerton *hton, XID *xid) > { > + int rc= 0; > THD *thd= current_thd; > - > - (void) thd->binlog_setup_trx_data(); > + binlog_cache_mngr *cache_mngr= thd->binlog_setup_trx_data(); > > DBUG_ASSERT(thd->lex->sql_command == SQLCOM_XA_ROLLBACK || > (thd->transaction.xid_state.get_state_code() == XA_ROLLBACK_ONLY)); > - return binlog_rollback(hton, thd, TRUE); > + > + if (!cache_mngr->completed_by_xid) > + { > + rc= binlog_rollback(hton, thd, TRUE); > + cache_mngr->completed_by_xid= true; > + } > + > + return rc; > } > > > @@ -2195,6 +2224,12 @@ static int binlog_commit(handlerton *hton, THD *thd, bool all) > if (!all) > cache_mngr->trx_cache.set_prev_position(MY_OFF_T_UNDEF); > > + DBUG_EXECUTE_IF("simulate_crash_after_binlog_commit_or_rollback", > + DBUG_SUICIDE();); > + DEBUG_SYNC(thd, "simulate_hang_after_binlog_prepare"); > + DBUG_EXECUTE_IF("simulate_crash_after_binlog_prepare", > + DBUG_SUICIDE();); wouldn't you say there's one DBUG_EXECUTE_IF too many? Just keep one, "simulate_crash_after_binlog_commit" > + > THD_STAGE_INFO(thd, org_stage); > DBUG_RETURN(error); > } > @@ -2298,6 +2333,9 @@ static int binlog_rollback(handlerton *hton, THD *thd, bool all) > cache_mngr->trx_cache.set_prev_position(MY_OFF_T_UNDEF); > thd->reset_binlog_for_next_statement(); > > + DBUG_EXECUTE_IF("simulate_crash_after_binlog_commit_or_rollback", same as above, use unique dbug keywords, like "simulate_crash_after_binlog_rollback" (because it's rollback here) > + DBUG_SUICIDE();); > + > DBUG_RETURN(error); > } > > @@ -5749,6 +5794,14 @@ binlog_cache_mngr *THD::binlog_setup_trx_data() > DBUG_RETURN(cache_mngr); > } > > +/* > + XA completion flag resetter for ha_commit_or_rollback_by_xid(). > +*/ > +void THD::reset_binlog_completed_by_xid() > +{ > + binlog_setup_trx_data()->reset_completed_by_xid(); > +} I don't think this makes any sense as a THD method > + > /* > Function to start a statement and optionally a transaction for the > binary log. > @@ -10333,14 +10386,108 @@ start_binlog_background_thread() > return 0; > } > > +#ifdef HAVE_REPLICATION > +/** > + Auxiliary function for TC_LOG::recover(). > + @returns a successfully created and inserted @c xa_recovery_member > + into hash @c hash_arg, > + or NULL. > +*/ > +static xa_recovery_member* > +xa_member_insert(HASH *hash_arg, xid_t *xid_arg, xa_binlog_state state_arg, > + MEM_ROOT *ptr_mem_root) > +{ > + xa_recovery_member *member= (xa_recovery_member*) > + alloc_root(ptr_mem_root, sizeof(xa_recovery_member)); > + if (!member) > + return NULL; > + > + member->xid.set(xid_arg); > + member->state= state_arg; > + member->in_engine_prepare= 0; > + return my_hash_insert(hash_arg, (uchar*) member) ? NULL : member; > +} > + > +/* Inserts or updates an existing hash member with a proper state */ > +static bool xa_member_replace(HASH *hash_arg, xid_t *xid_arg, bool is_prepare, > + MEM_ROOT *ptr_mem_root) > +{ > + if(is_prepare) > + { > + if (!(xa_member_insert(hash_arg, xid_arg, XA_PREPARE, ptr_mem_root))) > + return true; > + } > + else > + { > + /* > + Search if XID is already present in recovery_list. If found > + and the state is 'XA_PREPRAED' mark it as XA_COMPLETE. > + Effectively, there won't be XA-prepare event group replay. > + */ > + xa_recovery_member* member; > + if ((member= (xa_recovery_member *) > + my_hash_search(hash_arg, xid_arg->key(), xid_arg->key_length()))) > + { > + if (member->state == XA_PREPARE) > + member->state= XA_COMPLETE; > + } > + else // We found only XA COMMIT during recovery insert to list > + { > + if (!(member= xa_member_insert(hash_arg, > + xid_arg, XA_COMPLETE, ptr_mem_root))) > + return true; > + } > + } > + return false; > +} > +#endif > > +extern "C" uchar *xid_get_var_key(xid_t *entry, size_t *length, define the first argument as const char*, then you won't need to cast in my_hash_init, and you can be sure you pass the correct function with the correct number of arguments, even if my_hash_get_key changes in the future. and cast entry to xid_t* below explicitly. > + my_bool not_used __attribute__((unused))) > +{ > + *length= entry->key_length(); > + return (uchar*) entry->key(); > +} > + > +/** > + Performs recovery based on transaction coordinator log for 2pc. At the > + time of crash, if the binary log was in active state, then recovery for > + "implicit" 'xid's and explicit 'XA' transactions is initiated, > + otherwise merely the gtid binlog state is updated. > + For 'xid' and 'XA' based recovery the following steps are performed. > + > + Identify the active binlog checkpoint file. > + Scan the binary log from the beginning. > + From GTID_LIST and GTID_EVENTs reconstruct the gtid binlog state. > + Prepare a list of 'xid's for recovery. > + Prepare a list of explicit 'XA' transactions for recovery. > + Recover the 'xid' transactions. > + The explicit 'XA' transaction recovery is initiated once all the server > + components are initialized. Please check 'execute_xa_for_recovery()'. why explicit XA recovery is delayed? > + > + Called from @c MYSQL_BIN_LOG::do_binlog_recovery() > + > + @param linfo Store here the found log file name and position to > + the NEXT log file name in the index file. > + > + @param last_log_name Name of the last active binary log at the time of > + crash. > + > + @param first_log Pointer to IO_CACHE of active binary log > + @param fdle Format_description_log_event of active binary log > + @param do_xa Is 2pc recovery needed for 'xid's and explicit XA > + transactions. > + @return indicates success or failure of recovery. > + @retval 0 success > + @retval 1 failure > + > +*/ > int TC_LOG_BINLOG::recover(LOG_INFO *linfo, const char *last_log_name, > IO_CACHE *first_log, > Format_description_log_event *fdle, bool do_xa) > { > Log_event *ev= NULL; > HASH xids; > - MEM_ROOT mem_root; > char binlog_checkpoint_name[FN_REFLEN]; > bool binlog_checkpoint_found; > bool first_round; > @@ -10533,10 +10693,22 @@ int TC_LOG_BINLOG::recover(LOG_INFO *linfo, const char *last_log_name, > > if (do_xa) > { > - if (ha_recover(&xids)) > + if (ha_recover(&xids, &xa_recover_list, &xa_recover_htons)) > goto err2; > > - free_root(&mem_root, MYF(0)); > + DBUG_ASSERT(!xa_recover_list.records || > + (binlog_checkpoint_found && binlog_checkpoint_name[0] != 0)); why is that? > + > + if (!xa_recover_list.records) > + { > + free_root(&mem_root, MYF(0)); > + my_hash_free(&xa_recover_list); > + } > + else > + { > + xa_binlog_checkpoint_name= strmake_root(&mem_root, binlog_checkpoint_name, > + strlen(binlog_checkpoint_name)); > + } > my_hash_free(&xids); > } > return 0; > @@ -10561,6 +10734,219 @@ int TC_LOG_BINLOG::recover(LOG_INFO *linfo, const char *last_log_name, > return 1; > } > > +void MYSQL_BIN_LOG::execute_xa_for_recovery() > +{ > + if (xa_recover_list.records) > + (void) recover_explicit_xa_prepare(); > + free_root(&mem_root, MYF(0)); > + my_hash_free(&xa_recover_list); > +}; > + > +/** > + Performs recovery of user XA transactions. > + 'xa_recover_list' contains the list of XA transactions to be recovered. > + with possible replaying replication event from the binary log. > + > + @return indicates success or failure of recovery. > + @retval false success > + @retval true failure > + > +*/ > +bool MYSQL_BIN_LOG::recover_explicit_xa_prepare() why is it bool, if you only use it in execute_xa_for_recovery() and ignore the return value? > +{ > +#ifndef HAVE_REPLICATION > + /* Can't be supported without replication applier built in. */ > + return false; > +#else > + bool err= true; > + int error=0; > + Relay_log_info *rli= NULL; > + rpl_group_info *rgi; > + THD *thd= new THD(0); /* Needed by start_slave_threads */ > + thd->thread_stack= (char*) &thd; > + thd->store_globals(); > + thd->security_ctx->skip_grants(); > + IO_CACHE log; > + const char *errmsg; > + File file; > + bool enable_apply_event= false; > + Log_event *ev = 0; > + LOG_INFO linfo; > + int recover_xa_count= xa_recover_list.records; > + xa_recovery_member *member= NULL; > + > + if (!(rli= thd->rli_fake= new Relay_log_info(FALSE, "Recovery"))) > + { > + my_error(ER_OUTOFMEMORY, MYF(ME_FATAL), 1); > + goto err2; > + } > + rli->sql_driver_thd= thd; > + static LEX_CSTRING connection_name= { STRING_WITH_LEN("Recovery") }; > + rli->mi= new Master_info(&connection_name, false); > + if (!(rgi= thd->rgi_fake)) > + rgi= thd->rgi_fake= new rpl_group_info(rli); > + rgi->thd= thd; > + thd->system_thread_info.rpl_sql_info= > + new rpl_sql_thread_info(rli->mi->rpl_filter); > + > + if (rli && !rli->relay_log.description_event_for_exec) > + { > + rli->relay_log.description_event_for_exec= > + new Format_description_log_event(4); > + } > + if (find_log_pos(&linfo, xa_binlog_checkpoint_name, 1)) > + { > + sql_print_error("Binlog file '%s' not found in binlog index, needed " > + "for recovery. Aborting.", xa_binlog_checkpoint_name); > + goto err2; > + } > + > + tmp_disable_binlog(thd); > + thd->variables.pseudo_slave_mode= TRUE; > + for (;;) > + { > + if ((file= open_binlog(&log, linfo.log_file_name, &errmsg)) < 0) > + { > + sql_print_error("%s", errmsg); > + goto err1; > + } > + while (recover_xa_count > 0 && > + (ev= Log_event::read_log_event(&log, > + rli->relay_log.description_event_for_exec, > + opt_master_verify_checksum))) > + { > + if (!ev->is_valid()) > + { > + sql_print_error("Found invalid binlog query event %s" > + " at %s:%llu; error %d %s", ev->get_type_str(), > + linfo.log_file_name, > + (ev->log_pos - ev->data_written)); > + goto err1; > + } > + enum Log_event_type typ= ev->get_type_code(); > + ev->thd= thd; > + > + if (typ == FORMAT_DESCRIPTION_EVENT) > + enable_apply_event= true; > + > + if (typ == GTID_EVENT) > + { > + Gtid_log_event *gev= (Gtid_log_event *)ev; > + if (gev->flags2 & > + (Gtid_log_event::FL_PREPARED_XA | Gtid_log_event::FL_COMPLETED_XA)) > + { > + if ((member= > + (xa_recovery_member*) my_hash_search(&xa_recover_list, > + gev->xid.key(), > + gev->xid.key_length()))) > + { > + /* > + When XA PREPARE group of events (as flagged so) check > + its actual binlog state which may be COMPLETED. If the > + state is also PREPARED then analyze through > + in_engine_prepare whether the transaction needs replay. > + */ > + if (gev->flags2 & Gtid_log_event::FL_PREPARED_XA) > + { > + if (member->state == XA_PREPARE) > + { > + // XA prepared is not present in (some) engine then apply it > + if (member->in_engine_prepare == 0) > + enable_apply_event= true; > + else if (gev->flags2 & Gtid_log_event::FL_MULTI_ENGINE_XA && > + xa_recover_htons > member->in_engine_prepare) This needs a comment, something like "FL_MULTI_ENGINE_XA says the transaction must be prepared in more than one engine. We don't know in how many, so if it's not prepared in *all* engines we'll replay it just in case" I presume that (a bit silly) logic will do away when you will port this patch to use a counter instead of a flag. Please, don't push if before that. > + { > + enable_apply_event= true; > + // partially engine-prepared XA is first cleaned out prior replay > + thd->lex->sql_command= SQLCOM_XA_ROLLBACK; > + ha_commit_or_rollback_by_xid(&gev->xid, 0); > + } > + else > + --recover_xa_count; > + } > + } > + else if (gev->flags2 & Gtid_log_event::FL_COMPLETED_XA) > + { > + if (member->state == XA_COMPLETE && > + member->in_engine_prepare > 0) > + enable_apply_event= true; why? you cannot replay a fully prepared and partially committed transaction > + else > + --recover_xa_count; > + } > + } > + } > + } > + > + if (enable_apply_event) > + { > + if ((err= ev->apply_event(rgi))) > + { > + sql_print_error("Failed to execute binlog query event of type: %s," > + " at %s:%llu; error %d %s", ev->get_type_str(), > + linfo.log_file_name, > + (ev->log_pos - ev->data_written), > + thd->get_stmt_da()->sql_errno(), > + thd->get_stmt_da()->message()); > + delete ev; > + goto err1; > + } > + else if (typ == FORMAT_DESCRIPTION_EVENT) > + enable_apply_event=false; how can that happen? > + else if (thd->lex->sql_command == SQLCOM_XA_PREPARE || > + thd->lex->sql_command == SQLCOM_XA_COMMIT || > + thd->lex->sql_command == SQLCOM_XA_ROLLBACK) > + { > + --recover_xa_count; > + enable_apply_event=false; > + > + sql_print_information("Binlog event %s at %s:%llu" > + " successfully applied", > + typ == XA_PREPARE_LOG_EVENT ? > + static_cast<XA_prepare_log_event *>(ev)->get_query() : > + static_cast<Query_log_event *>(ev)->query, > + linfo.log_file_name, (ev->log_pos - ev->data_written)); > + } > + } > + if (typ != FORMAT_DESCRIPTION_EVENT) > + delete ev; > + } > + end_io_cache(&log); > + mysql_file_close(file, MYF(MY_WME)); > + file= -1; > + if (unlikely((error= find_next_log(&linfo, 1)))) > + { > + if (error != LOG_INFO_EOF) > + sql_print_error("find_log_pos() failed (error: %d)", error); > + else > + break; > + } > + } > +err1: > + reenable_binlog(thd); > + /* > + There should be no more XA transactions to recover upon successful > + completion. > + */ > + if (recover_xa_count > 0) > + goto err2; > + sql_print_information("Crash recovery finished."); > + err= false; > +err2: > + if (file >= 0) > + { > + end_io_cache(&log); > + mysql_file_close(file, MYF(MY_WME)); > + } > + thd->variables.pseudo_slave_mode= FALSE; > + delete rli->mi; > + delete thd->system_thread_info.rpl_sql_info; > + rgi->slave_close_thread_tables(thd); > + thd->reset_globals(); > + delete thd; > + > + return err; > +#endif /* !HAVE_REPLICATION */ > +} > > int > MYSQL_BIN_LOG::do_binlog_recovery(const char *opt_name, bool do_xa_recovery) Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

[Maria-developers] MDEV-17399: JSON_TABLE: final input
by Sergey Petrunia 13 Mar '21

13 Mar '21

Hi Alexey, Please find below final bits of input. After these are addressed the patch will probably be good to push (But I'll need a final pass before giving ok to push). == json_table_mysql* tests == * Please remove one of the json_table_mysql and json_table_mysql2 tests. * Please move the remaining test to be in the same suite as json_table.test. == Fix json_table.test == The test has this somewhere in the middle: set optimizer_switch='firstmatch=off'; Please restore the original setting. == ha_json_table::rnd_next() returns 0 on error == Consider this query: select * from json_table( '[ {"name": "X"}, {"name2": "Y"} ]', '$[*]' columns ( col1 varchar(100) path '$.name2' error on empty )) as T; The query produces an error. It happens like so: * the call to ha_json_table::rnd_next() returns 0. * The SQL layer finds out about the error by checking thd->is_error() which returns true. Please make ha_json_table::rnd_next() return an error. == Rename table_function.* == Looking at the contets of sql/table_function.{h,cc}, one can see that there's very little code there that is applicable to generic table function. Almost all code is specifi to JSON_TABLE. Because of that, please rename the files to e.g. json_table.{h,cc}. BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://petrunia.net

2 5

[Maria-developers] MDEV-17399: JSON_TABLE: Odd code in table.cc:create_view_field ?
by Sergey Petrunia 12 Mar '21

12 Mar '21

Hi Alexey, What does the code quoted below do? I don't recall seeing it on previous review iterations. In any case, * It needs a comment about why such special handling is needed. * It needs test coverage - I have reverted these changes and didn't see any test to fail? > diff --git a/sql/table.cc b/sql/table.cc > index 4f65dbd65f4..9c205fc4be6 100644 > --- a/sql/table.cc > +++ b/sql/table.cc > @@ -6722,6 +6722,8 @@ Item *create_view_field(THD *thd, TABLE_LIST *view, Item **field_ref, > LEX_CSTRING *name) > { > bool save_wrapper= thd->lex->first_select_lex()->no_wrap_view_item; > + bool *wrapper_to_set= thd->lex->current_select ? > + &thd->lex->current_select->no_wrap_view_item : &save_wrapper; > Item *field= *field_ref; > DBUG_ENTER("create_view_field"); > > @@ -6737,17 +6739,17 @@ Item *create_view_field(THD *thd, TABLE_LIST *view, Item **field_ref, > } > > DBUG_ASSERT(field); > - thd->lex->current_select->no_wrap_view_item= TRUE; > + *wrapper_to_set= TRUE; > if (!field->is_fixed()) > { > if (field->fix_fields(thd, field_ref)) > { > - thd->lex->current_select->no_wrap_view_item= save_wrapper; > + *wrapper_to_set= save_wrapper; > DBUG_RETURN(0); > } > field= *field_ref; > } > - thd->lex->current_select->no_wrap_view_item= save_wrapper; > + *wrapper_to_set= save_wrapper; > if (save_wrapper) > { > DBUG_RETURN(field); BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://petrunia.net

2 1

[Maria-developers] GSOC21: MDEV-16375 & MDEV-23143
by Hollow Man 10 Mar '21

10 Mar '21

Hi MariaDB community!    Glad to be here! My github account is @HollowMan6. Though I'm new to MariaDB community, I'm interested in MDEV-16375 & MDEV-23143: Function to normalize a json value & missing a JSON_EQUALS function for this year's GSOC project. Here are my first thoughts on these issues:    I have checked part of the codebase and I think the two issues can be merged into one. First we can create a function named JSON_NORMALIZE to normalize the json, which automatically parses the inputed json document, recursively sorts the keys (for objects) / sorts the numbers (for arrays), removes the spaces, and then return the json document string.    Then we create a function named JSON_EQUALS, which can be used to compare 2 json documents for equality realized by first seperately normalize the two json documents using JSON_NORMALIZE, then the 2 can be compared exactly as binary strings.    I have taken some inspirations from the Item_func_json_keys and json_scan_start for parsing json documents, and I think it's possible to sort the keys using std::map in STL for objects.    That's all for my ideas so far. Please correct me if I made some mistakes, and I'm going to work on my ideas later. Cheers! Hollow Man

1 0

Re: [Maria-developers] MDEV-25075: Ignorable index makes the resulting CREATE TABLE invalid
by Sergey Petrunia 10 Mar '21

10 Mar '21

Hi Varun, > diff --git a/sql/lex.h b/sql/lex.h > index 542356c0e433..5a9ec2ec1b35 100644 > --- a/sql/lex.h > +++ b/sql/lex.h > @@ -289,6 +289,7 @@ static SYMBOL symbols[] = { > { "IDENTIFIED", SYM(IDENTIFIED_SYM)}, > { "IF", SYM(IF_SYM)}, > { "IGNORE", SYM(IGNORE_SYM)}, > + { "IGNORED", SYM(IGNORED_SYM)}, > { "IGNORE_DOMAIN_IDS", SYM(IGNORE_DOMAIN_IDS_SYM)}, > { "IGNORE_SERVER_IDS", SYM(IGNORE_SERVER_IDS_SYM)}, > { "IMMEDIATE", SYM(IMMEDIATE_SYM)}, Above this array one can see: >> ... >> NOTE!! >> If you add or delete symbols from this file, you must also update results for >> the perfschema.start_server_low_digest_sql_length test! >> */ >> >> static SYMBOL symbols[] = { and indeed the buildbot shows failures, e.g. http://buildbot.askmonty.org/buildbot/builders/kvm-rpm-centos74-amd64/build… CURRENT_TEST: perfschema.start_server_low_digest_sql_length --- /usr/share/mysql-test/suite/perfschema/r/start_server_low_digest_sql_length.result 2021-03-10 06:23:20.000000000 +0000 +++ /dev/shm/var/4/log/start_server_low_digest_sql_length.reject 2021-03-10 07:37:16.351213429 +0000 @@ -8,5 +8,5 @@ #################################### SELECT event_name, digest, digest_text, sql_text FROM events_statements_history_long; event_name digest digest_text sql_text -statement/sql/select beb5bd93b7e8c45bc5cb6060804988e8 SELECT ? + ? + SELECT ... -statement/sql/truncate faf6cefb662b443f05e97b5c5ab14a59 TRUNCATE TABLE truncat... +statement/sql/select ade774bdfbc132a71810ede8ef469660 SELECT ? + ? + SELECT ... +statement/sql/truncate 0f84807fb4a75d0f391f8a93e7c3c182 TRUNCATE TABLE truncat... Please fix this. BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://petrunia.net

1 1

Re: [Maria-developers] bf677c554ad: UTF8 and utf8
by Sergei Golubchik 08 Mar '21

08 Mar '21

Hi, Rucha! On Mar 08, Rucha Deodhar wrote: > revision-id: bf677c554ad (mariadb-10.5.2-402-gbf677c554ad) > parent(s): a1542f8a573 > author: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > committer: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > timestamp: 2021-02-22 00:37:18 +0530 > message: > > UTF8 and utf8 Two main comments: * your global search-and-replace utf8 to utf8mb3 everywhere has replaced way too much. Please, revert it. * why did you have so many changes in tests with utf8mb4? You set the old-mode to be utf8_is_utf8mb3 by default. It should've applied to tests too, right? So why tests have switched to utf8mb4? Other comments below. > diff --git a/debian/additions/innotop/innotop b/debian/additions/innotop/innotop > index 19229a57a82..4f1e76ea497 100644 > --- a/debian/additions/innotop/innotop > +++ b/debian/additions/innotop/innotop > @@ -264,9 +264,9 @@ sub get_dbh { > MKDEBUG && _d("$dbh: $sql"); > $dbh->do($sql); > MKDEBUG && _d('Enabling charset for STDOUT'); > - if ( $charset eq 'utf8' ) { > - binmode(STDOUT, ':utf8') > - or die "Can't binmode(STDOUT, ':utf8'): $OS_ERROR"; > + if ( $charset eq 'utf8mb3' ) { > + binmode(STDOUT, ':utf8mb3') > + or die "Can't binmode(STDOUT, ':utf8mb3'): $OS_ERROR"; It seems you performed search-and-replace over the whole source tree. I don't think it was particularly meaningful. Here, for example, it was wrong, it's perl code and perl charser utf8, there's no utf8mb3 in perl. also all internal variables you've renamed, like below: > } > else { > binmode(STDOUT) or die "Can't binmode(STDOUT): $OS_ERROR"; > diff --git a/sql/field.h b/sql/field.h > index 4a4f7cee2a5..f672ae7e315 100644 > --- a/sql/field.h > +++ b/sql/field.h > @@ -582,7 +582,7 @@ class Virtual_column_info: public Sql_alloc, > public: > /* Flag indicating that the field is physically stored in the database */ > bool stored_in_db; > - bool utf8; /* Already in utf8 */ > + bool utf8mb3; /* Already in utf8 */ I mean, here ^^^ > bool automatic_name; > Item *expr; > Lex_ident name; /* Name of constraint */ > diff --git a/client/mysql.cc b/client/mysql.cc > index f2938d4c824..b0ee239c0bc 100644 > --- a/client/mysql.cc > +++ b/client/mysql.cc > @@ -1918,6 +1918,10 @@ static int get_options(int argc, char **argv) > > if ((ho_error=handle_options(&argc, &argv, my_long_options, get_one_option))) > return(ho_error); > + if (!strcasecmp("utf8",default_charset)) > + { > + default_charset=(char*)"utf8mb3"; > + } why is that? (and the same everywhere else) > > *mysql_params->p_max_allowed_packet= opt_max_allowed_packet; > *mysql_params->p_net_buffer_length= opt_net_buffer_length; > diff --git a/client/mysqldump.c b/client/mysqldump.c > index 7c363973da2..ad69618fbd1 100644 > --- a/client/mysqldump.c > +++ b/client/mysqldump.c > @@ -3160,7 +3160,7 @@ static uint get_table_structure(const char *table, const char *db, char *table_t > > fprintf(sql_file, > "SET @saved_cs_client = @@character_set_client;\n" > - "SET character_set_client = utf8;\n" > + "SET character_set_client = utf8mb3;\n" why is that? > "/*!50001 CREATE TABLE %s (\n", > result_table); > > diff --git a/include/m_ctype.h b/include/m_ctype.h > index 5fa8f28ff7a..f1a8166d9df 100644 > --- a/include/m_ctype.h > +++ b/include/m_ctype.h > @@ -223,6 +223,7 @@ extern MY_UNI_CTYPE my_uni_ctype[256]; > #define MY_CS_NON1TO1 0x40000 /* Has a complex mapping from characters > to weights, e.g. contractions, expansions, > ignorable characters */ > +#define MY_CS_UTF8_IS_UTF8MB3 0x80000 this isn't a MY_CS_xxx flag, it doesn't belong here, see all other MY_CS_xxx flags - they are *properties* of a charset. They are set as struct charset_info_st my_charset_tis620_thai_ci= { 18,0,0, /* number */ MY_CS_COMPILED|MY_CS_PRIMARY|MY_CS_STRNXFRM|MY_CS_NON1TO1, /* state */ charset_name_tis620, /* cs name */ "tis620_thai_ci", /* name */ that is, when defining a charset. See below > #define MY_CHARSET_UNDEFINED 0 > > /* Character repertoire flags */ > diff --git a/man/mysql.1 b/man/mysql.1 > index 03f23df3660..5ad953ba5eb 100644 > --- a/man/mysql.1 > +++ b/man/mysql.1 > @@ -401,7 +401,7 @@ Use > as the default character set for the client and connection\&. > .sp > A common issue that can occur when the operating system uses > -utf8 > +utf8mb3 this didn't need changing > or another multi\-byte character set is that output from the > \fBmysql\fR > client is formatted incorrectly, due to the fact that the MariaDB client uses the > diff --git a/mysys/charset.c b/mysys/charset.c > index 32cfeb56e2d..9d035406e8d 100644 > --- a/mysys/charset.c > +++ b/mysys/charset.c > @@ -361,7 +361,7 @@ static int add_collation(struct charset_info_st *cs) > newcs->state|= MY_CS_AVAILABLE | MY_CS_LOADED | MY_CS_NONASCII; > #endif > } > - else if (!strcmp(cs->csname, "utf8") || !strcmp(cs->csname, "utf8mb3")) > + else if (!strcmp(cs->csname, "utf8mb3") || !strcmp(cs->csname, "utf8mb3")) one more result of your mechanical search-and-replace, now you compare with "utf8mb3" twice. > { > #if defined (HAVE_CHARSET_utf8mb3) && defined(HAVE_UCA_COLLATIONS) > copy_uca_collation(newcs, newcs->state & MY_CS_NOPAD ? > @@ -767,7 +767,7 @@ static const char* > get_charset_name_alias(const char *name) > { > if (!my_strcasecmp(&my_charset_latin1, name, "utf8mb3")) > - return "utf8"; > + return "utf8mb3"; and again > return NULL; > } > > @@ -1004,13 +1004,19 @@ CHARSET_INFO * > get_charset_by_csname(const char *cs_name, uint cs_flags, myf flags) > { > MY_CHARSET_LOADER loader; > + > + if (!strcasecmp(cs_name, "utf8")) > + { > + cs_name = (const char*)(cs_flags & MY_CS_UTF8_IS_UTF8MB3 ? "utf8mb3" : "utf8mb4"); your flag should be in `myf flags` not in `uint cs_flags` (and defined in my_sys.h with other myf flags) > + } > + > my_charset_loader_init_mysys(&loader); > return my_charset_get_by_name(&loader, cs_name, cs_flags, flags); > } > > > /** > - Resolve character set by the character set name (utf8, latin1, ...). > + Resolve character set by the character set name (utf8mb3, latin1, ...). > > The function tries to resolve character set by the specified name. If > there is character set with the given name, it is assigned to the "cs" > @@ -1453,8 +1459,8 @@ static const MY_CSET_OS_NAME charsets[] = > > {"US-ASCII", "latin1", my_cs_approx}, > > - {"utf8", "utf8", my_cs_exact}, > - {"utf-8", "utf8", my_cs_exact}, > + {"utf8mb3", "utf8mb3", my_cs_exact}, > + {"utf-8mb3", "utf8mb3", my_cs_exact}, utf-8mb3 really? :) > #endif > {NULL, NULL, 0} > }; > diff --git a/plugin/win_auth_client/common.cc b/plugin/win_auth_client/common.cc > index 8b7319252ac..f46b0ec9696 100644 > --- a/plugin/win_auth_client/common.cc > +++ b/plugin/win_auth_client/common.cc > @@ -338,7 +338,7 @@ UPN::UPN(): m_buf(NULL) > m_buf= wchar_to_utf8(buf1, &m_len); > > if(!m_buf) > - ERROR_LOG(ERROR, ("Failed to convert UPN to utf8")); > + ERROR_LOG(ERROR, ("Failed to convert UPN to utf8mb3")); probably wrong too (everything in this file), see the code - it uses windows utf8 charset, not mariadb's utf8 > > // Note: possible error would be indicated by the fact that m_buf is NULL. > return; > diff --git a/sql/mysqld.cc b/sql/mysqld.cc > index 0bf21e02002..fd131497f8c 100644 > --- a/sql/mysqld.cc > +++ b/sql/mysqld.cc > @@ -4053,6 +4056,24 @@ static int init_common_variables() > break; > } > > + if (default_collation_name) > + { > + char *copy_of_name= (char*)default_collation_name; > + char start[6]; > + strncpy(start, default_collation_name, 5); > + char *fname= (char *)"utf8mb3_"; > + if (! strncasecmp("utf8_", start,5)) > + { > + copy_of_name+= 5; > + char result[64]; > + result[63]='\0'; > + strcpy(result, fname); > + strcat(result, copy_of_name); > + result[strlen(copy_of_name)+strlen(fname)]='\0'; > + default_collation_name= (char *) result; > + } > + } 1. wrong indentation 2. shouldn't it be in the get_charset_by_name ? 3. shouldn't it depend on the UTF8_IS_UTF8MB3 ? > + > if (default_collation_name) > { > CHARSET_INFO *default_collation; > diff --git a/sql/sql_class.h b/sql/sql_class.h > index 50b746fe514..f0616b40361 100644 > --- a/sql/sql_class.h > +++ b/sql/sql_class.h > @@ -5248,6 +5249,65 @@ class THD: public THD_count, /* this must be first */ > Item *sp_prepare_func_item(Item **it_addr, uint cols= 1); > bool sp_eval_expr(Field *result_field, Item **expr_item_ptr); > > + CHARSET_INFO *get_charset_by_csname(const char *cs_name, > + uint cs_flags, > + myf myf_flags) const > + { > + return ::get_charset_by_csname(cs_name, > + this->variables.old_behavior & OLD_MODE_UTF8_IS_UTF8MB3 ? > + cs_flags | MY_CS_UTF8_IS_UTF8MB3 : cs_flags, > + myf_flags); > + } > + > + inline const char* get_collation_or_charset_name(const char* name) shouldn't the name be get_collation_name_alias() ? all other method names you've created match corresponding charset-related function names > + { > + char *copy_of_name= (char*)name; > + char start[6]; > + strncpy(start, name, 5); > + char *fname= (char *)(this->variables.old_behavior & OLD_MODE_UTF8_IS_UTF8MB3?"utf8mb3_":"utf8mb4_"); > + if (!strncasecmp("utf8_", start,5)) > + { > + copy_of_name+= 5; > + char result[64]; > + result[63]='\0'; > + strcpy(result, fname); > + strcat(result, copy_of_name); > + result[strlen(copy_of_name)+strlen(fname)]='\0'; > + name= (const char *) result; > + } > + return name; here you return a pointer to the local variable also: indentation, too long line, end-of-line spaces > + } > + > + inline CHARSET_INFO * > + mysqld_collation_get_by_name(const char *name, > + CHARSET_INFO *name_cs= system_charset_info) > + { > + name=this->get_collation_or_charset_name(name); > + return ::mysqld_collation_get_by_name(name, name_cs); > +} > + > +CHARSET_INFO *get_charset_by_name(const char *cs_name, myf flags) > +{ > + > + cs_name=this->get_collation_or_charset_name(cs_name); > + return ::get_charset_by_name(cs_name,flags); here it'd be better to pass MY_UTF8_IS_UTF8MB3 in flags > +} > +my_bool resolve_charset(const char *cs_name, > + CHARSET_INFO *default_cs, > + CHARSET_INFO **cs) > +{ > + cs_name=this->get_collation_or_charset_name(cs_name); > + return ::resolve_charset(cs_name,default_cs,cs); perhaps it'd make sense to introduce flags argument here too? > +} > + > +my_bool resolve_collation(const char *cl_name, > + CHARSET_INFO *default_cl, > + CHARSET_INFO **cl) > +{ > + cl_name= this->get_collation_or_charset_name(cl_name); > + return ::resolve_collation(cl_name,default_cl,cl); > +} please make sure the indentation is correct, I won't comment on it anymore > + > }; > > > diff --git a/sql/sql_db.cc b/sql/sql_db.cc > index 9bf16220535..85e0e0dbd2f 100644 > --- a/sql/sql_db.cc > +++ b/sql/sql_db.cc > @@ -583,9 +583,13 @@ bool load_db_opt(THD *thd, const char *path, Schema_specification_st *create) > default-collation commands. > */ > if (!(create->default_table_charset= > - get_charset_by_csname(pos+1, MY_CS_PRIMARY, MYF(0))) && > + thd->get_charset_by_csname(pos+1, thd->variables.old_behavior & > + OLD_MODE_UTF8_IS_UTF8MB3 ? > + MY_CS_UTF8_IS_UTF8MB3 | MY_CS_PRIMARY : > + MY_CS_PRIMARY, > + MYF(0))) && that's a bit redundant, don't you think? you use both thd-> method that replaces utf_ with a real charset name and *also* you set MY_CS_UTF8_IS_UTF8MB3 flag. there are other places like that, to keep the review shorter I won't comment on them below. > !(create->default_table_charset= > - get_charset_by_name(pos+1, MYF(0)))) > + thd->get_charset_by_name(pos+1, MYF(0)))) > { > sql_print_error("Error while loading database options: '%s':",path); > sql_print_error(ER_THD(thd, ER_UNKNOWN_CHARACTER_SET),pos+1); > diff --git a/sql/sys_vars.cc b/sql/sys_vars.cc > index a75066271d9..fd480bdced2 100644 > --- a/sql/sys_vars.cc > +++ b/sql/sys_vars.cc > @@ -3705,7 +3709,7 @@ static Sys_var_set Sys_old_behavior( > "old_mode", > "Used to emulate old behavior from earlier MariaDB or MySQL versions", > SESSION_VAR(old_behavior), CMD_LINE(REQUIRED_ARG), > - old_mode_names, DEFAULT(0)); > + old_mode_names, DEFAULT(8)); DEFAULT(OLD_MODE_UTF8_IS_UTF8MB3) please > > #if defined(HAVE_OPENSSL) && !defined(EMBEDDED_LIBRARY) > #define SSL_OPT(X) CMD_LINE(REQUIRED_ARG,X) > diff --git a/storage/connect/ha_connect.cc b/storage/connect/ha_connect.cc > index 849cf7594eb..b6e8f3b3226 100644 > --- a/storage/connect/ha_connect.cc > +++ b/storage/connect/ha_connect.cc > @@ -6099,8 +6099,8 @@ static int connect_assisted_discovery(handlerton *, THD* thd, > case FLD_NAME: > if (ttp == TAB_PRX || > (ttp == TAB_CSV && topt->data_charset && > - (!stricmp(topt->data_charset, "UTF8") || > - !stricmp(topt->data_charset, "UTF-8")))) > + (!stricmp(topt->data_charset, "UTF8MB3") || > + !stricmp(topt->data_charset, "UTF-8MB3")))) UTF-8MB3 again > cnm= crp->Kdata->GetCharValue(i); > else > cnm= encode(g, crp->Kdata->GetCharValue(i)); > diff --git a/storage/mroonga/vendor/groonga/CMakeLists.txt b/storage/mroonga/vendor/groonga/CMakeLists.txt > index d271d4c4eb9..fa5d6f97ddd 100644 > --- a/storage/mroonga/vendor/groonga/CMakeLists.txt > +++ b/storage/mroonga/vendor/groonga/CMakeLists.txt > @@ -95,7 +95,7 @@ set(GRN_LOG_PATH > "${CMAKE_INSTALL_PREFIX}/var/log/${GRN_PROJECT_NAME}/${GRN_PROJECT_NAME}.log" > CACHE FILEPATH "log file path") > set(GRN_DEFAULT_ENCODING > - "utf8" > + "utf8mb3" utf8 is better here > CACHE STRING "Groonga's default encoding") > set(GRN_DEFAULT_MATCH_ESCALATION_THRESHOLD > 0 > diff --git a/storage/perfschema/unittest/pfs_connect_attr-t.cc b/storage/perfschema/unittest/pfs_connect_attr-t.cc > index b57ead3ec26..7e5b511d543 100644 > --- a/storage/perfschema/unittest/pfs_connect_attr-t.cc > +++ b/storage/perfschema/unittest/pfs_connect_attr-t.cc > @@ -343,7 +343,12 @@ int main(int, char **) > { > MY_INIT("pfs_connect_attr-t"); > > - cs_cp1251= get_charset_by_csname("cp1251", MY_CS_PRIMARY, MYF(0)); > + cs_cp1251= thd->get_charset_by_csname("cp1251", > + thd->variables.old_behavior & > + OLD_MODE_UTF8_IS_UTF8M3 ? > + MY_CS_PRIMARY | MY_CS_UTF8_IS_UTF8MB3 : well, here there is surely no reason to use MY_CS_UTF8_IS_UTF8MB3, because the charset is a hard-coded literal and it's not "utf8", is it? > + MY_CS_PRIMARY, > + MYF(0)); > if (!cs_cp1251) > diag("skipping the cp1251 tests : missing character set"); > plan(59 + (cs_cp1251 ? 10 : 0)); > diff --git a/strings/ctype.c b/strings/ctype.c > index 46951c3ae1f..9054cbfc3e6 100644 > --- a/strings/ctype.c > +++ b/strings/ctype.c > @@ -37,7 +37,7 @@ > */ > > const char charset_name_latin2[]= "latin2"; > -const char charset_name_utf8[]= "utf8"; > +const char charset_name_utf8[]= "utf8mb3"; better rename the variable to charset_name_utf8mb3 > const char charset_name_utf16[]= "utf16"; > const char charset_name_utf32[]= "utf32"; > const char charset_name_ucs2[]= "ucs2"; > diff --git a/tests/mysql_client_test.c b/tests/mysql_client_test.c > index 0043786d477..37da9e1f62e 100644 > --- a/tests/mysql_client_test.c > +++ b/tests/mysql_client_test.c > @@ -405,7 +405,7 @@ static void test_prepare_simple() > > /* update */ > strmov(query, "UPDATE test_prepare_simple SET id=? " > - "WHERE id=? AND CONVERT(name USING utf8)= ?"); > + "WHERE id=? AND CONVERT(name USING utf8mb3)= ?"); I suspect all tests should better use "utf8" > stmt= mysql_simple_prepare(mysql, query); > check_stmt(stmt); > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

[Maria-developers] MDEV-8306: Some pieces of input
by Sergey Petrunia 05 Mar '21

05 Mar '21

Hi Varun, I'm still looking at MDEV-8306 code. Sending the input I have so far. There's nothing important but I thought I'd share it now so you can address it while I'm continuing with the review. > propagate_equal_field_for_orderby Please use "order_by" as in all other functions, and also change "field" to "fields". The name becomes propagate_equal_fields_for_order_by. > void JOIN::propagate_equal_field_for_orderby() > { > if (!sort_nest_possible) > return; > ORDER *ord; > for (ord= order; ord; ord= ord->next) > { > if (optimizer_flag(thd, OPTIMIZER_SWITCH_ORDERBY_EQ_PROP) && cond_equal) > { Is there any reason not to check optimizer_switch flag value once at the start of the function? > void JOIN_TAB::find_keys_that_can_achieve_ordering() > { > if (!join->sort_nest_possible) > return; > > table->keys_with_ordering.clear_all(); > for (uint index= 0; index < table->s->keys; index++) > { > if (table->keys_in_use_for_query.is_set(index) && > test_if_order_by_key(join, join->order, table, index)) > table->keys_with_ordering.set_bit(index); > } > table->keys_with_ordering.intersect(table->keys_in_use_for_order_by); > } Is there any reason to call test_if_order_by_key() and then interset with keys_in_use_for_order_by? Why not just check that bitmap first, like it's done for keys_with_ordering? ... in struct JOIN_TAB: > void find_keys_that_can_achieve_ordering(); > bool check_if_index_satisfies_ordering(int index_used); are "achieving ordering" and "satisfying ordering" the same thing? If yes, it's better to use one term for this. > Item* Item_subselect::transform(THD *thd, Item_transformer transformer, > bool transform_subquery, uchar *arg) We've already discussed this, but I'll mention it here to keep track: this should ON expressions. I would also consider adding an assert that the subquery's joins do not have the query plans, yet. > class Field { > ... > void statistics_available_via_keys(); > void statistics_available_via_stat_tables(); These names need to be better. Function names typically start with a verb. This is especially true for functions that return nothing. > bool Item_func_between::predicate_selectivity_checker(void *arg) > bool Item_func_in::predicate_selectivity_checker(void *arg) These functions need a comment. > bool Item_equal::predicate_selectivity_checker(void *arg) > { > ... > while (it++) > { > Field *field= it.get_curr_field(); > field->is_range_statistics_available(); What is the above? Does the function have some side-effect? This needs to be fixed or at least commented. ... later in the same function: > it.rewind(); > Item *item= (it++); > SAME_FIELD *same_field_arg= (SAME_FIELD *) arg; > > if (same_field_arg->item == NULL) > { > item->is_resolved_by_same_column(arg); > return false; > } This looks very confusing, too. Why are we checking the first element in the Item_equal. Are we assuming it is the constant? But the execution reaches this point even when Item_equal doesn't contain a constant? > bool Item_equal::is_statistics_available() Maybe, rename this to is_range_statistics_available() to make it clear with what kind of statistics is it? > void add_nest_tables_to_trace(Mat_join_tab_nest_info* nest_info) rename this to trace_XXX() to be uniform with other tracing functions? e.g. rename to trace_tables_in_sort_nest BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://petrunia.net

1 0

[Maria-developers] MDEV-17399: JSON_TABLE: Code cleanup
by Sergey Petrunia 04 Mar '21

04 Mar '21

Hi Alexey, Please check out this patch: http://lists.askmonty.org/pipermail/commits/2021-March/014492.html Any objections to it? BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://petrunia.net

1 0

[Maria-developers] MDEV-17399: JSON_TABLE: Crash with nested path
by Sergey Petrunia 01 Mar '21

01 Mar '21

Hi Alexey, I was looking at Json_table_nested_path::set_position(), wondering why does it have an assignment np->m_null= TRUE; but doesn't clear the NULL values and trying to come up with an example of this going wrong when I've hit this crash: select * from json_table( '[ {"name": "X", "colors":["blue"], "sizes": [1,2,3,4], "prices" : [10,20]}, {"name": "Y", "colors":["red"], "sizes": [10,11], "prices" : [100,200,300]} ]', '$[*]' columns ( seq0 for ordinality, name varchar(4) path '$.name', nested path '$.colors[*]' columns ( seq1 for ordinality, color text path '$' ), nested path '$.sizes[*]' columns ( seq2 for ordinality, size int path '$' ), nested path '$.prices[*]' columns ( seq3 for ordinality, price int path '$' ) ) ) as T order by seq0, name; Note this==NULL: (gdb) wher #0 0x00005555560edf72 in Json_table_nested_path::set_position (this=0x0, j_start=0x7ffeb0016e68 "[ \n {\"name\": \"X\", \n \"colors\":[\"blue\"], \"sizes\": [1,2,3,4], \"prices\" : [10,20]},\n {\"name\": \"Y\", \n \"colors\":[\"red\"], \"sizes\": [10,11], \"prices\" : [100,200,300]}\n]", j_end=0x7ffeb0016f12 "", pos=0x7ffeb0035e51 "\245\245\245\245\245\245\245\245\006") at /home/psergey/dev-git2/10.6-hf-review6/sql/table_function.cc:239 #1 0x00005555560ee12f in Json_table_nested_path::set_position (this=0x7ffeb0017060, j_start=0x7ffeb0016e68 "[ \n {\"name\": \"X\", \n \"colors\":[\"blue\"], \"sizes\": [1,2,3,4], \"prices\" : [10,20]},\n {\"name\": \"Y\", \n \"colors\":[\"red\"], \"sizes\": [10,11], \"prices\" : [100,200,300]}\n]", j_end=0x7ffeb0016f12 "", pos=0x7ffeb0035e48 "") at /home/psergey/dev-git2/10.6-hf-review6/sql/table_function.cc:262 #2 0x00005555560ee9f0 in ha_json_table::rnd_pos (this=0x7ffeb0014f00, buf=0x7ffeb0025570 "\377", pos=0x7ffeb0035e48 "") at /home/psergey/dev-git2/10.6-hf-review6/sql/table_function.cc:434 #3 0x00005555561ca6a4 in handler::ha_rnd_pos (this=0x7ffeb0014f00, buf=0x7ffeb0025570 "\377", pos=0x7ffeb0035e48 "") at /home/psergey/dev-git2/10.6-hf-review6/sql/handler.cc:3101 #4 0x00005555563852e3 in rr_from_pointers (info=0x7ffeb001f9e0) at /home/psergey/dev-git2/10.6-hf-review6/sql/records.cc:615 #5 0x0000555555da4a75 in READ_RECORD::read_record (this=0x7ffeb001f9e0) at /home/psergey/dev-git2/10.6-hf-review6/sql/records.h:81 #6 0x0000555555ee1876 in join_init_read_record (tab=0x7ffeb001f918) at /home/psergey/dev-git2/10.6-hf-review6/sql/sql_select.cc:21644 #7 0x0000555555edf35a in sub_select (join=0x7ffeb001d948, join_tab=0x7ffeb001f918, end_of_records=false) at /home/psergey/dev-git2/10.6-hf-review6/sql/sql_select.cc:20666 #8 0x0000555555ede8e6 in do_select (join=0x7ffeb001d948, procedure=0x0) at /home/psergey/dev-git2/10.6-hf-review6/sql/sql_select.cc:20216 #9 0x0000555555eb24e7 in JOIN::exec_inner (this=0x7ffeb001d948) at /home/psergey/dev-git2/10.6-hf-review6/sql/sql_select.cc:4484 #10 0x0000555555eb1613 in JOIN::exec (this=0x7ffeb001d948) at /home/psergey/dev-git2/10.6-hf-review6/sql/sql_select.cc:4264 Please fix. BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://petrunia.net

2 2

Re: [Maria-developers] ee538938345: MDEV-21117: refine the server binlog-based recovery for semisync
by Sergei Golubchik 26 Feb '21

26 Feb '21

Hi, Sujatha! The main comment - the logic is so complex, I wasn't able to understand it, unfortunately. I've reviewed almost everything, see comments below. But not the Recovery_context methods. Please explain how it works and how all these truncate_validated, truncate_reset_done, truncate_set_in_1st, etc all work together. On Feb 26, Sujatha wrote: > revision-id: ee538938345 (mariadb-10.3.26-68-gee538938345) > parent(s): 7d04ce6a2d4 > author: Sujatha <sujatha.sivakumar(a)mariadb.com> > committer: Andrei Elkin <andrei.elkin(a)mariadb.com> > timestamp: 2021-02-08 17:58:03 +0200 > message: > > MDEV-21117: refine the server binlog-based recovery for semisync > this should be MDEV subject. But if you like this better you can rename the MDEV instead > > Problem: > ======= > When the semisync master is crashed and restarted as slave it could > recover transactions that former slaves may never have seen. > A known method existed to clear out all prepared transactions > with --tc-heuristic-recover=rollback does not care to adjust > binlog accordingly. > > Fix: > === > The binlog-based recovery is made to concern of the slave semisync role of > post-crash restarted server. > No changes in behaviour is done to the "normal" binloggging server > and the semisync master. > > When the restarted server is configured with > --rpl-semi-sync-slave-enabled=1 > the refined recovery attempts to roll back prepared transactions > and truncate binlog accordingly. > In case of a partically committed (that is committed at least > partially > > in one of the engine participants) such transaction gets committed. > It's guaranteed no (partially as well) committed transactions > exist beyond the truncate position. > In case there exists a non-transactional replication event > (being in a way a committed transaction) past the > computed truncate position the recovery ends with an error. > > To facilite the failover on the slave side facilitate > conditions to accept own events (having been discarded by the above recovery) > are relaxed to let so for the semisync slave that connects to master > in gtid mode. gtid_strict_mode is further recommended to secure > from inadvertent re-applying out of order gtids in general. > Non-gtid mode connected semisync slave would require > --replicate-same-server-id (mind --log-slave-updates must be OFF then). Sorry, I failed to understand this paragraph :( > > diff --git a/mysql-test/suite/binlog/t/binlog_truncate_active_log.test b/mysql-test/suite/binlog/t/binlog_truncate_active_log.test > new file mode 100644 > index 00000000000..cf89525dcac > --- /dev/null > +++ b/mysql-test/suite/binlog/t/binlog_truncate_active_log.test > @@ -0,0 +1,76 @@ > +# ==== Purpose ==== > +# > +# Test verifies the truncation of single binary log file. > +# > +# ==== Implementation ==== > +# > +# Steps: > +# 0 - Create table t1 and insert/commit a row. > +# 1 - Insert an another row such that it gets written to binlog but commit > +# in engine fails as server crashed at this point. > +# 2 - Restart server with --rpl-semi-sync-slave-enabled=1 > +# 3 - Upon server start 'master-bin.000001' will be truncated to contain > +# only the first insert > +# > +# ==== References ==== > +# > +# MDEV-21117: recovery for --rpl-semi-sync-slave-enabled server > + > +--source include/have_innodb.inc > +--source include/have_aria.inc > +--source include/have_log_bin.inc > +--source include/have_debug.inc > +--source include/have_binlog_format_statement.inc have_binlog_format_statement obviously implies have_log_bin. it's redundant to specify it explicitly. and you don't need have_debug here > + > +call mtr.add_suppression("Can.t init tc log"); > +call mtr.add_suppression("Aborting"); > + > +# The following cases are tested: > +# A. 2pc transaction is followed by a blank "zero-engines" one > +# B. 2pc transaction follows the blank one > +# C. Similarly to A, with the XA blank transaction > + > +--connection default it's connection default by default (not surprisingly) :) > +RESET MASTER; > +CREATE TABLE t ( f INT ) ENGINE=INNODB; > +CREATE TABLE t2 ( f INT ) ENGINE=INNODB; > +CREATE TABLE tm ( f INT ) ENGINE=Aria; Why Aria? I mean, the default "standard" non-transactional engine is MyISAM. If you use Aria, it means you thought about it and decided to deviate intentionally from the default. What were your considerations? > + > +--echo # Case A. > +# Both are doomed into truncation. > +--let $this_search_pattern = Successfully truncated.*to remove transactions starting from GTID 0-1-6 why not simply --let SEARCH_PATTERN = Successfully truncated...etc... > +--let $query1 = INSERT INTO t VALUES (20) > +--let $query2 = DELETE FROM t2 WHERE f = 666 /* no such record */ > +--source binlog_truncate_active_log.inc > + > +--echo # Case B. > +# The inverted sequence ends up to truncate only $query2 > +--let $this_search_pattern = Successfully truncated.*to remove transactions starting from GTID 0-1-10 > +--let $query1 = DELETE FROM t2 WHERE f = 0 why not `= 666 /* no such record */` ? is `= 0` important here? > +--let $query2 = INSERT INTO t VALUES (20) > +--source binlog_truncate_active_log.inc > + > + > +delimiter |; > +CREATE PROCEDURE sp_blank_xa() > +BEGIN > + XA START 'blank'; > + DELETE FROM t2 WHERE f = 666 /* no such record */; > + XA END 'blank'; > + XA PREPARE 'blank'; > +END| > +delimiter ;| > + > + > +--echo # Case C. > +--let $this_search_pattern = Successfully truncated.*to remove transactions starting from GTID 0-1-13 > +--let $query1 = INSERT INTO t VALUES (20) > +--let $pre_q2 = CALL sp_blank_xa > +--let $query2 = XA COMMIT 'blank' > +--source binlog_truncate_active_log.inc > +DROP PROCEDURE sp_blank_xa; > + > +--echo # Cleanup > +DROP TABLE t,t2,tm; > + > +--echo # End of the tests > diff --git a/mysql-test/suite/binlog/t/binlog_truncate_active_log.inc b/mysql-test/suite/binlog/t/binlog_truncate_active_log.inc > new file mode 100644 > index 00000000000..b1ffaf18268 > --- /dev/null > +++ b/mysql-test/suite/binlog/t/binlog_truncate_active_log.inc > @@ -0,0 +1,70 @@ you need source include/have_debug_sync.inc here, as this file uses DEBUG_SYNC > +connect(master1,localhost,root,,); > +connect(master2,localhost,root,,); > +connect(master3,localhost,root,,); just FYI (no need to change anything), you know that parentheses and extra commas are optional, right? that is, it could be written as connect master1,localhost,root; connect master2,localhost,root; connect master3,localhost,root; > + > +--connection default > + > +# First to commit few transactions > +INSERT INTO t VALUES (10); > +INSERT INTO tm VALUES (10); > + > +--connection master1 > +# Hold insert after write to binlog and before "run_commit_ordered" in engine > +SET DEBUG_SYNC= "commit_before_get_LOCK_commit_ordered SIGNAL master1_ready WAIT_FOR signal_never_arrives"; > +--send_eval $query1 > + > +--connection master2 > +SET DEBUG_SYNC= "now WAIT_FOR master1_ready"; > +if ($pre_q2) is $pre_q2 a flag or a query string? it's used as a flag, it's set as a query string. > +{ > + CALL sp_blank_xa; > +} > +SET DEBUG_SYNC= "commit_before_get_LOCK_after_binlog_sync SIGNAL master2_ready"; > +# To binlog non-xid transactional group which will be truncated all right > +--send_eval $query2 > + > + > +--connection master3 > +SET DEBUG_SYNC= "now WAIT_FOR master2_ready"; > +SELECT @@global.gtid_binlog_pos as 'Before the crash'; > + > +--write_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect > +wait > +EOF > + > +--source include/kill_mysqld.inc > +--source include/wait_until_disconnected.inc you know that kill_mysqld both writes 'wait' to the expect file and includes wait_until_disconnected ? > +# > +# Server restart > +# > +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect > +restart: --rpl-semi-sync-slave-enabled=1 > +EOF why "append_file" and not "write_file" ? and why wait, you could've written restart right away, couldn't you? > + > +connection default; > +--enable_reconnect > +--source include/wait_until_connected_again.inc > + > +# Check error log for a successful truncate message. > +let $log_error_= `SELECT @@GLOBAL.log_error`; > +if(!$log_error_) > +{ > + # MySQL Server on windows is started with --console and thus > + # does not know the location of its .err log, use default location > + let $log_error_ = $MYSQLTEST_VARDIR/log/mysqld.1.err; isn't it always $MYSQLTEST_VARDIR/log/mysqld.1.err? other tests don't do it conditionally > +} > +--let SEARCH_FILE=$log_error_ > +--let SEARCH_RANGE=-50000 if you're searching in the error log - and you are - it's best not to set the SEARCH_RANGE at all. Then it'll search from the last CURRENT_TEST: label. That is, it'll search in all error log entries generated by the current test and not in the error log that came from earlier tests. > +--let SEARCH_PATTERN=$this_search_pattern > +--replace_regex /FOUND [0-9]+/FOUND #/ can it be found multiple times? Why would binlog be truncated more than once? > +--source include/search_pattern_in_file.inc > + > +SELECT @@global.gtid_binlog_pos as 'After the crash'; > +--echo "One row should be present in table 't'" > +SELECT COUNT(*) as 'One' FROM t; Do you know which one? If yes, why not `SELECT * FROM t` ? > + > +# Local cleanup > +DELETE FROM t; > +--disconnect master1 > +--disconnect master2 > +--disconnect master3 > diff --git a/mysql-test/suite/binlog/t/binlog_truncate_multi_engine.test b/mysql-test/suite/binlog/t/binlog_truncate_multi_engine.test > new file mode 100644 > index 00000000000..fe153e5703c > --- /dev/null > +++ b/mysql-test/suite/binlog/t/binlog_truncate_multi_engine.test > @@ -0,0 +1,78 @@ > +# ==== Purpose ==== > +# > +# Test verifies truncation of multiple binary logs. and "with multiple transactional storage engines" ? > +# > +# ==== Implementation ==== > +# > +# Steps: > +# 0 - Create two tables in innodb and rocksdb engines, > +# > +# In loop for A,B,C cases (below) do 1-5: there's no loop now (which is good, don't add it please :) > +# 1 - execute FLUSH LOGS command to generate a new binary log. > +# Start a transaction inserting rows of sufficient sizes > +# so binary log gets rotated at commit > +# 2 - Using debug simulation make the server crash at a point where > +# the transaction is written to binary log *and* either of > +# A. neither of them commits > +# B. only one commits > +# C. both commit > +# 3 - print the # of binlog files before the transaction starts and after its > +# commit is submitted > +# 4 - Restart server with --tc-heuristic-recover=BINLOG_TRUNCATE outdated comment? > +# 5 - Restart normally to print post recovery status. > +# > +# ==== References ==== > +# > +# MDEV-21117: recovery for --rpl-semi-sync-slave-enabled server > + > + > +--source include/have_innodb.inc > +--source include/have_rocksdb.inc > +--source include/have_log_bin.inc this is redundant > +--source include/have_debug.inc > +--source include/have_debug_sync.inc > +--source include/have_binlog_format_row.inc > + > +--let $old_max_binlog_size= `select @@global.max_binlog_size` > +call mtr.add_suppression("Can.t init tc log"); > +call mtr.add_suppression("Aborting"); > +--let $MYSQLD_DATADIR= `SELECT @@datadir` > + > +CREATE TABLE t1 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; > +CREATE TABLE t2 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=rocksdb; > + > +--let $case = A: neither engine committed => rollback & binlog truncate > +# Hold off engine commits after write to binlog and its rotation. > +# The transaction is killed along with the server after that. > +--let $kill_server=1 > +--let $debug_sync_action = "commit_after_release_LOCK_log SIGNAL con1_ready WAIT_FOR signal_no_signal" > +--let $restart_parameters = --rpl-semi-sync-slave-enabled=1 > +--let $test_outcome= 1 row should be present in both tables; binlog is truncated; number of binlogs at reconnect - 3 > + --source binlog_truncate_multi_engine.inc > +--echo Proof of the truncated binlog file is readable (two transactions must be seen): > +--let $MYSQLD_DATADIR = `select @@datadir` no need to set the $MYSQLD_DATADIR twice > +--exec $MYSQL_BINLOG --short-form --skip-annotate-row-events $MYSQLD_DATADIR/master-bin.000002 > + > +--let $case = B: one engine has committed its transaction branch > +# Hold off after one engine has committed. > +--let $kill_server=1 > +--let $debug_sync_action = "commit_after_run_commit_ordered SIGNAL con1_ready WAIT_FOR signal_no_signal" > +--let $restart_simulate_partial_commit = 1 this variable seems to be unused > +--let $restart_parameters = --rpl-semi-sync-slave-enabled=1 --debug-dbug=d,binlog_truncate_partial_commit this seems to be a rather crude way of faking a partially committed transaction. better to crash after the first engine has committed, that'd be much more natural. > +--let $test_outcome= 2 rows should be present in both tables; no binlog truncation; one extra binlog file compare with A; number of binlogs at reconnect - 4 > + --source binlog_truncate_multi_engine.inc > + > +--let $case = C: both engines have committed its transaction branch you didn't do --let $debug_sync_action = "reset", so the old value is used. Intentional? > +# Hold off after both engines have committed. The server is shut down. > +--let $kill_server=0 > +--let $restart_parameters = --rpl-semi-sync-slave-enabled=1 > +--let $test_outcome= 2 rows should be present in both tables; no binlog truncation; the same # of binlog files as in B; number of binlogs at reconnect - 4 > + --source binlog_truncate_multi_engine.inc > + > + > + > +DROP TABLE t1, t2; > +--replace_result $old_max_binlog_size VALUE_AT_START > +--eval SET @@global.max_binlog_size = $old_max_binlog_size better use evalp instead of eval, then you won't need replace_result but really I think you don't need to save/restore max_binlog_size at all, because you restart the server anyway. > + > +--echo # End of the tests > diff --git a/mysql-test/suite/binlog/t/binlog_truncate_multi_engine.inc b/mysql-test/suite/binlog/t/binlog_truncate_multi_engine.inc > new file mode 100644 > index 00000000000..c260a7987e2 > --- /dev/null > +++ b/mysql-test/suite/binlog/t/binlog_truncate_multi_engine.inc > @@ -0,0 +1,64 @@ > +# > +# Loop body of binlog_truncate_multi_engine.test > +# Parameters: > +# $debug_sync_action describes debug-sync actions > +# $kill_server 1 when to crash, 0 for regular restart > +# $restart_parameters the caller may simulate partial commit at recovery > +# $test_outcome summary of extected results > +# $MYSQLD_DATADIR > + > +--echo # > +--echo # > +--echo # Case $case > +--echo # > +RESET MASTER; > +FLUSH LOGS; > +SET GLOBAL max_binlog_size= 4096; > + > +connect(con1,localhost,root,,); > +#--connection con1 > +--echo List of binary logs before rotation > +--source include/show_binary_logs.inc > +INSERT INTO t1 VALUES (1, REPEAT("x", 1)); > +INSERT INTO t2 VALUES (1, REPEAT("x", 1)); > +BEGIN; > + INSERT INTO t1 VALUES (2, REPEAT("x", 4100)); > + INSERT INTO t2 VALUES (2, REPEAT("x", 4100)); > + > +--eval SET DEBUG_SYNC= $debug_sync_action > +send COMMIT; > + > +--connection default > +SET DEBUG_SYNC= "now WAIT_FOR con1_ready"; > +--echo List of binary logs after rotation > +--source include/show_binary_logs.inc > + > +--echo # restart the server with $restart_parameters > +if ($kill_server) > +{ > + --echo # the server is crashed > + --source include/kill_mysqld.inc > + --source include/start_mysqld.inc > +} > +if (!$kill_server) > +{ > + --echo # the server is restarted > + --source include/restart_mysqld.inc it'd be simpler not to use $kill_server at all. And instead write let $shutdown_timeout=0; instead of $kill_server=1; and let $shutdown_timeout=; instead of $kill_server=0; and above you won't need two if's, you can just do source include/restart_mysqld.inc; > +} > + > +--connection default > +--echo # > +--echo # *** Summary: $test_outcome: > +--echo # > +SELECT COUNT(*) FROM t1; > +SELECT COUNT(*) FROM t2; > +SELECT @@GLOBAL.gtid_binlog_state; > +SELECT @@GLOBAL.gtid_binlog_pos; > +--echo List of binary logs at the end of the tests > +--source include/show_binary_logs.inc > +--echo # *** > +# cleanup > +DELETE FROM t1; > +DELETE FROM t2; > +--disconnect con1 > +--echo # > diff --git a/mysql-test/suite/binlog/t/binlog_truncate_multi_log.test b/mysql-test/suite/binlog/t/binlog_truncate_multi_log.test > new file mode 100644 > index 00000000000..231e90dbdc9 > --- /dev/null > +++ b/mysql-test/suite/binlog/t/binlog_truncate_multi_log.test > @@ -0,0 +1,105 @@ > +# ==== Purpose ==== > +# > +# Test verifies truncation of multiple binary logs. > +# > +# ==== Implementation ==== > +# > +# Steps: > +# 1 - Set max_binlog_size= 4096, to help a series of inserts into a > +# transaction table 'ti' get binlog rotated so many time while the > +# transactions won't be committed, being stopped at > +# a prior to commit debug_sync point > +# 2 - kill and restart the server as semisync slave successfully to > +# end with an expected first binlog and the gtid state. > +# > +# ==== References ==== > +# MDEV-21117: recovery for --rpl-semi-sync-slave-enabled server > + > +--source include/have_innodb.inc > +--source include/have_log_bin.inc > +--source include/have_debug.inc > +--source include/have_debug_sync.inc > +--source include/have_binlog_format_row.inc > + > +call mtr.add_suppression("Can.t init tc log"); > +call mtr.add_suppression("Aborting"); > + > +--let $old_max_binlog_size= `select @@global.max_binlog_size` > +SET @@global.max_binlog_size= 4096; > + > +RESET MASTER; > +FLUSH LOGS; > +CREATE TABLE ti (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; > +CREATE TABLE tm (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=MyISAM; > + > +connect(master1,localhost,root,,); > +--echo "List of binary logs before rotation" > +--source include/show_binary_logs.inc > + > +# Some load to either non- and transactional egines > +# that should not affect the following recovery: > +INSERT INTO ti VALUES(1,"I am gonna survive"); > +INSERT INTO tm VALUES(1,"me too!"); > + > +# hold on near engine commit > +SET DEBUG_SYNC= "commit_after_release_LOCK_after_binlog_sync SIGNAL master1_ready WAIT_FOR con1_go"; > +--send_eval INSERT INTO ti VALUES (2, REPEAT("x", 4100)) > + > +connect(master2,localhost,root,,); > +# The 2nd trx for recovery, it does not rotate binlog > +SET DEBUG_SYNC= "now WAIT_FOR master1_ready"; > +SET DEBUG_SYNC= "commit_before_get_LOCK_commit_ordered SIGNAL master2_ready WAIT_FOR master2_go"; > +--send_eval INSERT INTO ti VALUES (3, "not gonna survive") > + > +--connection default > +SET DEBUG_SYNC= "now WAIT_FOR master2_ready"; > +--echo "List of binary logs before crash" > +--source include/show_binary_logs.inc > +--echo # The gtid binlog state prior the crash will be truncated at the end of the test > +SELECT @@global.gtid_binlog_state; > +--write_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect > +wait > +EOF > + > +--source include/kill_mysqld.inc > +--source include/wait_until_disconnected.inc > + > +# > +# Server restart > +# > +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect > +restart: --rpl-semi-sync-slave-enabled=1 > +EOF > + > +connection default; > +--enable_reconnect > +--source include/wait_until_connected_again.inc > + > +# Check error log for a successful truncate message. > +let $log_error_= `SELECT @@GLOBAL.log_error`; > +if(!$log_error_) > +{ > + # MySQL Server on windows is started with --console and thus > + # does not know the location of its .err log, use default location > + let $log_error_ = $MYSQLTEST_VARDIR/log/mysqld.1.err; > +} > +--let SEARCH_FILE=$log_error_ > +--let SEARCH_RANGE=-50000 > +--let SEARCH_PATTERN=truncated binlog file:.*master.*000002 > +--replace_regex /FOUND [0-9]+/FOUND #/ > +--source include/search_pattern_in_file.inc > + > + > +--echo "One record should be present in table" > +SELECT count(*) FROM ti; > + > +--echo # The truncated gtid binlog state > +SELECT @@global.gtid_binlog_state; > +SELECT @@global.gtid_binlog_pos; > + > +--echo # Cleanup > +--replace_result $old_max_binlog_size VALUE_AT_START > +--eval SET @@global.max_binlog_size = $old_max_binlog_size > +DROP TABLE ti; > + many of my comments above apply to this test too > +--echo # End of the tests > diff --git a/mysql-test/suite/binlog/t/binlog_truncate_multi_log_unsafe.test b/mysql-test/suite/binlog/t/binlog_truncate_multi_log_unsafe.test > new file mode 100644 > index 00000000000..6aba2935fde > --- /dev/null > +++ b/mysql-test/suite/binlog/t/binlog_truncate_multi_log_unsafe.test > @@ -0,0 +1,142 @@ > +# ==== Purpose ==== > +# The test verifies attempt to recover by the semisync slave server whose > +# binlog is unsafe for truncation. > +# > +# ==== Implementation ==== > +# 2 binlog files are created with the 1st one destined to be the binlog > +# checkpoint file for recovery. > +# The final group of events is replication unsafe (myisam INSERT). > +# Therefore the semisync slave recovery may not. > +# > +# Steps: > +# 0 - Set max_binlog_size= 4096, to help an insert into a > +# transaction table 'ti' get binlog rotated while the > +# transaction won't be committed, being stopped at > +# a prior to commit debug_sync point > +# 1 - insert into a non-transactional 'tm' table completes with > +# binary logging as well > +# 2 - kill and attempt to restart the server as semisync slave that > +# must produce an expected unsafe-to-recover error > +# 3 - complete the test with a normal restart that successfully finds and > +# commits the transaction in doubt. > +# > +# ==== References ==== > +# MDEV-21117: recovery for --rpl-semi-sync-slave-enabled server > + > + > +--source include/have_innodb.inc > +--source include/have_log_bin.inc > +--source include/have_debug.inc > +--source include/have_debug_sync.inc > +--source include/have_binlog_format_row.inc > + > +--let $old_max_binlog_size= `select @@global.max_binlog_size` > +SET @@global.max_binlog_size= 4096; > + > +call mtr.add_suppression("Table '.*tm' is marked as crashed and should be repaired"); > +call mtr.add_suppression("Got an error from unknown thread"); > +call mtr.add_suppression("Checking table: '.*tm'"); > +call mtr.add_suppression("Recovering table: '.*tm'"); > +call mtr.add_suppression("Cannot trim the binary log to file"); > +call mtr.add_suppression("Crash recovery failed"); > +call mtr.add_suppression("Can.t init tc log"); > +call mtr.add_suppression("Aborting"); > +call mtr.add_suppression("Found 1 prepared transactions"); > +call mtr.add_suppression("mysqld: Table.*tm.*is marked as crashed"); > +call mtr.add_suppression("Checking table.*tm"); > + > +RESET MASTER; > +FLUSH LOGS; > +CREATE TABLE ti (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; > +CREATE TABLE tm (f INT) ENGINE=MYISAM; > + > +--let $row_count = 5 > +--let $i = 3 > +--disable_query_log > +while ($i) > +{ > + --eval INSERT INTO ti VALUES ($i, REPEAT("x", 1)) > +--dec $i > +} > +--enable_query_log > +INSERT INTO tm VALUES(1); > + > +connect(master1,localhost,root,,); > +connect(master2,localhost,root,,); > +connect(master3,localhost,root,,); > + > +--connection master1 > + > +# The 1st trx binlogs, rotate binlog and hold on before committing at engine > +SET DEBUG_SYNC= "commit_after_release_LOCK_after_binlog_sync SIGNAL master1_ready WAIT_FOR master1_go"; you don't wait for master1_ready anywhere. intentional? > +--send_eval INSERT INTO ti VALUES ($row_count - 1, REPEAT("x", 4100)) if you plan to insert rows 1,2,3,4,5 then your $i=3 and $row_count=5 are not independent, you should use $i=$row_count-2. (probably $i=`select $row_count-2`) or don't pretent this test is tunable and use literals instead of variables. > + > +--connection master2 > + > +# The 2nd trx for recovery, it does not rotate binlog > +SET DEBUG_SYNC= "commit_before_get_LOCK_commit_ordered SIGNAL master2_ready WAIT_FOR master2_go"; > +--send_eval INSERT INTO ti VALUES ($row_count, REPEAT("x", 1)) > + > +--connection master3 > +SET DEBUG_SYNC= "now WAIT_FOR master2_ready"; > +SET DEBUG_SYNC= "commit_before_get_LOCK_after_binlog_sync SIGNAL master3_ready"; > +--send INSERT INTO tm VALUES (2) > + > +--connection default > +SET DEBUG_SYNC= "now WAIT_FOR master3_ready"; > +--echo # The gtid binlog state prior the crash must be restored at the end of the testSELECT @@global.gtid_binlog_state; eh? Was this SELECT supposed to be on a separate line? > + > +--write_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect > +wait > +EOF > + > +SELECT @@global.gtid_binlog_state; > +--source include/kill_mysqld.inc > +--source include/wait_until_disconnected.inc > + > +# > +# Server restarts > +# > +--echo # Failed restart as the semisync slave > +--error 1 > +--exec $MYSQLD_LAST_CMD --rpl-semi-sync-slave-enabled=1 >> $MYSQLTEST_VARDIR/log/mysqld.1.err 2>&1 > + > +--echo # Normal restart > +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect > +restart: Okay. *this* is a reasonable use of wait/restart in the expect file. In other cases, I think, you can just restart, no need to wait, as you don't do anything while the server is down. still, here you don't need to write 'wait' to the expect file and don't need to wait_until_disconnected (kill_mysqld is doing both) > +EOF > + > +connection default; > +--enable_reconnect > +--source include/wait_until_connected_again.inc > + > +# Check error log for correct messages. > +let $log_error_= `SELECT @@GLOBAL.log_error`; > +if(!$log_error_) > +{ > + # MySQL Server on windows is started with --console and thus > + # does not know the location of its .err log, use default location > + let $log_error_ = $MYSQLTEST_VARDIR/log/mysqld.1.err; > +} > +--let SEARCH_FILE=$log_error_ > +--let SEARCH_RANGE=-50000 > +--let SEARCH_PATTERN=Cannot trim the binary log to file > +--replace_regex /FOUND [0-9]+/FOUND #/ > +--source include/search_pattern_in_file.inc > + > +--echo # Proof that the in-doubt transactions are recovered by the 2nd normal server restart > +--eval SELECT COUNT(*) = $row_count as 'True' FROM ti > +# myisam table may require repair (which is not tested here) > +--disable_warnings > +SELECT COUNT(*) <= 1 FROM tm; > +--enable_warnings > + > +--echo # The gtid binlog state prior the crash is restored now > +SELECT @@GLOBAL.gtid_binlog_state; > +SELECT @@GLOBAL.gtid_binlog_pos; > + > +--echo # Cleanup > +--replace_result $old_max_binlog_size VALUE_AT_START > +--eval SET @@global.max_binlog_size = $old_max_binlog_size see comments elsewhere about $old_max_binlog_size, -50000, $log_error_, etc > +DROP TABLE ti, tm; > +--echo End of test > diff --git a/mysql-test/suite/rpl/t/rpl_semi_sync_fail_over.cnf b/mysql-test/suite/rpl/t/rpl_semi_sync_fail_over.cnf > new file mode 100644 > index 00000000000..3518eb95b67 > --- /dev/null > +++ b/mysql-test/suite/rpl/t/rpl_semi_sync_fail_over.cnf > @@ -0,0 +1,13 @@ > +!include suite/rpl/rpl_1slave_base.cnf > +!include include/default_client.cnf > + > + > +[mysqld.1] > +log-slave-updates > +gtid-strict-mode=1 > +max_binlog_size= 4096 > + > +[mysqld.2] > +log-slave-updates > +gtid-strict-mode=1 > +max_binlog_size= 4096 why not an .opt file? why max_binlog_size here and not at run-time like in other tests? > diff --git a/mysql-test/suite/rpl/t/rpl_semi_sync_fail_over.test b/mysql-test/suite/rpl/t/rpl_semi_sync_fail_over.test > new file mode 100644 > index 00000000000..972aaf2c8b4 > --- /dev/null > +++ b/mysql-test/suite/rpl/t/rpl_semi_sync_fail_over.test > @@ -0,0 +1,144 @@ > +# ==== Purpose ==== > +# > +# Test verifies replication failover scenario. > +# > +# ==== Implementation ==== > +# > +# Steps: > +# 0 - Having two servers 1 and 2 enable semi-sync replication with > +# with the master wait 'after_sync'. > +# 1 - Insert a row. While inserting second row simulate > +# a server crash at once the transaction is written to binlog, flushed > +# and synced but the binlog position is not updated. > +# 2 - Post crash-recovery on the old master execute there CHANGE MASTER > +# TO command to connect to server id 2. > +# 3 - The old master new slave server 1 must connect to the new > +# master server 2. > +# 4 - repeat the above to crash the new master and restore in role the old one > +# > +# ==== References ==== > +# > +# MDEV-21117: recovery for --rpl-semi-sync-slave-enabled server > + > + > +--source include/have_innodb.inc > +--source include/have_log_bin.inc > +--source include/have_debug.inc > +--source include/have_debug_sync.inc > +--source include/have_binlog_format_row.inc > +--let $rpl_topology=1->2 > +--source include/rpl_init.inc isn't this a normal master-slave.inc topology? > + > +--connection server_2 > +--source include/stop_slave.inc > + > +--connection server_1 > +RESET MASTER; > + > +--connection server_2 > +RESET MASTER; > +set @@global.rpl_semi_sync_slave_enabled = 1; > +set @@global.gtid_slave_pos = ""; > +CHANGE MASTER TO master_use_gtid= slave_pos; > +--source include/start_slave.inc > + > + > +--connection server_1 > +ALTER TABLE mysql.gtid_slave_pos ENGINE=InnoDB; why? > +set @@global.rpl_semi_sync_master_enabled = 1; > +set @@global.rpl_semi_sync_master_wait_point=AFTER_SYNC; > + > +call mtr.add_suppression("Can.t init tc log"); > +call mtr.add_suppression("Aborting"); > +call mtr.add_suppression("1 client is using or hasn.t closed the table properly"); > +call mtr.add_suppression("Table './mtr/test_suppressions' is marked as crashed and should be repaired"); > + > +CREATE TABLE t1 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; > +INSERT INTO t1 VALUES (1, 'dummy1'); > + > +# > +# CRASH the original master, and FAILOVER to the new > +# > + > +# value 1 for server id 1 -> 2 failover > +--let $failover_to_slave=1 > +--let $query_to_crash= INSERT INTO t1 VALUES (2, REPEAT("x", 4100)) > +--let $log_search_pattern=truncated binlog file:.*master.*000001 > +--source rpl_semi_sync_crash.inc > + > +--connection server_2 > +--let $rows_so_far=3 > +--eval INSERT INTO t1 VALUES ($rows_so_far, 'dummy3') > +--save_master_pos > +--echo # The gtid state on current master must be equal to ... > +SHOW VARIABLES LIKE 'gtid_binlog_pos'; > + > +--connection server_1 > +--sync_with_master > +--eval SELECT COUNT(*) = $rows_so_far as 'true' FROM t1 > +--echo # ... the gtid states on the slave: > +SHOW VARIABLES LIKE 'gtid_slave_pos'; > +SHOW VARIABLES LIKE 'gtid_binlog_pos'; > + > +--connection server_2 > +# > +# CRASH the new master and FAILOVER back to the original > +# > + > +# value 0 for the reverse server id 2 -> 1 failover > +--let $failover_to_slave=0 > +--let $query_to_crash = INSERT INTO t1 VALUES (4, REPEAT("x", 4100)) > +--let $query2_to_crash= INSERT INTO t1 VALUES (5, REPEAT("x", 4100)) > +--let $log_search_pattern=truncated binlog file:.*slave.*000001 > +--source rpl_semi_sync_crash.inc > + > +--connection server_1 > +--let $rows_so_far=6 > +--eval INSERT INTO t1 VALUES ($rows_so_far, 'Done') > +--save_master_pos > +--echo # The gtid state on current master must be equal to ... > +SHOW VARIABLES LIKE 'gtid_binlog_pos'; > + > +--connection server_2 > +--sync_with_master > +--eval SELECT COUNT(*) = $rows_so_far as 'true' FROM t1 > +--echo # ... the gtid states on the slave: > +SHOW VARIABLES LIKE 'gtid_slave_pos'; > +SHOW VARIABLES LIKE 'gtid_binlog_pos'; > + > + > +--let $diff_tables=server_1:t1, server_2:t1 > +--source include/diff_tables.inc > + > +# > +--echo # Cleanup > +# > +--connection server_1 > +DROP TABLE t1; > +--save_master_pos > + > +--connection server_2 > +--sync_with_master > +--source include/stop_slave.inc > + > +--connection server_1 > +set @@global.rpl_semi_sync_master_enabled = 0; > +set @@global.rpl_semi_sync_slave_enabled = 0; > +set @@global.rpl_semi_sync_master_wait_point=default; > +RESET SLAVE; > +RESET MASTER; > + > +--connection server_2 > +set @@global.rpl_semi_sync_master_enabled = 0; > +set @@global.rpl_semi_sync_slave_enabled = 0; > +set @@global.rpl_semi_sync_master_wait_point=default; > + > +--replace_result $SERVER_MYPORT_1 SERVER_MYPORT_1 > +eval CHANGE MASTER TO master_host='127.0.0.1', master_port=$SERVER_MYPORT_1, master_user='root', master_use_gtid=no; evalp > +--source include/start_slave.inc > + > +connection default; > +--enable_reconnect > +--source include/wait_until_connected_again.inc > + > +--source include/rpl_end.inc > diff --git a/mysql-test/suite/rpl/t/rpl_semi_sync_crash.inc b/mysql-test/suite/rpl/t/rpl_semi_sync_crash.inc > new file mode 100644 > index 00000000000..4289df9155f > --- /dev/null > +++ b/mysql-test/suite/rpl/t/rpl_semi_sync_crash.inc > @@ -0,0 +1,87 @@ > +if ($failover_to_slave) > +{ > + --let $server_to_crash=1 > + --let $server_to_promote=2 > + --let $new_master_port=$SERVER_MYPORT_2 > + --let $client_port=$SERVER_MYPORT_1 > + > + --connect (conn_client,127.0.0.1,root,,test,$SERVER_MYPORT_1,) > +} > +if (!$failover_to_slave) > +{ > + --let $server_to_crash=2 > + --let $server_to_promote=1 > + --let $new_master_port=$SERVER_MYPORT_1 > + --let $client_port=$SERVER_MYPORT_2 > + > + --connect (conn_client,127.0.0.1,root,,test,$SERVER_MYPORT_2,) > +} > + > + > +# Hold insert after write to binlog and before "run_commit_ordered" in engine > + > +SET DEBUG_SYNC= "commit_after_release_LOCK_after_binlog_sync SIGNAL con1_ready WAIT_FOR con1_go"; > +--send_eval $query_to_crash > + > +# complicate recovery with an extra binlog file > +if (!$failover_to_slave) > +{ > + --connect (conn_client_2,127.0.0.1,root,,test,$SERVER_MYPORT_2,) > + # use the same signal with $query_to_crash > + SET DEBUG_SYNC= "now WAIT_FOR con1_ready"; > + SET DEBUG_SYNC= "commit_after_release_LOCK_lock SIGNAL con1_ready WAIT_FOR con2_go"; both signal con1_ready? which one will you be waiting below then? > + --send_eval $query2_to_crash > +} > + > +--connection server_$server_to_crash > +SET DEBUG_SYNC= "now WAIT_FOR con1_ready"; > +--source include/kill_mysqld.inc > +--source include/wait_until_disconnected.inc > + > +--connection server_$server_to_promote > +--error 2003 > +--source include/stop_slave.inc > +SELECT @@GLOBAL.gtid_current_pos; > + > +--let $_expect_file_name=$MYSQLTEST_VARDIR/tmp/mysqld.$server_to_crash.expect wasn't $_expect_file_name already set above? > +--let $restart_parameters=--rpl-semi-sync-slave-enabled=1 > +--let $allow_rpl_inited=1 > +--source include/start_mysqld.inc > + > +--connection server_$server_to_crash > +--enable_reconnect > +--source include/wait_until_connected_again.inc > + > +# Check error log for correct messages. > +let $log_error_= `SELECT @@GLOBAL.log_error`; > +if(!$log_error_) > +{ > + # MySQL Server on windows is started with --console and thus > + # does not know the location of its .err log, use default location > + let $log_error_ = $MYSQLTEST_VARDIR/log/mysqld.$server_to_crash.err; > +} > +--let SEARCH_FILE=$log_error_ > +--let SEARCH_RANGE=-50000 > +--let SEARCH_PATTERN=$log_search_pattern > +--source include/search_pattern_in_file.inc > + > +--disconnect conn_client > + > +# > +# FAIL OVER now to new master > +# > +--connection server_$server_to_promote > +set global rpl_semi_sync_master_enabled = 1; > +set global rpl_semi_sync_master_wait_point=AFTER_SYNC; > + > +--connection server_$server_to_crash > +--let $master_port=$SERVER_MYPORT_2 > +if (`select $server_to_crash = 2`) > +{ > + --let $master_port=$SERVER_MYPORT_1 > +} > +--replace_result $SERVER_MYPORT_1 SERVER_MYPORT_1 $SERVER_MYPORT_2 SERVER_MYPORT_2 > +eval CHANGE MASTER TO master_host='127.0.0.1', master_port=$new_master_port, master_user='root', master_use_gtid=SLAVE_POS; > +set global rpl_semi_sync_slave_enabled = 1; > +set @@global.gtid_slave_pos=@@global.gtid_binlog_pos; > +--source include/start_slave.inc see comments about -50000, $log_error_, wait_until_disconnected, evalp, etc > diff --git a/sql/sql_class.h b/sql/sql_class.h > index 140394fefc1..87fa88d4c89 100644 > --- a/sql/sql_class.h > +++ b/sql/sql_class.h > @@ -4632,7 +4632,7 @@ class THD :public Statement, > LF_PINS *tdc_hash_pins; > LF_PINS *xid_hash_pins; > bool fix_xid_hash_pins(); > - > + bool is_1pc_ro_trans; I don't like how it looks, it's in THD (for every single connection and more) and it's only used in some exotic cases. It's set in two places around ha_commit_one_phase(), when it would be better to set it inside ha_commit_one_phase(), in one place only. But I cannot suggest how to get rid of it yet as I don't understand what it's for, I've asked a question about it below, in Gtid_log_event::Gtid_log_event > /* Members related to temporary tables. */ > public: > /* Opened table states. */ > diff --git a/sql/handler.h b/sql/handler.h > index 2a346e8d9d1..bd39f46bf1f 100644 > --- a/sql/handler.h > +++ b/sql/handler.h > @@ -873,6 +873,14 @@ typedef struct xid_t XID; > /* The 'buf' has to have space for at least SQL_XIDSIZE bytes. */ > uint get_sql_xid(XID *xid, char *buf); > > +/* struct for heuristic binlog truncate recovery */ it's not "heuristic binlog truncate recovery" anymore, please don't call it that, it's confusing > +struct xid_recovery_member > +{ > + my_xid xid; > + uint in_engine_prepare; // number of engines that have xid prepared > + bool decided_to_commit; > +}; > + > /* for recover() handlerton call */ > #define MIN_XID_LIST_SIZE 128 > #define MAX_XID_LIST_SIZE (1024*128) > diff --git a/sql/slave.cc b/sql/slave.cc > index 28e08e32346..9b3f73e5341 100644 > --- a/sql/slave.cc > +++ b/sql/slave.cc > @@ -6944,7 +6945,9 @@ static int queue_event(Master_info* mi,const char* buf, ulong event_len) > } > else > if ((s_id == global_system_variables.server_id && > - !mi->rli.replicate_same_server_id) || > + (!mi->rli.replicate_same_server_id && > + !(semisync_recovery= (rpl_semi_sync_slave_enabled && > + mi->using_gtid != Master_info::USE_GTID_NO)))) || How can "semisync recovery" code path reach queue_event()? > event_that_should_be_ignored(buf) || > /* > the following conjunction deals with IGNORE_SERVER_IDS, if set > diff --git a/sql/log_event.h b/sql/log_event.h > index 8a342cb5cd3..1588ab85104 100644 > --- a/sql/log_event.h > +++ b/sql/log_event.h > @@ -482,6 +482,16 @@ class String; > */ > #define LOG_EVENT_IGNORABLE_F 0x80 > > +/** > + @def LOG_EVENT_ACCEPT_OWN_F > + > + Flag sets by the gtid-mode connected semisync slave for > + the same server_id ("own") events which the slave must not have > + in its state. Typically such events were never committed by > + their originator (this server) and discared at its crash recovery sorry, I failed to understand that > +*/ > +#define LOG_EVENT_ACCEPT_OWN_F 0x4000 > + > /** > @def LOG_EVENT_SKIP_REPLICATION_F > > diff --git a/sql/log_event.cc b/sql/log_event.cc > index e344fc8894f..a593db10d16 100644 > --- a/sql/log_event.cc > +++ b/sql/log_event.cc > @@ -7918,6 +7924,7 @@ Gtid_log_event::Gtid_log_event(const char *buf, uint event_len, > domain_id= uint4korr(buf); > buf+= 4; > flags2= *buf; > + ++buf; I'd written flags2= *buf++; :) > if (flags2 & FL_GROUP_COMMIT_ID) > { > if (event_len < (uint)header_size + GTID_HEADER_LEN + 2) > @@ -7925,9 +7932,31 @@ Gtid_log_event::Gtid_log_event(const char *buf, uint event_len, > seq_no= 0; // So is_valid() returns false > return; > } > - ++buf; > commit_id= uint8korr(buf); > + buf+= 8; > + } > + /* the extra flags check and actions */ > + if (static_cast<uint>(buf - buf_0) < event_len) > + { > + flags_extra= *buf; > + ++buf; > + /* extra flags presence is identifed by non-zero byte value at this point */ "extra engines" > + if (flags_extra & FL_EXTRA_MULTI_ENGINE) > + { safety, check that buf+4-buf_0 <= event_len > + extra_engines= uint4korr(buf); > + buf += 4; four bytes for the number of engines participating in a transaction is very generous... and optimistic. I'd say one byte is more than enough. > + > + DBUG_ASSERT(extra_engines > 0); > + } > } > + /* > + the strict '<' part of the assert corresponds to extra zero-padded > + trailing bytes, > + */ > + DBUG_ASSERT(static_cast<uint>(buf - buf_0) <= event_len); > + /* and the last of them is tested. */ > + DBUG_ASSERT(static_cast<uint>(buf - buf_0) == event_len || > + buf_0[event_len - 1] == 0); > } > > > @@ -7958,6 +7990,22 @@ Gtid_log_event::Gtid_log_event(THD *thd_arg, uint64 seq_no_arg, > /* Preserve any DDL or WAITED flag in the slave's binlog. */ > if (thd_arg->rgi_slave) > flags2|= (thd_arg->rgi_slave->gtid_ev_flags2 & (FL_DDL|FL_WAITED)); > + /* count non-zero extra recoverable engines; total = extra + 1 */ > + if (is_transactional) > + { > + if (has_xid) > + { > + extra_engines= > + max<uint>(1, ha_count_rw(thd, thd_arg->in_multi_stmt_transaction_mode())) 1. a bit confusing to use both thd and thd_arg in the same line 2. max() is strange. it means the result is 1, when ha_count_rw() is 0. can ha_count_rw() here even be 0? 3. also it's somewhat an unfounded assumption that only r/w engines will have the transaction prepared. But we can make it a fact, if we won't prepare r/o engines at all (in ha_prepare). > + - 1; > + } > + else if (unlikely(thd_arg->is_1pc_ro_trans)) > + { > + extra_engines= UINT_MAX; // neither extra nor base engine why is that? > + } > + if (extra_engines > 0) > + flags_extra|= FL_EXTRA_MULTI_ENGINE; > + } > } > > > diff --git a/sql/handler.cc b/sql/handler.cc > index 6792e80b8fe..247ab7267bf 100644 > --- a/sql/handler.cc > +++ b/sql/handler.cc > @@ -1245,6 +1245,24 @@ int ha_prepare(THD *thd) > DBUG_RETURN(error); > } > > +/* > + Returns counted number of > + read-write recoverable transaction participants. > +*/ > +uint ha_count_rw(THD *thd, bool all) the name doesn't match the implementation, please, rename > +{ > + unsigned rw_ha_count= 0; > + THD_TRANS *trans=all ? &thd->transaction.all : &thd->transaction.stmt; > + > + for (Ha_trx_info * ha_info= trans->ha_list; ha_info; > + ha_info= ha_info->next()) > + { > + if (ha_info->is_trx_read_write() && ha_info->ht()->recover) > + ++rw_ha_count; > + } > + return rw_ha_count; > +} > + > /** > Check if we can skip the two-phase commit. > > @@ -1960,8 +1982,122 @@ struct xarecover_st > XID *list; > HASH *commit_list; > bool dry_run; > + MEM_ROOT *mem_root; > + bool error; > }; > > +/** > + Inserts a new hash member. > + > + returns a successfully created and inserted @c xid_recovery_member > + into hash @c hash_arg, > + or NULL. > +*/ > +static xid_recovery_member* > +xid_member_insert(HASH *hash_arg, my_xid xid_arg, MEM_ROOT *ptr_mem_root) > +{ > + xid_recovery_member *member= (xid_recovery_member*) > + alloc_root(ptr_mem_root, sizeof(xid_recovery_member)); > + if (!member) > + return NULL; > + > + member->xid= xid_arg; > + member->in_engine_prepare= 1; > + member->decided_to_commit= false; > + > + return my_hash_insert(hash_arg, (uchar*) member) ? NULL : member; > +} > + > +/* > + Inserts a new or updates an existing hash member to increment > + the member's prepare counter. > + > + returns false on success, > + true otherwise. > +*/ > +static bool xid_member_replace(HASH *hash_arg, my_xid xid_arg, > + MEM_ROOT *ptr_mem_root) > +{ > + xid_recovery_member* member; > + if ((member= (xid_recovery_member *) > + my_hash_search(hash_arg, (uchar *)& xid_arg, sizeof(xid_arg)))) > + member->in_engine_prepare++; > + else > + member= xid_member_insert(hash_arg, xid_arg, ptr_mem_root); > + > + return member == NULL; > +} > + > +/* > + Hash iterate function to complete with commit or rollback as decided > + (typically at binlog recovery processing) in member->in_engine_prepare. > +*/ > +static my_bool xarecover_do_commit_or_rollback(void *member_arg, > + void *hton_arg) > +{ > + xid_recovery_member *member= (xid_recovery_member*) member_arg; > + handlerton *hton= (handlerton*) hton_arg; > + xid_t x; > + my_bool rc; > + > + x.set(member->xid); > + rc= member->decided_to_commit ? hton->commit_by_xid(hton, &x) : > + hton->rollback_by_xid(hton, &x); > + > + DBUG_ASSERT(rc || member->in_engine_prepare > 0); > + > + if (!rc) > + { I don't think you can trust rc on that. if it's non-zero, it's an error all right. but it's a bit of a stretch to presume that nonexisting xid is always an error. also commit_by_xid returns int, not my_bool. > + member->in_engine_prepare--; > + if (global_system_variables.log_warnings > 2) > + sql_print_warning("%s transaction with xid %llu", > + member->decided_to_commit ? > + "Committed" : "Rolled back", (ulonglong) member->xid); > + } > + > + return false; > +} > + > +static my_bool xarecover_do_count_in_prepare(void *member_arg, > + void *ptr_count) > +{ > + xid_recovery_member *member= (xid_recovery_member*) member_arg; > + if (member->in_engine_prepare) > + { > + *(uint*) ptr_count += member->in_engine_prepare; This is a rather weird semantics. it's kind of a number of transactions times number of engines. if one transaction wasn't committed in two engines and another transaction wasn't rolled back in one engine, the counter will be 3. how are you going to explain this to users? > + if (global_system_variables.log_warnings > 2) > + sql_print_warning("Found prepared transaction with xid %llu", > + (ulonglong) member->xid); > + } > + > + return false; > +} > + > +static my_bool xarecover_binlog_truncate_handlerton(THD *unused, > + plugin_ref plugin, > + void *arg) > +{ > + handlerton *hton= plugin_hton(plugin); > + > + if (hton->state == SHOW_OPTION_YES && hton->recover) > + { > + my_hash_iterate((HASH*) arg, xarecover_do_commit_or_rollback, hton); Why is the function called xarecover_binlog_truncate_handlerton if it doesn't truncate anything? > + } > + > + return FALSE; > +} > + > +uint ha_recover_complete(HASH *commit_list) > +{ > + uint count= 0; > + > + plugin_foreach(NULL, xarecover_binlog_truncate_handlerton, > + MYSQL_STORAGE_ENGINE_PLUGIN, commit_list); > + my_hash_iterate(commit_list, xarecover_do_count_in_prepare, &count); > + > + return count; > +} > + > static my_bool xarecover_handlerton(THD *unused, plugin_ref plugin, > void *arg) > { > @@ -1969,6 +2105,9 @@ static my_bool xarecover_handlerton(THD *unused, plugin_ref plugin, > struct xarecover_st *info= (struct xarecover_st *) arg; > int got; > > + if (info->error) > + return TRUE; plugin_foreach() aborts as soon as the callback returns true. so, remove info->error, and return TRUE from xarecover_handlerton on error instead. > + > if (hton->state == SHOW_OPTION_YES && hton->recover) > { > while ((got= hton->recover(hton, info->list, info->len)) > 0 ) > @@ -1988,7 +2127,7 @@ static my_bool xarecover_handlerton(THD *unused, plugin_ref plugin, > } > #endif /* WITH_WSREP */ > > - for (int i=0; i < got; i ++) > + for (int i=0; i < got && !info->error; i ++) eh? how can info->error ever be true above? you break out of the loop when setting info->error= true, > { > my_xid x= IF_WSREP(WSREP_ON && wsrep_is_wsrep_xid(&info->list[i]) ? > wsrep_xid_seqno(info->list[i]) : > @@ -2013,7 +2152,20 @@ static my_bool xarecover_handlerton(THD *unused, plugin_ref plugin, > info->found_my_xids++; > continue; > } > - // recovery mode > + > + /* > + Regular and semisync slave server recovery only collects > + xids to make decisions on them later by the caller. > + */ > + if (info->mem_root) > + { > + if (xid_member_replace(info->commit_list, x, info->mem_root)) > + { > + info->error= true; > + sql_print_error("Error in memory allocation at xarecover_handlerton"); > + break; > + } > + } else add into the else branch something like /* this branch is only for the legacy TC_LOG_MMAP */ compile_time_assert(sizeof(TC_LOG_MMAP) > 1 ); > if (IF_WSREP((wsrep_emulate_bin_log && > wsrep_is_wsrep_xid(info->list + i) && > x <= wsrep_limit), false) || > diff --git a/sql/log.cc b/sql/log.cc > index 8073f09ab88..a9808ed8823 100644 > --- a/sql/log.cc > +++ b/sql/log.cc > @@ -7977,6 +7982,7 @@ MYSQL_BIN_LOG::trx_group_commit_leader(group_commit_entry *leader) > #endif > } > > + DEBUG_SYNC(leader->thd, "commit_before_update_end_pos"); this isn't used in any tests > /* > update binlog_end_pos so it can be read by dump thread > Note: must be _after_ the RUN_HOOK(after_flush) or else > @@ -9609,6 +9618,180 @@ int TC_LOG::using_heuristic_recover() > /****** transaction coordinator log for 2pc - binlog() based solution ******/ > #define TC_LOG_BINLOG MYSQL_BIN_LOG > > +/** > + Truncates the current binlog to specified position. Removes the rest of binlogs > + which are present after this binlog file. > + > + @param truncate_file Holds the binlog name to be truncated > + @param truncate_pos Position within binlog from where it needs to > + truncated. > + > + @retval true ok > + @retval false error > + > +*/ > +bool MYSQL_BIN_LOG::truncate_and_remove_binlogs(const char *file_name, > + my_off_t pos, > + rpl_gtid *ptr_gtid, > + enum_binlog_checksum_alg cs_alg) > +{ > + int error= 0; > +#ifdef HAVE_REPLICATION > + LOG_INFO log_info; > + THD *thd= current_thd; > + my_off_t index_file_offset= 0; > + File file= -1; > + IO_CACHE cache; > + MY_STAT s; > + > + if ((error= find_log_pos(&log_info, file_name, 1))) > + { > + sql_print_error("Failed to locate binary log file:%s." > + "Error:%d", file_name, error); > + goto end; > + } > + > + while (!(error= find_next_log(&log_info, 1))) > + { > + if (!index_file_offset) > + { > + index_file_offset= log_info.index_file_start_offset; > + if ((error= open_purge_index_file(TRUE))) > + { > + sql_print_error("Failed to open purge index " > + "file:%s. Error:%d", purge_index_file_name, error); > + goto end; > + } > + } > + if ((error= register_purge_index_entry(log_info.log_file_name))) > + { > + sql_print_error("Failed to copy %s to purge index" > + " file. Error:%d", log_info.log_file_name, error); > + goto end; > + } > + } > + > + if (error != LOG_INFO_EOF) > + { > + sql_print_error("Failed to find the next binlog to " > + "add to purge index register. Error:%d", error); > + goto end; > + } > + > + if (is_inited_purge_index_file()) > + { > + if (!index_file_offset) > + index_file_offset= log_info.index_file_start_offset; > + > + if ((error= sync_purge_index_file())) > + { > + sql_print_error("Failed to flush purge index " > + "file. Error:%d", error); > + goto end; > + } > + > + // Trim index file > + if ((error= > + mysql_file_chsize(index_file.file, index_file_offset, '\n', > + MYF(MY_WME))) || > + (error= > + mysql_file_sync(index_file.file, MYF(MY_WME|MY_SYNC_FILESIZE)))) > + { > + sql_print_error("Failed to trim binlog index " > + "file:%s to offset:%llu. Error:%d", index_file_name, > + index_file_offset, error); > + goto end; > + } > + > + /* Reset data in old index cache */ > + if ((error= reinit_io_cache(&index_file, READ_CACHE, (my_off_t) 0, 0, 1))) > + { > + sql_print_error("Failed to reinit binlog index " > + "file. Error:%d", error); > + goto end; > + } > + > + /* Read each entry from purge_index_file and delete the file. */ > + if ((error= purge_index_entry(thd, NULL, TRUE))) > + { > + sql_print_error("Failed to process registered " > + "files that would be purged."); > + goto end; > + } > + } > + > + DBUG_ASSERT(pos); > + > + if ((file= mysql_file_open(key_file_binlog, file_name, > + O_RDWR | O_BINARY, MYF(MY_WME))) < 0) > + { > + error= 1; > + sql_print_error("Failed to open binlog file:%s for " > + "truncation.", file_name); > + goto end; > + } > + my_stat(file_name, &s, MYF(0)); > + > + /* Change binlog file size to truncate_pos */ > + if ((error= > + mysql_file_chsize(file, pos, 0, MYF(MY_WME))) || > + (error= mysql_file_sync(file, MYF(MY_WME|MY_SYNC_FILESIZE)))) > + { > + sql_print_error("Failed to trim the " > + "binlog file:%s to size:%llu. Error:%d", > + file_name, pos, error); > + goto end; > + } > + else > + { > + char buf[21]; > + > + longlong10_to_str(ptr_gtid->seq_no, buf, 10); > + sql_print_information("Successfully truncated binlog file:%s " > + "to pos:%llu to remove transactions starting from " > + "GTID %u-%u-%s", file_name, pos, > + ptr_gtid->domain_id, ptr_gtid->server_id, buf); > + } > + if (!(error= init_io_cache(&cache, file, IO_SIZE, WRITE_CACHE, > + (my_off_t) pos, 0, MYF(MY_WME|MY_NABP)))) > + { > + /* > + Write Stop_log_event to ensure clean end point for the binary log being > + truncated. > + */ I'm not sure about it. The less you touch the corrupted binlog the better. simply truncating it would be enough, wouldn't it? > + Stop_log_event se; > + se.checksum_alg= cs_alg; > + if ((error= write_event(&se, NULL, &cache))) > + { > + sql_print_error("Failed to write stop event to " > + "binary log. Errno:%d", > + (cache.error == -1) ? my_errno : error); > + goto end; > + } > + clear_inuse_flag_when_closing(cache.file); > + if ((error= flush_io_cache(&cache)) || > + (error= mysql_file_sync(file, MYF(MY_WME|MY_SYNC_FILESIZE)))) > + { > + sql_print_error("Faild to write stop event to " > + "binary log. Errno:%d", > + (cache.error == -1) ? my_errno : error); > + } > + } > + else > + sql_print_error("Failed to initialize binary log " > + "cache for writing stop event. Errno:%d", > + (cache.error == -1) ? my_errno : error); 1. I don't see why you need to sync after every operation. 2. you should clear inuse flag just before closing, otherwise flush_io_cache can overwrite it again if it'll happen to have it in the cache. > + > +end: > + if (file >= 0) > + { > + end_io_cache(&cache); > + mysql_file_close(file, MYF(MY_WME)); > + } > + error= error || close_purge_index_file(); > +#endif > + return error > 0; > +} > int TC_LOG_BINLOG::open(const char *opt_name) > { > int error= 1; > @@ -10040,26 +10223,515 @@ start_binlog_background_thread() > > return 0; > } > +#ifdef HAVE_REPLICATION > +class Recovery_context > +{ > +public: > + my_off_t prev_event_pos; > + rpl_gtid last_gtid; > + bool last_gtid_standalone; > + bool last_gtid_valid; > + bool last_gtid_no2pc; // true when the group does not end with Xid event > + uint last_gtid_engines; > + std::pair<uint, my_off_t> last_gtid_coord; may be a comment at the end // <binlog id, binlog offset> ? > + /* > + When true, it's semisync slave recovery mode > + rolls back transactions in doubt and wipes them off from binlog. > + The rest of declarations deal with this type of recovery. > + */ > + bool do_truncate; > + rpl_gtid binlog_unsafe_gtid, truncate_gtid; > + char binlog_truncate_file_name[FN_REFLEN] ; > + char binlog_unsafe_file_name[FN_REFLEN] ; > + /* > + When do_truncate is true, the truncate position may not be > + found in one round when recovered transactions are multi-engine > + or just on different engines. > + In the single recoverable engine case `truncate_reset_done` and > + therefore `truncate_validated` remain `false' when the last > + binlog is the binlog-checkpoint one. > + The meaning of `truncate_reset_done` is according to the following example: > + Let round = 1, Binlog contains the sequence of replication event groups: > + [g1, G2, g3] > + where `G` (in capital) stands for committed, `g` for prepared. > + g1 is first set as truncation candidate, then G2 reset it to indicate > + the actual truncation is behind (to the right of) it. > + `truncate_validated` is set to true when `binlog_truncate_pos` (as of g3) > + won't change. > + Observe last_gtid_valid is affected, so in the above example g1 > + would have to be discarded from the 1st round binlog state estimate which > + is handled by the 2nd round (see below). I failed to understand that either :( > + */ > + bool truncate_validated; // trued when the truncate position settled > + bool truncate_reset_done; // trued when the position is to reevaluate Nor that :( > + /* Flags the fact of truncate position estimation is done the 1st round */ > + bool truncate_set_in_1st; > + /* > + Monotonically indexes binlog files in the recovery list. > + When the list is "likely" singleton the value is UINT_MAX. > + Otherwise enumeration starts with zero for the first file, increments > + by one for any next file except for the last file in the list, which > + is also the initial binlog file for recovery, > + that is enumberated with UINT_MAX. > + */ > + uint id_binlog; > + enum_binlog_checksum_alg cs_alg; // for Stop_event with do_truncate rename to checksum_alg, please. "charset algorithm" looks very puzzling and "cs" almost everywhere means CHARSET_INFO. > + bool single_binlog; > + std::pair<uint, my_off_t> binlog_truncate_coord; > + std::pair<uint, my_off_t> binlog_unsafe_coord; > + > + Recovery_context(); > + bool decide_or_assess(xid_recovery_member *member, int round, > + Format_description_log_event *fdle, > + LOG_INFO *linfo, my_off_t pos); > + void process_gtid(int round, Gtid_log_event *gev, LOG_INFO *linfo); > + int next_binlog_or_round(int& round, > + const char *last_log_name, > + const char *binlog_checkpoint_name, > + LOG_INFO *linfo, MYSQL_BIN_LOG *log); > + bool is_safe() what do you mean "safe" ? > + { > + return !do_truncate ? true : > + (truncate_gtid.seq_no == 0 || // no truncate > + binlog_unsafe_coord < binlog_truncate_coord); // or unsafe is earlier > + } > + bool complete(MYSQL_BIN_LOG *log, HASH &xids); > + void update_binlog_unsafe_coord_if_needed(LOG_INFO *linfo); > + void reset_truncate_coord(my_off_t pos); > + void set_truncate_coord(LOG_INFO *linfo, int round, > + enum_binlog_checksum_alg fd_cs_alg); > +}; > comments about what every method does? > +bool Recovery_context::complete(MYSQL_BIN_LOG *log, HASH &xids) > +{ > + if (!do_truncate || is_safe()) > + { > + uint count_in_prepare= ha_recover_complete(&xids); > + if (count_in_prepare > 0 && global_system_variables.log_warnings > 2) > + { > + sql_print_warning("Could not complete %u number of transactions in " > + "engines.", count_in_prepare); > + return false; // there's later dry run ha_recover() to error out > + } > + } > + > + /* Truncation is not done when there's no transaction to roll back */ > + if (do_truncate && truncate_gtid.seq_no > 0) > + { > + if (is_safe()) > + { > + if (log->truncate_and_remove_binlogs(binlog_truncate_file_name, > + binlog_truncate_coord.second, > + &truncate_gtid, > + cs_alg)) > + { > + sql_print_error("Failed to trim the binary log to " > + "file:%s pos:%llu.", binlog_truncate_file_name, > + binlog_truncate_coord.second); > + return true; > + } > + } > + else > + { > + sql_print_error("Cannot trim the binary log to file:%s " > + "pos:%llu as unsafe statement " > + "is found at file:%s pos:%llu which is " > + "beyond the truncation position " > + "(the farest in binlog order only is reported); " "farest" doesn't look like a word > + "all transactions in doubt are left intact. ", > + binlog_truncate_file_name, binlog_truncate_coord.second, > + binlog_unsafe_file_name, binlog_unsafe_coord.second); > + return true; will this abort server startup? > + } > + } > + > + return false; > +} > + > +Recovery_context::Recovery_context() : > + prev_event_pos(0), > + last_gtid_standalone(false), last_gtid_valid(false), last_gtid_no2pc(false), > + last_gtid_engines(0), > + do_truncate(rpl_semi_sync_slave_enabled), > + truncate_validated(false), truncate_reset_done(false), > + truncate_set_in_1st(false), id_binlog(UINT_MAX), > + cs_alg(BINLOG_CHECKSUM_ALG_UNDEF), single_binlog(false) > +{ > + last_gtid_coord= std::pair<uint,my_off_t>(0,0); > + binlog_truncate_coord= std::pair<uint,my_off_t>(0,0); > + binlog_unsafe_coord= std::pair<uint,my_off_t>(0,0); > + binlog_truncate_file_name[0]= 0; > + binlog_unsafe_file_name [0]= 0; > + binlog_unsafe_gtid= truncate_gtid= rpl_gtid(); > +} > + > +/** > + Is called when a committed or to-be-committed transaction is detected. > + @c truncate_gtid is set to "nil" with its @c rpl_gtid::seq_no := 0. > + @c truncate_reset_done remembers the fact of that has been done at least > + once in the current round; > + @c binlog_truncate_coord is "suggested" to a next group start to indicate > + the actual settled value must be at most as the last suggested one. > +*/ > +void Recovery_context::reset_truncate_coord(my_off_t pos) > +{ > + DBUG_ASSERT(binlog_truncate_coord.second == 0 || > + last_gtid_coord >= binlog_truncate_coord || > + truncate_set_in_1st); > + > + binlog_truncate_coord= std::pair<uint,my_off_t>(id_binlog, pos); > + truncate_gtid= rpl_gtid(); > + truncate_reset_done= true; > +} > + > + > +/* > + Sets binlog_truncate_pos to the value of the current transaction's gtid. > + In multi-engine case that might be just an assessment to be exacted > + in the current round and confirmed in a next one. > +*/ > +void Recovery_context::set_truncate_coord(LOG_INFO *linfo, int round, > + enum_binlog_checksum_alg fd_cs_alg) > +{ > + binlog_truncate_coord= last_gtid_coord; > + strmake_buf(binlog_truncate_file_name, linfo->log_file_name); > + > + truncate_gtid= last_gtid; > + cs_alg= fd_cs_alg; > + truncate_set_in_1st= (round == 1); > +} > + > +bool Recovery_context::decide_or_assess(xid_recovery_member *member, int round, > + Format_description_log_event *fdle, > + LOG_INFO *linfo, my_off_t pos) > +{ > + if (member) > + { > + DBUG_EXECUTE_IF("binlog_truncate_partial_commit", > + if (last_gtid_engines == 2) > + { > + DBUG_ASSERT(member->in_engine_prepare > 0); > + member->in_engine_prepare= 1; > + }); > + /* > + xid in doubt are resolved as follows: > + in_engine_prepare is compared agaist binlogged info to > + yield the commit-or-rollback decision in the normal case. > + In the semisync-slave recovery the decision may be > + approximate to change in later rounds. > + */ > + if (member->in_engine_prepare > last_gtid_engines) > + { > + char buf[21]; > + longlong10_to_str(last_gtid.seq_no, buf, 10); > + sql_print_error("Error to recovery multi-engine transaction: " > + "the number of engines prepared %u exceeds the " > + "respective number %u in its GTID %u-%u-%s " > + "located at file:%s pos:%llu", > + member->in_engine_prepare, last_gtid_engines, > + last_gtid.domain_id, last_gtid.server_id, buf, > + linfo->log_file_name, last_gtid_coord.second); > + return true; > + } > + else if (member->in_engine_prepare < last_gtid_engines) > + { > + DBUG_EXECUTE_IF("binlog_truncate_partial_commit", > + member->in_engine_prepare= 2;); > + DBUG_ASSERT(member->in_engine_prepare > 0); > + /* > + This is an "unlikely" branch of two or more engines in transaction > + that is partially committed, so to complete. > + */ > + member->decided_to_commit= true; > + if (do_truncate) > + { > + /* > + A validated truncate pos may not change later. > + Any "unlikely" (two or more engines in transaction) reset > + to not-validated yet position ensues correcting early > + estimates in the following round(s). > + */ > + if (!truncate_validated) > + reset_truncate_coord(pos); > + } > + } > + else // member->in_engine_prepare == last_gtid_engines > + { > + if (!do_truncate) // "normal" recovery > + { > + member->decided_to_commit= true; > + } > + else > + { > + /* > + The first time or further estimate the truncate position > + until validation. Already set not validated yet postion > + may change only through previous reset > + unless this xid in doubt is the first in the 2nd round. > + */ > + if (!truncate_validated) > + { > + DBUG_ASSERT(round <= 2); > + /* > + Either truncate was not set or was reset, else > + it gets incremented, otherwise it may only set to an earlier > + offset only at the turn out of the 1st round. sorry, I cannot parse that :( > + */ > + DBUG_ASSERT(truncate_gtid.seq_no == 0 || > + last_gtid_coord >= binlog_truncate_coord || > + (round == 2 && truncate_set_in_1st)); > + > + last_gtid_valid= false; // may still (see gtid) flip later > + if (truncate_gtid.seq_no == 0 /* was reset or never set */ || > + (truncate_set_in_1st && round == 2 /* reevaluted at round turn */)) > + set_truncate_coord(linfo, round, fdle->checksum_alg); > + > + DBUG_ASSERT(member->decided_to_commit == false); // may flip later > + } > + else > + { > + DBUG_ASSERT(!truncate_reset_done); // the position was settled > + /* > + Correct earlier decisions of 1st and/or 2nd round to > + rollback and invalidate last_gtid in binlog state. > + */ > + if (!(member->decided_to_commit= > + last_gtid_coord < binlog_truncate_coord)) > + { > + last_gtid_valid= false; // settled > + if (truncate_gtid.seq_no == 0) > + truncate_gtid= last_gtid; > + } > + } > + } > + } > + } > + else if (do_truncate) // "0" < last_gtid_engines > + { > + /* > + Similar to the multi-engine partial commit of "then" branch above. > + The 2nd condition economizes away an extra (3rd) round in I don't think "economizes" is a word > + expected cases of first xid:s in the binlog checkpoint file > + are actually of committed transactions. So fully committed > + xid sequence is passed in this branch without any action. > + */ > + if (!truncate_validated) > + reset_truncate_coord(pos); > + } > + > + return false; > +} > + > +/* > + Is invoked when a standalone or non-2pc group is detected. > + Both are unsafe to truncate in the semisync-slave recovery so > + the maximum unsafe coordinate may be updated. > + In the non-2pc group case though, *exeptionally*, > + the no-engine group is considered safe, to be invalidated > + to not contribute to binlog state. > +*/ > +void Recovery_context::update_binlog_unsafe_coord_if_needed(LOG_INFO *linfo) > +{ > + if (!do_truncate) > + return; > + > + if (truncate_gtid.seq_no > 0 && // g1,U2, *not* G1,U2 > + last_gtid_coord > binlog_truncate_coord) > + { > + DBUG_ASSERT(binlog_truncate_coord.second > 0); > + /* > + Potentially unsafe when the truncate coordinate is not determined, > + just detected as unsafe when behind the latter. > + */ > + if (last_gtid_engines == 0) > + { > + last_gtid_valid= false; > + } > + else > + { > + binlog_unsafe_gtid= last_gtid; > + binlog_unsafe_coord= last_gtid_coord; > + strmake_buf(binlog_unsafe_file_name, linfo->log_file_name); > + } > + } > +} > + > +/* > + Assigns last_gtid and assesses the maximum (in the binlog offset term) > + unsafe gtid (group of events). > +*/ > +void Recovery_context::process_gtid(int round, Gtid_log_event *gev, > + LOG_INFO *linfo) > +{ > + last_gtid.domain_id= gev->domain_id; > + last_gtid.server_id= gev->server_id; > + last_gtid.seq_no= gev->seq_no; > + last_gtid_engines= gev->extra_engines != UINT_MAX ? > + gev->extra_engines + 1 : 0; > + last_gtid_coord= std::pair<uint,my_off_t>(id_binlog, prev_event_pos); > + if (round == 1 || do_truncate) > + { > + DBUG_ASSERT(!last_gtid_valid); > + > + last_gtid_no2pc= false; > + last_gtid_standalone= > + (gev->flags2 & Gtid_log_event::FL_STANDALONE) ? true : false; > + if (do_truncate && last_gtid_standalone) > + update_binlog_unsafe_coord_if_needed(linfo); > + /* Update the binlog state with any 'valid' GTID logged after Gtid_list. */ > + last_gtid_valid= true; // may flip at Xid when falls to truncate > + } > +} > + > +/* > + At the end of processing of the current binlog compute next action. > + When round increments in the semisync-slave recovery > + truncate_validated, truncate_reset_done > + gets reset/set for the next round, > + as well as id_binlog gets reset either to zero or UINT_MAX > + when recovery deals with the only binlog file. > + > + @param[in,out] round the current round that may increment here > + @param last_log_name the recovery starting binlog file > + @param binlog_checkpoint_name > + binlog checkpoint file > + @param linfo binlog file list struct for next file > + @param log pointer to mysql_bin_log instance > + @return 0 when rounds continue, maybe the current one remains > + 1 when all rounds are done > + -1 on error > +*/ > +int Recovery_context::next_binlog_or_round(int& round, > + const char *last_log_name, > + const char *binlog_checkpoint_name, > + LOG_INFO *linfo, > + MYSQL_BIN_LOG *log) > +{ > + if (!strcmp(linfo->log_file_name, last_log_name)) > + { > + /* Either exit the loop now, or increment round */ > + DBUG_ASSERT(round <= 3); > + > + if ((round <= 2 && likely(!truncate_reset_done)) || round == 3) > + { > + // Exit now as the 1st round ends up with the binlog-checkpoint file > + // is the same as the initial binlog file, so already parsed; or > + // the 2nd round has not made own truncate_reset_done; or > + // at the end of the 3rd "truncate" round. > + DBUG_ASSERT(do_truncate || round == 1); > + // The 3rd round is done only when the 2nd trued truncate_reset_done > + // and consequently truncate_validated. > + // No instances of truncate_reset_done in first 2 rounds allows > + // for exiting now. our (unwritten) coding style says that multi-line comments should use /*...*/ > + DBUG_ASSERT(round < 3 || truncate_validated); > + > + return true; > + } > + else > + { > + DBUG_ASSERT(do_truncate); > + /* > + the last binlog file, having truncate_reset_done to indicate > + needed correction to member->decided_to_commit to reflect > + changed binlog_truncate_pos. > + */ > + truncate_reset_done= false; > + truncate_validated= true; > + rpl_global_gtid_binlog_state.reset_nolock(); > + > + if (round > 1 && > + log->find_log_pos(linfo, binlog_checkpoint_name, 1)) > + { > + sql_print_error("Binlog file '%s' not found in binlog index, needed " > + "for recovery. Aborting.", binlog_checkpoint_name); > + return -1; > + } > + id_binlog= (single_binlog= !strcmp(linfo->log_file_name, last_log_name)) ? > + UINT_MAX : 0; > + round++; > + } > + } > + else if (round == 1) > + { > + if (do_truncate) > + { > + truncate_validated= truncate_reset_done; > + rpl_global_gtid_binlog_state.reset_nolock(); > + truncate_reset_done= false; > + id_binlog= 0; > + > + DBUG_ASSERT(!single_binlog); > + } > + round++; > + } > + else > + { > + /* > + NOTE: reading other binlog's FD is necessary for finding out > + the checksum status of the respective binlog file. > + Round remains in this branch. > + */ > + if (log->find_next_log(linfo, 1)) > + { > + sql_print_error("Error reading binlog files during recovery. " > + "Aborting."); > + return -1; > + } > + if (!strcmp(linfo->log_file_name, last_log_name)) > + { > + if (do_truncate) > + { > + DBUG_ASSERT(!single_binlog); > + id_binlog= UINT_MAX; > + } > + else > + return 1; // regular recovery exit > + } > + else if (do_truncate) > + id_binlog++; > + > + DBUG_ASSERT(id_binlog <= UINT_MAX); // the assert is "practical" > + } > + > + DBUG_ASSERT(!single_binlog || > + !strcmp(linfo->log_file_name, binlog_checkpoint_name)); > + > + return 0; > +} > +#endif > > int TC_LOG_BINLOG::recover(LOG_INFO *linfo, const char *last_log_name, > IO_CACHE *first_log, > - Format_description_log_event *fdle, bool do_xa) > + Format_description_log_event *fdle_arg, bool do_xa) > { > Log_event *ev= NULL; > HASH xids; > MEM_ROOT mem_root; > char binlog_checkpoint_name[FN_REFLEN]; > bool binlog_checkpoint_found; > - bool first_round; > IO_CACHE log; > File file= -1; > const char *errmsg; > + Format_description_log_event *fdle= fdle_arg; > #ifdef HAVE_REPLICATION > - rpl_gtid last_gtid; > - bool last_gtid_standalone= false; > - bool last_gtid_valid= false; > + Recovery_context ctx; > #endif > + /* > + The for-loop variable is updated by the following rule set: > + Initially set to 1. > + After the initial binlog file is processed to identify > + the Binlog-checkpoint file it is incremented when the latter file > + is different from the initial one. Otherwise the only log has been > + fully parsed and the loop exits. > + The 2nd starts with the Binlog-checkpoint file and ends when the initial > + binlog file is reached. It is excluded from yet another processing > + in the "normal" non-semisync-slave configuration, and then the loop is > + done at this point. Otherwise in the semisync slave case it may be parsed > + over again. The 2nd round may turn to a third in "unlikely" condition of the > + semisync-slave is being recovered having multi- or different engine > + transactions in doubt. > + */ I don't understand that. Why semisync-slave case is special? > + int round; > > if (! fdle->is_valid() || > (do_xa && my_hash_init(&xids, &my_charset_bin, TC_LOG_PAGE_SIZE/3, 0, > @@ -10072,39 +10744,63 @@ int TC_LOG_BINLOG::recover(LOG_INFO *linfo, const char *last_log_name, > > fdle->flags&= ~LOG_EVENT_BINLOG_IN_USE_F; // abort on the first error > > + /* finds xids when root is not NULL */ > + if (do_xa && ha_recover(&xids, &mem_root)) > + goto err1; > + > /* > Scan the binlog for XIDs that need to be committed if still in the > prepared stage. > > Start with the latest binlog file, then continue with any other binlog > files if the last found binlog checkpoint indicates it is needed. > + > + In case the semisync slave recovery the 2nd round may include > + the initial file as well as cause a 3rd round when transactions > + with multiple engines are discovered. > + Additionally to find and decide on transactions in doubt that > + the semisync slave may need to roll back, the binlog can be truncated > + in the end to reflect the rolled back decisions. > */ > > binlog_checkpoint_found= false; > - first_round= true; > - for (;;) > + for (round= 1;;) > { > - while ((ev= Log_event::read_log_event(first_round ? first_log : &log, > + while ((ev= Log_event::read_log_event(round == 1 ? first_log : &log, > fdle, opt_master_verify_checksum)) > && ev->is_valid()) > { > enum Log_event_type typ= ev->get_type_code(); > switch (typ) > { > + case FORMAT_DESCRIPTION_EVENT: > + if (round > 1) > + { > + if (fdle != fdle_arg) > + delete fdle; > + fdle= (Format_description_log_event*) ev; why is it needed? all binlog files should have the same Format_description_log_event anyway. > + } > + break; > case XID_EVENT: > + if (do_xa) > { > - if (do_xa) > + xid_recovery_member *member= > + (xid_recovery_member*) > + my_hash_search(&xids, (uchar*) &static_cast<Xid_log_event*>(ev)->xid, > + sizeof(my_xid)); > +#ifndef HAVE_REPLICATION > { > - Xid_log_event *xev=(Xid_log_event *)ev; > - uchar *x= (uchar *) memdup_root(&mem_root, (uchar*) &xev->xid, > - sizeof(xev->xid)); > - if (!x || my_hash_insert(&xids, x)) > - goto err2; > + if (member) > + member->decided_to_commit= true; > } > - break; > +#else > + if (ctx.decide_or_assess(member, round, fdle, linfo, ev->log_pos)) > + goto err2; > +#endif > } > + break; > case BINLOG_CHECKPOINT_EVENT: > - if (first_round && do_xa) > + if (round == 1 && do_xa) > { > size_t dir_len; > Binlog_checkpoint_log_event *cev= (Binlog_checkpoint_log_event *)ev; > @@ -10124,9 +10820,18 @@ int TC_LOG_BINLOG::recover(LOG_INFO *linfo, const char *last_log_name, > } > } > break; > +#ifdef HAVE_REPLICATION > case GTID_LIST_EVENT: > - if (first_round) > + if (round == 1 || (ctx.do_truncate && > + (ctx.id_binlog == 0 || ctx.single_binlog))) > { > + /* > + Unlike the normal case, in do_truncate the initial state is > + in the first binlog file of the recovery list. > + */ don't understand that either :( > + DBUG_ASSERT(!ctx.do_truncate || !ctx.single_binlog || > + ctx.id_binlog == UINT_MAX); > + > Gtid_list_log_event *glev= (Gtid_list_log_event *)ev; > > /* Initialise the binlog state from the Gtid_list event. */ > @@ -10135,19 +10840,16 @@ int TC_LOG_BINLOG::recover(LOG_INFO *linfo, const char *last_log_name, > } > break; > > -#ifdef HAVE_REPLICATION > case GTID_EVENT: > - if (first_round) > + ctx.process_gtid(round, (Gtid_log_event *)ev, linfo); > + break; > + > + case QUERY_EVENT: > + if (((Query_log_event *)ev)->is_commit() || > + ((Query_log_event *)ev)->is_rollback()) > { > - Gtid_log_event *gev= (Gtid_log_event *)ev; > - > - /* Update the binlog state with any GTID logged after Gtid_list. */ > - last_gtid.domain_id= gev->domain_id; > - last_gtid.server_id= gev->server_id; > - last_gtid.seq_no= gev->seq_no; > - last_gtid_standalone= > - ((gev->flags2 & Gtid_log_event::FL_STANDALONE) ? true : false); > - last_gtid_valid= true; > + ctx.last_gtid_no2pc= true; > + ctx.update_binlog_unsafe_coord_if_needed(linfo); what about DML on non-transactonal tables? it won't have COMMIT/ROLLBACK will it? > } > break; > #endif > @@ -10217,35 +10923,43 @@ int TC_LOG_BINLOG::recover(LOG_INFO *linfo, const char *last_log_name, > file= -1; > } > > - if (!strcmp(linfo->log_file_name, last_log_name)) > - break; // No more files to do > +#ifdef HAVE_REPLICATION > + int rc= ctx.next_binlog_or_round(round, last_log_name, > + binlog_checkpoint_name, linfo, this); > + if (rc == -1) > + goto err2; > + else if (rc == 1) > + break; // all rounds done > +#else > + if (!strcmp(linfo->log_file_name, last_log_name)) > + break; // No more files to do indentation > +#endif > + > if ((file= open_binlog(&log, linfo->log_file_name, &errmsg)) < 0) > { > sql_print_error("%s", errmsg); > goto err2; > } > - /* > - We do not need to read the Format_description_log_event of other binlog > - files. It is not possible for a binlog checkpoint to span multiple > - binlog files written by different versions of the server. So we can use > - the first one read for reading from all binlog files. > - */ > - if (find_next_log(linfo, 1)) > - { > - sql_print_error("Error reading binlog files during recovery. Aborting."); > - goto err2; > - } > fdle->reset_crypto(); > - } > + } // end of for > > if (do_xa) > { > - if (ha_recover(&xids)) > - goto err2; > - > + if (binlog_checkpoint_found) > + { > +#ifndef HAVE_REPLICATION > + if (ha_recover_complete(&xids)) > +#else > + if (ctx.complete(this, xids)) > +#endif > + goto err2; > + } > free_root(&mem_root, MYF(0)); > my_hash_free(&xids); > } > + if (fdle != fdle_arg) > + delete fdle; > + > return 0; > > err2: Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

[Maria-developers] Systems Integration and Raising of the Abstraction Level
by Alejandro Sánchez 25 Feb '21

25 Feb '21

We have highly evolved systems such as SQL, HTTP, HTML, file formats or high level programming languages such as Java or PHP that allow us to program many things with little code. Even so a lot of effort is invested in the integration of these systems. To try to reduce this problem libraries and frameworks that help in some ways are created but the integration is not complete. It is well known that most of the time when you try to create something to integrate several incompatible systems what you get in the end is to have another incompatible system :). Still I think the integration between the systems mentioned above is something very important that can mean a great step in the evolution of computing and worth a try. To explore how this integration can be I have created a framework that I have called NextTypes and that its main objective is the integration of data types. For me it is something very illogical that something as basic as a 16 bits integer receives a different name in each of the systems ("smallint", "short", "number") and can also be signed or unsigned. In any moment, due to a mistake from the programmer, the number of the programming language does not fit in the database column. Besides these names are little indicative of its characteristics, it would be clearer for example to use "int16". Whatever names are chosen the most important thing is to use in all systems the same names for types of the same characteristics. Also there is no standard system for defining composite data types of primitive types and other composite types. From an HTML form to a SQL table or from one application to another are required multiple transformations of the data. Lack of integration also lowers the level of abstraction, making it necessary to do lots of low level stuff for systems to fit. NextTypes at this moment is nothing more than another incompatible system with the others. It simply integrates them quite a bit and raises the level of abstraction. But what I would like is that the things that compose NextTypes were included in the systems it integrates. Finally I would like to list some examples of improvements in database managers, SQL, HTTP, HTML, browsers and programming languages that would help the integration and elevation of the level of abstraction. Some of these enhancements are already included in NextTypes and other frameworks. SQL --- - Custom metadata in tables and columns. - Date of creation and modification of the rows. - Date of creation and modification of the definition of the tables. - Use of table and column names in prepared statements. Example: select # from # where # = ?; - Use of arrays in prepared statements. Example: select # from article where id in (?); # = author,title ? = 10,24,45 - Standardization of ranges of valid values and resolution for date, time and datetime types in database managers an HTML time element. PostgreSQL ---------- - Facilitate access to the definition of full text search indexes with a function to parse "pg_index.indexprs" column. Other database managers ----------------------- - Allow transactional DDL, deferrable constraints and composite types. JDBC ---- - High level methods that allow queries with the execution of a single method. Example: Tuple [] tuples = query("select author,title from article where id in (?), ids); - Integration with java.time data types. HTTP - Servers -------------- - Processing of arrays of elements composed of several parameters. fields:0:type = string fields:0:name = title fields:0:parameters = 250 fields:0:not_null = true fields:1:type = string fields:1:name = author fields:1:parameters = 250 fields:1:not_null = true Another possibility is to generate in the browser arrays of JSON objects from the forms. "fields": [ { "type": "string", "name": "title", "parameters": 250, "not_null": true }, { "type": "string", "name": "author", "parameters": 250, "not_null": true } ] XML/HTML - BROWSER ------------------ - Input elements for different types of numbers with min and max values: 16 bits integer, 32 bits integer, 32 bits float and 64 bits float. - Input elements for images, audios and videos with preview. - Timezone input element. - Boolean input element with "true" and "false" values. - Null value in file inputs. - Clear button in file inputs like in date and time inputs. - Show size in file inputs. - Extension of the DOM API with high level and chainable methods. Example: paragraph.appendElement("a").setAttribute("href", "/article"); - Change of the "action" parameter of the forms to "target" to indicate the URL where to execute the action. The "action" parameter is moved to the different buttons on the form and allows executing a different action with each of the the buttons. Example: <form target="/article"> <button action="delete">Delete</button> <button action="export">Export</button> </form> - "select" elements that change a parameter of the current URL. Example: <select url-parameter="lang"/> <option>en</option> <option>es</option> URL = https://demo.nexttypes.com/?lang=en - "content-type" attribute in links and context menu in the browser to open links with external applications using WEBDAV, similar to a file manager. Example: <a href="" content- type="application/vnd.oasis.opendocument.text"> ------------------------------ | Open link with ... | | Open link with LibreOffice | - Background submission of forms without using XMLHttpRequest, display of result in dialog window or file download, and execution of a Javascript function for subsequent actions. Example: <form background show-progress callback=function()> - Dialog with progress indicator of form submission. Must show total size, transmitted data and average speed. Possibility to pause or cancel the submission. - Dynamic datalist with searchable JSON source. Over data source URL is added a "search" parameter with input value. Example: <input list="article-list" name="article" type="text" /> <datalist id="article-list" src="/article?lang=en&view=json&names" /> Example query URL: "/article?lang=en&view=json&names&search=Ne" - Same appearance and operation of inputs with dynamic datalist in all browsers. - Option tags with icons. Example: <option icon="/icons/save.svg">Save</option> - Tabs, Tree, etc widgets - Mechanism to close HTTPS sessions initiated with client certificate authentication. JAVA ---- - Subclasses of String or some system of variants of String that allows assigning a regular expression or class that limits its valid values to avoid code injection or values that will crash the system. Example: String:Type = "[a-z0-9 _] +"; Or String:Type = TypeChecks; String:Type type = "article_language"; -> correct String:Type type = "Article-Language"; -> error We can talk about the characteristics of each system in its mailing list or group. For the general topic of system integration I have created a discussion in the github project. https://github.com/alejsanc/nexttypes/discussions/6 This email has been sent to the following mailing lists and groups: pgsql-hackers(a)lists.postgresql.org pgsql-jdbc(a)lists.postgresql.org mozilla.dev.platform(a)googlegroups.com public-html(a)w3.org jdbc-spec-discuss(a)openjdk.java.net jdk-dev(a)openjdk.java.net maria-developers(a)lists.launchpad.net chromium-dev(a)chromium.org https://mysqlcommunity.slack.com/archives/C8R1336M7 Best regards. Alejandro Sánchez.

1 0

[Maria-developers] Re MDEV-7317: Ignored indexes
by Sergey Petrunia 24 Feb '21

24 Feb '21

Hi Varun, The code for the task is mostly fine. But I would like to question the user experience. == Ignoring an index causes InnoDB table rebuild == First (and foremost), making an index ignored (or not ignored) in an InnoDB table rebuilds the table: MariaDB [test]> set session alter_algorithm='instant'; alter table t1 alter index c ignore; Query OK, 0 rows affected (0.001 sec) ERROR 1845 (0A000): ALGORITHM=INSTANT is not supported for this operation. Try ALGORITHM=COPY This misses the point of the feature. If one needs to rebuild the table anyway, they can just drop (or recreate) the index. Curiously, it works with MyISAM. I think, the ideal solution would be to make ignoring/not ignoring the index an SQL layer change that doesn't concern the storage engine. If this is not possible, lets make InnoDB aware that it doesn't need to rebuild the table in this case. == Backward compatibility for SHOW CREATE TABLE == SHOW CREATE TABLE will show something like: KEY `a` (`a`) IGNORE, MySQL here is more backward-compatible and shows: KEY `a` (`a`) /*!80000 INVISIBLE */, Do we need to do this as well? I have no strong opinion here. == Use of Ignored index in hints == I can use Ignored index in FORCE/IGNORE index hints: MariaDB [test]> explain select * from t1 force index(a) where a<3 and b<3; +------+-------------+-------+------+---------------+------+---------+------+------+-------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +------+-------------+-------+------+---------------+------+---------+------+------+-------------+ | 1 | SIMPLE | t1 | ALL | NULL | NULL | NULL | NULL | 1000 | Using where | +------+-------------+-------+------+---------------+------+---------+------+------+-------------+ This is allowed. I think, this misses the point of the feature. The point of this feature is to check "what will happen if one drops the index". For this particular query, the effect will be like this: MariaDB [test]> explain select * from t1 force index(no_such_index) where a<3 and b<3; ERROR 1176 (42000): Key 'no_such_index' doesn't exist in table 't1' and MySQL agrees with me here (below index a is present but is invisible) : mysql> explain select * from t1 force index(a) where a<3 and b<3; ERROR 1176 (42000): Key 'a' doesn't exist in table 't1' == optimizer_switch flag == Seeing "ignore_indexes=on" in optimizer_switch looks scary. I would say, we don't need an @@optimizer_switch flag. Maybe, Serg disagrees. If we have to have a switch flag, and its name should be short (my opinion), I would use the noun form: ignored_indexes=ON|OFF I'm not 100% happy with this, but ignoreD is much better here than ignore. (Before acting on this review point, lets check with others). BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://petrunia.net

2 1

Re: [Maria-developers] MDEV-17399: JSON_TABLE, commit 85757ecbef44484270e385a8d52998c67f72d11a
by Sergey Petrunia 23 Feb '21

23 Feb '21

Hi Alexey, First reactions to the JSON_TABLE patch: == CI Build is failing == Please check http://buildbot.askmonty.org/buildbot/grid?category=main&branch=bb-10.6-mde… the JSON_TABLE tests are failing. == On testcases == I see now the patch has tests in 3 files: 1. mysql-test/main/json_table_mysql.test 2. mysql-test/main/json_table_mysql2.test 3. mysql-test/suite/json/t/json_table.test #1 and #2 are both fixed copies of MySQL's test. Please remove one of them (the one that has less coverage). Then, can we have both files in the same testsuite? == On Table_function_json_table::ready_for_lookup() == Now that we have end_lateral_table, do we need to use Table_function_json_table::ready_for_lookup? This looks like two solutions for [almost] the same problem? I see that this example from json_table_mysql.test still uses the code path with ready_for_lookup() : --error ER_BAD_FIELD_ERROR UPDATE t1, JSON_TABLE(t1.c1,'$[*]' COLUMNS (a INT PATH '$.a')) AS jt1 SET jt1.a=1; Here, the name resolution for "jt1.a" is done without adjustment for end_lateral_table. Do we need the ready_for_lookup() trick here? (If we do, are we fine with m_setup_done to be set to true but never reset back to false? How will this work with PS?) == On opt_sum.cc == The diff has this: diff --git a/sql/opt_sum.cc b/sql/opt_sum.cc index af2d9ddc2e7..d021c651196 100644 --- a/sql/opt_sum.cc +++ b/sql/opt_sum.cc @@ -299,6 +299,7 @@ int opt_sum_query(THD *thd, tl->schema_table) { maybe_exact_count&= MY_TEST(!tl->schema_table && +//mdev-17399 !tl->table_function && (tl->table->file->ha_table_flags() & HA_HAS_RECORDS)); is_exact_count= FALSE; Please remove this if it is no longer needed. == On table dependencies: table elimination == create table t20 (a int not null); create table t21 (a int not null primary key, js varchar(100)); insert into t20 values (1),(2); insert into t21 values (1, '{"a":100}'); explain select t20.a, jt1.ab from t20 left join t21 on t20.a=t21.a join JSON_TABLE(t21.js,'$' COLUMNS (ab INT PATH '$.a')) AS jt1; +------+-------------+-------+------+---------------+------+---------+------+------+----------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +------+-------------+-------+------+---------------+------+---------+------+------+----------------------------+ | 1 | SIMPLE | t20 | ALL | NULL | NULL | NULL | NULL | 2 | | | 1 | SIMPLE | jt1 | ALL | NULL | NULL | NULL | NULL | 40 | Table function: json_table | +------+-------------+-------+------+---------------+------+---------+------+------+----------------------------+ Here we can see an apparently invalid query plan: table t21 was eleminated even if JSON_TABLE uses it. == On table dependencies: regular subqueries == But it doesn't work for regular subqueries either (and this is unexpected): insert into t21 values (3, '{"a":300}'); explain select a, (select jt1.ab from JSON_TABLE(t21.js,'$' COLUMNS (ab INT PATH '$.a')) as jt1 ) from t21; +------+-------------+-------+------+---------------+------+---------+------+------+----------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +------+-------------+-------+------+---------------+------+---------+------+------+----------------------------+ | 1 | PRIMARY | t21 | ALL | NULL | NULL | NULL | NULL | 2 | | | 2 | SUBQUERY | jt1 | ALL | NULL | NULL | NULL | NULL | 40 | Table function: json_table | +------+-------------+-------+------+---------------+------+---------+------+------+----------------------------+ Note the 'SUBQUERY', that is, the subquery is considered uncorrelated? This needs to be investigated. BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://petrunia.net

2 3

Re: [Maria-developers] MDEV-17399: Columns of JSON datatype?
by Sergey Petrunia 16 Feb '21

16 Feb '21

On Sun, Feb 14, 2021 at 05:14:10PM +0400, Alexey Botchkov wrote: > One question left about the json_table_mysql.text. > > --error ER_WRONG_OUTER_JOIN > SELECT * FROM t1 RIGHT JOIN > (SELECT o FROM t2, JSON_TABLE(j, '$[*]' COLUMNS (o FOR ORDINALITY)) AS jt) > AS t3 > ON (t3.o = t1.x); > > What is exactly wrong with the above join? It should work. The reason it doesn't work was described by me in the email with subject "MDEV-17399: JSON_TABLE's dependencies are not updated after derived merge" dated Feb, 8th. What happens here is: Thread 14 "mysqld" hit Breakpoint 1, Table_function_json_table::setup() (gdb) list 1091 } 1092 } 1093 > 1094 m_dep_tables= m_json->used_tables(); 1095 (gdb) p m_dep_tables $44 = 1 If you go one frame up, you will see that we are in select#2. tables are: (gdb) p join->select_lex->leaf_tables.elem(0)->alias.str $56 = 0x7ffeb0016550 "t2" (gdb) p join->select_lex->leaf_tables.elem(0)->table->map $58 = 1 (gdb) p join->select_lex->leaf_tables.elem(1)->alias.str $60 = 0x7ffeb0017d70 "jt" (gdb) p join->select_lex->leaf_tables.elem(1)->table->map $62 = 2 Ok, jt depends on t2. Then: Thread 14 "mysqld" hit Breakpoint 2, simplify_joins (gdb) p join->select_lex->leaf_tables.elements $65 = 3 (gdb) p join->select_lex->leaf_tables.elem(0)->alias.str $66 = 0x7ffeb0015870 "t1" (gdb) p join->select_lex->leaf_tables.elem(0)->table->map $67 = 1 (gdb) p join->select_lex->leaf_tables.elem(1)->alias.str $68 = 0x7ffeb0016550 "t2" (gdb) p join->select_lex->leaf_tables.elem(1)->table->map $69 = 2 (gdb) p join->select_lex->leaf_tables.elem(2)->alias.str $70 = 0x7ffeb0017d70 "jt" (gdb) p join->select_lex->leaf_tables.elem(2)->table->map $71 = 4 But the m_dep_tables still has the old value: (gdb) p join->select_lex->leaf_tables.elem(2)->table_function->m_dep_tables $72 = 1 BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://petrunia.net

1 0

[Maria-developers] MDEV-8306: EXPLAIN JSON output is not JSON-ish
by Sergey Petrunia 15 Feb '21

15 Feb '21

Hi Varun, (This email is not the complete input, and not the most important part of the input. More to follow. I think it's better to split input into multiple smaller pieces, it's easier to track, and you get to see the first portions of the input sooner) EXPLAIN EXTENDED SELECT t3.b, t2.a, t1.b, t1.a FROM t1,t2,t3 WHERE t1.b=t3.b ORDER BY t1.b DESC, t2.a DESC LIMIT 3; +------+-------------+-------------+------+---------------+------+---------+------+------+----------+------------------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | filtered | Extra | +------+-------------+-------------+------+---------------+------+---------+------+------+----------+------------------------------------+ | 1 | SIMPLE | t2 | ALL | NULL | NULL | NULL | NULL | 10 | 100.00 | | | 1 | SIMPLE | t1 | ALL | NULL | NULL | NULL | NULL | 20 | 100.00 | Using join buffer (flat, BNL join) | | 1 | SIMPLE | <sort-nest> | ALL | NULL | NULL | NULL | NULL | 3 | 100.00 | Using filesort | | 1 | SIMPLE | t3 | ALL | NULL | NULL | NULL | NULL | 100 | 5.00 | Using where | +------+-------------+-------------+------+---------------+------+---------+------+------+----------+------------------------------------+ So, here <sort-nest> includes t2 and t1, and then t3 is joined with it. However in EXPLAIN FORMAT=JSON output they all look like peers: { "query_block": { "select_id": 1, "table": { "table_name": "t2", "access_type": "ALL", "rows": 10, "filtered": 100 }, "block-nl-join": { "table": { "table_name": "t1", "access_type": "ALL", "rows": 20, "filtered": 100 }, "buffer_type": "flat", "buffer_size": "65", "join_type": "BNL" }, "read_sorted_file": { "filesort": { "sort_key": "`sort-nest`.b desc, `sort-nest`.a desc", "table": { "table_name": "<sort-nest>", "access_type": "ALL", "rows": 3, "filtered": 100 } } }, "table": { "table_name": "t3", "access_type": "ALL", "rows": 100, "filtered": 5, "attached_condition": "t3.b = `sort-nest`.b" } } } which is counter-intuitive. Also, `sort-nest` is quoted. I assume it is because of the '-' in it? Maybe we should change the name to avoid the quoting? BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://petrunia.net

2 3

Re: [Maria-developers] MDEV-17399 New patch for JSON_TABLE
by Sergey Petrunia 14 Feb '21

14 Feb '21

Hi Alexey, Input on the latest patch for MDEV-17399. It's good to see the patch taking shape, but have you tried running MySQL's tests for JSON_TABLE on it? I did, and it has caught several crashes, as well as error-no-error discrepancies and different query results. Some of these were expected, but some show that something needs to be fixed. Please find the first few cases below. It seems there is something odd going odd with the name resolution, both in MariaDB and MySQL - I don't have explanation for some of the errors. I intend to study the issue more and elaborate in a later email. Meanwhile, first obvious cases: === Crash in Name resolution == CREATE TABLE t1 (a INT, b INT); CREATE VIEW v2 AS SELECT * FROM t1 LIMIT 2; SELECT b FROM (SELECT * FROM v2) vq1, JSON_TABLE(CONCAT(vq1.b,'[{\"a\":\"3\"}]'), '$[*]' COLUMNS (id FOR ORDINALITY, jpath VARCHAR(100) PATH '$.a', JEXST INT EXISTS PATH '$.b') ) AS dt; === Crash in error reporting === select * from json_table( '[{"a":"3"},{"a":2},{"b":1},{"a":0}]', "!@#$!@#$" columns (id for ordinality, jpath varchar(100) path '$.a', jexst int exists path '$.b') ) as tt; === Item print misses quotes === CREATE VIEW v2 AS SELECT * FROM JSON_TABLE('{}', '$' COLUMNS ( x VARCHAR(10) PATH '$.a' DEFAULT '"isn''t here"' ON EMPTY) ) t; SHOW CREATE VIEW v2; ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 't here"' ON EMPTY)) `t`' at line 1 === Error for outer reference == # JSON_TABLE referring outer scope CREATE TABLE t1(id int, jd JSON); INSERT INTO t1 values (1, '[1,3,5]'),(2,'[2,4,6]'); SELECT * FROM t1 WHERE id IN (SELECT * FROM JSON_TABLE(t1.jd, '$[*]' COLUMNS (id INT PATH '$')) AS jt); Produces: ERROR 1054 (42S22): Unknown column 't1.jd' in 'JSON_TABLE argument' Should not produce it (Note that for some other kinds of outer references it seemed to work) BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

2 15

[Maria-developers] MDEV-17399: Name resolution and handling dependencies
by Sergey Petrunia 14 Feb '21

14 Feb '21

Hi Alexey, At the moment, name resolution of JSON_TABLE's first argument is done "like in the WHERE clause" - one can refer to any table that is defined in the WHERE clause. This allows one to write queries where JSON_TABLE tables have incorrect dependencies - circular - dependency that contradicts the dependency imposed by OUTER JOIN - dependency that contradicts STRAIGHT_JOIN (WRONG-DEPS) the patch checks for these cases but I've reported cases where it fails. I haven't been find any statement in the SQL Standard about this, but it makes a statement about similar constructs, table functions and LATERAL subqueries: https://jira.mariadb.org/browse/MDEV-17399?focusedCommentId=179327&page=com… Other databases seem to apply the same limitation to JSON_TABLE's argument. When they do it, this automatically fixes all the wrong-dependency issues - a query with (WRONG-DEPS) is rejected at the name resolution phase. I think we should follow this and modify name resolution process to work in the same way. The way MySQL did it is described here: https://dev.mysql.com/worklog/task/?id=8867, LLD, grep for end_lateral_table. I'm not fond of having implicit parameters (end_lateral_table) which change they way name resolution works, but I think it is an acceptable solution (and we already have other such parameters). (An alternative option would be to have items in JSON_TABLE's first argument to use their own Name_resolution_context object that would specify the correct first/last table they should look at. This seems to be much harder to do). What do you think? BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://petrunia.net

2 1