- developers - lists.mariadb.org

Re: [Maria-developers] 37b6f5cb8ef: decouple commit/rollback code from binlog hton
by Sergei Golubchik 23 Nov '21

23 Nov '21

Hi, Nikita! On Nov 15, Nikita Malyavin wrote: > revision-id: 37b6f5cb8ef (mariadb-10.5.2-484-g37b6f5cb8ef) > parent(s): f3603cbfaeb > author: Nikita Malyavin > committer: Nikita Malyavin > timestamp: 2021-01-29 03:47:52 +1000 > message: > > decouple commit/rollback code from binlog hton let's squash it with "[6/8] store cache managers in a list" Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] ad6e7b87107: introduce cache flipping
by Sergei Golubchik 23 Nov '21

23 Nov '21

Hi, Nikita! Interesting trick. On Nov 13, Nikita Malyavin wrote: > revision-id: ad6e7b87107 (mariadb-10.5.2-477-gad6e7b87107) > parent(s): 85fbd867b4c > author: Nikita Malyavin > committer: Nikita Malyavin > timestamp: 2021-01-27 14:17:39 +1000 > message: > > introduce cache flipping Could you add few lines from the Cache_flip_event_log comment here too please? > diff --git a/sql/log.h b/sql/log.h > index 74c409e1ac7..1e422ae4524 100644 > --- a/sql/log.h > +++ b/sql/log.h > @@ -424,10 +424,9 @@ class Event_log: public MYSQL_LOG > > bool open( > const char *log_name, > - enum_log_type log_type, > const char *new_name, ulong next_file_number, > enum cache_type io_cache_type_arg); > - IO_CACHE *get_log_file() { return &log_file; } > + virtual IO_CACHE *get_log_file() { return &log_file; } > > int write_description_event(enum_binlog_checksum_alg checksum_alg, > bool encrypt, bool dont_set_created); > @@ -435,6 +434,73 @@ class Event_log: public MYSQL_LOG > bool write_event(Log_event *ev, binlog_cache_data *data, IO_CACHE *file); > }; > > +/** > + A single-reader, multiple-writer non-blocking layer for Event_log. > + Provides IO_CACHE for writing and IO_CACHE for reading.\ > + > + Writers use an overrided get_log_file version for their writes, while readers > + should use flip() to initiate reading. > + flip() swaps pointers to allow non-blocking reads. > + > + Writers can block other writers and a reader with a mutex, but a reader only > + swaps two pointers under a lock, so it won't block writers. > + > + TODO should be unnecessary after MDEV-24676 is done > + */ > +class Cache_flip_event_log: public Event_log { > + IO_CACHE alt_buf; > + IO_CACHE *current, *alt; > +public: > + > + Cache_flip_event_log() : Event_log(), alt_buf{}, > + current(&log_file), alt(&alt_buf) {} > + bool open(enum cache_type io_cache_type_arg) > + { > + log_file.dir= mysql_tmpdir; > + alt_buf.dir= log_file.dir; > + bool res= Event_log::open(NULL, NULL, 0, io_cache_type_arg); > + if (res) > + return res; > + > + name= my_strdup(key_memory_MYSQL_LOG_name, "online-alter-binlog", > + MYF(MY_WME)); > + if (!name) > + return false; > + > + res= init_io_cache(&alt_buf, -1, LOG_BIN_IO_SIZE, io_cache_type_arg, 0, 0, > + MYF(MY_WME | MY_NABP | MY_WAIT_IF_FULL)) != 0; > + return res; > + } > + > + /** > + Swaps current and alt_log. Can be called only from the reader thread. > + @return a new IO_CACHE pointer to read from. > + */ > + IO_CACHE *flip() > + { > + IO_CACHE *tmp= current; > + reinit_io_cache(alt, WRITE_CACHE, 0, 0, 0); > + mysql_mutex_lock(get_log_lock()); > + reinit_io_cache(current, READ_CACHE, 0, 0, 0); > + current= alt; > + mysql_mutex_unlock(get_log_lock()); > + alt= tmp; > + > + return alt; > + } > + > + IO_CACHE *get_log_file() override > + { > + return current; may be mysql_mutex_assert_owner(get_log_lock()) ? > + } > + > + void cleanup() > + { > + end_io_cache(&alt_buf); > + Event_log::cleanup(); > + } > +}; > + Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] c80991c79f7: MDEV-25785 Add support for OpenSSL 3.0
by Sergei Golubchik 21 Nov '21

21 Nov '21

Hi, Vladislav! few questions below: On Nov 18, Vladislav Vaintroub wrote: > revision-id: c80991c79f7 (mariadb-10.6.1-213-gc80991c79f7) > parent(s): cee33f1ab7c > author: Vladislav Vaintroub > committer: Vladislav Vaintroub > timestamp: 2021-11-09 02:04:22 +0100 > message: > > MDEV-25785 Add support for OpenSSL 3.0 > > Summary of changes > > - MD_CTX_SIZE is increased > > - EVP_CIPHER_CTX_buf_noconst(ctx) does not work anymore, points > to nobody knows where. The assumption made previously was that > (since the function does not seem to be documented) > was that it points to the last partial source block. > Add own partial block buffer for NOPAD encryption instead > > - SECLEVEL in CipherString in openssl.cnf > had been downgraded to 0, from 1, to make TLSv1.0 and TLSv1.1 possible The definition of SECLEVEL is: Level 0 Everything is permitted. This retains compatibility with previous versions of OpenSSL. Level 1 The security level corresponds to a minimum of 80 bits of security. Any parameters offering below 80 bits of security are excluded. As a result RSA, DSA and DH keys shorter than 1024 bits and ECC keys shorter than 160 bits are prohibited. All export cipher suites are prohibited since they all offer less than 80 bits of security. SSL version 2 is prohibited. Any cipher suite using MD5 for the MAC is also prohibited. Note that signatures using SHA1 and MD5 are also forbidden at this level as they have less than 80 security bits. Only at level 3 does it say "TLS versions below 1.1 are not permitted." I don't see why you had to change from 1 to 0. Do we have "signatures using SHA1 and MD5"? > - ctx_buf buffer now must be aligned to 16 bytes with openssl( > previously with WolfSSL only), ot crashes will happen > > - updated aes-t , to be better debuggable > using function, rather than a huge multiline macro > added test that does "nopad" encryption piece-wise, to test > replacement of EVP_CIPHER_CTX_buf_noconst > > diff --git a/cmake/ssl.cmake b/cmake/ssl.cmake > index 7c2488be8bd..64c93ff9b4f 100644 > --- a/cmake/ssl.cmake > +++ b/cmake/ssl.cmake > @@ -139,9 +139,20 @@ MACRO (MYSQL_CHECK_SSL) > SET(SSL_INTERNAL_INCLUDE_DIRS "") > SET(SSL_DEFINES "-DHAVE_OPENSSL") > > + FOREACH(x INCLUDES LIBRARIES DEFINITIONS) > + SET(SAVE_CMAKE_REQUIRED_${x} ${CMAKE_REQUIRED_${x}}) > + ENDFOREACH() why do you set/restore them? > + # Silence "deprecated in OpenSSL 3.0" > + IF((NOT OPENSSL_VERSION) # 3.0 not determined by older cmake > + OR NOT(OPENSSL_VERSION VERSION_LESS "3.0.0")) > + SET(SSL_DEFINES "${SSL_DEFINES} -DOPENSSL_API_COMPAT=0x10100000L") > + SET(CMAKE_REQUIRED_DEFINITIONS -DOPENSSL_API_COMPAT=0x10100000L) > + ENDIF() This is just postponing the inevitable. They'll drop the old API eventually. As far as I understand the internals were changed in a way that doesn't fit the old API. Is there some safe subset of OpenSSL API that works both in 1.0 and in 3.0 ? It might be more future proof to use only that. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

3 4

Re: [Maria-developers] 4492c869f31: MDEV-26238: Remove inconsistent behaviour of --default-* options in my_print_defaults
by Sergei Golubchik 18 Nov '21

18 Nov '21

Hi, Rucha! Thanks! Looks good now. One comment about tests: On Nov 18, Rucha Deodhar wrote: > revision-id: 4492c869f31 (mariadb-10.6.1-212-g4492c869f31) > parent(s): 3f3ec40c91b > author: Rucha Deodhar > committer: Rucha Deodhar > timestamp: 2021-11-08 16:51:33 +0530 > message: > > MDEV-26238: Remove inconsistent behaviour of --default-* options > in my_print_defaults ... > diff --git a/mysql-test/main/my_print_defaults.test b/mysql-test/main/my_print_defaults.test > index bfd4e563826..b52deaf09ff 100644 > --- a/mysql-test/main/my_print_defaults.test > +++ b/mysql-test/main/my_print_defaults.test > @@ -87,20 +85,74 @@ read_buffer_size=1M ... > +--echo # checking that --defaults* option only works when mentioned at beginning > + > +--echo # Testing --defaults-file at beginning only > +--exec $MYSQL_MY_PRINT_DEFAULTS --defaults-file=$MYSQLTEST_VARDIR/tmp/tmp1.cnf --mysqld > +--remove_file $MYSQLTEST_VARDIR/tmp/tmp3.cnf How does it test that "--defaults* option only works at beginning"? You test that --defaults-file works at the beginning all right. This is redundant, as there are lots of tests in this file that have --defaults-file, but it doesn't hurt. But you don't test the "only" part. You don't test anything that you've changed - I suspect your new test would happily pass without any changes in my_print_defaults.c To test that --defaults-file works *only* at the beginning, you need to verify that it fails not at the beginning. Like error 1 exec $MYSQL_MY_PRINT_DEFAULTS --myslqd --defaults-file=$MYSQLTEST_VARDIR/tmp/tmp1.cnf also with -c error 1 exec $MYSQL_MY_PRINT_DEFAULTS -c $MYSQLTEST_VARDIR/tmp/tmp1.cnf --mysqld This should fail even at the beginning. The point is - you need to test the behavior that you've changed. As your test has introduced failures - you need to test for these failures. But all your tests succeed. > +--echo # Testing --defaults-extra-file works at beginning only > +--exec $MYSQL_MY_PRINT_DEFAULTS --defaults-file=$MYSQLTEST_VARDIR/tmp/tmp1.cnf --defaults-extra-file=$MYSQLTEST_VARDIR/tmp/tmp2.cnf --mysqld Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] MDEV-27036: do not output empty subqueries in traces
by Sergey Petrunia 16 Nov '21

16 Nov '21

Hi Sergei, > commit eb7d6304a2aa168e09d48f6884dadf8c9f868feb > Author: Sergei Krivonos <sergei.krivonos(a)mariadb.com> > Date: Mon Nov 15 01:37:04 2021 +0200 > > do not output empty subqueries in traces This reads as if the trace can have an empty clause describing a subquery. This is very confusing. Actually, the patch removes potential empty "subqueries" arrays. Do you have a testcase for this? I'm trying to find one with grep -rn -A1 '"subqueries"' mysql-test/{main,suite} and I can't. > diff --git a/sql/sql_explain.cc b/sql/sql_explain.cc > index 499b6dbc97a..9a9bf786d62 100644 > --- a/sql/sql_explain.cc > +++ b/sql/sql_explain.cc > @@ -25,6 +25,8 @@ > #include "opt_range.h" > #include "sql_expression_cache.h" > > +#include <algorithm> > + Is this needed? It seems to work without it. > + bool Explain_node::has_json_printable_connection_objects(Explain_query *query) const > + { > + auto n= children.elements(); > + if (n) > + { > + for (; n--;) > + { Why such a convoluted loop inside an if statement? Can you make the code simpler by using for (n=0; n < children.elements(); n++) { if (is_connection_printable_in_json(children.at(i)->connection_type)) return true; } return false; ? > + auto node= query->get_node(children.at(n)); > + /* Derived tables are printed inside Explain_table_access objects */ > + > + if (!is_connection_printable_in_json(node->connection_type)) > + continue; > + > + return true; > + } > + } > + return false; > + } BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://petrunia.net

1 0

Re: [Maria-developers] MDEV-27036: duplicated keys in best_access_path
by Sergey Petrunia 16 Nov '21

16 Nov '21

Hi Sergei, > commit d2b4782200ccfd0850f36d25c75082974e4b86e1 > Author: Sergei Krivonos <sergei.krivonos(a)mariadb.com> > Date: Sun Nov 14 10:37:20 2021 +0200 > > MDEV-27036: duplicated keys in best_access_path > > diff --git a/sql/sql_select.cc b/sql/sql_select.cc > index 3272f8b7f65..29e9c0efe36 100644 > --- a/sql/sql_select.cc > +++ b/sql/sql_select.cc > @@ -7611,16 +7611,14 @@ best_access_path(JOIN *join, > } > else > { > + trace_access_idx.add("used_range_estimates", false); > if (table->quick_keys.is_set(key)) > { > - trace_access_idx.add("used_range_estimates",false) > - .add("cause", > - "not better than ref estimates"); > + cause= "not better than ref estimates"; Issue #1: you make this change here, in the branch marked as ReuseRangeEstimateForRef-2. But there's very similar code right above, in ReuseRangeEstimateForRef-1 branch: /* quick_range couldn't use key! */ records= (double) s->records/rec; trace_access_idx.add("used_range_estimates", false) .add("cause", "not available"); Issue #2: this patch causes some information to be removed from the trace! How about an alternative solution: change the name from "cause" here to "reason"? > } > else > { > - trace_access_idx.add("used_range_estimates", false) > - .add("cause", "not available"); > + cause= "not available"; > } > } > } BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://petrunia.net

1 0

[Maria-developers] MDEV-26929: Please enable optimizer trace run in buildbot JSON
by Sergey Petrunia 09 Nov '21

09 Nov '21

Hi Elena, The patch for https://jira.mariadb.org/browse/MDEV-26929 has been merged from 10.4 into all versions up to 10.8. Could you enable the optimizer trace run in the buildbot please? BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://petrunia.net

2 2

Re: [Maria-developers] 80749146b7d: MDEV-26238: Remove inconsistent behaviour of --default-* options in my_print_defaults
by Sergei Golubchik 07 Nov '21

07 Nov '21

Hi, Rucha! On Nov 05, Rucha Deodhar wrote: > > On Nov 04, Rucha Deodhar wrote: > > > revision-id: 80749146b7d (mariadb-10.6.1-77-g80749146b7d) > > > parent(s): 76149650764 > > > author: Rucha Deodhar > > > committer: Rucha Deodhar > > > timestamp: 2021-09-12 02:11:19 +0530 > > > message: > > > > > > MDEV-26238: Remove inconsistent behaviour of --default-* options > > > in my_print_defaults > > > > > > diff --git a/extra/my_print_defaults.c b/extra/my_print_defaults.c > > > index b7f52382721..607509a309e 100644 > > > --- a/extra/my_print_defaults.c > > > +++ b/extra/my_print_defaults.c > > > @@ -141,50 +124,39 @@ static int get_options(int *argc,char ***argv) > > ... > > > + /* > > > + We already process --defaults* options at the beginning in > > > + get_defaults_options(). So skip --defaults* options and > > > + pass remaining options to handle_options(). > > > + */ > > > + if (is_prefix(org_argv[1], "--no-defaults")) > > > + args_used--; > > > > I'm sorry, I don't understand. Why do you need that? > > I made it to keep the output consistent with previous version. > > If --no-defaults is present in the command line, then it is counted in > args_used when get_defaults_options() is called. Later we skip > "args_used" number of arguments so that only non-default options > mentioned after --default-* options are passed to get_options(). > > The above code decreases this count when --no-defaults is present so > that we don't skip it unlike --defaults-* options and pass it to > get_options(). It is required to pass --no-defaults to get_options() > because if it is not passed then it produces output which looks like > --help ( if no other options are there in addition to --no-defaults). > If it is passed to get_options(), then --no-defaults produces no > output like it does in 10.6. But it's correct, isn't it? If you run my_print_defaults with no groups, it'll print the help text. Try, for example, my_print_defaults --defaults-file=var/my.cnf (presuming you're in mysql-test dir). You'll get the help text, because you didn't specify any groups. Or my_print_defaults --defaults-extra-file=var/my.cnf But my_print_defaults --no-defaults doesn't do it. Doesn't behave like other *defaults* options. We can easily preserve the inconsisent "if --no-defaults, no group is required" behavior with, like if (my_no_defaults) cleanup_and_exit(0); (and add my_no_defaults to my_default.h). But I don't think it's necessary. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 50f43d46299: MDEV-14938 make buildbot to include galera into bintars
by Sergei Golubchik 05 Nov '21

05 Nov '21

Hi, Alexey! On Nov 05, Alexey Bychko wrote: > revision-id: 50f43d46299 (mariadb-10.2.40-106-g50f43d46299) > parent(s): 05c3dced861 > author: Alexey Bychko > committer: Alexey Bychko > timestamp: 2021-10-22 16:30:56 +0700 > message: > > MDEV-14938 make buildbot to include galera into bintars > > this commit adds cmake module to process external directory with built > galera to include galera library and binaries into server build. > it introduces new cmake variable EXT_GALERA_PATH representing relative > or absolute path to compiled galera tree. No compiled galera tree. This is useless. MDEV description says Currently bintars that buildbot produces don't contain libgalera_smm.so. Publishing scripts repackage bintars to add it. We want to produce correct packages in buildbot and release exactly what buildbot builds. It means, buildbot need to produce *exactly* what `prep` script was generating. `prep` script does not build galera. This is a completely separate step, `prep` only packages pre-existing galera files. It's enough to assume that there's ${EXT_GALERA_PATH}/garbd and ${EXT_GALERA_PATH}/libgalera_smm.so. Or if you want it to be really universal, make two different variables, cmake -DBINTAR_GARBD=/path/to/garbd -DBINTAR_LIBGALERA=/path/to/libgalera_smm.so This will work with a compiled galera tree, with installed galera package, with just garbd and libgalera_smm.so in $HOME, with everything. > diff --git a/CMakeLists.txt b/CMakeLists.txt > index 67216e0e443..1107013860c 100644 > --- a/CMakeLists.txt > +++ b/CMakeLists.txt > @@ -153,6 +153,7 @@ INCLUDE(mysql_version) > INCLUDE(cpack_source_ignore_files) > INCLUDE(install_layout) > INCLUDE(wsrep) > +INCLUDE(galera_external) > > # Add macros > INCLUDE(character_sets) > diff --git a/cmake/galera_external.cmake b/cmake/galera_external.cmake > new file mode 100644 > index 00000000000..64ddfc9c498 > --- /dev/null > +++ b/cmake/galera_external.cmake > @@ -0,0 +1,75 @@ > +# EXT_GALERA_PATH > +# a path to built galera git tree > +# or a path to unpacked binary tarball > + > +IF(WIN32 OR NOT EXT_GALERA_PATH) why do you need WIN32 check here? > + RETURN() > +ENDIF() > + > +IF(NOT EXISTS ${EXT_GALERA_PATH}) > + MESSAGE(FATAL_ERROR "Galera dir ${EXT_GALERA_PATH} does not exist!") > +ENDIF() > + > +SET(GALERA_PATH ${CMAKE_SOURCE_DIR}/GALERASYM) > +IF(EXISTS ${GALERA_PATH}) > + FILE({REMOVE ${GALERA_PATH}) > +ENDIF() > +EXECUTE_PROCESS(COMMAND ${CMAKE_COMMAND} -E create_symlink ${EXT_GALERA_PATH} ${GALERA_PATH}) Why do you need this symlink? > + > +FIND_LIBRARY(GALERA_LIB > + NAMES galera_smm galera_enterprise_smm > + PATHS ${GALERA_PATH} > + NO_DEFAULT_PATH > + ) > + > +IF(NOT GALERA_LIB) > + MESSAGE(FATAL_ERROR "Galera library not found in ${GALERA_PATH}") > +ENDIF() > + > +INSTALL(PROGRAMS > + ${GALERA_LIB} > + DESTINATION lib > + RENAME libgalera_smm.so > + COMPONENT Server > + ) > + > +INSTALL(PROGRAMS > + ${GALERA_PATH}/garb/garbd > + DESTINATION bin > + COMPONENT Server > + ) > + > +INSTALL(PROGRAMS > + ${GALERA_PATH}/garb/files/garb-systemd > + DESTINATION bin > + COMPONENT Server > + ) > + > +INSTALL(FILES > + ${GALERA_PATH}/garb/files/garb.cnf > + DESTINATION docs/galera > + COMPONENT Server > + ) MDEV description says Currently bintars that buildbot produces don't contain libgalera_smm.so. Publishing scripts repackage bintars to add it. We want to produce correct packages in buildbot and release exactly what buildbot builds. It means, buildbot need to produce *exactly* what `prep` script was generating. Meaning three galera files. Like: -rwxr-xr-x mariadb-10.2.40-linux-glibc_214-x86_64/bin/garbd -rw-r--r-- mariadb-10.2.40-linux-glibc_214-x86_64/lib/libgalera_smm.so lrwxrwxrwx mariadb-10.2.40-linux-glibc_214-x86_64/lib/galera/libgalera_smm.so -> ../libgalera_smm.so Nothing else. Also, please, verify that the generated directory inside the tarball matches the pattern above. And that the tarbal name does too. > + > +INSTALL(FILES > + ${GALERA_PATH}/AUTHORS > + ${GALERA_PATH}/COPYING > + ${GALERA_PATH}/README > + DESTINATION docs/galera/doc > + COMPONENT Server > + ) > + > +INSTALL(FILES > + ${GALERA_PATH}/asio/LICENSE_1_0.txt > + DESTINATION docs/galera/doc > + RENAME LICENSE.asio > + COMPONENT Server > + ) > + > +INSTALL(FILES > + ${GALERA_PATH}/chromium/LICENSE > + DESTINATION docs/galera/doc > + RENAME LICENSE.chromium > + COMPONENT Server > + ) > + > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] MDEV-23766: Optimizer tracing code is prone to producing invalid JSON
by Sergey Petrunia 05 Nov '21

05 Nov '21

Hi Serg, On Tue, Nov 02, 2021 at 02:19:05AM +0200, Sergei Krivonos wrote: > Applied fixes by your tests patch 638930f85f… <https://github.com/MariaDB/server/commit/638930f85ff40bd39354da81e53e7278a4…> > #1: Please check out the commit adding unit tests: http://lists.askmonty.org/pipermail/commits/2021-November/014773.html It turns out it is not that hard to add them. Any comments/ways to improve this? #2: In the above commit, note that two last unit tests show a scenario that is not currently checked. #3: What is the point of functions Json_writer::on_add_str(), Json_writer::on_start_array, Json_writer::on_start_object() ? I would understand if all the checking code was isolated in these functions. But it isn't. Please either remove the functions or explain the reasoning. #4 See the input below about the commit comment. Please change (feel free to force-push into the branch after that). I think this is all the input, and the patch will be ready for pushing once all of this is addressed. > > Item #2: please improve the commit comment as was requested in my previous > > e-mail: > > > > On Thu, Oct 28, 2021 at 04:58:35PM +0300, Sergey Petrunia wrote: > >> First, input on the commit comment. Please consider it as a request applying > >> to ALL further commits to the MariaDB codebase. > >> > >> > >>> MDEV-23766: implemented requested debug checks > >> > >> Imagine somebody looking at this in a few years. Will they know what checks > >> were requested? They might get a clue by looking at the Jira text, but the > >> idea to make the commit comments concise and self-contained descriptions. > >> > >> Something like this: > >> > >> Line #1: one-line description of the patch: > >> > >>>> MDEV-23766: Make attempts to write invalid JSON cause assertion failures > >> > >> Subsequent lines: > >> > >>>> Make JSON writing API (class Json_writer) check the produced JSON document > >>>> is valid. The following checks are made: > >>>> - JSON objects must contain named members > >>>> - JSON arrays must contain unnamed members. > >>>> - (TODO: add other restrictions we're enforcing). BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://petrunia.net

1 0