- developers - lists.mariadb.org

Re: [MariaDB commits] [PATCH 1/4] MDEV-31273: Replace Log_event::writer with function parameter
by Kristian Nielsen 16 Oct '23

16 Oct '23

Hi Monty, [Following up on review on IRC] Kristian Nielsen via commits <commits(a)lists.mariadb.org> writes: > Replace Log_event::writer with just passing the writer object as a > function parameter to Log_event::write(). We wondered how the compiled code will look when passing the writer object as a parameter to Log_event::write() vs. temporarily storing it in the Log_event object. Interestingly, the code is actually "better" after my patch despite an extra parameter being passed along. This is because the compiler can cache the function parameter in a register across nested calls. While with a class member, the compiler cannot determine that the member will not be changed by the nested calls, so the value needs to be repeatedly re-loaded from memory. Some more details on this finding below. My reason for the change was probably more general, based on this code: > - ev->writer= this; > - int res= ev->write(); > - IF_DBUG(ev->writer= 0,); // writer must be set before every Log_event::write > + int res= ev->write(this); It's clear that the intention with this code is to pass the Log_event_writer object `this` to the ev->write() method, and it's a strange way to do it to temporarily set it as a member in the object rather than using a function parameter. Needing an IF_DBUG() to enforce/check that this is done correctly is a warning sign, and I was worried that as I did further changes I would introduce subtle bugs this way. It is interesting that this makes the code clearer not only for the developer (me), but also for the compiler in this case. Function parameter passing is quite well optimized on amd64, but it was still good to see this in detail in this case. And as you predicted we learned something from it! As we discussed, I will extend the commit message to explain these reasons better. Thanks for the suggestion; it's always good in the reviews to question the reason for changes like this. To check the compiled code in detail, I did a release build (BUILD/compile-pentium64-max) with and without this patch, and disassembled the User_var_log_event::write() function, which has a number of calls now passing the writer object as a function parameter: > - return write_header(event_length) || > - write_data(buf, sizeof(buf)) || > - write_data(name, name_len) || > - write_data(buf1, buf1_length) || > - write_data(pos, val_len) || > - write_data(&flags, unsigned_len) || > - write_footer(); > + return write_header(writer, event_length) || > + write_data(writer, buf, sizeof(buf)) || > + write_data(writer, name, name_len) || > + write_data(writer, buf1, buf1_length) || > + write_data(writer, pos, val_len) || > + write_data(writer, &flags, unsigned_len) || > + write_footer(writer); It looks like this would need to pass the `writer` argument many times to called functions. However, write_data() and write_footer() are actually inline functions: bool write_data(const uchar *buf, size_t data_length) { return writer->write_data(buf, data_length); } bool write_footer() { return writer->write_footer(); } So it's actually only write_header() where we now pass an extra parameter. For write_data() and write_footer() it was already being passed as the (implicit) `this` parameter. With the patch, the compiler saves the `writer` parameter in register r12 and uses it for each call: movq %rsi, %r12 ... movq %r12, %rsi # `writer` pointer movq %rbx, %rdi # `this` pointer call _ZN9Log_event12write_headerEP16Log_event_writerm@PLT ... movl $4, %edx # `data_length` parameter movq %r12, %rdi # `writer` pointer call _ZN16Log_event_writer10write_dataEPKhm@PLT ... leaq 137(%rbx), %rsi # `buf` parameter movq %r14, %rdx # `data_length` parameter movq %r12, %rdi # `writer` pointer call _ZN16Log_event_writer10write_dataEPKhm@PLT ... movq %r12, %rdi # `writer` pointer call _ZN16Log_event_writer12write_footerEv@PLT Before the patch, the code for User_var_log_event::write() loads this->writer for each call except write_header(): movq %rbx, %rdi # Here we don't pass `writer`, write_header() loads it from Log_event::writer call _ZN9Log_event12write_headerEm@PLT ... movq 88(%rbx), %rdi # this->writer leaq -126(%rbp), %rsi # `buf` parameter movl $4, %edx # `data_length` parameter call _ZN16Log_event_writer10write_dataEPKhm@PLT ... movq 88(%rbx), %rdi # this->writer leaq 153(%rbx), %rsi # `buf` parameter movq %r13, %rdx # `data_length` parameter call _ZN16Log_event_writer10write_dataEPKhm@PLT ... movq 88(%rbx), %rdi # this->writer call _ZN16Log_event_writer12write_footerEv@PLT I think the reason for the repeated loads of this->writer from 88(%rbx) is because the compiler cannot be sure that this->writer will not be modified by calls to write_data(). So it needs to reload the pointer each time. With the patch the pointer is a function parameter which cannot be modified by nested calls, so it can be saved in a register. Thus, we save some memory loads with my patch for each call of write_data() / write_footer(), which should be a bit faster (though perhaps not noticeably so). In general, passing parameters to functions is cheap (on amd64), as they are passed in registers, and register-to-register moves are very fast on high-performance CPUs, while memory loads have a relative high latency, even when they hit the cache. And like we see here, using local variables helps the compiler do alias analysis and cache values in registers, while this is a lot harder for the compiler to do for class members. - Kristian.

1 0

Re: d4c173ae6df: MDEV-27595 Backport SQL service, introduced by MDEV-19275.
by Sergei Golubchik 16 Oct '23

16 Oct '23

Hi, Alexey, On Oct 16, Alexey Botchkov wrote: > revision-id: d4c173ae6df (mariadb-10.4.30-117-gd4c173ae6df) > parent(s): 837b777bfd7 > author: Alexey Botchkov > committer: Alexey Botchkov > timestamp: 2023-09-05 13:18:12 +0400 > message: > > MDEV-27595 Backport SQL service, introduced by MDEV-19275. > > Fixes to make SQL SERVICE working. > > Conflicts: > storage/spider/spd_table.cc ^^^ better remove these two lines. normally git removes them automatically, and there's no spd_table.cc in this commit anyway. Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

1 0

Re: 837b777bfd7: MDEV-27595 Backport SQL service, introduced by MDEV-19275.
by Sergei Golubchik 16 Oct '23

16 Oct '23

Hi, Alexey, On Oct 16, Alexey Botchkov wrote: > revision-id: 837b777bfd7 (mariadb-10.4.30-116-g837b777bfd7) > parent(s): a49b9314c16 > author: Alexey Botchkov > committer: Alexey Botchkov > timestamp: 2023-09-05 13:18:12 +0400 > message: > > MDEV-27595 Backport SQL service, introduced by MDEV-19275. > > The SQL SERVICE backported into the 10.4. > > diff --git a/plugin/test_sql_service/CMakeLists.txt b/plugin/test_sql_service/CMakeLists.txt > new file mode 100644 > index 00000000000..615508bdc4e > --- /dev/null > +++ b/plugin/test_sql_service/CMakeLists.txt > @@ -0,0 +1,18 @@ > +# Copyright (C) 2013 Alexey Botchkov and SkySQL Ab > +# > +# This program is free software; you can redistribute it and/or modify > +# it under the terms of the GNU General Public License as published by > +# the Free Software Foundation; version 2 of the License. > +# > +# This program is distributed in the hope that it will be useful, > +# but WITHOUT ANY WARRANTY; without even the implied warranty of > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > +# GNU General Public License for more details. > +# > +# You should have received a copy of the GNU General Public License > +# along with this program; if not, write to the Free Software > +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1335 USA > + > +SET(SOURCES test_sql_service.c) > + > +MYSQL_ADD_PLUGIN(test_sql_service ${SOURCES} MODULE_ONLY) COMPONENT Test > diff --git a/plugin/test_sql_service/COPYING b/plugin/test_sql_service/COPYING > new file mode 100644 > index 00000000000..6e475df5526 > --- /dev/null > +++ b/plugin/test_sql_service/COPYING > @@ -0,0 +1,339 @@ > + GNU GENERAL PUBLIC LICENSE No need to, we already have one in the CMAKE_SOURCE_DIR, why do you want to copy it just for one plugin? Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

1 0

Re: About Release Model
by Kristian Nielsen 16 Oct '23

16 Oct '23

Hi Sergei, Sergei Golubchik <serg(a)mariadb.org> writes: > You've said that it's unclear how the release model works, when we > release what and where a feature can go. > > I've documented the current release model in the knowledge base, > https://mariadb.com/kb/en/mariadb-release-model/ > would you like to take a look and see if it answers your questions? This is a fine overview page, and if this info was not previously available, then it's surely good to have it documented. However, it doesn't help explain how to meaningfully participate as a core developer in the releases. Where does the discussion happen about the progress and planning of the release, and addressing problems? How to partipate in those discussions? How to push a patch into a preview release, or a release, how to make a release? Etc. My assumption is that this is all restricted to employees? But even that isn't explained on the page. - Kristian.

2 3

Question about lacking git commit message in dc9498b
by Otto Kekäläinen 12 Oct '23

12 Oct '23

Hi Sergei and others! While reviewing changes in MariaDB 10.11.5 I was wondering about this commit: https://github.com/mariadb/server/commit/dc9498beb6f79f08edd02fb7f7153e5150… It has only the default message from issuing 'git revert' and a human written explanation *why* the revert was done is missing. AFAIK the reverted commit https://github.com/MariaDB/server/commit/1d0e3d80d84c2b375a444cdbb41c28afd7… was correct and good - so why was it reverted?

2 1

Re: 6cb526bd363: MDEV-32123: require_secure_transport doesn't allow TCP connections
by Sergei Golubchik 10 Oct '23

10 Oct '23

Hi, Dmitry, Looks pretty good! Basically no comments, some minor nitpicking below, feel free to ignore. On Oct 10, Dmitry Shulga wrote: > revision-id: 6cb526bd363 (mariadb-11.2.1-9-g6cb526bd363) > parent(s): 872ed5342d8 > author: Dmitry Shulga > committer: Dmitry Shulga > timestamp: 2023-10-03 18:07:35 +0700 > message: > > MDEV-32123: require_secure_transport doesn't allow TCP connections > > In case the option require_secure_transport is on the user can't > establish a secure ssl connection over TCP protocol. Inability to set up > a ssl session over TCP was caused by the fact that a type of client's > connection was checked before ssl handshake performed (ssl handshake > happens at the function acl_authenticate()). At that moment vio type has > the value VIO_TYPE_TCPIP for client connection that uses TCP transport. > In result, checking for allowable vio type for fails despite the fact > that SSL session being established. To fix the issue move checking of > vio type for allowable values inside the function > parse_client_handshake_packet() > right after client's capabilities discovered that SSL is not requested > by the client. > > diff --git a/mysql-test/main/require_secure_transport.result b/mysql-test/main/require_secure_transport.result > index 94474d0fa0e..8fa273cbae4 100644 > --- a/mysql-test/main/require_secure_transport.result > +++ b/mysql-test/main/require_secure_transport.result > @@ -1,8 +1,13 @@ > -CREATE TABLE t1 (t int(1)); > SET GLOBAL require_secure_transport=ON; > -ERROR HY000: Connections using insecure transport are prohibited while --require_secure_transport=ON. > +ERROR 08004: Connections using insecure transport are prohibited while --require_secure_transport=ON. I wonder why sqlstate has changed now, as far as I remember it was the previous patch that changed it. Was the test skipped until now? > +SELECT (VARIABLE_VALUE <> '') AS have_ssl FROM INFORMATION_SCHEMA.SESSION_STATUS WHERE VARIABLE_NAME='Ssl_cipher'; > +have_ssl > +1 > +disconnect with_ssl; > connection default; > SET GLOBAL require_secure_transport=OFF; > +SELECT (VARIABLE_VALUE <> '') AS have_ssl FROM INFORMATION_SCHEMA.SESSION_STATUS WHERE VARIABLE_NAME='Ssl_cipher'; > +have_ssl > +0 > disconnect without_ssl; > connection default; > -DROP TABLE t1; > diff --git a/mysql-test/main/require_secure_transport.test b/mysql-test/main/require_secure_transport.test > index e238e732423..f3da73a5167 100644 > --- a/mysql-test/main/require_secure_transport.test > +++ b/mysql-test/main/require_secure_transport.test > @@ -1,15 +1,19 @@ > -- source include/have_ssl_communication.inc > -CREATE TABLE t1 (t int(1)); > SET GLOBAL require_secure_transport=ON; > --disable_query_log > --error ER_SECURE_TRANSPORT_REQUIRED > connect without_ssl,localhost,root,,,,,TCP NOSSL; > + > +connect with_ssl,localhost,root,,,,,TCP SSL; > --enable_query_log I'd suggest to remove disable/enable_query_log and let connect commmand show in the result file. It'll make them easier to read. You'll need replace_result before the failing connect to replace the port away. > +SELECT (VARIABLE_VALUE <> '') AS have_ssl FROM INFORMATION_SCHEMA.SESSION_STATUS WHERE VARIABLE_NAME='Ssl_cipher'; > +disconnect with_ssl; > + > connection default; > SET GLOBAL require_secure_transport=OFF; > --disable_query_log > connect without_ssl,localhost,root,,,,,TCP NOSSL; > --enable_query_log > +SELECT (VARIABLE_VALUE <> '') AS have_ssl FROM INFORMATION_SCHEMA.SESSION_STATUS WHERE VARIABLE_NAME='Ssl_cipher'; > disconnect without_ssl; > connection default; > -DROP TABLE t1; Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

1 0

Re: 217a7a7ceff: online alter: rework savepoints
by Sergei Golubchik 09 Oct '23

09 Oct '23

Hi, Nikita, On Sep 29, Nikita Malyavin wrote: > revision-id: 217a7a7ceff (mariadb-11.2.1-10-g217a7a7ceff) > parent(s): ad6a305d84d > author: Nikita Malyavin > committer: Nikita Malyavin > timestamp: 2023-09-23 11:46:36 +0400 > message: > > online alter: rework savepoints > > Use standard handlerton functions for savepoint add/rollback. > > To identify the savepoint, the pointer passed is used. > > Every table that has online alter in progress maintains a list of > savepoints independently. > > Also this removes setting a value to a global variable savepoint_alloc_size > without any protection, which was a race condition bug. > > --- a/sql/online_alter.cc > +++ b/sql/online_alter.cc > @@ -6,7 +6,19 @@ > > static handlerton *online_alter_hton; > > -using Online_alter_cache_list= ilist<online_alter_cache_data>; > +using sv_id_t= void *; typedef, please > + > +struct Online_alter_cache_list: ilist<online_alter_cache_data> > +{ > + sv_id_t savepoint_id= 0; > +}; > @@ -281,13 +308,11 @@ static int online_alter_log_init(void *p) > { > online_alter_hton= (handlerton *)p; > online_alter_hton->db_type= DB_TYPE_ONLINE_ALTER; > - online_alter_hton->savepoint_offset= sizeof(my_off_t); > + online_alter_hton->savepoint_offset= 1; you can even use 0, I guess. Otherwise it'll still be 8, as the server will need to align allocation, I suppose. But I suggest to try to swap the implementation. Make online_alter_hton->savepoint_offset=sizeof(void*); and in savepoint_set create a list or an array of all offsets and store it in the savepoint. Something like this: struct { uint savepoints; my_off_t log_prev_pos[1]; } oasv; int online_alter_savepoint_set(handlerton *hton, THD *thd, oasv **sv) { uint count= 0, i=0; for (auto &cache: cache_list) count++; DBUG_ASSERT(count); *sv= malloc(sizeof(oasv) + sizeof(my_off_t)*(count - 1)); (*sv)->savepoints= count; for (auto &cache: cache_list) (*sv)->log_prev_pos[i]= cache.get_byte_position(); } and so on. You just need to make sure you always use push_back and not push_front for adding caches to the cache_list. instead of a structure with a counter you can use an array with MY_OFF_T_UNDEF (not 0!) at the end. > online_alter_hton->close_connection= online_alter_close_connection; > > - online_alter_hton->savepoint_set= // Done by online_alter_savepoint_set > - [](handlerton *, THD *, void *){ return 0; }; > - online_alter_hton->savepoint_rollback= // Done by online_alter_savepoint_rollback > - [](handlerton *, THD *, void *){ return 0; }; > + online_alter_hton->savepoint_set= online_alter_savepoint_set; > + online_alter_hton->savepoint_rollback= online_alter_savepoint_rollback; > online_alter_hton->savepoint_rollback_can_release_mdl= > [](handlerton *hton, THD *thd){ return true; }; Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 2

Re: ad6a305d84d: online alter: use thd->ha_data to store cache_list
by Sergei Golubchik 09 Oct '23

09 Oct '23

Hi, Nikita, On Sep 29, Nikita Malyavin wrote: > revision-id: ad6a305d84d (mariadb-11.2.1-9-gad6a305d84d) > parent(s): 132c56f1325 > author: Nikita Malyavin > committer: Nikita Malyavin > timestamp: 2023-09-23 11:46:36 +0400 > message: > > online alter: use thd->ha_data to store cache_list nice > diff --git a/sql/online_alter.cc b/sql/online_alter.cc > index 3255fc06c3e..5ad2156cd21 100644 > --- a/sql/online_alter.cc > +++ b/sql/online_alter.cc > @@ -6,6 +6,7 @@ > > static handlerton *online_alter_hton; > > +using Online_alter_cache_list= ilist<online_alter_cache_data>; please, use typedef for that > class online_alter_cache_data: public Sql_alloc, public ilist_node<>, > public binlog_cache_data > @@ -246,7 +268,11 @@ int online_alter_savepoint_rollback(THD *thd, LEX_CSTRING name) > > static int online_alter_close_connection(handlerton *hton, THD *thd) > { > - DBUG_ASSERT(thd->online_alter_cache_list.empty()); > + Ha_data &ha= thd->ha_data[online_alter_hton->slot]; > + auto *cache_list= (Online_alter_cache_list*)ha.ha_ptr; why not thd_get_ha_data()? and don't forget thd_set_ha_data(thd, online_alter_hton, 0); THD can be reused, I suppose, so let's play safe. > + > + DBUG_ASSERT(!cache_list || cache_list->empty()); > + delete cache_list; > return 0; > } Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 3

Re: MariaDB 10.11.5 mtr --big results from amd64/arm64/armhf/ppc64el/riscv64/s390x
by Otto Kekäläinen 09 Oct '23

09 Oct '23

Hi! > > I just wanted to share the summary of mariadb-test-run with --big from > > all available architectures on Launchpad > > (https://launchpad.net/~otto/+archive/ubuntu/mariadb/+builds?build_text=&bui…) > > I have tried to fix type_test_double and maria.maria-purge in 10.6. > With 'tried', I mean that I have not been able to repeat the test, but > I have a clue what is going on. > I will push later today. > Any chance you can try to run bb-10.6-monty on launchpad to test my fixes? I cherry-picked these two commits: https://github.com/MariaDB/server/commit/8e093d7597664290b0b5e0eb401b3a180f… https://github.com/MariaDB/server/commit/630cb99823f8e83545ffa16d60c9ac75cd… .. and pushed to https://launchpad.net/~otto/+archive/ubuntu/mariadb/+builds?build_text=&bui… and indeed both tests are now fixed: maria.maria-purge w1 [ pass ] 2134 type_test.type_test_double w1 [ pass ] 59 (from https://launchpadlibrarian.net/691041067/buildlog_ubuntu-mantic-amd64.maria…) Thanks Monty! Last failing that remains after Monty's and Daniel's fixes is: - unit.conc_connection: CONC-640/MDEV-30885

1 0

MariaDB 10.11.5 mtr --big results from amd64/arm64/armhf/ppc64el/riscv64/s390x
by Otto Kekäläinen 07 Oct '23

07 Oct '23

Hi! I just wanted to share the summary of mariadb-test-run with --big from all available architectures on Launchpad (https://launchpad.net/~otto/+archive/ubuntu/mariadb/+builds?build_text=3D&= build_state=3Dall): s390x Completed: Failed 13/5644 tests, 99.77% were successful. Failing test(s): innodb.row_size_error_log_warnings_3 disks.disks_notembedded type_test.type_test_double maria.maria-purge unit.conc_connection main.func_json_notembedded riscv64 Completed: Failed 6/6289 tests, 99.90% were successful. Failing test(s): disks.disks_notembedded type_test.type_test_double unit.conc_connection ppc64el Completed: Failed 9/6291 tests, 99.86% were successful. Failing test(s): disks.disks_notembedded type_test.type_test_double maria.maria-purge unit.conc_connection armhf Too many failed: Failed 19/1767 tests, 98.92% were successful. Failing test(s): federated.federatedx_versioning spider/bg.basic_sql spider/bg.spider_fixes_part spider.basic_sql spider.spider_fixes_part versioning.online versioning.partition period.versioning arm64 completed: Failed 9/6291 tests, 99.86% were successful. Failing test(s): disks.disks_notembedded type_test.type_test_double maria.maria-purge unit.conc_connection amd64 Completed: Failed 9/6291 tests, 99.86% were successful. Failing test(s): disks.disks_notembedded type_test.type_test_double maria.maria-purge unit.conc_connection As you can see, these seem to affect all architectures: - maria.maria-purge: MDEV-31349, easily repeatable failure on all architectures on Debian CI, Salsa CI and Launchpad - type_test.type_test_double: MDEV-22243 wasn't actually fixed yet, and failure visible on all architectures and platforms tested - unit.conc_connection: CONC-640/MDEV-30885 visible on all architectures tested - disk.disk{_notembedded}: MDEV-21587 Of these, Daniel Black is currently fixing the item 4 in the list (disk.disk) at https://github.com/MariaDB/server/pull/2777. For the items 1-3 help would be needed. The armhf situation does not map into a single MDEV. Do we have any Raspberry Pi enthusiasts here who would like to run the mariadb-test-run suite on their Pi and try to fix some of the failures? Overall it is nice to see nearly 6k tests pass on so many architectures! The Debian builds at https://buildd.debian.org/status/package.php?mariadb only run post-build the main suite (~1060 tests), and some architectures don't run any tests (e.g. m68k), just the build, but still nice to see the architecture support situation being very good at the moment for 10.11.5 across official and in-official Debian architectures as well :) Many thanks in particular to Daniel, Marko, Sergei, Tuukka, Faustin and Andrew for helping to hunt for fixes to the MariaDB code and tests to have wide architecture support! - Otto

1 0