- developers - lists.mariadb.org

Re: d20e4e10353: MDEV-36536 Add option to not collect statistics for long char/varchars
by Sergei Golubchik 23 Apr '25

23 Apr '25

Hi, Michael, On Apr 22, Michael Widenius wrote: > revision-id: d20e4e10353 (mariadb-10.6.21-79-gd20e4e10353) > parent(s): 4f0a0dbd104 > author: Michael Widenius > committer: Michael Widenius > timestamp: 2025-04-22 10:53:08 +0300 > message: > > MDEV-36536 Add option to not collect statistics for long char/varchars > > This is needed to make it easy for users to automatically ignore long > char and varchars when using ANALYZE TABLE PERSISTENT. These fields > can cause problems as they will consume 'CHARACTERS * > MAX_CHARACTER_LENGTH * 2 * number_of_rows' space on disk during > analyze, which can easily be much bigger than the analyzed table. > > This commit adds a new user variable, analyze_max_length, default > value 4G. Any field that is bigger than this in bytes, will be ignored > by ANALYZE TABLE PERSISTENT unless it is specified in FOR COLUMNS(). > > While doing this patch, I noticed that we do not skip GEOMETRY columns > from ANALYZE TABLE, like we do with BLOB. This should be fixed when > merging to the 'main' branch. At the same time we should add a > resonable default value for analyze_max_length, probably 1024, like we > have for max_sort_length. > > diff --git a/mysql-test/main/mysqld--help.test b/mysql-test/main/mysqld--help.test > index 971983fd66c..44449cd3ad5 100644 > --- a/mysql-test/main/mysqld--help.test > +++ b/mysql-test/main/mysqld--help.test > @@ -20,14 +20,14 @@ exec $MYSQLD_BOOTSTRAP_CMD --symbolic-links=0 --log-bin=foo --lower-case-table-n > > perl; > # Variables which we don't want to display in the result file since > - # their paths may vary: > + # their paths may vary or they may use a default of 4294967295 : why? what's wrong with 4294967295? > @skipvars=qw/basedir open-files-limit general-log-file log plugin-dir plugin-maturity > log-slow-queries pid-file slow-query-log-file log-basename > datadir slave-load-tmpdir tmpdir socket thread-pool-size > large-files-support lower-case-file-system system-time-zone > collation-server character-set-server log-tc-size table-cache > table-open-cache table-open-cache-instances max-connections > - server-uid tls-version version.*/; > + server-uid tls-version version.* analyze-max-length/; > > # Plugins which may or may not be there: > @plugins=qw/innodb archive blackhole federated partition s3 > diff --git a/sql/sql_class.h b/sql/sql_class.h > index 69b021cd41d..aac433181a4 100644 > --- a/sql/sql_class.h > +++ b/sql/sql_class.h > @@ -731,6 +731,7 @@ typedef struct system_variables > ha_rows select_limit; > ha_rows max_join_size; > ha_rows expensive_subquery_limit; > + ulong analyze_max_length; don't use ulong, please. uint or ulonglong. > ulong auto_increment_increment, auto_increment_offset; > #ifdef WITH_WSREP > /* > diff --git a/sql/sys_vars.cc b/sql/sys_vars.cc > index 0d4a8e4c6fe..4ae14a58feb 100644 > --- a/sql/sys_vars.cc > +++ b/sql/sys_vars.cc > @@ -445,6 +445,20 @@ static Sys_var_double Sys_analyze_sample_percentage( > CMD_LINE(REQUIRED_ARG), VALID_RANGE(0, 100), > DEFAULT(100)); > > +/* > + The max length have to be UINT_MAX32 to not remove GEOMETRY fields > + from analyze. > +*/ > + > +static Sys_var_ulong Sys_analyze_max_length( > + "analyze_max_length", > + "Fields that require more storage than analyze_max_length and are not " better "Fields which length in bytes more than this are skipped by ANALYZE TABLE PERSISTENT unless explicitly listed in the FOR COLUMNS () clause" "require more storage" is not very clear > + "listed in ANALYZE ... FOR COLUMNS () will automatically be skipped by " > + "ANALYZE TABLE PERSISTENT", > + SESSION_VAR(analyze_max_length), > + CMD_LINE(REQUIRED_ARG), VALID_RANGE(32, UINT_MAX32), > + DEFAULT(UINT_MAX32), BLOCK_SIZE(1)); > + > static Sys_var_ulong Sys_auto_increment_increment( > "auto_increment_increment", > "Auto-increment columns are incremented by this", > Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 1

Remaining InnoDB parts for MDEV-34705 binlog-in-engine?
by Kristian Nielsen 21 Apr '25

21 Apr '25

Hi Marko, I have been making some good progress on the new binlog implementation, I implemented all of what we discussed regarding the InnoDB side I believe, and also got a lot done on the integration with the server side code. I wanted to ask you what outstanding issues on the InnoDB side remain from your point of view, if any? The page format now is implemented in the new design, as described in a separate mail: https://lists.mariadb.org/hyperkitty/list/developers@lists.mariadb.org/thre… All the InnoDB-specific page overhead is gone, only a 4-byte CRC32 remains at the end of each page. The first page is reserved as a header page. The new binlog code no longer uses the InnoDB buffer pool, nor create any InnoDB tablespace structures (except for the two reserved tablespace IDs for the redo records). The code has its own buffer pool replacement (called the "page fifo") and own background flushing. The binlog hooks into the InnoDB crash recovery code following your suggestions, and just the single "memcpy" record is needed. Recovery is implemented (and tested somewhat), and has been carefully designed to recover in all cases (barring any bugs of course) while requiring only the two reserved tablespace IDs. I have two outstanding InnoDB-related issues I know about: 1. In order for a replication setup to be crash-safe, I must ensure that the slave does not receive an event until it has become durable on the master. Otherwise, when the master crashes and comes back up, an event might be applied on the slave that no (longer) exist on the master. For this, I need something in the InnoDB redo log code that will callback into the binlog after a specific LSN has been durably written to disk. This callback will then notify the dump threads (slave connections) somehow that the corresponding part of the binlog can now be safely read. 2. Integration with mariabackup. With the new binlog, mariabackup can backup the binlog together with the rest of the InnoDB data in a consistent way, something that I think is not possible currently? This will also remove the need for mariabackup to extract the binlog position with SHOW MASTER STATUS or for InnoDB to store the binlog position in the undo segments. When the binlog is backed up, the correct GTID position can be obtained after restoring a master backup simply by checking the binlog (SELECT @@gtid_binlog_pos). (For a slave backup, the position is similarly available from the backup of the mysql.gtid_slave_pos table, as before). I am not yet sure of the details of mariabackup support. On the high level, it will involve copying the binlog-NNNNNN.ibb into the backup along with the other InnoDB tablespace files; and applying redo records to those binlog files as part of --prepare. One important detail is to handle that we have only two reserved tablespace IDs for binlog files. The current recovery code requires that only the two most recent binlog files need recovery. We could try to ensure that the same applies duing mariabackup --prepare. An idea is to use the fact that binlog files are strictly append-only. Around the point where mariabackup decides the LSN where it stops copying the redo log that becomes the backup snapshot, we can determine the last binlog file that exists. We will check this just before and just after determining the LSN, and repeat if a new binlog file was created just at that point (which would otherwise leave doubt if the file was created before or after that LSN). Then after the snapshot LSN is determined, we can just copy all the binlog files up to and including (but not after) the snapshot point. This way mariabackup --prepare should be able to run the same binlog recovery code that runs in crash recovery, only needing to consider the two most recent of the copied binlog files. I can imagine other ways to do mariabackup of binlog files, but this is one idea that could be feasible. What do you think? Anything other outstanding issues with the new binlog implementation regarding InnoDB that you are aware of? - Kristian.

1 0

Design of new binlog file format (MDEV-34705)
by Kristian Nielsen 21 Apr '25

21 Apr '25

As the MDEV-34705 binlog-in-engine project matures, I'm finalising the file format of the new binlog, and will present it here in case of any final comments or request for changes. The file format of course is an important part of the design, as file formats in general are more difficult to change going forward than many other aspects of the code. Following previous discussions on the list with Marko, the file format of the new binlog is now simplified, and mostly independent of InnoDB. The InnoDB buffer pool is no longer used for binlog files, and InnoDB recovery uses separate code to recover binlog pages than other tablespace pages. The binlog implementation reserves two special tablespace IDs in the redo log to denote binlog files, but otherwise binlog files share little code with other InnoDB tablespaces. 1. Binlog file names -------------------- The innodb binlog files have a fixed naming scheme: binlog-000000.ibb binlog-000001.ibb binlog-000002.ibb ... By default they are written to the data directory, but a separate directory can be configured with --binlog-directory. This, there is no longer any need for a binlog index file. New binlog files are pre-allocated to the configured --max-binlog-size in the background. Unused pages contain zeros until written to. Each binlog file is always written strictly append-only, bytes are never overwritten or written out-of-order. 2. Binlog file header --------------------- Each binlog file is page-based. The page size is 4k, fixed in the initial release, but planned to be configurable in a future release. Pages are checksummed (same algorithm as InnoDB full_crc32). The last four bytes of each page contain the CRC32, rest is available for data (no special page header is needed). The first page of the file is reserved as a file header. It contains the following data (little-endian format): Offset Length Data 0 4 "Magic" file identifier (0x010dfefe) 4 4 Log2 of page size (ie. 12 for 4k pages) 8 4 Major version number (0) 12 4 Minor version number (0) 16 8 The file number from the binlog name, consistency check 24 8 File length, in number of pages 32 8 LSN corresponding to the start of this binlog file 40 8 The "GTID state interval" N. Used to speed up GTID lookup. 48 8 File number containing out-of-band data referenced from this file. 56 8 File number containing XA data referenced from this file (for future expansion to support XA). 508 4 Extra crc32 checksum, to support down to 512 bytes page size. The idea with version numbers is that new minor versions are readable by older code, while new major versions are not and older code must refuse to read the file. The LSN in the header is used to do recovery correctly. Any redo record with smaller LSN will be ignored during recovery and not applied to this file. This way only two tablespace IDs are needed for recovery of binlog files. The GTID state interval (--innodb-binlog-state-interval) is a replacement for the GTID indexes of the legacy binlog. Every N pages in a binlog file, the current binlog GTID state is written. This allows to quickly find a GTID position for the slave to start at, by doing binary search on the GTID state records and then scanning ahead over the last few (<=N) pages to find the starting position. The first GTID state in the file stores the full state, the remaining pages only store the difference to the first state. The out-of-band (OOB) file number reference is used to manage when large transactions are written into the binlog files interleaved with other transaction data. If a large transaction starts binlogging in binlog-000005.ibb say, and then completes and writes the commit record to file binlog-000008.ibb, then the file header for file_no=8 will contain oob-reference=5, to mark that file_no=5 is needed by dump threads(slave connections) reading from file_no=8. An extra crc32 checksum is stored at offset 512 in the header page. This is to be later able to support 512-byte (or larger) configurable page size. This makes it possible to read the first 512 bytes of the file, verify the integrity, and read the actual page size at offset 4. The initial release is planned to have fixed 4k page size. 3. Binlog data page format -------------------------- Overall, the binlog contains a logical sequence of "records". Conceptually, the binlog is one single stream of such records; one record can span two files, and file boundary is not meaningful for the binlog content. Each binlog record is written as a single InnoDB mini-transaction (mtr). This means binlog records will be recovered atomically during InnoDB crash recovery, a partial record will not be found at the end of a binlog file. It also means the size of one record is limited by the maximum size of an mtr. Each record is split into one or more "chunks". A chunk always fits in one page. Each binlog page (apart from the first header page) contains one or more chunks. Then follows 0-3 filler bytes 0xff to pad any remaining space, and last the 4 bytes of CRC32 checksum. Here is the byte layout of a page: Offset Length Data 0 len+3 <type> <len1> <len2> <data 0> <data 1> ... <data len-1> len+3 A+3+B+3 <type> <len1> <len2> <data 0> <data 1> ... <data len-1> ... 4089 1 <filler (optional)> 4090 1 <filler (optional)> 4091 1 <filler (optional)> 4092 4 <CRC0> <CRC1> <CRC2> <CRC3> Thus, each chunk has a simple format with one byte denoting the chunk type, two bytes (little-endian) denoting the length, and then the data. Chunk types: 0 Empty (not yet used data, has no length field) 1 Commit record 2 GTID state record, ie. for --innodb-binlog-state-interval 3 Out-of-band binlog data for large transactions 4 Dummy record (used to fill up the last page when FLUSH BINARY LOGS) 0xff Filler byte, to pad page when there is no room for a whole chunk (no length field). Here for example is a commit record, type=1: 00001000: 41c8 0000 918e 0168 a201 0000 A......h.... 00001010: 0026 0000 0000 0000 0008 0001 0000 0000 .&.............. 00001020: 0000 0000 0000 0029 0000 0000 0000 918e .......)........ 00001030: 0168 0201 0000 00a1 0000 0000 0000 0000 .h.............. 00001040: 0007 0000 0000 0000 0004 0000 2300 0000 ............#... 00001050: 0000 0101 0000 2054 0000 0000 0603 7374 ...... T......st 00001060: 6404 0800 0800 0800 818c 0000 0000 0000 d............... 00001070: 0074 6573 7400 4352 4541 5445 2054 4142 .test.CREATE TAB 00001080: 4c45 2074 3120 2861 2049 4e54 204e 4f54 LE t1 (a INT NOT 00001090: 204e 554c 4c2c 2062 2049 4e54 204e 4f54 NULL, b INT NOT 000010a0: 204e 554c 4c2c 2063 2054 4558 542c 2050 NULL, c TEXT, P 000010b0: 5249 4d41 5259 204b 4559 2861 2c20 6229 RIMARY KEY(a, b) 000010c0: 2920 454e 4749 4e45 3d49 6e6e 6f44 42 ) ENGINE=InnoDB The 0x41 is the type, and the 0xc8 0x00 is the length=0x00c8. Then follows the raw event data, in the same format used in the existing binlog (sql/log_event.h). The type byte has two additional flag bits (hence 0x41 and not 0x01): Bit 6 (0x40) This is the last chunk of a record Bit 7 (0x80) This is a continuation chunk (not first) of a record Thus a commit record can consist of a single chunk: type=0x41 , or two chunks: type=0x01 type=0xc1 , or more than two chunks: type=0x01 type=0x81 type=0x81 ... type=0xc1 Chunks are thus used to split a record across pages (a page will not contain more than one chunk for a single record). 4. Chunk data contents ---------------------- The commit record, type=1, has the following header data, consisting of 5 64-bit numbers: num_oob Number of out-of-band records in this transaction oob_first_file_no The file_no of the first oob record oob_first_offset The offset into oob_first_file_no oob_last_file_no The file_no of the last oob record oob_last_offset The offset into oob_last_file_no For small event groups with no out-of-band data, only num_oob=0 is stored. The numbers are stored in a compressed format that saves space for small numbers, see storage/innobase/include/ut0compr_int.h After follows the raw binlog event data. For a small transaction with no out-of-band records, this is the whole event group (GTID, Query/rows, XID/Commit). For large transactions it is just the GTID event. See above for example commit record. The GTID state record, type=2, is just a list of GTIDs, similar to the existing Gtid_list_log_event: number_of_gtids domain_id_0, server_id_0, seq_no_0 domain_id_1, server_id_1, seq_no_1 ... domain_id_N-1, server_id_N-1, seq_no_N-1 Like for commit records, the numbers are stored compressed to save space for small numbers. Here is an example GTID state record containing one GTID 0-1-575 (see ut0compr_int.h for how to interpret the compressed numbers 0x8 0x0 0x8 0x11f9 as 1, 0, 1, 575): 00001000: 4205 0008 0008 f911 B....... 00001010: The OOB data record, type=3, stores one node in a binary tree of event group data. The out-of-band data is structured as binary trees in a way that can be written strictly append-only, and read efficiently by dump threads/connected slaves. The OOB node is defined by 5 numbers: index The identification of the node (nodes are numbered 0, 1, 2...) left_file_no The file_no of the left child of the node left_offset The offset into left_file_no of the left child. Zero denotes a leaf node. right_file_no The file_no of the right child of the node right_offset The offset into the right_file_no. After this follows the raw event data. 5. Conclusion ------------- So this is how the new binlog format is planned to look, barring any last-minute design change requirements or design review comments. I tried to include a good amount of details, feel free to ask for any of the specific details that I did omit, if interested. Hope this helps, - Kristian.

1 0

Re: [JIRA] (MDEV-36629) Change SLAVE to REPLICA in Server error messages
by Kristian Nielsen 18 Apr '25

18 Apr '25

Taking this discussion out of the MDEV-36629 specifics, as this is a general issue. It concerns a problem with how documentation is now being incorrectly updated: "Alex Hanshaw (Jira)" <jira(a)mariadb.org> writes: > Alex Hanshaw commented on MDEV-36629: > Understood. For context Vikas had told the Doc team "I understand the > compatibility issue - so we need to make both options work and > document only the new terminology." When the Doc team asked about the Ouch. This will need to be reverted ASAP, before more damage is done. The documentation is for documenting how the server works. All aspects of how it works. Not some distorted version, whether good- or ill-intended. We must certainly _not_ deliberately remove documentation of existing syntax and functionality! For example I see that documentation of the syntax START SLAVE was removed. This is just plain wrong. START SLAVE is working syntax in replication, and will remain so for the foreseeable future. It needs to be documented for users to be able to understand how it works. Please take this seriously, and ensure that such changes are stopped immediately, and all incorrect changes reverted. - Kristian. >> Change SLAVE to REPLICA in Server error messages >> ------------------------------------------------ >> >> Key: MDEV-36629 >> URL: https://jira.mariadb.org/browse/MDEV-36629 >> Project: MariaDB Server >> Issue Type: Task >> Components: Replication >> Reporter: Shwetha CM >> Assignee: Sergei Golubchik >> Priority: Major >> Fix For: N/A

2 1

Re: 6d4479d0e32: MDEV-36565 Assertion `src != ((void *)0)' failed in my_casedn_8bit
by Sergei Golubchik 16 Apr '25

16 Apr '25

Hi, Alexander, I don't know, this looks like a strange place to fix this. I'd say that either Item_func_xml_extractvalue shouldn't return {0, 0} or, if we'll treat it as a valid String, caseup() and casedn() should be able to handle it. But in a similarly-looking bug MDEV-35945 Assertion `src != ((void *)0)' failed in my_caseup_8bit you fixed it to use { "", 0 } for an empty string. It's commit c69fb1a6273. Then I'd suggest to do the same here. Let's avoid {0, 0} ? On Apr 16, Alexander Barkov wrote: > revision-id: 6d4479d0e32 (mariadb-11.4.5-25-g6d4479d0e32) > parent(s): d3c9a2ee212 > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2025-04-15 19:30:44 +0400 > message: > > MDEV-36565 Assertion `src != ((void *)0)' failed in my_casedn_8bit > > diff --git a/sql/item_strfunc.cc b/sql/item_strfunc.cc > index 2e3c33059d3..5df25c7f237 100644 > --- a/sql/item_strfunc.cc > +++ b/sql/item_strfunc.cc > @@ -1998,7 +1998,16 @@ String *Item_str_conv::val_str(String *str) > goto err; > > null_value= false; > - len= converter(collation.collation, (char*) res->ptr(), res->length(), > + /* > + Avoid passing nullptr as the second parameter to converter(): > + - val_str() of some Items (e.g. Item_func_xml_extractvalue) can return > + Strings with {Ptr=0,str_length=0} meaning an empty string. > + - But my_charset_handler_st virtual function caseup() and casedn() > + (which "converter" point to) do not expect {src=nullptr,srclen=0} input. > + */ > + DBUG_ASSERT(res->ptr() != nullptr || res->length() == 0); > + len= !res->ptr() ? 0/*Avoid passing nullptr*/ : > + converter(collation.collation, (char*) res->ptr(), res->length(), > (char*) str->ptr(), alloced_length); > DBUG_ASSERT(len <= alloced_length); > str->set_charset(collation.collation); > Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

1 0

Re: MDEV-36290: ALTER TABLE with multi-master can cause data loss (PR #3967)
by Kristian Nielsen 16 Apr '25

16 Apr '25

Brandon Nesterenko <notifications(a)github.com> writes: > MDEV-36290: ALTER TABLE with multi-master can cause data loss > https://github.com/MariaDB/server/pull/3967 Hi Brandon, Here is my initial review of the MDEV-36290 draft patch. Overall, it is clear that this is not something that's appropriate for 10.6. It implements a new way to apply row events, based on column names (if available) rather than column number. It is something for next dev version, 12.x. > commit de27bfbafc22601d53ddc0567401fcb26d28a91c > Author: Brandon Nesterenko <brandon.nesterenko(a)mariadb.com> > Date: Thu Apr 3 12:37:09 2025 -0600 > > MDEV-36290: ALTER TABLE with multi-master can cause data loss Please do not use the term "data loss" in the description (I know it is not you who introduced the term ;-). The issue here is conflicting queries run in parallel against multiple masters in a multi-master setup. In general, the support of multi-master in asynchronous replication requires the user to ensure that no conflicting queries are run, or to ensure that such conflicts will not cause undesirable behaviour. The behaviour addressed here is just one of many ways that conflicting ALTER TABLE could cause different data on slave and master. Using the term "data loss" can cause misunderstanding about what the problem is and what the change does. Of course, every case that is handled more correctly is in principle better. In this case, we introduce the facility to configure (--binlog_row_metadata=FULL) replication so that it is possible to replicate between tables on master and slave with different order of the columns. Not sure how useful such an option is in practice, seems to be very few users that will really need this, and such need has to be weighed against the increase in complexity in the code and potential for bugs. With that said, the approach in the patch generally looks reasonable. A few initial detail comments below, as you mentioned this is still a draft patch. As you also mentioned to me earlier, the logic in the patch is not trivial. I did not yet try to understand in detail all parts of the logic and the different corner cases. > + unsigned char *get_optional_metadata_str() > + { > + return m_optional_metadata; > + } > + > + unsigned int get_optional_metadata_len() > + { > + return m_optional_metadata_len; > + } > + Generally I do not like such do-nothing accessor functions. They just add abstraction and reduce grepability without doing anything useful. That's a style preference of course. > + /* > + Maps column index from master to slave. This is determined using the field > + names (provied by optional metadata when the master is configured with > + binlog_row_metadata=FULL). > + */ > + std::map<uint, uint> master_to_slave_index_map; > + > + /* > + When using field names to map from master->slave columns, this keeps track > + of column indices on the master which aren't present on the slave. > + > + It is used to skip columns when checking type-conversions and unpacking > + row data. > + */ > + std::set<uint> master_unmatched_cols; Is using std::map/std::set going to be expensive here, this is operations that will happen on every row-based event (when --binlog-row-metadata=FULL)? The std:: functions can imply a lot of memory allocations that are not very visible, and we usually try to avoid them in performance-relevant server code. master_to_slave_index_map is just a lookup table of numbers 0..#columns-1, right? So it doesn't need to be an associative array, a normal array is sufficient, and doesn't need to be re-allocated each event. And master_unmatched_cols is again just a set of numbers 0..#columns-1, so a bitmap could be used. - Kristian.

1 0

CI failures visible on GitHub (affecting contributors)
by Otto Kekäläinen 15 Apr '25

15 Apr '25

Hi, I see that on the 11.8 branch (https://github.com/MariaDB/server/commits/11.8/) which is supposed to be fully green and passing CI as it is soon to be released as a new LTS version, actually isn't consistently green. Out of the latest 10 CI runs the jobs passing rates are: 12/!2 11/12 12/!2 12/!2 8/8 10/12 8/12 7/8 8/8 11/12 Questions: - What is the current plan to get CI to be consistently green? - How is it possible that a commit is pushed on the 11.8 branch even though CI is not passing? Isn't branch protection preventing it? - If there is a way to get commits on 11.8 branch with failures bypassing branch protection, are all developers committed to follow-up on their commits and fix them before they do any other work? Or what is the plan for how to ensure post-push test failures are fixed? - The jobs buildbot/amd64-debian-11-msan-clang-16 and continuous-integration/appveyor/branch seem to be the most common failing ones. Can they be removed now to make the CI consistently green? And apply full branch protection so that they, or any other jobs, can't be added back to unless they are actually green? - Why is the CI occasionally tunning 8 jobs instead of 12? I also see plenty of red at https://buildbot.mariadb.org/#/grid?branch=11.8. Hopefully we get as much green as possible for the start of the 11.8 maintenance so that any potential regressions in future stable maintenance releases of 11.8 series are easier to catch. Most of the issues seem to be test issues and not real MariaDB bugs. I have also done a bunch of testing on Debian QA systems for MariaDB 11.8 and I have only found a couple small issues, so overall 11.8 is looking good! - Otto

2 1

MariaDB 11.8.1 fully compatible with Y2038
by Otto Kekäläinen 09 Apr '25

09 Apr '25

Hi, I just wanted to report that I tested building MariaDB in 2025 and running MTR in 2037 and 2038 using libfaketime[1]. I can confirm no test started failing in 2038, thus MariaDB had no Y2038 issues in my testing. [1] https://salsa.debian.org/mariadb-team/mariadb-server/-/merge_requests/74

1 0

MariaDB Memory Map
by o.halaiev＠gmail.com 08 Apr '25

08 Apr '25

Hi MariaDB team I'm Oleksii Halaev, studying Python and DataScience I'm currently working on a GSoC 2025 proposal and would like to contribute to MariaDB with an idea I'm developing - it's called ‘MariaDB Memory Map’ The idea is to create an interactive visualisation tool that shows how data changes in the MariaDB database over time - how it is added, modified, deleted and how it ‘travels’ between tables Think of it as a sort of ‘life map’ of your data I believe this can be a useful tool for developers and DBAs to gain a deeper understanding of their databases, make debugging easier, and even track down data anomalies Here is my suggestion: https://drive.google.com/file/d/1vOMF0MZRTueEKkavWdwvEIP6_t2UgJLA/view?usp=… I'd really appreciate any feedback, advice or thoughts - and of course, if there's anyone out there who might be interested in mentoring this project, I'd be happy to chat and learn more Thanks for your time! I look forward to hearing from you GitHub: https://github.com/Alekcentijules Email: o.halaiev(a)gmail.com

2 1

Re: 0ed7370d133: MDEV-24726 Assertion `0' failed in Field_varstring_compressed::key_cmp
by Sergei Golubchik 04 Apr '25

04 Apr '25

Hi, Aleksey, Despite the email subject, this is a combined diff of three commits, git diff 071c20fd23b2 2fecd944d6dc On Apr 04, Aleksey Midenkov wrote: > diff --git a/sql/field.cc b/sql/field.cc > index 8b96c473ab6..d6be66cf399 100644 > --- a/sql/field.cc > +++ b/sql/field.cc > @@ -2516,7 +2521,7 @@ Field *Field::new_key_field(MEM_ROOT *root, TABLE *new_table, > uchar *new_null_ptr, uint new_null_bit) > { > Field *tmp; > - if ((tmp= make_new_field(root, new_table, table == new_table))) > + if ((tmp= make_new_field({root, new_table, table == new_table}))) it's a creative trick, I admit, but let's not do it. looks very weird, we don't have it anywhere else. it's not needed either, because you don't need to pass down a new argument, see below > { > tmp->ptr= new_ptr; > tmp->null_ptr= new_null_ptr; > @@ -8278,12 +8281,28 @@ int Field_varstring::cmp_binary(const uchar *a_ptr, const uchar *b_ptr, > } > > > -Field *Field_varstring::make_new_field(MEM_ROOT *root, TABLE *new_table, > - bool keep_type) > +Field *Field_varstring::make_new_field(make_new_field_args args) > { > - Field_varstring *res= (Field_varstring*) Field::make_new_field(root, > - new_table, > - keep_type); > + Field_varstring *res= (Field_varstring*) Field::make_new_field(args); > + if (res) > + res->length_bytes= length_bytes; > + return res; > +} > + > + > +Field *Field_varstring_compressed::make_new_field(make_new_field_args args) > +{ > + Field_varstring *res; > + if (args.tmp_field) tmp field is a field of a tmp table. Unfortunately, INTERNAL_TMP_TABLE value of new_table->s->tmp_table is very overloaded and no longer means internal temporary table anymore. I think we'll need to fix it someday. But a smaller fix for today could be, like @@ -1113,6 +1113,11 @@ struct TABLE_SHARE return (tmp_table == SYSTEM_TMP_TABLE) ? 0 : table_map_id; } + bool is_optimizer_tmp_table() + { + return tmp_table == INTERNAL_TMP_TABLE && !db.length && table_name.length; + } + using the fact that optimizer tmp tables are all named "(temporary)" or something like that and have no db. Using this one can do a fix, like @@ -8281,11 +8279,23 @@ int Field_varstring::cmp_binary(const uchar *a_ptr, con> Field *Field_varstring::make_new_field(MEM_ROOT *root, TABLE *new_table, bool keep_type) { - Field_varstring *res= (Field_varstring*) Field::make_new_field(root, + Field_varstring *res; + if (new_table->s->optimizer_tmp_table()) + { + res= new (root) Field_varstring(ptr, field_length, length_bytes, null_ptr, + null_bit, Field::NONE, &field_name, + new_table->s, charset()); + if (res) + res->init_for_make_new_field(new_table, orig_table); + } + else + { + res= (Field_varstring*) Field::make_new_field(root, new_table, keep_type); if (res) res->length_bytes= length_bytes; + } return res; } > + { > + /* Compressed fields can't have keys (see Field_varstring_compressed::key_cmp()). */ > + res= new (args.root) Field_varstring(*this); > + res->init_new_field(args); > + /* See Column_definition::create_length_to_internal_length_string() */ > + res->field_length--; > + } > + else > + res= (Field_varstring*) Field::make_new_field(args); > if (res) > res->length_bytes= length_bytes; > return res; Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

1 0

MDEV-36107: Proposal for Enhancements in MariaDB Operator Handling
by Harshit Vijay Kumar 02 Apr '25

02 Apr '25

Dear MariaDB Developer Community, I hope you are doing well. I am *Harshit Vijay Kumar*, a final-year undergraduate student in Computer Science. I have been researching *MDEV-36107 <https://jira.mariadb.org/browse/MDEV-36107>* for *GSoC 2025* and, during my exploration of operator operations in MariaDB, I came across some behaviors that caught my attention. To further investigate, I compared these behaviors with MySQL to understand their differences. *Observations: * *(1) Handling Large Number Multiplications Using Exponents* - Currently, multiplying two large integers results in an *overflow*, returning 0 instead of using an exponent-based representation. SELECT 18014398509481984 * 18014398509481984; *MariaDB Output:* 0 *SQL Output:* 3.2451855365842673e+32 - Interestingly, if one of the operands is a *floating-point number*, the result is computed without overflow: SELECT 18014398509481984 * 18014398509481984399405765456789087654390954.0; *Output:* 324518553658426733978210646499548233026655740664701555572736.0 - This inconsistency between integer-only and mixed-type operations is confusing. A standardized approach, possibly leveraging exponent notation, could improve clarity. *(2) Rounding Floating-Point Outputs* - MariaDB currently returns full precision in floating-point calculations, which can lead to unnecessary precision in practical use cases. SELECT 1.0376728938476536278903984765 + 1.039876453672890938476536278902345678908765434567890; *MariaDB Output:* 2.07754934752054456636693475540234567891 *SQL Output:* 2.077549347520544 - A rounding mechanism would allow for concise representation and reduce storage overhead, especially for users unaware of excessive precision in their datasets. *(3) Eliminating Unnecessary Zeros in Floating-Point Representation* SELECT 1.0000 * 1.0000000000; *MariaDB Output:* 1.00000000000000 - While useful in some cases, the inclusion of excessive trailing zeros unnecessarily increases storage usage. A more compact representation should be considered, where only *significant digits* are retained. *(4) Implementation of == Equality Operator* - Currently, MariaDB allows the *assignment operator (=)* for comparisons. However, since = is primarily used for assignments, this dual-purpose behavior can be confusing. - Proposal: Introduce a dedicated *== equality operator* for clear and intuitive comparisons across *INT, FLOAT, BOOL, and CHAR* types. - Expected behavior: - If values match → return 1 - If values don’t match → return 0 Testing & Next Steps To implement these improvements, I plan to add *test cases* in mysql-test covering all edge cases for these operators. I will follow the approach used in the following implementations: - func_equal.result <https://github.com/MariaDB/server/blob/main/mysql-test/main/func_equal.resu…> - func_equal.test <https://github.com/MariaDB/server/blob/main/mysql-test/main/func_equal.test> I would really appreciate your insights on these ideas. If any of these behaviors have intentional design choices, I would love to understand the rationale behind them. Looking forward to your feedback! Best Regards, Harshit Vijay Kumar

1 0

Re: 0a0c5f349c6: MDEV-36425 Extend read_only to also block share locks and super user
by Sergei Golubchik 02 Apr '25

02 Apr '25

Hi, Monty, On Apr 01, Michael Widenius wrote: > revision-id: 0a0c5f349c6 (mariadb-11.8.1-42-g0a0c5f349c6) > parent(s): 21321025e5e > author: Michael Widenius > committer: Michael Widenius > timestamp: 2025-03-31 20:09:03 +0300 > message: > > MDEV-36425 Extend read_only to also block share locks and super user > > The main purpose of this allow one to use the --read-only > option to ensure that no one can issue a query that can > block replication. > > The --read-only option can now take 4 different values: > 0 No read only (as before). > 1 Blocks changes for users without the 'READ ONLY ADMIN' > privilege (as before). > 2 Blocks in addition LOCK TABLES and SELECT IN SHARE MODE > for not 'READ ONLY ADMIN' users. > 3 Blocks in addition 'READ_ONLY_ADMIN' users for all the > previous statements. > diff --git a/mysql-test/main/read_only.result b/mysql-test/main/read_only.result > index 65cc12ffce9..89390ad65d5 100644 > --- a/mysql-test/main/read_only.result > +++ b/mysql-test/main/read_only.result > @@ -69,11 +69,14 @@ set global read_only=1; > connection con1; > select @@global.read_only; > @@global.read_only > -0 > +OFF > unlock tables ; > +Timeout in wait_condition.inc for SELECT @@global.read_only= 1 > +Id User Host db Command Time State Info Progress > +5 test localhost test Query 0 starting show full processlist 0.000 Forgot to fix? It should wait for @@global.read_only='READ_ONLY' now > select @@global.read_only; > @@global.read_only > -1 > +READ_ONLY > connection default; > reap; > connection default; > diff --git a/sql/lock.cc b/sql/lock.cc > index 1296e5455d5..8f9f97f88b7 100644 > --- a/sql/lock.cc > +++ b/sql/lock.cc > @@ -179,19 +198,48 @@ lock_tables_check(THD *thd, TABLE **tables, uint count, uint flags) > > /* > Prevent modifications to base tables if READ_ONLY is activated. > - In any case, read only does not apply to temporary tables. > + In any case, read only does not apply to temporary tables or slave > + threads. > */ > - if (!(flags & MYSQL_LOCK_IGNORE_GLOBAL_READ_ONLY) && !t->s->tmp_table) > + if (unlikely(opt_readonly) && > + !(flags & MYSQL_LOCK_IGNORE_GLOBAL_READ_ONLY) && !t->s->tmp_table && > + !thd->slave_thread) > { > - if (t->reginfo.lock_type >= TL_FIRST_WRITE && > - !ignore_read_only && opt_readonly && !thd->slave_thread) > + switch (opt_readonly) // Impossible you have "Impossible" comment on a wrong line > { > - my_error(ER_OPTION_PREVENTS_STATEMENT, MYF(0), "--read-only"); > - DBUG_RETURN(1); > + case READONLY_OFF: > + DBUG_ASSERT(0); > + break; > + case READONLY_ON: // Compatibility > + if (!(thd->security_ctx->master_access & PRIV_IGNORE_READ_ONLY) && > + t->reginfo.lock_type >= TL_FIRST_WRITE) > + { > + mariadb_error_read_only(); > + DBUG_RETURN(1); I'd make mariadb_error_read_only() to return 1. Then you can write DBUG_RETURN(mariadb_error_read_only()); here and also in all other places, check it out, it's always followed by return 1. > + } > + break; > + case READONLY_NO_LOCK: > + if (!(thd->security_ctx->master_access & PRIV_IGNORE_READ_ONLY) && > + (thd->lex->sql_command == SQLCOM_LOCK_TABLES || > + t->reginfo.lock_type >= TL_FIRST_WRITE || > + t->reginfo.lock_type == TL_READ_WITH_SHARED_LOCKS)) may be t->reginfo.lock_type >= TL_READ_WITH_SHARED_LOCKS ? > + { > + mariadb_error_read_only(); > + DBUG_RETURN(1); > + } > + break; > + case READONLY_NO_LOCK_NO_ADMIN: > + if (thd->lex->sql_command == SQLCOM_LOCK_TABLES || > + t->reginfo.lock_type >= TL_FIRST_WRITE || > + t->reginfo.lock_type == TL_READ_WITH_SHARED_LOCKS) and here > + { > + mariadb_error_read_only(); > + DBUG_RETURN(1); > + } > + break; > } > } > } > - > /* > Locking of system tables is restricted: > locking a mix of system and non-system tables in the same lock > diff --git a/sql/sql_class.h b/sql/sql_class.h > index d65cec37154..34ee8391009 100644 > --- a/sql/sql_class.h > +++ b/sql/sql_class.h > @@ -3448,9 +3452,12 @@ class THD: public THD_count, /* this must be first */ > */ > inline bool is_read_only_ctx() rename to, for example, check_read_only_with_errror() > { > - return opt_readonly && > - !(security_ctx->master_access & PRIV_IGNORE_READ_ONLY) && > - !slave_thread; > + if (likely(!opt_readonly) || slave_thread || > + ((security_ctx->master_access & PRIV_IGNORE_READ_ONLY) && > + opt_readonly != READONLY_NO_LOCK_NO_ADMIN)) > + return false; > + mariadb_error_read_only(); > + return true; > } > > private: > diff --git a/sql/sys_vars.cc b/sql/sys_vars.cc > index 0247afd91a6..545b4674da1 100644 > --- a/sql/sys_vars.cc > +++ b/sql/sys_vars.cc > @@ -3253,15 +3253,24 @@ static bool fix_read_only(sys_var *self, THD *thd, enum_var_type type) > transition (especially when transitioning from false to true) and > synchronizes both booleans in the end. > */ > -static Sys_var_on_access_global<Sys_var_mybool, > + > +const char *read_only_mode_names[]= > +{"OFF", "READ_ONLY", "READ_ONLY_NO_LOCK", "READ_ONLY_NO_LOCK_NO_ADMIN"}; > + > +static Sys_var_on_access_global<Sys_var_enum, > PRIV_SET_SYSTEM_GLOBAL_VAR_READ_ONLY> > Sys_readonly( > "read_only", > - "Make all non-temporary tables read-only, with the exception for " > - "replication (slave) threads and users with the 'READ ONLY ADMIN' " > - "privilege", > - GLOBAL_VAR(read_only), CMD_LINE(OPT_ARG), DEFAULT(FALSE), > - NO_MUTEX_GUARD, NOT_IN_BINLOG, > + "Do not allow changes to non-temporary tables. Options are:" > + "OFF changes allowed. " > + "READ_ONLY blocks changes for users without the 'READ ONLY ADMIN' " "Disallow changes for users without the READ ONLY ADMIN privilege" > + "privilege. " > + "READ_ONLY_NO_LOCK blocks in addition LOCK TABLES and " > + "SELECT IN SHARE MODE for not ADMIN users. " "Additionally disallows LOCK TABLES and SELECT IN SHARE MODE" > + "READ_ONLY_NO_LOCK_NO_ADMIN blocks in addition 'ADMIN' users. " "Disallows also for users with READ_ONLY ADMIN privilege" > + "Replication (slave) threads are not affected by this option", > + GLOBAL_VAR(read_only), CMD_LINE(OPT_ARG), > + read_only_mode_names, DEFAULT(0), NO_MUTEX_GUARD, NOT_IN_BINLOG, > ON_CHECK(check_read_only), ON_UPDATE(fix_read_only)); > > // Small lower limit to be able to test MRR Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 1

Inquiry About GSoC 2025 and Proposal Submission
by Harsh Gupta 31 Mar '25

31 Mar '25

*Dear MariaDB Developers,* I hope you are doing well. My name is *Harsh Gupta*, and I am a *second-year B.Tech Computer Science and Design student at Madhav Institute of Technology and Science, Gwalior*. I am passionate about databases and open-source development, and I am eager to contribute to MariaDB through *Google Summer of Code 2025*. I have gone through the project ideas listed for GSoC and found the MDEV-28395 <https://jira.mariadb.org/browse/MDEV-28395> LOAD DATA plugins particularly interesting. I have experience with *C++ and database systems*, and I have been exploring MariaDB’s codebase to get familiar with its architecture. I have prepared a draft proposal outlining my approach to this project, including my implementation plan and timeline. I would appreciate your feedback to refine it further before submitting it officially. Could you please review my proposal and provide any suggestions for improvement? Also, I would love to connect with potential mentors to discuss the project's scope and feasibility. Looking forward to your guidance and feedback! Best regards, *Harsh Gupta* 📧 harshgupta9329(a)gmail.com

1 0

Re: b7dbe187b1a: MDEV-34680 Asynchronous and Buffered Logging for Audit Plugin.
by Sergei Golubchik 31 Mar '25

31 Mar '25

Hi, Alexey, See few comments below. But the main question, could you have used FILE or IO_CACHE instead of a custom implementation? On Mar 08, Alexey Botchkov wrote: > revision-id: b7dbe187b1a (mariadb-11.8.1-8-gb7dbe187b1a) > parent(s): 6f1161aa34c > author: Alexey Botchkov > committer: Alexey Botchkov > timestamp: 2025-03-02 16:27:23 +0400 > message: > > MDEV-34680 Asynchronous and Buffered Logging for Audit Plugin. > > Buffering added to the file logger. > Two buffers are created so writing to the file doesn't > block other threads from logging. > Operations isolated with two mutexes. > diff --git a/include/mysql/service_logger.h b/include/mysql/service_logger.h > index 35c2eb1e3a9..adce7bc9038 100644 > --- a/include/mysql/service_logger.h > +++ b/include/mysql/service_logger.h > @@ -64,41 +64,44 @@ extern struct logger_service_st { > void (*logger_init_mutexes)(); > LOGGER_HANDLE* (*open)(const char *path, > unsigned long long size_limit, > - unsigned int rotations); > + unsigned int rotations, size_t buffer_size); incompatible API change. MUST change the major service version. > int (*close)(LOGGER_HANDLE *log); > int (*vprintf)(LOGGER_HANDLE *log, const char *fmt, va_list argptr) > ATTRIBUTE_FORMAT_FPTR(printf, 2, 0); > int (*printf)(LOGGER_HANDLE *log, const char *fmt, ...) > ATTRIBUTE_FORMAT_FPTR(printf, 2, 3); > - int (*write)(LOGGER_HANDLE *log, const char *buffer, size_t size); > + int (*write)(LOGGER_HANDLE *log, const void *data, size_t size); > int (*rotate)(LOGGER_HANDLE *log); > + int (*sync)(LOGGER_HANDLE *log); > } *logger_service; > > #ifdef MYSQL_DYNAMIC_PLUGIN > > #define logger_init_mutexes logger_service->logger_init_mutexes > -#define logger_open(path, size_limit, rotations) \ > - (logger_service->open(path, size_limit, rotations)) > +#define logger_open(path, size_limit, rotations, buffer_size) \ > + (logger_service->open(path, size_limit, rotations, buffer_size)) > #define logger_close(log) (logger_service->close(log)) > #define logger_rotate(log) (logger_service->rotate(log)) > +#define logger_rsync(log) (logger_service->sync(log)) > #define logger_vprintf(log, fmt, argptr) (logger_service->\ > vprintf(log, fmt, argptr)) > #define logger_printf (*logger_service->printf) > -#define logger_write(log, buffer, size) \ > - (logger_service->write(log, buffer, size)) > +#define logger_write(log, data, size) \ > + (logger_service->write(log, data, size)) > #else > > void logger_init_mutexes(); > LOGGER_HANDLE *logger_open(const char *path, > unsigned long long size_limit, > - unsigned int rotations); > + unsigned int rotations, size_t buffer_size); > int logger_close(LOGGER_HANDLE *log); > int logger_vprintf(LOGGER_HANDLE *log, const char *fmt, va_list argptr) > ATTRIBUTE_FORMAT(printf, 2, 0); > int logger_printf(LOGGER_HANDLE *log, const char *fmt, ...) > ATTRIBUTE_FORMAT(printf, 2, 3); > - int logger_write(LOGGER_HANDLE *log, const char *buffer, size_t size); > + int logger_write(LOGGER_HANDLE *log, const void *data, size_t size); > int logger_rotate(LOGGER_HANDLE *log); > + int logger_sync(LOGGER_HANDLE *log); > #endif > > > diff --git a/mysys/file_logger.c b/mysys/file_logger.c > index a753c049f68..5464e43eddf 100644 > --- a/mysys/file_logger.c > +++ b/mysys/file_logger.c > @@ -145,85 +230,108 @@ static int do_rotate(LOGGER_HANDLE *log) > namebuf[log->path_len]= 0; > result= my_rename(namebuf, logname(log, log->path, 1), MYF(0)); > log->file= my_open(namebuf, LOG_FLAGS, MYF(0)); > + log->in_file= 0; > exit: > errno= my_errno; > return log->file < 0 || result; > } > > > -/* > - Return 1 if we should rotate the log > -*/ > - > -my_bool logger_time_to_rotate(LOGGER_HANDLE *log) > -{ > - my_off_t filesize; > - if (log->rotations > 0 && > - (filesize= my_tell(log->file, MYF(0))) != (my_off_t) -1 && > - ((ulonglong) filesize >= log->size_limit)) > - return 1; > - return 0; > -} > - > - > int logger_vprintf(LOGGER_HANDLE *log, const char* fmt, va_list ap) > { > - int result; > char cvtbuf[1024]; > size_t n_bytes; > > - flogger_mutex_lock(&log->lock); > - if (logger_time_to_rotate(log) && do_rotate(log)) > - { > - result= -1; > - errno= my_errno; > - goto exit; /* Log rotation needed but failed */ > - } > - > n_bytes= my_vsnprintf(cvtbuf, sizeof(cvtbuf), fmt, ap); > if (n_bytes >= sizeof(cvtbuf)) > n_bytes= sizeof(cvtbuf) - 1; > > - result= (int)my_write(log->file, (uchar *) cvtbuf, n_bytes, MYF(0)); > - > -exit: > - flogger_mutex_unlock(&log->lock); > - return result; > + return logger_write(log, cvtbuf, n_bytes); > } > > > -static int logger_write_r(LOGGER_HANDLE *log, my_bool allow_rotations, > - const char *buffer, size_t size) > +int logger_write(LOGGER_HANDLE *log, const void *data, size_t size) > { > int result; > + size_t in_buffer; > + > + mysql_mutex_lock(&log->lock); > > - flogger_mutex_lock(&log->lock); > - if (allow_rotations && logger_time_to_rotate(log) && do_rotate(log)) > + if (log->buffer_left >= size) > + { > + memcpy(log->buffer_pos, data, size); > + log->buffer_pos+= size; > + log->buffer_left-= size; > + mysql_mutex_unlock(&log->lock); > + return size; > + } > + > + mysql_mutex_lock(&log->lock_buffer); > + > + in_buffer= log->buffer_pos - log->buffer; > + > + { /* swap buffers. */ > + uchar *tmp_buf= log->buffer; > + log->buffer= log->idle_buffer; > + log->idle_buffer= tmp_buf; > + } > + > + set_buffer_pos_and_left(log); > + > + /* Now other threads can write to the new buffer. */ > + mysql_mutex_unlock(&log->lock); > + > + if ((in_buffer && > + my_write(log->file, log->idle_buffer, in_buffer, MYF(0)) != in_buffer)|| > + my_write(log->file, (uchar *) data, size, MYF(0)) != size) > { > result= -1; > errno= my_errno; > - goto exit; /* Log rotation needed but failed */ > + goto exit_buf; > } > + else > + result= 0; > > - result= (int)my_write(log->file, (uchar *) buffer, size, MYF(0)); > + log->in_file+= in_buffer + size; > + if (log->rotations > 0 && log->in_file >= log->size_limit) > + { > + if (do_rotate(log)) > + { > + result= -1; > + errno= my_errno; > + goto exit_buf; /* Log rotation needed but failed */ > + } > + } > > -exit: > - flogger_mutex_unlock(&log->lock); > +exit_buf: > + mysql_mutex_unlock(&log->lock_buffer); > return result; > } > > > -int logger_write(LOGGER_HANDLE *log, const char *buffer, size_t size) > -{ > - return logger_write_r(log, TRUE, buffer, size); > -} > - > int logger_rotate(LOGGER_HANDLE *log) > { > int result; > - flogger_mutex_lock(&log->lock); > - result= do_rotate(log); > - flogger_mutex_unlock(&log->lock); > + mysql_mutex_lock(&log->lock); > + mysql_mutex_lock(&log->lock_buffer); > + result= flush_buffer(log) || do_rotate(log); I'd though you can switch buffers here too and flush/rotate only under the lock_buffer mutex > + mysql_mutex_unlock(&log->lock_buffer); > + mysql_mutex_unlock(&log->lock); > + return result; > +} > + > + > +int logger_sync(LOGGER_HANDLE *log) Why did you introduce an explicit sync API? > +{ > + int result; > + mysql_mutex_lock(&log->lock); > + mysql_mutex_lock(&log->lock_buffer); > + if ((result= flush_buffer(log))) > + { > + result= fsync(log->file); > + } > + mysql_mutex_unlock(&log->lock_buffer); > + mysql_mutex_unlock(&log->lock); > return result; > } > > diff --git a/plugin/server_audit/server_audit.c b/plugin/server_audit/server_audit.c > index 5fc0e3373f3..e51df7ab6b5 100644 > --- a/plugin/server_audit/server_audit.c > +++ b/plugin/server_audit/server_audit.c > @@ -1393,26 +1401,15 @@ static void change_connection(struct connection_info *cn, > static int write_log(const char *message, size_t len, int take_lock) > { > int result= 0; > + > if (take_lock) > - { > - /* Start by taking a read lock */ > mysql_prlock_rdlock(&lock_operations); > - } > > if (output_type == OUTPUT_FILE) > { > if (logfile) > { > - my_bool allow_rotate= !take_lock; /* Allow rotate if caller write lock */ > - if (take_lock && logger_time_to_rotate(logfile)) > - { > - /* We have to rotate the log, change above read lock to write lock */ > - mysql_prlock_unlock(&lock_operations); > - mysql_prlock_wrlock(&lock_operations); > - allow_rotate= 1; > - } > - if (!(is_active= (logger_write_r(logfile, allow_rotate, message, len) == > - (int) len))) > + if (!(is_active= (logger_write(logfile, message, len) == (int) len))) nice, rotation is indeed the logger's headache, not the caller's > { > ++log_write_failures; > result= 1; > @@ -2899,6 +2906,36 @@ static void update_file_rotations(MYSQL_THD thd __attribute__((unused)), > } > > > +static void update_file_buffer_size(MYSQL_THD thd __attribute__((unused)), > + struct st_mysql_sys_var *var __attribute__((unused)), > + void *var_ptr __attribute__((unused)), const void *save) > +{ > + file_buffer_size= *(unsigned int *) save; > + > + error_header(); > + fprintf(stderr, "Log file buffer size was changed to '%d'.\n", file_buffer_size); > + > + if (!logging || output_type != OUTPUT_FILE) > + return; > + > + ADD_ATOMIC(internal_stop_logging, 1); > + if (!maria_55_started || !debug_server_started) > + mysql_prlock_wrlock(&lock_operations); > + > + stop_logging(); > + if (start_logging()) this could be an API call in the logger. Like "reinit(new params)" and it'll close, change params, open - all under a log mutex, which doesn't need to be destroyed and reinitialized. > + { > + error_header(); > + fprintf(stderr, "Logging was disabled..\n"); > + CLIENT_ERROR(1, "Logging was disabled.", MYF(ME_WARNING)); > + } > + > + if (!maria_55_started || !debug_server_started) > + mysql_prlock_unlock(&lock_operations); > + ADD_ATOMIC(internal_stop_logging, -1); > +} > + > + > static void update_file_rotate_size(MYSQL_THD thd __attribute__((unused)), > struct st_mysql_sys_var *var __attribute__((unused)), > void *var_ptr __attribute__((unused)), const void *save) Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 1

Introduction & Interest in GSoC 2025 with MariaDB
by H.NAVEENKUMAR 30 Mar '25

30 Mar '25

*Hello MariaDB Developers,* I hope you're all doing well. My name is *Naveen Kumar*, and I am currently a student with a strong interest in open-source development, databases, and backend engineering. I have experience working with *Python, JavaScript, MySQL, MongoDB, and full-stack development*. I recently registered for *Google Summer of Code (GSoC) 2025* with MariaDB and am eager to contribute to the project. I am still exploring the available project ideas and would appreciate guidance on how to choose the best fit based on my skills. It would be great if mentors or experienced contributors could suggest beginner-friendly issues or key areas to focus on while getting familiar with the MariaDB codebase. Looking forward to learning, collaborating, and making meaningful contributions to the community! Best regards, *Naveen Kumar*

2 1

MariaDB 11.8.1 in Debian
by Otto Kekäläinen 23 Mar '25

23 Mar '25

Hi, I just wanted to let you know that MariaDB 11.8.1 just landed in Debian testing, which will be the upcoming Debian 13 "Trixie" release. I will continue with various quality assurance tasks so that eventually when 11.8.2 comes out it will be as polished as possible from Debian/Ubuntu perspective. Hopefully .2 does not have any big changes, but just bug fixes.

2 1

Prepared statements analytics problem
by Dmitrii Kochetov 19 Mar '25

19 Mar '25

Most database monitoring systems, such as Newrelic, Datadog, Dolphie, Ottertune, Releem, Solarwinds, use data from the performance_schema database from the events_statements_summary_by_digest and events_statements_* tables to get query analytics for databases. This information allows to calculate system performance metrics, in particular Latency. Also, the data from these tables allows you to get a complete picture of the database operation - what queries are executed, their execution statistics, to get query templates based on digest_text, and based on SQL_TEXT field from events_statements_* or QUERY_SAMPLE_TEXT from events_statements_summary_by_digest (for MySQL >= 8) to get a sample query for digest and optimize it. The prepared statements analytics are contained in a single table prepared_statements_instances, which is managed by Performance_schema as follows: - Statement preparation If the statement is successfully instrumented, a new row is added to the prepared_statements_instances table. If the statement cannot be instrumented, Performance_schema_prepared_statements_lost status variable is incremented. - Prepared statement execution Execution of a EXECUTE command for an instrumented prepared statement instance updates the corresponding prepared_statements_instances table row. - Prepared statement deallocation Execution of a CLOSE command for an instrumented prepared statement instance removes the corresponding prepared_statements_instances table row. To avoid resource leaks, removal occurs even if the prepared statement instruments described previously are disabled. In this regard, there are the following problems with analytics for prepared statements compared to regular queries: 1) Most applications work with prepared statements using the following method when processing a single query - creating a prepared statement, executing the prepared statement, and then dropping the prepared statement. This leads to the fact that query statistics exist for a very short period of time, during which it may not be possible to fetch them. 2) The second problem of prepared statement analytics in comparison with regular queries is the absence of the query hash (digest) and information about the schema_name in which the prepared statement is executed, which makes it difficult to identify unique queries, since a separate record in the Prepared_Statements_instances table is created for each creation of a prepared statement. 3) Execution of prepared statements is possible in two protocols: Binary and text. Execution of a prepared statement in the binary protocol does not create an entry in the events_statements_* tables, but execution in the text protocol creates an entry in the specified tables, which leads to different behaviors of the same prepared statement but in different protocols. 4) For prepared statements, there is no option in getting an example request without a ? placeholder. The SQL_TEXT field in Prepared_Statements_instances contains The prepared statement text, with ? placeholder markers, and in some cases (when executing in the text protocol) it is possible to find only the execute command of this statement, which further requires mapping ? placeholder markers with parameters from the execute request. But there are limitations that the binary protocol does not store EXECUTE requests for prepared statements, and most of the interactions with the database occur through it, through drivers for different programming languages, and another limitation may be the use of variables in the EXECUTE command, the value of which still needs to be obtained. Proposed solution: 1) Records should not be deleted from the Prepared_Statements_instances table after dropping prepared statements. If the table is full, the counter Performance_schema_prepared_statements_lost and If no row has the statement digest value for the statement that just completed, and the table is full, the statistics for the statement that just completed are added to a special “catch-all” row with DIGEST = NULL, which is created if necessary. 2) Each statement must be unique by the fields schema_name, in which the given statement is executed and its hash, digest, which will be the same for the same prepared statement. 3) In the tables events_statements_* must be recorded information about the fact of PREPARE, EXECUTE and DEALLOCATE execution for prepared statement, where in the DIGEST_TEXT field is stored the call of the execute command, and in SQL_TEXT - the normalized prepared query, in which ? placeholder markers are replaced by concrete values.

1 1

MariaDB 11.8(.1) to Debian
by Otto Kekäläinen 15 Mar '25

15 Mar '25

Hi, Just FYI, I am done with the February maintenance releases to Debian and Ubuntu, and this weekend started working on importing MairaDB 11.8(.1) to Debian. Draft is visible at https://salsa.debian.org/mariadb-team/mariadb-server/-/merge_requests/107

1 2

Using GitHub directly from inside Emacs
by Otto Kekäläinen 15 Mar '25

15 Mar '25

Hi! I don't use Emacs myself, but I learned last weekend at FOSDEM that some Emacs users love using the Emacs extension Magit [1] with the Forge [2] feature to list and review GitHub Pull Requests directly from Emacs without opening a browser. Just curious to know if any Emacs users here (Monty?) have tried Magit for interacting with https://github.com/MariaDB/server/pulls? There are currently 240 open PRs and 157 have zero reviews [3] from people with GitHub permissions to give reviews. To get more contributors, we need to have more people participating in reviewing/approving incoming contributions. Also some parts of the contributions process are not very optimal, and the process is likely to become smoother only if enough of current core developers at least occasionally "eat their own cooking" and experience (at least parts) of the contribution process. - Otto [1] https://magit.vc/ [2] https://magit.vc/manual/forge.html [3] https://github.com/MariaDB/server/pulls?q=is%3Apr+is%3Aopen+review%3Anone

2 3

Review of MDEV-35915 Implement Global temporary tables, part 2
by Michael Widenius 12 Mar '25

12 Mar '25

Review of Global temporary tables, part 2 commit a9478ca44b44118cf70de493569d64afbb5cd925 Author: Nikita Malyavin <nikitamalyavin(a)gmail.com> Date: Fri Nov 22 02:15:56 2024 +0100 Fix deallocation of Sql_cmd_dml::result Made as a part of MDEV-34309 Sql_cmd_dml::result is using mem_root to control memory allocation, but Sql_alloc allows to mark the memory as freed via destructor. Normal runs have DTRACE disabled, so accesses in MYSQL_DML_DONE are turned into no-op, and that is the bug is not shown. Enabling DTRACE brings a crash yet in a bootstrap. The fix is to carefully set Sql_cmd_dml::result to nullptr whenever it is freed, or is marked as freed. Please include the title of MDEV-354309 in the commit description @@ -34253,6 +34253,7 @@ bool Sql_cmd_dml::prepare(THD *thd) MYSQL_DML_START(thd); + result= nullptr; // It could've been set since the last run. lex->context_analysis_only|= CONTEXT_ANALYSIS_ONLY_DERIVED; if (open_tables_for_query(thd, lex->query_tables, &table_count, 0, How +++ b/sql/sql_delete.cc @@ -1882,6 +1882,7 @@ bool Sql_cmd_delete::execute_inner(THD *thd) { select_result *sel_result= NULL; delete result; + result= nullptr; /* This is DELETE ... RETURNING. It will return output to the client */ if (thd->lex->analyze_stmt) { @@ -1929,10 +1930,7 @@ bool Sql_cmd_delete::execute_inner(THD *thd) } if (result) - { res= false; - delete result; - } Why remove the delete. If the delete may free wrong things, then the newly added result is also likely to free wrong things. Anyway, the code is clearly wrong here. One cannot set res to 0 if res is 1. This means we are ignoring a possible error from the upper level (probably OOM) and we are probably going to assert later in the code that checks if we can send ok. Try to remove the seteting of res= false and run mtr. Same logics goes for all changes in Sql_cmd_update::execute_inner(THD *thd diff --git a/sql/sql_update.cc b/sql/sql_update.cc if (unlikely(res)) + if (unlikely(res) && result) result->abort_result_set(); else { The above is wrong. Should be: + if (unlikely(res)) { if (result) result->abort_result_set(); } else { As otherwise you may call send_explain() when you shouldn't ------------------ Suggestion: - Move delete of result to unprepare. This will ensure that result is always 0 on Sql_cmd_dml::prepare() and excute_innert and you can replace delete result with 'DBUG_ASSERT(result == 0)' in most of your changes. - Remove setting res= 0 if we have a result and check with mtr. Conclusion: - As this is not cricial and you are going on vacation, I will take over this issue --------- commit 6a000835f5f58aba221ab89c120adae4be16abf3 Author: Nikita Malyavin <nikitamalyavin(a)gmail.com> Date: Thu Feb 20 16:12:37 2025 +0100 fix show create diff --git a/sql/sql_show.cc b/sql/sql_show.cc index da0081fd1f8..201aa8e3110 100644 --- a/sql/sql_show.cc +++ b/sql/sql_show.cc @@ -1413,8 +1413,13 @@ mysqld_show_create(THD *thd, TABLE_LIST *table_list) if (table_list->table && table_list->table->s->tmp_table != NO_TMP_TABLE) { TMP_TABLE_SHARE *share= (TMP_TABLE_SHARE*)table_list->table->s; - if (share->from_share && table_list->table->file->records() == 0) - thd->drop_temporary_table(table_list->table, NULL, true); + if (share->from_share) + { + if (table_list->table->file->info(HA_STATUS_VARIABLE) + || (table_list->table->file->records() == 0 + && thd->drop_temporary_table(table_list->table, NULL, true))) + error= TRUE; + } Please move || and && to previous lines so we have the test lined up nicely in the editor. This is how most of the MariaDB code is written. commit 55ff0fb18bbe4539db27ea73cc2b795e53521772 Author: Nikita Malyavin <nikitamalyavin(a)gmail.com> Date: Thu Feb 20 16:12:49 2025 +0100 add rpl assertions if (thd->temporary_tables - && thd->temporary_tables->global_temporary_tables_count) // TODO this may be buggy on slave + && thd->temporary_tables->global_temporary_tables_count) { move && up to previous line. @@ -6051,6 +6053,7 @@ my_bool open_global_temporary_table(THD *thd, TABLE_SHARE *source, if (thd->mdl_context.clone_ticket(&share->mdl_request)) return TRUE; table->reginfo.lock_type= TL_IGNORE; + share->table_creation_was_logged= 1; } The above means that we assume that slave have the same global temporary table, which is probably not the case. Better to set share->table_creation_was_logged=0 for the local temporary tables we are creating which means that any update of the table through the GTT will use row based replication. Please also add a function comment before my_bool open_global_temporary_table(THD *thd, TABLE_SHARE *source, TABLE_LIST *out_table, MDL_ticket *mdl_ticket) commit a64bbaf483ad714493db401f68049ce85a886c96 Author: Nikita Malyavin <nikitamalyavin(a)gmail.com> Date: Thu Feb 27 01:07:45 2025 +0100 CREATE: use frm header to store GTT bits diff --git a/sql/temporary_tables.cc b/sql/temporary_tables.cc index 45bca833d17..ea46c4035ba 100644 --- a/sql/temporary_tables.cc +++ b/sql/temporary_tables.cc @@ -1324,10 +1324,10 @@ int THD::commit_global_tmp_tables() All_share_tables_list::Iterator tables_it(share->all_tmp_tables); TABLE *table= tables_it++; DBUG_ASSERT(table); - if (!share->on_commit_delete) + if (!share->on_commit_delete()) table->file->info(HA_STATUS_VARIABLE); // update records() stat - if (share->on_commit_delete || table->file->records() == 0) + if (share->on_commit_delete() || table->file->records() == 0) error= drop_tmp_table_share(share, NULL, true); } I still think it would be better to not drop the temporary file based on file->records(). The reason is that some temporary table will be slow to drop and if there are users that is using these for many queries things will be slower than they need to be. - Not critical, can be discussed and changed later. commit 21f9ac1e56d6c81a431e5bdbc9b6b89c8b565ca2 Author: Nikita Malyavin <nikitamalyavin(a)gmail.com> Date: Thu Feb 27 01:12:34 2025 +0100 minor (typo, style) fixes -struct Closefrm_context +static my_bool open_gtt_on_error(TABLE *table) { - TABLE *table; - ~Closefrm_context() { closefrm(table); } -}; + closefrm(table); + return TRUE; +} Change return to void as this should never fail. Yes, it will require a few changes in callers, but it better to do return TRUE; in the caller than have the reviewer have to look at what another function is returning. +++ b/sql/temporary_tables.cc @@ -329,7 +329,9 @@ bool THD::internal_open_temporary_table(TABLE_LIST *tl, TABLE **table, { if (tmp_share->from_share && sql_command_flags() & (CF_ADMIN_COMMAND |CF_SCHEMA_CHANGE)) - DBUG_RETURN(false); + DBUG_RETURN(false); /* We want to use real global temporary table + when ALTER/DROP is executed. + */ Move the comment before the DBUG_RETURN() commit 9e1f5413f619bf4f88eef0f0a50fb7df4b8e7d44 Author: Nikita Malyavin <nikitamalyavin(a)gmail.com> Date: Sat Mar 1 01:27:04 2025 +0100 fix ALTER TABLE long wait @@ -11676,7 +11677,7 @@ do_continue:; */ if (alter_info->requested_lock != Alter_info::ALTER_TABLE_LOCK_EXCLUSIVE && thd->mdl_context.upgrade_shared_lock(mdl_ticket, MDL_SHARED_NO_WRITE, - lock_wait_timeout)) + thd->variables.lock_wait_timeout)) goto err_new_table_cleanup; Add a comment why we should not use lock_wait_timeout here. This is needed as we are using it a few lines above and it is important for coders to know why the value is not reused here. commit f7629d745fe729ea445688f5b710a97108da7c76 Author: Nikita Malyavin <nikitamalyavin(a)gmail.com> Date: Sat Mar 1 02:09:20 2025 +0100 Don't drop local data tables for ON COMMIT PRESERVE when records()==0 connect con1,localhost,root,,; +insert t values(1); +connection default; +alter table t add j int; +ERROR HY000: Lock wait timeout exceeded; try restarting transaction +set statement lock_wait_timeout=0 for drop table t; +ERROR HY000: Lock wait timeout exceeded; try restarting transaction We have to add better error messages for known cases. "Cannot alter global temporary tables as we have open instances of it" "Cannot alter global temporary tables as there are other users for it" Can be done after vacation. diff --git a/mysql-test/main/global_temporary_table.test b/mysql-test/main/global_temporary_table.test index 0edc6eee493..17195eaf3ea 100644 --- a/mysql-test/main/global_temporary_table.test +++ b/mysql-test/main/global_temporary_table.test +--} insert into t1 values(1, 'one'); + select * from t1; No reason for having an extra empty line here. This commit is using: bool THD::use_real_global_temporary_share() const { return (sql_command_flags() & (CF_ALTER_TABLE | CF_SCHEMA_CHANGE | CF_STATUS_COMMAND) && lex->sql_command != SQLCOM_CREATE_TABLE) || lex->sql_command == SQLCOM_CREATE_VIEW; } Please add a comment and move |, && and || to the end of lines. Please also fix the indentation. You can also add a new flag CF_USE_LOCAL_GTT table and mark those commands for which using GTT will open a local GTT copy. In sql_show.cc:get_all_tables(THD *thd, TABLE_LIST *tables, COND *cond) there is this code: if (thd->use_real_global_temporary_share()) continue; This looks wrong. We are ignoring all temporary tables if the above case is true. To ignore GTT ables we have to verify that the found share is a GTT table. Shouldn't we also test for tmp_share->from_share ? Why are we testing for use_real_global_temporary_share() in this code? This is called only be DML. Shouldn't that be safe for GTT tables? For what commands are thd->use_real_global_temporary_share() not true here? commit 718afd78dae4a37ed009ab349dfe9bca68fbc3d7 Author: Nikita Malyavin <nikitamalyavin(a)gmail.com> Date: Tue Mar 4 18:30:07 2025 +0100 Extract THD::has_open_global_temporary_tables() No comments commit cc35b9cb54e74a2f0fbbf4a2400599c7fddb750d Author: Nikita Malyavin <nikitamalyavin(a)gmail.com> Date: Tue Mar 4 21:01:07 2025 +0100 Add CF_ALTER_TABLE No comments commit 381243f4e75f61a31305c656f9b6be20cec042d2 Author: Nikita Malyavin <nikitamalyavin(a)gmail.com> Date: Wed Mar 5 00:18:42 2025 +0100 add global_temporary_tables plugin, move relevant code to temporary_tables.cc + MariaDB_PLUGIN_MATURITY_STABLE Should probably not be stable ;) Please change to BETA. We just have to ensure that this does not cause problems in Jenkin. If it does, you can change it to GAMMA or STABLE (depending on what Jenkins supports) commit a7fa6f835998bc33f1528420df5fe3f1b8a5dbfd Author: Nikita Malyavin <nikitamalyavin(a)gmail.com> Date: Wed Mar 5 00:50:50 2025 +0100 refactor drop_tmp_table_share + while ((tab= it++)) { + if (tab != table && tab->query_id != 0) + { + /* Found a table instance in use. This table cannot be dropped. */ + my_error(ER_CANT_REOPEN_TABLE, MYF(0), table->alias.c_ptr()); + result= true; + goto end; + } Suggested comment change: /* Found a table instance in use. This table cannot be dropped. */ -> /* Found a table instance in use (query_id != 0). This table cannot be dropped. */ ------- commit bb9ff1b2ec45b437bd39b6dacef8885fadeb818b Author: Nikita Malyavin <nikitamalyavin(a)gmail.com> Date: Wed Mar 5 02:51:23 2025 +0100 refactor THD::internal_open_temporary_table No comments commit 72914bfd92ae7390f8e7924944c92a1dc9591e1a Author: Nikita Malyavin <nikitamalyavin(a)gmail.com> Date: Wed Mar 5 22:06:02 2025 +0100 enforce limitations + if (create_info->table_options & HA_OPTION_GLOBAL_TEMPORARY_TABLE) + { + const char *error= NULL; + for (const Key &fk: create_info->alter_info->key_list) + if (fk.type == Key::FOREIGN_KEY) + { + error= "FOREIGN KEY"; + break; + } You can replace the above loop with: if (thd->lex->alter_info_flags & ALTER_ADD_FOREIGN_KEY) Note also that in your old code, later errors could overwrite earlier ones. Changing this to a normal if, like the others, will solve that. commit a2831f8e8efcfecd548b8ced6e19c3fc4d6966f9 Author: Nikita Malyavin <nikitamalyavin(a)gmail.com> Date: Thu Mar 6 20:55:00 2025 +0100 information_schema support mysql-test/main/global_temporary_table.result +GLOBAL TEMPORARY +show table status where temporary='Y'; +Name Engine Version Row_format Rows Avg_row_length Data_length Max_data_length Index_length Data_free Auto_increment Create_time Update_time Check_time Collation Checksum Create_options Comment Max_index_length Temporary Please add a test that show the output from information_schema.tables for all fields that are stable for mtr. By the way, why does the above show table status not show any data while +select TABLE_TYPE from information_schema.tables where table_name = 't1'; +TABLE_TYPE +TEMPORARY +GLOBAL TEMPORARY Show two rows just above show table status; At least 'TEMPORARY' should have 'Temporary' set to 'Y'' commit a580c6bc85791829c850d7eedc03d62a5ee99ffd Author: Nikita Malyavin <nikitamalyavin(a)gmail.com> Date: Thu Mar 6 23:43:38 2025 +0100 VIEW support Views already open the correct tables automatically, so it just works. Almost ;) + || lex->sql_command == SQLCOM_CREATE_VIEW ; .... commit d4c00419ebb39a07f87eabf0eee3660a768cbbed Author: Nikita Malyavin <nikitamalyavin(a)gmail.com> Date: Fri Mar 7 02:16:23 2025 +0100 AS SELECT support + if (table->s->tmp_table && + table->s->db_create_options & + (HA_OPTION_GLOBAL_TEMPORARY_TABLE|HA_OPTION_ON_COMMIT_DELETE_ROWS)) + table= NULL; // It will be deleted on commit + This will not work because of the following code: if (WSREP(thd) && table->file->ht->db_type == DB_TYPE_INNODB) You either have to add a goto or fix all references to 'table'. Note this code later in the same function. if (m_plock) { MYSQL_LOCK *lock= *m_plock; *m_plock= NULL; m_plock= NULL; if (create_info->pos_in_locked_tables) { /* If we are under lock tables, we have created a table that was originally locked. We should add back the lock to ensure that all tables in the thd->open_list are locked! */ table->mdl_ticket= create_info->mdl_ticket; This should be safe as long as on cannot use LOCK TABLES on gtt tables. Please add a check for that! Another thing to check is with GALERA that local gtt tables are not replicated. commit 45e2f14bcbdd61c1ea5712f5b79b07a096aa46d2 Author: Nikita Malyavin <nikitamalyavin(a)gmail.com> Date: Mon Mar 10 20:16:16 2025 +0100 CREATE TABLE ... LIKE support sql_table.cc: local_create_info.table_options&= ~HA_OPTIONS_TO_RESET_CREATE_LIKE; Looking at the code, I belive the above is always 0 here. Please replace with DBUG_ASSERT(local_create_info.table_options == 0); commit 1c44e4dabe22555a3b517db7f67c1e4d8b03fc70 Author: Nikita Malyavin <nikitamalyavin(a)gmail.com> Date: Mon Mar 10 21:49:11 2025 +0100 RENAME support +++ b/sql/sql_parse.cc @@ -6351,6 +6351,13 @@ check_rename_table(THD *thd, TABLE_LIST *first_table, check_grant(thd, INSERT_ACL | CREATE_ACL, &new_list, FALSE, 1, FALSE))) return 1; + + if (table->table && + table->table->s->db_create_options & HA_OPTION_GLOBAL_TEMPORARY_TABLE) + { + my_error(ER_LOCK_WAIT_TIMEOUT, MYF(0)); + return 1; + } We need a better error here. Something like: "Cannot rename a global temporary tables as this connection has open instances of it" commit b0f02644c22522c325bfbddd847030d61237b4e8 Author: Nikita Malyavin <nikitamalyavin(a)gmail.com> Date: Tue Mar 11 00:48:11 2025 +0100 FLUSH support, forbid LOCK FLUSH TABLE on Global temporary table is basically no-op, since it never has open instances. - It is ok to ignore flush table on GTT tables. There is no point in doing that. Same goes for temporary tables. +/** + Check if any global temporary tables are still opened in this session. +*/ +static int flush_check_for_global_temporary_tables(THD *thd, TABLE_LIST *tl) +{ + if (thd->rgi_slave || !thd->has_open_global_temporary_tables()) + return 0; + + for (;tl; tl= tl->next_global) + { + TABLE_SHARE *share= thd->find_tmp_table_share(tl); + if (share && share->db_create_options & HA_OPTION_GLOBAL_TEMPORARY_TABLE) + { + my_error(ER_LOCK_WAIT_TIMEOUT, MYF(0)); + return 1; + } + } + return 0; +} I think you can remove the above. +flush table t; +lock table t write; +ERROR 42S02: Unknown table 't' I think it is wrong that we get an err for the lock. We should fix it so that we open the table local GTT table and do not take a lock. (In other words, it should work as normal lock tables for tmp tables). @@ -5970,6 +5970,12 @@ my_bool open_global_temporary_table(THD *thd, TABLE_SHARE *source, { DBUG_ASSERT(!thd->rgi_slave); // slave won't use global temporary tables + if (thd->lex->sql_command == SQLCOM_LOCK_TABLES) + { + my_error(ER_BAD_TABLE_ERROR, MYF(0), source->table_name.str); + return TRUE; + } + See above. We should remove the above and instead open the table normally. Author: Nikita Malyavin <nikitamalyavin(a)gmail.com> Date: Tue Mar 11 02:08:10 2025 +0100 Replication support Please do not use 'show binlog events' as this will cause result conflicts with any changes in the binlog format and server changes. You have now things like Server ver: 11.8.0-MariaDB-debug-log, Binlog ver: 4 in the test output use instead source include/show_binlog_events.inc; One can also does not need to use ../.. to access main tests. Here is a diff for the test: +++ b/mysql-test/suite/binlog/t/global_temporary_table.test @@ -4,7 +4,7 @@ set @old_binlog_annotate_row_events= @@global.binlog_annotate_row_events; set global binlog_annotate_row_events= OFF; set session binlog_annotate_row_events= OFF; -source ../../main/global_temporary_table.test; +source main/global_temporary_table.test; +source include/show_binlog_events.inc; -show binlog events; set global binlog_annotate_row_events= @old_binlog_annotate_row_events; ----- +bool is_user_tmp_table(TMP_TABLE_SHARE *share) +{ + if (share->db_create_options & HA_OPTION_GLOBAL_TEMPORARY_TABLE) + return false; + return share->tmp_table == TRANSACTIONAL_TMP_TABLE || + share->tmp_table == NON_TRANSACTIONAL_TMP_TABLE; +} The normal way to test for temporary table is: share->tmp_table != NO_TMP_TABLE; see is_temporary_table() This should be safe also for the above function. Add () around the return functions, to get editors to indent it correctly by default. I think the above function needs some comments. Please explain how things are: if (share->db_create_options & HA_OPTION_GLOBAL_TEMPORARY_TABLE) is true, this is a GTT definition table and tmp_table would be NO_TMP_TABLE. If this is a local GTT tables, then (share->db_create_options & HA_OPTION_GLOBAL_TEMPORARY_TABLE) is false and share->from_share is true. It would be good if the above function would have good enough comments to explain all the above. We need to describe exactly how things are in some please in the source. Could you add the description how to detect GTT tables and local GTT tables and how they differ from normal temporary tables at the start of temporary_tables.cc Especially how share->db_create_options and share->tmp_table are set. If you add this function first in tempoary_table.cc, it may explain how things are: bool is_user_tmp_table(TMP_TABLE_SHARE *share) { if (share->db_create_options & HA_OPTION_GLOBAL_TEMPORARY_TABLE) { /* This is either a global temporary table (GTT) definition table or local GTT table created from the GTT definition. If (share->tmp_table != NO_TMP_TABLE) then this is local GTT table. In this case share->from_share is also set and points to the global GTT definition. */ return false; } /* Return true of the table is non GTT temporary table */ return (share->tmp_table != NO_TMP_TABLE) }

1 0

Re: [PATCH] Fix redundant ER_PRIOR_COMMIT_FAILED in parallel replication
by Kristian Nielsen 11 Mar '25

11 Mar '25

Hi Monty, thanks for review! Michael Widenius <michael.widenius(a)gmail.com> writes: > Review of patch: > > wait_for_prior_commit() can be called multiple times per event group, > only do my_error() the first time the call fails. > > > --- a/sql/sql_class.cc > +++ b/sql/sql_class.cc > @@ -8365,6 +8365,18 @@ wait_for_commit::wait_for_prior_commit2(THD > *thd, bool allow_kill) > } > > > +void > +wait_for_commit::prior_commit_error(THD *thd) > +{ > + /* > + Only raise a "prior commit failed" error if we didn't already raise > + an error. > + */ > + if (!thd->get_stmt_da()->is_set()) > + my_error(ER_PRIOR_COMMIT_FAILED, MYF(0)); > +} > + > + > /* > Wakeup anyone waiting for us to have committed. > > diff --git a/sql/sql_class.h b/sql/sql_class.h > index 4cee31b296e..69b021cd41d 100644 > --- a/sql/sql_class.h > +++ b/sql/sql_class.h > @@ -2398,8 +2398,8 @@ struct wait_for_commit > return wait_for_prior_commit2(thd, allow_kill); > else > { > - if (wakeup_error) > - my_error(ER_PRIOR_COMMIT_FAILED, MYF(0)); > + if (unlikely(wakeup_error)) > + prior_commit_error(thd); > return wakeup_error; > } > } > The only small objection I have to this patch is that if we get first > one error and then a wait_for_prior_commit error, then we will not get > the second error. > I would suggest that instead of checking for if > (!thd->get_stmt_da()->is_set()) instead add a new flag > 'got_prior_commit_error' to check if we have already > got a prior commit error for this thread. > Adding bool get_prior_commit_error into THD after bool > wakeup_blocked; has no storage cost. It is actually deliberate to skip the ER_PRIOR_COMMIT_FAILED if we already gave a different error. The reason is that the ER_PRIOR_COMMIT_FAILED is not a real error - it is an internal error thrown only to make sure that a following transaction does not succeed after a prior transaction failed, which would result in inconsistent commit order. Therefore, if a worker thread already got a real error, for example duplicate key or row not found, then it seems preferable to keep that error rather than override it with a default, internal-only error. (Arguably it might make sense to not write ER_PRIOR_COMMIT_FAILED errors to the error log at all, as they do not seem very useful; they are not an error that the user should do anything about, on the contrary they indicate that the worker correctly handled an earlier error. But that would be a separate patch, probably). > It up to you to decide which solution to choose. Ok, then I will keep this, but I will add a comment explaining why the original error is preferred over ER_PRIOR_COMMIT_FAILED. > This should be pushed to 10.6 to avoid the 63*4 warnings we can get > with parallel=4. Ok, will do then. > I will try to spend some time today to try to simulate the case where > we get new calls to wait_for_prior_commit after signaling a previous > failure from it. I hope you find a way to reproduce, I did not manage to do so when I tried. It would be good to find the root cause of those repeated ER_PRIOR_COMMIT_FAILED. - Kristian.

1 0

Re: 0cc1c382dfc: MDEV-34712 Add support to sha2 and pbkdf2 key derivation in file_key_management
by Sergei Golubchik 10 Mar '25

10 Mar '25

Hi, Yuchen, On Mar 10, Yuchen Pei wrote: > revision-id: 0cc1c382dfc (mariadb-11.8.1-9-g0cc1c382dfc) > parent(s): 3a81664cb80 > author: Yuchen Pei > committer: Yuchen Pei > timestamp: 2025-03-05 13:33:53 +1100 > message: > > MDEV-34712 Add support to sha2 and pbkdf2 key derivation in file_key_management > > Add two new variables: > > file_key_management_digest={sha1|sha224|sha256|sha384|sha512} > > This specify the digest function to use in key derivation of the key > used for decryption of the keyfile > > file_key_management_use_pbkdf2=N > > This specify whether pbkdf2 is used in the key derivation, and if > so (N != 0), how many thousands iterations. > diff --git a/include/mysql/service_my_crypt.h b/include/mysql/service_my_crypt.h > index 1a4ebcfd067..3a58d0828fc 100644 > --- a/include/mysql/service_my_crypt.h > +++ b/include/mysql/service_my_crypt.h > @@ -70,6 +73,10 @@ extern struct my_crypt_service_st { > unsigned int (*my_aes_get_size)(enum my_aes_mode mode, unsigned int source_length); > unsigned int (*my_aes_ctx_size)(enum my_aes_mode mode); > int (*my_random_bytes)(unsigned char* buf, int num); > + void (*my_bytes_to_key)(const unsigned char *salt, const unsigned char *input, > + unsigned int input_len, unsigned char *key, > + unsigned char *iv, enum my_digest digest, > + unsigned int use_pbkdf2); backward compatible API change - you need to increment the service version > } *my_crypt_service; > > #ifdef MYSQL_DYNAMIC_PLUGIN > diff --git a/mysql-test/suite/encryption/t/filekeys_encfile.test b/mysql-test/suite/encryption/t/filekeys_encfile.test > index a0611a38ec4..89e13834a79 100644 > --- a/mysql-test/suite/encryption/t/filekeys_encfile.test > +++ b/mysql-test/suite/encryption/t/filekeys_encfile.test > @@ -1 +1,34 @@ > +# # The encrypted key files were generated using the following: > + > +# let $plain_keyfile=$MYSQL_TMP_DIR/keys.txt; > +# let $enc_keyfile_prefix=$MTR_SUITE_DIR/t/filekeys-data; > + > +# write_file $plain_keyfile; > +# # > +# # this is a comment > +# # > +# 1;770A8A65DA156D24EE2A093277530142 > +# 2;4D92199549E0F2EF009B4160F3582E5528A11A45017F3EF8 > +# # another comment > +# 33;B374A26A71490437AA024E4FADD5B497FDFF1A8EA6FF12F6FB65AF2720B59CCF > +# 4;18420B5CBA31CCDFFE9716E91EB61374D05914F3ADE23E03 --> ignored > + > +# 5;966050D7777350B6FD5CCB3E5F648DA45C63BEFB6DEDDFA13443F156B7D35C84 > +# 6;B5EA210C8C09EF20DB95EC584714A89F # and yet another > + > +# EOF > + > +# exec openssl enc -aes-256-cbc -md sha1 -pass pass:secret -in $plain_keyfile -out $enc_keyfile_prefix-sha1.enc; > +# exec openssl enc -aes-256-cbc -md sha224 -pass pass:secret -in $plain_keyfile -out $enc_keyfile_prefix-sha224.enc; > +# exec openssl enc -aes-256-cbc -md sha256 -pass pass:secret -in $plain_keyfile -out $enc_keyfile_prefix-sha256.enc; > +# exec openssl enc -aes-256-cbc -md sha384 -pass pass:secret -in $plain_keyfile -out $enc_keyfile_prefix-sha384.enc; > +# exec openssl enc -aes-256-cbc -md sha512 -pass pass:secret -in $plain_keyfile -out $enc_keyfile_prefix-sha512.enc; > +# exec openssl enc -aes-256-cbc -md sha1 -pbkdf2 -pass pass:secret -in $plain_keyfile -out $enc_keyfile_prefix-sha1-pbkdf2.enc; better say -iter 1000 instead of -pbkdf2 (to match plugin parameters) And change some tests to use a non-default value, please > +# exec openssl enc -aes-256-cbc -md sha224 -pbkdf2 -pass pass:secret -in $plain_keyfile -out $enc_keyfile_prefix-sha224-pbkdf2.enc; > +# exec openssl enc -aes-256-cbc -md sha256 -pbkdf2 -pass pass:secret -in $plain_keyfile -out $enc_keyfile_prefix-sha256-pbkdf2.enc; > +# exec openssl enc -aes-256-cbc -md sha384 -pbkdf2 -pass pass:secret -in $plain_keyfile -out $enc_keyfile_prefix-sha384-pbkdf2.enc; > +# exec openssl enc -aes-256-cbc -md sha512 -pbkdf2 -pass pass:secret -in $plain_keyfile -out $enc_keyfile_prefix-sha512-pbkdf2.enc; > + > +# echo Done; > + > source filekeys_goodtest.inc; > diff --git a/mysys_ssl/my_crypt.cc b/mysys_ssl/my_crypt.cc > index 346eff51b1b..ef74121d530 100644 > --- a/mysys_ssl/my_crypt.cc > +++ b/mysys_ssl/my_crypt.cc > @@ -382,4 +384,39 @@ int my_random_bytes(uchar *buf, int num) > return MY_AES_OK; > } > > +static inline const EVP_MD *get_digest(enum my_digest digest) > +{ > + switch (digest) > + { > + case MY_DIGEST_SHA1: return EVP_sha1(); > + case MY_DIGEST_SHA224: return EVP_sha224(); > + case MY_DIGEST_SHA256: return EVP_sha256(); > + case MY_DIGEST_SHA384: return EVP_sha384(); > + case MY_DIGEST_SHA512: return EVP_sha512(); > + default: > + { > + assert(0); > + return NULL; > + } > + } > +} > + > +void my_bytes_to_key(const unsigned char *salt, const unsigned char *input, > + uint input_len, unsigned char *key, unsigned char *iv, > + enum my_digest digest, uint use_pbkdf2) > +{ > + if (use_pbkdf2 == 0) > + EVP_BytesToKey(EVP_aes_256_cbc(), get_digest(digest), salt, > + input, input_len, 1, key, iv); > + else > + { > + uchar keyiv[KEY_LENGTH + IV_LENGTH]; > + PKCS5_PBKDF2_HMAC((const char*) input, input_len, salt, 8, > + use_pbkdf2 * 1000, get_digest(digest), I suggest not to do *1000. It doesn't help with anything and introduces a limitation that we don't need. Perhaps somebody has a policy of using 2^N iterations in PBKDF2? Also, `openssl enc -iter` doesn't do *1000 > + KEY_LENGTH + IV_LENGTH, keyiv); > + memcpy(key, keyiv, KEY_LENGTH); > + memcpy(iv, keyiv + KEY_LENGTH, IV_LENGTH); > + } > +} > + > } > diff --git a/plugin/file_key_management/file_key_management_plugin.cc b/plugin/file_key_management/file_key_management_plugin.cc > index 8702785a1fd..f37cfe05960 100644 > --- a/plugin/file_key_management/file_key_management_plugin.cc > +++ b/plugin/file_key_management/file_key_management_plugin.cc > @@ -54,10 +68,22 @@ static MYSQL_SYSVAR_ENUM(encryption_algorithm, encryption_algorithm, > "Encryption algorithm to use" recommendation, > NULL, NULL, 0, &encryption_algorithm_typelib); > > +static MYSQL_SYSVAR_ENUM(digest, digest, > + PLUGIN_VAR_RQCMDARG | PLUGIN_VAR_READONLY, > + "Digest to use", "Digest algorithm used for decrypting the key file. The value of the openssl enc -md option" > + NULL, NULL, 0, &digest_typelib); > + > +static MYSQL_SYSVAR_UINT(use_pbkdf2, use_pbkdf2, > + PLUGIN_VAR_RQCMDARG | PLUGIN_VAR_READONLY, > + "Use pbkdf2", "Number of PBKDF2 iteration used for decrypting the key file. Use 0 to disable PBKDF2, otherwise it's a value of the openssl enc -iter option" > + NULL, NULL, 0, 0, UINT_MAX / 1000, 0); > + > static struct st_mysql_sys_var* settings[] = { > MYSQL_SYSVAR(filename), > MYSQL_SYSVAR(filekey), > MYSQL_SYSVAR(encryption_algorithm), > + MYSQL_SYSVAR(digest), > + MYSQL_SYSVAR(use_pbkdf2), > NULL > }; Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

1 0

Has anyone implemented InnoDB redo log-based physical replication?
by Otto Kekäläinen 04 Mar '25

04 Mar '25

Hi, Has anyone attempted or seen any existing implementations "physical" replication between two MariaDB instances, where a secondary read-only instance would follow the primary by directly reading and applying page changes from the primary's redo logs? The secondary instance would start (and restart if necessary) using a full copy of the data from the primary instance, but while running keep itself current by reading redo logs from the primary instance, and apply pages based on the increasing LSN number. I call this "physical" replication to distinguish it from normal logical replication based on e.g. binlogs, and this would obviously only work for InnoDB. Thanks, Otto

2 2

Review request for MariaDB 2025 Q1 uploads to Debian and Ubuntu
by Otto Kekäläinen 24 Feb '25

24 Feb '25

Hi! Thanks Vicentiu for reviewing https://salsa.debian.org/mariadb-team/mariadb-server/-/merge_requests/102 The next uploads in line are ready for review at: https://salsa.debian.org/mariadb-team/mariadb-server/-/merge_requests/98 Prepare MariaDB Server 1:10.11.11-0+deb12u1 minor maintenance release for Debian stable (Bookworm) https://salsa.debian.org/mariadb-team/mariadb-server/-/merge_requests/106 Prepare upload to MariaDB 10.11.9 to Ubuntu 24.04 "Noble" https://salsa.debian.org/mariadb-team/mariadb-server/-/merge_requests/96 Prepare upload to MariaDB 10.6.19 to Ubuntu 22.04 "Jammy" https://salsa.debian.org/mariadb-team/mariadb-10.5/-/merge_requests/20 New upstream version MariaDB 10.5.28 for Debian 11 "Bullseye" Once I am done with these, I will start testing MariaDB 11.8 with the goal of uploading it into Debian in the coming weeks.

1 0

Re: [PATCH] MDEV-36099 Ensure that creation and usage of temporary tables in replication is predictable
by Kristian Nielsen 23 Feb '25

23 Feb '25

Hi Monty, > From: Monty <monty(a)mariadb.org> > > The purpose of this commit is to ensure that creation and changes of > temporary tables are properly and predicable logged to the binary > log. It also fixes some bugs where ROW logging was used in MIXED mode, > when STATEMENT would be a better (and expected) choice. Here is my review of the patch for MDEV-36099 predictable binlog format for temporary tables: Overall this seems like solid work, thanks! As Elena also remarked, this does significant changes to a very tricky and bug-ridden area of replication, that of temporary tables, this is scary stuff. But the patch does look good and relatively simple. A main change with this patch is that now all use of temporary tables will by default be binlogged in row mode. I think this is a reasonable change. There are still remaining problems with temporary tables in statement-based mode; and it also helps with parallel replication, which cannot run in parallel statement-logged temporary tables. And you provided --create-temporary-table-binlog-formats as a way to enable backwards-compatible behaviour. I think it is important to mention this prominently in the release notes. There will invariably be some users that manipulate a large number of rows with temporary tables, eg. BEGIN; CREATE TEMPORARY TABLE tmp AS SELECT a, b, c FROM large_table1 JOIN large_table2 ON ... INSERT INTO other_table SELECT b, c FROM tmp WHERE a <100; DROP TEMPORARY TABLE tmp; COMMIT; This will change from binlogging a few hundred bytes of Query events to potentially many megabytes of row events, which could cause problems in some cases. I think the change is still justified from the benefits it gives and the relative few cases where this will cause serious problems, but it is still important to try to document it as well as possible. I read the code part of the patch through in detail, but I did not spot anything in it that looks wrong. There may be things missing from the patch that are hard to spot in a review, I think that's unavoidable in a change like this. But I did try to think of possible issues without finding anything, and the patch does look quite simple compared to what it is doing. So looks good I think. Just a few questions on details I did not understand, and some minor suggestions for changes. Questions: > diff --git a/sql/sql_parse.cc b/sql/sql_parse.cc > index c2f842aea56..3d1df5ac42f 100644 > --- a/sql/sql_parse.cc > +++ b/sql/sql_parse.cc > @@ -531,6 +531,7 @@ void init_update_queries(void) > CF_SCHEMA_CHANGE; > sql_command_flags[SQLCOM_CREATE_SEQUENCE]= (CF_CHANGES_DATA | > CF_REEXECUTION_FRAGILE | > + CF_FORCE_ORIGINAL_BINLOG_FORMAT | > CF_AUTO_COMMIT_TRANS | > CF_SCHEMA_CHANGE); I did not get the reason for this change. I assume some bug fix found during test case writing? > --- a/sql/sql_class.h > +++ b/sql/sql_class.h > @@ -817,6 +817,7 @@ typedef struct system_variables > /* Flags for slow log filtering */ > ulong log_slow_rate_limit; > ulong binlog_format; ///< binlog format for this thd (see enum_binlog_format) > + ulong create_temporary_table_binlog_formats; Is there any way to make this take less space than 8 bytes (on 64-bit)? It goes into every THD, and only two bits are used. > + > + Temporary tables will be logged only on CREATE in STMT format > + or on INSERT if all changes to the table is in the binlog. > */ > if ((WSREP_EMULATE_BINLOG(thd) || mysql_bin_log.is_open()) && > - (likely(!error) || thd->transaction->stmt.modified_non_trans_table || > - thd->log_current_statement())) > + (table->s->using_binlog() || > + ((in_create_table && > + (!table->s->tmp_table || thd->binlog_create_tmp_table())))) && > + (likely(!error) || > + (!in_create_table && > + (thd->transaction->stmt.modified_non_trans_table || > + thd->log_current_statement())))) This conditional is now very hard to understand. I think I managed to understand all parts of it, except the part "thd->log_current_statement()", but that is unchanged with your patch. This if () would now benefit from being split into multiple parts, each of which explained in a comment. > diff --git a/mysql-test/main/statistics.result b/mysql-test/main/statistics.result > index bb6469c46da..ebd9cb7f36e 100644 > --- a/mysql-test/main/statistics.result > +++ b/mysql-test/main/statistics.result > @@ -1278,7 +1278,7 @@ Table Op Msg_type Msg_text > mysql.column_stats analyze error Invalid argument > ANALYZE TABLE mysql.column_stats; > Table Op Msg_type Msg_text > -mysql.column_stats analyze status OK > +mysql.column_stats analyze status Table is already up to date > SELECT * FROM mysql.table_stats; > db_name table_name cardinality > SELECT * FROM mysql.column_stats; I did not understand why this .result change happens after your patch? Minor things: > diff --git a/mysql-test/suite/rpl/t/create_or_replace.inc b/mysql-test/suite/rpl/t/create_or_replace.inc > index e8fa95cb94a..b4e91d7b412 100644 > --- a/mysql-test/suite/rpl/t/create_or_replace.inc > +++ b/mysql-test/suite/rpl/t/create_or_replace.inc > @@ -4,7 +4,24 @@ > --let $rpl_topology=1->2 > --source include/rpl_init.inc > > -# Create help tables > +select @@binlog_format, @@create_temporary_table_binlog_formats; > + > +# Copy create_temporary_table_binlog_formats from master to slave > +# This is done to get same results as with older versions of MariaDB. > +# The slave will work even if this is not done. However in mixed > +# format on slave temporary tables would not be logged to in the > +# slaves binary log > +let $format=`select @@create_temporary_table_binlog_formats`; > +connection server_2; > +--eval set @@global.create_temporary_table_binlog_formats='$format' > +# Ensure that the slave threads uses the new values. > +stop slave; > +start slave; Here, use --source include/stop_slave.inc and --source include/start_slave.inc. For consistent with other test cases. > - Rename of not binlogged temporary tables where binlogged to binary log > which caused replication to stop. s/where/were/ > + /* > + Mark if query was logged as statement. Used to check it tmp table changes > + are logged. > + */ > + bool query_binlogged_as_stmt; s/check it table/check if table/ + /* + This is set for all normal tables and for temporary tables that has + the CREATE binary logged. + + 0 table table is not in binary log (not logged temporary table) + 1 table create was logged (normal table or logged temp table) + 2 table create was logged but not all changes are in the binary log. + ROW LOGGING will be used for the table. s/that has the CREATE/that have the CREATE/ s/table table is not/table is not/ > @@ -794,6 +821,20 @@ void THD::mark_tmp_table_as_free_for_reuse(TABLE *table) > > DBUG_ASSERT(table->s->tmp_table); > > + /* > + Ensure that table changes where either binary logged or the table > + is marked as not up to date. > + */ s/where/were/ Thanks, Monty! Reviewed-by: Kristian Nielsen <knielsen(a)knielsen-hq.org> - Kristian.

2 1

MDEV-33671: MTR Safe Process Hardcodes RLIMIT_NOFILE
by mohanadkhaled87＠gmail.com 11 Feb '25

11 Feb '25

Hello I am Mohanad and this is my first issue here I started working on this issue a while ago but only began testing it recently. As per the comment in this issue (https://jira.mariadb.org/browse/MDEV-33671?focusedCommentId=281856&page=com…) "You can run ulimit from Perl, between the fork and exec." I implemented this by using the BSD::Resource library, as it appears to be part of the core Perl libraries and does not introduce an external dependency and as a sanity check I added ```perl my $has_bsd_resource = eval { require BSD::Resource; BSD::Resource->import(qw(setrlimit getrlimit RLIMIT_NOFILE)); 1; }; ``` Would this be an okay to use ? Also, I have manually verified the implementation by printing the limit values. However, is it possible to add test files or automate testing for this issue in MTR ?

1 0

Current status on MDEV-34705 binlog-in-engine
by Kristian Nielsen 09 Feb '25

09 Feb '25

Hi Brandon (and other interested), It is great that you want to look closer at the MDEV-34705, binlog-in-engine project. I wanted to write up a high-level summary of the current status to help you get an overview. For reference, this is a project to implement a new binlog format that solves some of the complexity and performance problems with the old format that is not integrated with the storage engine transaction log (eg. InnoDB binlog). The code is being developed on the branch knielsen_binlog_in_engine on Github, https://github.com/MariaDB/server/commits/knielsen_binlog_in_engine There has been some previous design discussions on the mailing list, eg.: https://lists.mariadb.org/hyperkitty/list/developers@lists.mariadb.org/thre… https://lists.mariadb.org/hyperkitty/list/developers@lists.mariadb.org/thre… - Kristian. Current status: - Feature is off by default, enabled by --binlog-storage-engine=innodb. - Binlog files are now identified solely with their file_no, starting from 0 (no more master-bin.index). Binlog file names are hard-coded by the engine (binlog-NNNNNN.ibb), directory can be selected with --binlog-directory. - Binlog files are pre-allocated to --max-binlog-size, thus their size is fixed. Event groups and even events can span multiple binlog files. - New binlog format is page-based. On server start, binary search is used to find the current position (no need for scanning a whole binlog file). GTID start position is likewise found by binary search, no need for separate GTID index. - Binlog files are written strictly append-only, data is never updated after initial write. The InnoDB redo log is used to ensure that files are recovered into a consistent state with the engine data after a crash. - Large event groups (larger than --binlog-cache-size) are split into pieces that are binlogged "out-of-band", meaning they are written in separate records during transaction execution, before commit. The final commit record merely has a reference to the out-of-band data. Out-of-band data for rolled-back transactions remain as "garbage" until binlog purge. - The binlog dump thread re-assembles out-of-band data into consecutive event groups for the slaves. Thus, slaves see event data in the same format as before, no slave code changes needed. Out-of-band data is structured as complete binary trees for efficient reading. - Basic slave replication is working, not much is tested though. Currently GTID mode only, might make sense to not support filename/position mode for simplicity. - RESET MASTER, FLUSH BINARY LOGS, PURGE BINARY LOGS are implemented. - Initial benchmarking show good speedup (x2) in durable configuration, potentially huge (3 to sevelal hundred times faster) speedup in crash-safe configuration. A main challenge with this project is that the binlog internals are visible in so many places to users, and to other parts of the code. Here is an (incomplete) list of major issues still to implement: - The current implementation is using the normal InnoDB buffer pool and checkpoint mechanism. See the mailing list discussions for Marko's ideas for implementing a simpler integration with the InnoDB redo log. - When --sync-binlog=1, implement that binlog events are not sent to slaves until durably written to redo log. When --sync-binlog=0, implement that events are not sent until redo log is (non-durably) written (--innodb-flush-log-at-trx-commit=2). - mysqlbinlog ability to read new binlog format. The code is there, mostly a matter of finding a good way to share code between the server/InnoDB and mysqlbinlog.cc. - Disable all parts of the legacy binlog when --binlog-storage-engine is set (in current code some parts still remain active). - SHOW BINLOG EVENTS. - Binlog checksum and encryption. Will checksum and encrypt the individual pages, not use the legacy per-event checksums or encryption. - Storage engine API. The functionality is implemented, but the design of the API and interfaces might need review and cleanup, ie. maybe parts to be implemented as services (include/service_versions.h etc.) - Lots of smaller ToDo:s, some marked in comments in the code. - There is a --suite=binlog_in_engine, but a lot more testing is neeed. Some parts may be initially not supported in the first release, but implemented later: - Semi-synchronous replication. Should be straight-forward to implement later. Might make sense to use GTID to send back-and-forth instead of filename/offset. AFTER_SYNC no longer makes sense, but could consider implementing AFTER_PREPARE. - External XA, to be implemented following MDEV-32020. The out-of-band mechanism will be used so that XA COMMIT can reference the binlog data and replicate without the need for any transactions to be in XA PREPARED state on the slave or in backups.

1 0

Next step on MDEV-34705, implement binlog in InnoDB
by Kristian Nielsen 30 Jan '25

30 Jan '25

Hi Marko, Time flies, somehow it's already more than a year since our first discussions on implementing the binlog in InnoDB and avoiding the extra fsync() and complexity of the two-phase commit between InnoDB and binlog. But things have progressed, and I have now reached the point where most of the basic groundwork is implemented. Event groups are binlogged to InnoDB tablespaces. Binlog dump thread can read the binlog and send to slave, and replication is working. Large event groups are split into pieces, bounding the amount of data that needs to be atomically written in mini-transactions and at commit time. There are still many details left, but mostly in the server-layer replication code which should be manageable, just will take some time to get completed. I think now is a good time for you to take a first real look at the InnoDB part of the changes, I would really value your input. The main part of the InnoDB code is in two files: 1. handler/handler0binlog.cc for the high-level part that deals mostly with the new binlog file format and interfacing to the server layer. 2. fsp/fsp0binlog.cc for the low-level part that most tightly interacts with the InnoDB mini-transactions and buffer pool. The most interesting part for you to look at is fsp/fsp0binlog.cc (~1k lines), though I'm happy to hear comments on any part of the patch, of course. The code is pushed to GitHub in the branch knielsen_binlog_in_engine: https://github.com/MariaDB/server/commits/knielsen_binlog_in_engine and I've also attached the complete patch. This is my first major patch for InnoDB, so there will undoubtedly be a number of style changes required. But the overall structure of the code should now be close to what I imagine would be the final result, with some pending ToDo steps marked in comments in the code, and detailed in the below list, some of which we discussed a bit already. I hope you will take a look at the patch and let me know of any questions or other things you need from me. Maybe we can also find a chance to discuss further if you will come to FOSDEM start of February, or I could visit sometimes in Finland. - Kristian. ---- Some known outstanding issues in the InnoDB part: - We previously discussed removing some of the page header overhead for binlog tablespaces. Currently the code just leaves alone the first FIL_PAGE_DATA bytes (38) and the last FIL_PAGE_DATA_END (8 IIRC). - We discussed previously to write the current LSN at the start of the tablespace, and use this in recovery to handle that we have only two tablespace IDs that are reused. So we need code in recovery that checks the LSN at the start of the tablespace, and skips redo records with LSN smaller than this. - We want to avoid the double-write buffer for binlog pages, at least for the first page write (most pages will only be written as full pages). You mentioned an idea to completely avoid the double-write buffer and instead do some specific code for recovery in the uncommon case where a partial binlog page is written to disk due to low commit activity. - The flushing of binlog pages to disk currently happens in a dedicated thread in the background. I'd welcome ideas on how to do this differently. It is good to flush binlog pages quickly and re-use their buffer pool entries for something better. Also writing the pages to disk quickly (not necessarily fsync()'ing) makes the data readable by mysqlbinlog. - Checksum and encryption should use the standard InnoDB mechanism. I assume checksum is already handled in the code through using the buffer pool and mini-transactions to read/write pages. Not sure about encryption. I need to implement that the code handles checksum and decryption when reading the pages manually from the file (not through buffer pool).

2 10

2025 Q1 release - near final
by Daniel Black 29 Jan '25

29 Jan '25

The issue where this is tracked is: https://jira.mariadb.org/browse/MDEV-35789 Current blockers: https://jira.mariadb.org/issues/?filter=31019&jql=project%20in%20(mdev)%20A… The "fix version" in the above url list all are 11.4+ so all LTS versions should be stable. Branches bb-10.5-release (LTS) bb-10.6-release (LTS) bb-10.11-release (LTS) bb-11.4-release (LTS) bb-11.7-release (stable for first time) bb-11.8-release (gamma - no LTS announcement made) If there's regression particularly impacting please ensure the issue has a blocker priority, with a good business case reason why it should block, and make as blocks MDEV-35789. In addition to compiling the source yourselves: Containers: quay.ion/mariadb-foundation/mariadb-devel:{release} list: https://quay.io/repository/mariadb-foundation/mariadb-devel?tab=tags note: * pushes to *-release branch also gains the same tag as the main 10.X, 11.X tags. Check commit at the beginning of the server startup to determine actual commit. Repos for testing (packages may not last more than a few weeks at urls): https://ci.mariadb.org/ contains for deb and rpm distros a repo file in the form: bb-{version}-release-latest-{arch}-{distro}-{version}-{deb|rpm}-autobake.{sources|repo}

1 0

Parameterized queries
by E.S. Rosenberg 27 Jan '25

27 Jan '25

Hi, Recently I found out about parameterized queries in MySQL, it seems to have been supported at least since 8.0.x but it is not listed as an incompatibility between mariadb and mysql yet the package spec for COM_QUERY on mariadb does not mention any support for parameterized queries. So is this an incompatibility, some out of date docs or am I just looking at the wrong com_query page? Refs: https://dev.mysql.com/doc/dev/mysql-server/8.0.40/page_protocol_com_query.h… https://mariadb.com/kb/en/com_query/ Thanks! Eli

4 9

Re: 6a4b85ce1cc: MDEV-20281 "[ERROR] Failed to write to mysql.slow_log:" without error reason
by Sergei Golubchik 25 Jan '25

25 Jan '25

Hi, Oleksandr, Ok to push after fixing spagetty (either by fixing a typo or by changing the code pattern as suggested below). On Jan 25, Oleksandr Byelkin wrote: > revision-id: 6a4b85ce1cc (mariadb-10.5.27-202-g6a4b85ce1cc) > parent(s): d261fa5c703 > author: Oleksandr Byelkin > committer: Oleksandr Byelkin > timestamp: 2025-01-24 17:24:17 +0100 > message: > > MDEV-20281 "[ERROR] Failed to write to mysql.slow_log:" without error reason > > Add "backup" (in case of absence issued by error) reasons for failed logging. > > @@ -998,6 +1047,9 @@ bool Log_to_csv_event_handler:: > close_log_table(thd, &open_tables_backup); > thd->time_zone_used= save_time_zone_used; > DBUG_RETURN(result); > +store_err: // spagetty, but reduce size of the code > + cause= "Can't write data (possible incorrect log table structure)"; > + goto err; > } typo. it's "spaghetti" but may be better to avoid it altogether by rewriting as cause= "..."; if (field[0]->store(...)) goto err; if (field[1]->store(...)) goto err; if (field[2]->store(...)) goto err; ... cause= NULL; Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

1 0

Re: 0b403e03939: MDEV-35661 Assertion `!cache_list ||...' fails in online_alter_close_connection
by Sergei Golubchik 22 Jan '25

22 Jan '25

Hi, Nikita, On Jan 19, Nikita Malyavin wrote: > revision-id: 0b403e03939 (mariadb-11.7.1-16-g0b403e03939) > parent(s): 40e502eca4c > author: Nikita Malyavin > committer: Nikita Malyavin > timestamp: 2025-01-03 00:54:52 +0100 > message: > > MDEV-35661 Assertion `!cache_list ||...' fails in > online_alter_close_connection > > After MDEV-35182 (88adcbf3) fix, online_alter still wasn't fully ready > for `XA PREPARE` to be skipped. > > In fact, according to XA standard, even xa_commit() call may happen > without preceding xa_prepare(), so this stage is completely optional. > > This commit removes any relying on xa_prepare() call. The only thing > that will be done there is an equivalent of "statement-level commit", > which basically just flushes a pending rows event to an online alter > cache. > > This is done by always storing an online alter cache in the XA_data > object whenever XA is active, rather than in THD::ha_data. Why? I thought it'd be simpler to keep the cache in the THD and take it off it in commit or rollback. That is, almost as now, but commit/rollback won't rely on prepare being called before. Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 1

Re: 33e36944232: MDEV-30111 InnoDB: Failing assertion: update->n_fields == 0 in row_ins_sec_index_entry_by_modify
by Sergei Golubchik 22 Jan '25

22 Jan '25

Hi, Alexander, On Jan 22, Alexander Barkov wrote: > revision-id: 33e36944232 (mariadb-10.6.20-19-g33e36944232) > parent(s): 25be7da2024 > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2024-11-14 10:19:24 +0400 > message: > > MDEV-30111 InnoDB: Failing assertion: update->n_fields == 0 in row_ins_sec_index_entry_by_modify > > Also fixes: > > MDEV-32190 Index corruption with unique key and nopad collation (without DESC or HASH keys) > > The code in strings/strcoll.inl when comparing an empty string > to a string like 0x0001050001 did not take into account > that the leftmost weight in the latter can be zero, while > there are some more weights can follow the zero weight. > > Rewriting the code to treat the shorter string as smaller than > a longer string. > > --- a/strings/strcoll.inl > +++ b/strings/strcoll.inl > @@ -192,10 +192,10 @@ MY_FUNCTION_NAME(strnncoll)(CHARSET_INFO *cs __attribute__((unused)), > >0 >0 Two weights were scanned, check weight difference. > */ > if (!a_wlen) > - return b_wlen ? -b_weight : 0; > + return b_wlen ? -1 : 0; > > if (!b_wlen) > - return b_is_prefix ? 0 : a_weight; > + return b_is_prefix ? 0 : +1; > > if ((res= (a_weight - b_weight))) > return res; Sorry, I don't understand it. The function returns >0, 0, or <0. It shouldn't matter whether it returns -b_weight or -1, a_weight or +1. Except if the weight can be <= 0 itself. But in that case, if ((res= (a_weight - b_weight))) return res; is rather suspicious too. Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

1 0

On the question on producing bintars going forward
by Kristian Nielsen 22 Jan '25

22 Jan '25

Hi Vicentiu, I read through your proposal for how to handle MariaDB bintars. I know a lot of the old history on this (like from working in the MySQL build team back in 2005), I also know first-hand of the tough spot between what people think is "easily possible" and the practicalities of packaging for real linux systems. So I hope my input can be useful in some way: 1. To me, providing bintars is not important. Nor is providing any binaries for that matter. MariaDB is open source, the most important thing is it is easy to compile. Others can provide binaries as needed, and in fact distros are already doing so very well. So I personally am fine with not releasing bintars going forward. I know binaries are considered important by others. 2. I agree a clear definition is lacking of exactly what binaries should be/provide. My opinion for bintars is that they are a way to use MariaDB easily similar to compiling the source, but without the need for having installed a compiler toolchain. So bintars should be the result of a mostly default compile of the source, on a stable/oldstable linux distro like Debian or some redhat with minimal optional dependencies installed. Most work should go into making default compile do the right thing, as opposed to endless tweaking in build scripts or Buildbot for how bintars are compiled. 3. In your proposal, I think there is too much focus (out of good intentions, I'm sure) about solving all possible problems. No install method is perfect for all use cases. Bintars provide very simple install or even easy-of-use without installing, but they have other limitations. Let us not try to make bintars "perfect". Rather make them simple to produce, and simple to use in the limited scenarious where they can work well. 4. Let us firmly give up on building static binaries. Static binaries do not work well, and have not done so in many years. Here is a reference for a problem with linking libc statically: https://sourceware.org/glibc/wiki/FAQ#Even_statically_linked_programs_need_… And mixing static and dynamic linking creates some very serious problems, see MDEV-32251 and MDEV-25633. We should just build a default `cmake` binary with dynamic linking against the required library dependencies, do the build on a minimal linux where most optional dependencies are not installed. This will then work well on user's systems that have compatible libraries. On systems that have not, there are other options for running MariaDB. 5. I think we should _definitely_ not maintain an infrastructure for building our own static versions of various system libraries / 3rd-party libraries, see also (4). 6. I agree with concerns about too much effort needed to maintain bintars for many years. Let us be pragmatic, make things simple, and only do what can be done with small amount of effort. Build the bintars on stable linux systems for compatibility, and use default installs as much as possible for simplicity. When those systems become too old to easily continue building bintars, just stop building them. Existing users can easily transition to compiling themselves if they really need, thanks to (2); or use another of the different options available for running MariaDB. And finally Vicentiu, let me commend you for taking responsibility of the task of producing bintars and working so constructively on getting to a working result. It is a task the difficulty of which is easily underappreciated. So these are my opinions and suggestions, hope this helps. - Kristian.

2 1

Re: d47b1ebc06d: MDEV-35326: Memory Leak in init_io_cache_ext upon SHUTDOWN
by Sergei Golubchik 10 Jan '25

10 Jan '25

Hi, Oleksandr, On Jan 09, Oleksandr Byelkin wrote: > revision-id: d47b1ebc06d (mariadb-10.5.27-143-gd47b1ebc06d) > parent(s): 3bbbeae7924 > author: Oleksandr Byelkin > committer: Oleksandr Byelkin > timestamp: 2025-01-08 14:14:10 +0100 > message: > > MDEV-35326: Memory Leak in init_io_cache_ext upon SHUTDOWN > > The problems were that: > 1) resources was freed "asimetric" normal execution in send_eof, > in case of error in destructor. > 2) destructor was not called in case of SP for result objects. > (so if the last SP execution ended with error resorces was not > freeded on reinit before execution (cleanup() called before next > execution) and destructor also was not called due to lack of > delete call for the object) > > All result method revised and freeing resources made "symetric". > > Destructor of result object called for SP. > > Added skipped invalidation in case of error in insert. > > Removed misleading naming of reset(thd) (could be mixed with > with reset()) > > diff --git a/sql/sp_head.cc b/sql/sp_head.cc > --- a/sql/sp_head.cc > +++ b/sql/sp_head.cc > @@ -3665,6 +3665,18 @@ int sp_lex_keeper::cursor_reset_lex_and_exec_core(THD *thd, uint *nextp, > return res; > } > > +sp_lex_keeper::~sp_lex_keeper() > +{ > + if (m_lex_resp) > + { > + /* Prevent endless recursion. */ > + m_lex->sphead= NULL; > + if (m_lex->result) you don't need this if(), the standard explicitly says that delete of a NULL pointer is valid. > + delete m_lex->result; > + lex_end(m_lex); > + delete m_lex; > + } > +} > > /* > sp_instr class functions > diff --git a/sql/sql_class.cc b/sql/sql_class.cc > --- a/sql/sql_class.cc > +++ b/sql/sql_class.cc > @@ -3192,10 +3193,22 @@ bool select_send::send_eof() > Handling writing to file > ************************************************************************/ > > +bool select_to_file::free_recources() > +{ > + if (file >= 0) > + { > + (void) end_io_cache(&cache); > + bool error= mysql_file_close(file, MYF(MY_WME)); > + file= -1; > + return error; > + } > + return FALSE; > +} > + > bool select_to_file::send_eof() > { > int error= MY_TEST(end_io_cache(&cache)); > - if (unlikely(mysql_file_close(file, MYF(MY_WME))) || > + if (unlikely(free_recources()) || free_recources() incluses end_io_cache(), you don't need to call it separately above. > unlikely(thd->is_error())) > error= true; > > @@ -5934,7 +5935,15 @@ class select_to_file :public select_result_interceptor { > { path[0]=0; } > ~select_to_file(); > bool send_eof() override; > + void abort_result_set() override; > void cleanup() override; > + /* > + It is separated from cleanup() becaouse: 1. typo 2. if you rename cleanup() to reset_for_next_ps_execution() then you won't need that comment, because, obviously, "freeing resources" is distinct from "resetting for next ps execution" but it's not obvious that "freeing resources" is distinct from "cleanup" > + 1. cleanup can not be called (conventional execution) > + 2. cleanup not only free recources but reset counters (some sends > + counter updates in destructor) > + */ > + bool free_recources(); > }; > > > diff --git a/sql/sql_delete.cc b/sql/sql_delete.cc > --- a/sql/sql_delete.cc > +++ b/sql/sql_delete.cc > @@ -1422,6 +1422,13 @@ void multi_delete::abort_result_set() > { > DBUG_ENTER("multi_delete::abort_result_set"); > > + /**************************************************************************** > + > + NOTE: if you change here be aware that almost the same code is in > + multi_delete::send_eof(). > + > + ***************************************************************************/ Let's talk about it separately. > + > /* the error was handled or nothing deleted and no side effects return */ > if (error_handled || > (!thd->transaction->stmt.modified_non_trans_table && !deleted)) > diff --git a/sql/sql_update.cc b/sql/sql_update.cc > --- a/sql/sql_update.cc > +++ b/sql/sql_update.cc > @@ -2743,6 +2743,13 @@ void multi_update::abort_result_set() > (!thd->transaction->stmt.modified_non_trans_table && !updated))) > return; > > + /**************************************************************************** > + > + NOTE: if you change here be aware that almost the same code is in > + multi_delete::send_eof(). multi_update::send_eof() > + > + ***************************************************************************/ > + > /* Something already updated so we have to invalidate cache */ > if (updated) > query_cache_invalidate3(thd, update_tables, 1); Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

1 0

Re: 331430a6a73: MDEV-33658 2/2 Cannot add a foreign key on a table with a matching long UNIQUE
by Sergei Golubchik 09 Jan '25

09 Jan '25

Hi, Nikita, On Jan 01, Nikita Malyavin wrote: > diff --git a/sql/sql_class.cc b/sql/sql_class.cc > index de688b4cb31..2ffceb3932b 100644 > --- a/sql/sql_class.cc > +++ b/sql/sql_class.cc > @@ -217,6 +218,16 @@ Foreign_key::Foreign_key(const Foreign_key &rhs, MEM_ROOT *mem_root) > > bool is_foreign_key_prefix(Key *a, Key *b) > { > + ha_key_alg a_alg= a->key_create_info.algorithm; > + ha_key_alg b_alg= b->key_create_info.algorithm; > + > + // The real algorithm will be BTREE is none was given by user. Technically, this is not correct, the algorithm will be whatever the engine wants. For MEMORY engine it'll be HASH. But only InnoDB can do foreign keys now, and for InnoDB th3 default key algorithm is BTREE. Let's reflect it in the comment, like, for example (also fixed a typo "is" -> "if") // The real algorithm in InnoDB will be BTREE if none was given by user > + a_alg= a_alg == HA_KEY_ALG_UNDEF ? HA_KEY_ALG_BTREE : a_alg; > + b_alg= b_alg == HA_KEY_ALG_UNDEF ? HA_KEY_ALG_BTREE : b_alg; > + > + if (a_alg != b_alg) > + return false; > + > /* Ensure that 'a' is the generated key */ > if (a->generated) > { > diff --git a/sql/sql_table.cc b/sql/sql_table.cc > index 3f035defbd3..d7aeb5d7cbf 100644 > --- a/sql/sql_table.cc > +++ b/sql/sql_table.cc > @@ -3556,6 +3556,303 @@ key_add_part_check_null(const handler *file, KEY *key_info, > } > > > +static > +my_bool key_check_without_overlaps(THD *thd, HA_CREATE_INFO *create_info, > + Alter_info *alter_info, > + Key &key) > +{ > + DBUG_ENTER("key_check_without_overlaps"); > + if (!key.without_overlaps) > + DBUG_RETURN(FALSE); > + // append_system_key_parts is already called, so we should check all the > + // columns except the last two. > + const auto &period_start= create_info->period_info.period.start; > + const auto &period_end= create_info->period_info.period.end; > + List_iterator<Key_part_spec> part_it_forwarded(key.columns); > + List_iterator<Key_part_spec> part_it(key.columns); > + part_it_forwarded++; > + part_it_forwarded++; > + while (part_it_forwarded++) > + { > + Key_part_spec *key_part= part_it++; > + > + if (period_start.streq(key_part->field_name) > + || period_end.streq(key_part->field_name)) > + { > + my_error(ER_KEY_CONTAINS_PERIOD_FIELDS, MYF(0), key.name.str, > + key_part->field_name.str); > + DBUG_RETURN(TRUE); > + } > + } It was simpler to have this check inside append_system_key_parts(), why did you move it? > + > + if (key.key_create_info.algorithm == HA_KEY_ALG_HASH || > + key.key_create_info.algorithm == HA_KEY_ALG_LONG_HASH) > + > + { > + my_error(ER_KEY_CANT_HAVE_WITHOUT_OVERLAPS, MYF(0), key.name.str); > + DBUG_RETURN(TRUE); > + } <...> > + KEY_CREATE_INFO *key_cinfo= &key.key_create_info; > + > + if (is_hash_field_needed) > + { > + if (key_cinfo->algorithm == HA_KEY_ALG_UNDEF) > + key_cinfo->algorithm= HA_KEY_ALG_LONG_HASH; Perhaps you shouldn't set key algorithm in add_hash_field() then? or, rather replace the assignment with an assert? > + > + if (key_cinfo->algorithm != HA_KEY_ALG_HASH && > + key_cinfo->algorithm != HA_KEY_ALG_LONG_HASH) > + { > + my_error(ER_TOO_LONG_KEY, MYF(0), max_key_length); > + DBUG_RETURN(TRUE); > + } > + } > + } > + } > + DBUG_RETURN(FALSE); > +} > + > + > /* > Preparation for table creation Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 2

MDEV-29744 fix required when merging to 11.4
by Kristian Nielsen 09 Jan '25

09 Jan '25

Hi Sanja, others who might need to merge 10.6->10.11->11.4: The patch for MDEV-29744 in 10.6 needs a fix when merged to 11.4, otherwise test binlog.max_binlog_total_size (at least) will fail. I have attached the required fix. I could not push this fix to 11.4 myself, as branch protection is broken and rejected my push on a false positive. - Kristian.

1 0

Re: 87188cd3309: MDEV-29744: Fix incorrect locking order of LOCK_log/LOCK_commit_ordered and LOCK_global_system_variables
by Sergei Golubchik 08 Jan '25

08 Jan '25

Hi, Kristian, Mostly ok, but I'd prefer not to access max_binlog_size without any protection, so suggest to do something like ulong saved= max_binlog_size; mysql_mutex_unlock(&LOCK_global_system_variables); mysql_bin_log.set_max_size(saved); mysql_mutex_lock(&LOCK_global_system_variables); On Jan 08, Kristian Nielsen wrote: > revision-id: 87188cd3309 (mariadb-10.6.20-47-g87188cd3309) > parent(s): 7372ecc3965 > author: Kristian Nielsen > committer: Kristian Nielsen > timestamp: 2024-12-05 09:51:17 +0100 > message: > > MDEV-29744: Fix incorrect locking order of LOCK_log/LOCK_commit_ordered and LOCK_global_system_variables > > --- a/sql/sys_vars.cc > +++ b/sql/sys_vars.cc > @@ -1660,7 +1660,9 @@ Sys_max_binlog_stmt_cache_size( > > static bool fix_max_binlog_size(sys_var *self, THD *thd, enum_var_type type) > { > + mysql_mutex_unlock(&LOCK_global_system_variables); > mysql_bin_log.set_max_size(max_binlog_size); > + mysql_mutex_lock(&LOCK_global_system_variables); > return false; > } > static Sys_var_on_access_global<Sys_var_ulong, Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 1

Why is innodb-log-file-mmap is available only on amd64/arm64/ppc64el but not other Linux architectures?
by Otto Kekäläinen 08 Jan '25

08 Jan '25

Hi, The documentation at https://mariadb.com/kb/en/innodb-system-variables/#innodb_log_file_mmap says this is supposed to be enabled by default on Linux, regardless of the architecture. However, in my testing it is not available on all architectures. Can somebody help me read the code path of innodb-log-file-mmap to figure out why the visibility/availability of this option it is currently architecture-dependent? ## Context In Debian I maintain an extra test called "configuration-trace" which basically dumps the output of these four commands into a file: /usr/bin/mariadb --print-defaults /usr/bin/mariadb --verbose --help /usr/sbin/mariadbd --print-defaults /usr/sbin/mariadbd --verbose --help Then, using the static file containing the text output of these files, it compares to the output of these commands on a running system to detect if the MariaDB defaults or features randomly changed (for example due to changes in dependencies or platform). This "configuration-trace" is also a handy way to detect what changed upstream in minor MariaDB releases, which aren't supposed to have much changes. The update of the static text output files for MariaDB 11.4.4 were done in https://salsa.debian.org/mariadb-team/mariadb-server/-/commit/e2851fcfca9e8… and they show among others a new variable innodb-log-file-mmap ## Problem The issues I am running into is that this new innodb-log-file-mmap only seems to exist on amd64 and arm64? I discovered this in the Debian CI system (autpkgtests) at https://ci.debian.net/packages/m/mariadb/ which is passing for amd64/arm64/ppc64el, but failing on other platforms: https://ci.debian.net/packages/m/mariadb/unstable/armel/55869348/ https://ci.debian.net/packages/m/mariadb/unstable/armhf/55869349/ https://ci.debian.net/packages/m/mariadb/unstable/i386/55869350/ https://ci.debian.net/packages/m/mariadb/unstable/loong64/55869351/ https://ci.debian.net/packages/m/mariadb/unstable/s390x/55869354/ (The output also shows that mariadbd --verbose --help prints out wsrep-provider twice, once wsrep-provider=ON and once wsrep-provider=none, but that is a different issue.) On systems where it isn't available, shouldn't the option still show up (but as permanently disabled)? It seems odd to have system variables appear or disappear depending on the Linux architecture.

2 8

Re: b16319bdd0f: MDEV-35469 Heap tables are calling mallocs to often
by Sergei Golubchik 30 Dec '24

30 Dec '24

Hi, Monty, > commit b16319bdd0f > Author: Michael Widenius <monty(a)mariadb.org> > AuthorDate: Thu Nov 21 12:28:57 2024 +0200 > Commit: Michael Widenius <monty(a)mariadb.org> > CommitDate: Fri Dec 27 13:53:37 2024 +0200 > > diff --git a/storage/heap/hp_block.c b/storage/heap/hp_block.c > index 324efc8b4af..44bca363746 100644 > --- a/storage/heap/hp_block.c > +++ b/storage/heap/hp_block.c > @@ -75,9 +75,11 @@ int hp_get_new_block(HP_SHARE *info, HP_BLOCK *block, size_t *alloc_length) > Next time we allocate data for X rows. > When level 1 is full, we allocate data for HP_PTRS_IN_NOD at level 2 and 1 > + X rows at level 0. > */ > *alloc_length= (sizeof(HP_PTRS) * ((i == block->levels) ? i : i - 1) + > (ulonglong)block->records_in_block * block->recbuffer); > + /* Alloc in blocks of powers of 2 */ > + *alloc_length= MY_MAX(*alloc_length, block->alloc_size); What will happen to block->alloc_size - *alloc_length bytes? they'll be unused? > if (!(root=(HP_PTRS*) my_malloc(hp_key_memory_HP_PTRS, *alloc_length, > MYF(MY_WME | > (info->internal ? > diff --git a/storage/heap/hp_create.c b/storage/heap/hp_create.c > index c07a1e968c4..29812f08107 100644 > --- a/storage/heap/hp_create.c > +++ b/storage/heap/hp_create.c > @@ -15,11 +15,23 @@ > Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1335 USA */ > > #include "heapdef.h" > +#include <my_bit.h> > > static int keys_compare(void *heap_rb, const void *key1, const void *key2); > static void init_block(HP_BLOCK *block,uint reclength,ulong min_records, > ulong max_records); > > + > +/* > + In how many parts are we going to do allocations of memory and indexes > + If we assigne 1M to the heap table memory, we will allocate roughly > + (1M/16) bytes per allocaiton > +*/ > +int heap_allocation_parts= 16; static. technically, it could be `static const`, But I presume the compiler can figure out the `const` part if the variable is static. And e.g. use >> 4. Otherwise it has to assume it can be modified in some other file. > + > +/* min block allocation */ > +ulong heap_min_allocation_block= 16384; same. static. > + > /* Create a heap table */ > > int heap_create(const char *name, HP_CREATE_INFO *create_info, > @@ -170,7 +182,8 @@ int heap_create(const char *name, HP_CREATE_INFO *create_info, > share->keydef= (HP_KEYDEF*) (share + 1); > share->key_stat_version= 1; > keyseg= (HA_KEYSEG*) (share->keydef + keys); > - init_block(&share->block, visible_offset + 1, min_records, max_records); > + init_block(&share->block, hp_memory_needed_per_row(visible_offset + 1), visible_offset is already MY_MAX(reclength, sizeof (char*)). perhaps you don't need visible_offset and should use hp_memory_needed_per_row(reclength) here? > + min_records, max_records); > /* Fix keys */ > memcpy(share->keydef, keydef, (size_t) (sizeof(keydef[0]) * keys)); > for (i= 0, keyinfo= share->keydef; i < keys; i++, keyinfo++) > @@ -266,44 +279,90 @@ static int keys_compare(void *heap_rb_, const void *key1_, > heap_rb->search_flag, not_used); > } > > + > +/* > + Calculate length needed for storing one row > +*/ > + > +size_t hp_memory_needed_per_row(size_t reclength) > +{ > + /* Data needed for storing record + pointer to records */ > + reclength= MY_MAX(reclength, sizeof(char*) + 1); > + reclength= MY_ALIGN(reclength, sizeof(char*)); > + return reclength; > +} > + > +/* > + Calculate the number of rows that fits into a given memory size > +*/ > + > +ha_rows hp_rows_in_memory(size_t reclength, size_t index_size, > + size_t memory_limit) > +{ > + /* We need one extra byte for the delete flag */ > + reclength= hp_memory_needed_per_row(reclength+1); you have +1 for the delete flag in hp_memory_needed_per_row() > + if ((memory_limit - sizeof(HP_PTRS)) < (index_size + reclength)) > + return 0; /* Wrong arguments */ Why do you need this if()? If its condition is true the expression below will return 0 naturally, you don't need a separate check for that. > + return (ha_rows) ((memory_limit - sizeof(HP_PTRS)) / > + (index_size + reclength)); > +} > + > + > static void init_block(HP_BLOCK *block, uint reclength, ulong min_records, > ulong max_records) > { > diff --git a/storage/heap/hp_write.c b/storage/heap/hp_write.c > index 5469784c8c1..cb079eac757 100644 > --- a/storage/heap/hp_write.c > +++ b/storage/heap/hp_write.c > @@ -145,6 +145,8 @@ static uchar *next_free_record_pos(HP_SHARE *info) > DBUG_PRINT("exit",("Used old position: %p", pos)); > DBUG_RETURN(pos); > } > + if (!(block_pos=(info->records % info->block.records_in_block))) > + { why? > if ((info->records > info->max_records && info->max_records) || > (info->data_length + info->index_length >= info->max_table_size)) > { Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 2

Re: 448e82f9f0d: MDEV-35720 Add query_time to statistics
by Sergei Golubchik 30 Dec '24

30 Dec '24

Hi, Monty, Only a couple of comments about the test case: > commit 448e82f9f0d > Author: Michael Widenius <monty(a)mariadb.org> > Date: Fri Dec 27 19:26:55 2024 +0200 > > diff --git a/mysql-test/main/log_state.test b/mysql-test/main/log_state.test > --- a/mysql-test/main/log_state.test > +++ b/mysql-test/main/log_state.test > @@ -321,6 +321,48 @@ SET GLOBAL general_log_file = @old_general_log_file; > > --enable_ps_protocol > > +--echo # > +--echo # MDEV-35720 Add query_time to statistics > +--echo # > + > +connect (con2,localhost,root,,); > +let $s1=`show status like "query_time"`; > +select sleep(3) into @a; > +let $s2=`show status like "query_time"`; > +let $s3=`show global status like "query_time"`; > +--disable_query_log > +--eval SET @s1= SUBSTRING_INDEX('$s1', '\t', -1); > +--eval SET @s2= SUBSTRING_INDEX('$s2', '\t', -1); > +--eval SET @s3= SUBSTRING_INDEX('$s3', '\t', -1); Eh, that's not how one is supposed to do it. I mean, it works, but the easier approach is let $s1=query_get_value(show status like "query_time", Value); or purely in SQL set @s1=(select variable_value from information_schema.global_status where variable_name='query_time'); > +select @s1 >= 0.00 and @s1 <= 2.00 as "should be true"; > +--eval select @s2 >= 2.00 and @s2 < 10.00 as "should be true"; > +--eval select @s3 >= 3.00 as "should be true"; 1. you don't need eval here 2. I expect this test, like almost all other timing tests will be horribly unstable in buildbot and will be eventually disabled. No, I don't have a recipe for a stable timing test. > +--enable_query_log > + > +# select @s1,@s2,@s3; > +disconnect con2; > +connection default; > + > +delimiter |; > +create procedure p1() > +begin > +select sleep(1) into @a; > +select sleep(2) into @a; > +end| > +delimiter ;| > + > +connect (con2,localhost,root,,); > +call p1(); > +let $s1=`show status like "query_time"`; > +--disable_query_log > +--eval SET @s1= SUBSTRING_INDEX('$s1', '\t', -1); > +--eval select @s1 > 2.00 and @s1 < 10.00 as "should be true"; > +--enable_query_log > + > +# select @s1; > +disconnect con2; > +connection default; > + > # > # Cleanup > # Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 1

Re: dfc1632da73: Change default MEM_ROOT allocations to decrease calls to malloc
by Sergei Golubchik 27 Dec '24

27 Dec '24

Hi, Monty, Please, cherry-pick it into 10.11 and push as bb-10.11-merge. There will be *a lot* of conflicts in memroot code because of your "Added support to MEM_ROOT for write protected memory" change, it'd be best if you resolve your conflicts yourself. On Dec 22, Michael Widenius wrote: > revision-id: dfc1632da73 (mariadb-10.6.20-60-gdfc1632da73) > parent(s): 3297680579f > author: Michael Widenius > committer: Michael Widenius > timestamp: 2024-12-17 10:45:25 +0200 > message: > > Change default MEM_ROOT allocations to decrease calls to malloc ... > --- a/mysys/my_alloc.c > +++ b/mysys/my_alloc.c > @@ -173,7 +180,7 @@ void *alloc_root(MEM_ROOT *mem_root, size_t length) > DBUG_ENTER("alloc_root"); > - DBUG_PRINT("enter",("root: %p", mem_root)); > + DBUG_PRINT("enter",("root: %p length: %ld", mem_root, (long) length)); FYI, my_vsnprintf() supports standard "%zu" to print size_t values. Not important here, of course. > DBUG_ASSERT(alloc_root_inited(mem_root)); Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 1

Re: 2ba3bbb2463: Updated misc_session_status.test to not fail if select does not fail
by Sergei Golubchik 27 Dec '24

27 Dec '24

Hi, Monty, On Dec 22, Michael Widenius wrote: > revision-id: 2ba3bbb2463 (mariadb-10.6.20-61-g2ba3bbb2463) > parent(s): dfc1632da73 > author: Michael Widenius > committer: Michael Widenius > timestamp: 2024-12-17 10:45:25 +0200 > message: > > Updated misc_session_status.test to not fail if select does not fail > > The test with memory restrictions randomly works or fails depending on > server configurations. > As the test was there to just check if the server crashes when run with > small memory configurations, I disabled testing if the query would fail > or not. The test still has its original purpose. This is very questionable. The test would still serve its original purpose indeed if it'll hit OOM at least somewhere. But with your change it's not clear that it'll ever do. The query might as well succeed everywhere and we'll never know. Or it might start succeeding in the future. That is, technically you are right, the test still does its job (presuming you've verified that it hits OOM at least on some builders). But your change basically guarantees that it'll become meaningless in the future. Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 1

Re: 6c5c1d0bf00: MDEV-35469 Heap tables are calling mallocs to often
by Sergei Golubchik 23 Dec '24

23 Dec '24

Hi, Monty, On Dec 21, Michael Widenius wrote: > revision-id: 6c5c1d0bf00 (mariadb-10.6.20-57-g6c5c1d0bf00) > parent(s): 671f80c7388 > author: Michael Widenius > committer: Michael Widenius > timestamp: 2024-12-17 10:45:25 +0200 > message: > > MDEV-35469 Heap tables are calling mallocs to often > > Heap tables are allocated blocks to store rows according to > my_default_record_cache (mapped to the server global variable > read_buffer_size). > This causes performance issues when the record length is big > (> 1000 bytes) and the my_default_record_cache is small. > > Changed to instead split the default heap allocation to 1/16 of the > allowed space and not use my_default_record_cache anymore when creating > the heap. The allocation is also aligned to be just under a power of 2. > > For some test that I have been running, which was using record length=633, > the speed of the query doubled thanks to this change. > > Other things: > - Fixed calculation of max_records passed to hp_create() to take > into account padding between records. > > diff --git a/storage/heap/hp_create.c b/storage/heap/hp_create.c > index c07a1e968c4..fcfcf607822 100644 > --- a/storage/heap/hp_create.c > +++ b/storage/heap/hp_create.c ... > + of the block. > + */ > + records_in_block= > + (my_round_up_to_next_power((uint32) > + (records_in_block * recbuffer + > + sizeof(HP_PTRS) + MALLOC_OVERHEAD)/2) / > + recbuffer); Strictly speaking, it should be (my_round_up_to_next_power(records_in_block * recbuffer + extra) - extra ) / recbuffer And why /2 ? Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 1

Re: 07364d701b3: Added status variable "Max_memory_used" to SHOW STATUS
by Sergei Golubchik 22 Dec '24

22 Dec '24

Hi, Michael, On Dec 22, Michael Widenius wrote: > revision-id: 07364d701b3 (mariadb-10.6.20-64-g07364d701b3) > parent(s): 13fed908291 > author: Michael Widenius > committer: Michael Widenius > timestamp: 2024-12-17 10:45:26 +0200 > message: > > Added status variable "Max_memory_used" to SHOW STATUS > > This shows the maximum memory allocations used by the current connection. > The value for @@global.max_memory_used is 0 as we are not collecting this > value as it would cause a notable performance issue registering this for > all threads for every memory allocation Showing 0 is very confusing. I suggest not to print this variable in SHOW GLOBAL STATUS at all (probably possible, by returning a zero-length SHOW_ARRAY). Or show it as NULL. Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

1 0

Re: 13fed908291: Added more information to errors reported by report_reply_packet()
by Sergei Golubchik 22 Dec '24

22 Dec '24

Hi, Monty, On Dec 22, Michael Widenius wrote: > revision-id: 13fed908291 (mariadb-10.6.20-63-g13fed908291) > parent(s): ca5ac5ac490 > author: Michael Widenius > committer: Michael Widenius > timestamp: 2024-12-17 10:45:26 +0200 > message: > > Added more information to errors reported by report_reply_packet() > > In particular ""Read semi-sync reply magic number error" now prints out > what was wrong with the packet. > > @@ -622,7 +626,8 @@ int Repl_semi_sync_master::report_reply_packet(uint32 server_id, > log_file_len = packet_len - REPLY_BINLOG_NAME_OFFSET; > if (unlikely(log_file_len >= FN_REFLEN)) > { > - sql_print_error("Read semi-sync reply binlog file length too large"); > + sql_print_error("Read semi-sync reply binlog file length too large: %llu", > + (longlong) log_file_pos); You used unsigned "%llu" but cast to a signed longlong. Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

1 0

Can find sponsor: Would anyone be interested in patching mha to be MariaDB gtid compatible?
by highclass99 22 Dec '24

22 Dec '24

Hello, Would anyone be interested in patching mha to be MariaDB gtid compatible? I think I would be able to find a sponsor to pay a bit for it. reference links) https://github.com/yoshinorim/mha4mysql-manager https://github.com/yoshinorim/mha4mysql-node https://github.com/charlesdirk/MariaDB-MHA -> could not get this to work: if anyone got it to work please let me know how to get it to work. Thank you. Have a nice day.

3 3

Re: 3297680579f: Remove end newlines from CHECK TABLE output.
by Sergei Golubchik 21 Dec '24

21 Dec '24

Hi, Michael, On Dec 21, Michael Widenius wrote: > revision-id: 3297680579f (mariadb-10.6.20-59-g3297680579f) > parent(s): 4038ec2cc2c > author: Michael Widenius > committer: Michael Widenius > timestamp: 2024-12-17 10:45:25 +0200 > message: > > Remove end newlines from CHECK TABLE output. > > InnoDB has in some cases (not always) an newline at the end of error > messages. This messes up the CHECK TABLE output. > Fixed CHECK TABLE to remove the end newlines in its result set. I'd rather fix InnoDB to not do that, instead of working around InnoDB bug that garbles the CHECK TABLE output Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

1 0

Re: e0afb5f08b5: cleanup: further replace thd->alloc() with operator new
by Sergei Golubchik 10 Dec '24

10 Dec '24

Hi, Nikita, Despite the commit has in the title, comments below apply to four commits: $ git log --oneline 7d5165894282^..0f6fd23e1604 0f6fd23e160 cleanup: add new(thd) to Sql_alloc and Item e0afb5f08b5 cleanup: further replace thd->alloc() with operator new 8daea2e14b2 cleanup: add operator new(size_t, const THD*) for single-object allocations 7d516589428 cleanup: add operator new[](size_t, const THD*) On Dec 06, Nikita Malyavin wrote: > diff --git a/sql/sql_class.h b/sql/sql_class.h > index aff02df69ec..299c8298606 100644 > --- a/sql/sql_class.h > +++ b/sql/sql_class.h > @@ -1279,11 +1279,5 @@ class Query_arena > { return state == STMT_CONVENTIONAL_EXECUTION; } > > template <typename T=char> > - inline T* alloc(size_t size) const > - { > - return (T*)alloc_root(mem_root, sizeof(T)*size); > - } > - > - template <typename T=char> > inline T* calloc(size_t size) const Hmm, you didn't remove thd->calloc<>() I understand why, but that kind of makes the API inconsistent, perhaps it would've been cleaner to keep thd->alloc<>() and only implement new (thd) and new (arena) for Sql_alloc? May be not only for Sql_alloc. There are 1. pod types. they can be alloc'ed an calloc'ed, I think thd->alloc/thd->calloc is fine for them 2. non-pod types, can be allocated on the heap and on a mem_root. your new (thd)/delete (thd) operators are best for them. 3. non-pod types that can only be allocated on a mem_root. Sql_alloc can be used for them, if it'll disallow non-placement new. I feel like we don't need to make a distinction between 2 and 3, in that case Sql_alloc can simply go away. > { > diff --git a/plugin/feedback/feedback.cc b/plugin/feedback/feedback.cc > index bc8a98c11ae..dcc2e4417e5 100644 > --- a/plugin/feedback/feedback.cc > +++ b/plugin/feedback/feedback.cc > @@ -103,22 +103,22 @@ static COND* make_cond(THD *thd, TABLE_LIST *tables, LEX_STRING *filter) > nrc->resolve_in_table_list_only(tables); > nrc->select_lex= tables->select_lex; > > - res= new (thd) Item_cond_or(thd); > + res= new (thd) Item_cond_or(thd); > if (!res) > return OOM; > > for (; filter->str; filter++) > { > - Item_field *fld= new (thd) Item_field(thd, nrc, db, table, > + Item_field *fld= new (thd) Item_field(thd, nrc, db, table, > field); > - Item_string *pattern= new (thd) Item_string(thd, filter->str, > + Item_string *pattern= new (thd) Item_string(thd, filter->str, > (uint) filter->length, cs); you might want to go through the diff and fix the indentation > - Item_string *escape= new (thd) Item_string(thd, "\\", 1, cs); > + Item_string *escape= new (thd) Item_string(thd, "\\", 1, cs); > > if (!fld || !pattern || !escape) > return OOM; > > - Item_func_like *like= new (thd) Item_func_like(thd, fld, pattern, > + Item_func_like *like= new (thd) Item_func_like(thd, fld, pattern, > escape, 0); > > if (!like) > diff --git a/sql/item.h b/sql/item.h > index 631fd1b19f8..b0cf03c1435 100644 > --- a/sql/item.h > +++ b/sql/item.h > @@ -857,8 +857,9 @@ class Item :public Value_source, > static void *operator new(size_t size); > > public: > - static void *operator new(size_t size, MEM_ROOT *mem_root) throw () > + static void *operator new(size_t size, MEM_ROOT *mem_root) noexcept > { return alloc_root(mem_root, size); } > + static void *operator new(size_t size, const THD *thd) noexcept; > static void operator delete(void *ptr,size_t size) { TRASH_FREE(ptr, size); } > static void operator delete(void *ptr, MEM_ROOT *mem_root) {} Better inherit it from Sql_alloc > diff --git a/sql/item_cmpfunc.cc b/sql/item_cmpfunc.cc > index d86fdf5c5d6..e0adc5c3bb4 100644 > --- a/sql/item_cmpfunc.cc > +++ b/sql/item_cmpfunc.cc > @@ -2007,7 +2007,7 @@ bool Item_func_interval::fix_length_and_dec(THD *thd) > > if (not_null_consts) > { > - intervals= thd->alloc<interval_range>(rows - 1); > + intervals= new(current_thd) interval_range[rows - 1]; current_thd ? > if (!intervals) > return true; > > diff --git a/sql/item_subselect.cc b/sql/item_subselect.cc > index 008384a6590..21364286260 100644 > --- a/sql/item_subselect.cc > +++ b/sql/item_subselect.cc > @@ -6086,8 +6086,8 @@ bool Ordered_key::init(int col_idx) > > // TIMOUR: check for mem allocation err, revert to scan > > - key_columns= thd->alloc<Item_field*>(1); > - compare_pred= thd->alloc<Item_func_lt*>(1); > + key_columns= new(thd) Item_field*();; > + compare_pred= new(thd) Item_func_lt*();; ;; > > key_columns[0]= new (thd) Item_field(thd, tbl->field[col_idx]); > /* Create the predicate (tmp_column[i] < outer_ref[i]). */ > diff --git a/sql/opt_subselect.h b/sql/opt_subselect.h > index 8b4dd1f0092..37902c6eebe 100644 > --- a/sql/opt_subselect.h > +++ b/sql/opt_subselect.h > @@ -357,7 +357,7 @@ uint get_number_of_tables_at_top_level(JOIN *join); > the tuple. > */ > > -class SJ_TMP_TABLE : public Sql_alloc > +class SJ_TMP_TABLE why? > { > public: > /* > diff --git a/sql/sql_alloc.h b/sql/sql_alloc.h > index f5d2d4e8b1a..e643a9f817d 100644 > --- a/sql/sql_alloc.h > +++ b/sql/sql_alloc.h > @@ -23,17 +23,21 @@ > class Sql_alloc > { > public: > - static void *operator new(size_t size) throw () > + static void *operator new(size_t size) noexcept > { > return thd_alloc(_current_thd(), size); > } can this be disallowed completely? > - static void *operator new[](size_t size) throw () > + static void *operator new[](size_t size) noexcept > { > return thd_alloc(_current_thd(), size); > } > - static void *operator new[](size_t size, MEM_ROOT *mem_root) throw () > + > + static void *operator new[](size_t size, const THD *thd) noexcept; > + static void *operator new(size_t size, const THD *thd) noexcept; > + > + static void *operator new[](size_t size, MEM_ROOT *mem_root) noexcept > { return alloc_root(mem_root, size); } > - static void *operator new(size_t size, MEM_ROOT *mem_root) throw() > + static void *operator new(size_t size, MEM_ROOT *mem_root) noexcept > { return alloc_root(mem_root, size); } > static void operator delete(void *ptr, size_t size) { TRASH_FREE(ptr, size); } > static void operator delete(void *, MEM_ROOT *){} > diff --git a/sql/sql_class.cc b/sql/sql_class.cc > index 7847389dabc..37869da63ae 100644 > --- a/sql/sql_class.cc > +++ b/sql/sql_class.cc > @@ -8831,3 +8831,46 @@ LEX_CSTRING make_string(THD *thd, const char *start_ptr, > size_t length= end_ptr - start_ptr; > return {strmake_root(thd->mem_root, start_ptr, length), length}; > } > +void* operator new(size_t size, const Query_arena *thd) noexcept > +{ > + return alloc_root(thd->mem_root, size); > +} > + > +void operator delete[](void *ptr, const Query_arena *thd) noexcept > +{} > + > +void operator delete(void *ptr, const Query_arena *thd) noexcept > +{} if you put them into sql_class.h - will they be inlined? Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 3

MDEV-34705: Storing binlog in InnoDB
by Kristian Nielsen 06 Dec '24

06 Dec '24

I am in the early stages of a project to hopefully implement a new binlog format that is managed transactionally by InnoDB. This is a complex change, and I want to solicit comments early, so I encourage feedback. The TLDR; of this: - The current binlog format stores events separately from InnoDB. This requires a complex and _very_ constly 2-phase commit protocol between the binlog and InnoDB for every commit. - The binlog format is rather naive, and not well suited as a high-performance transactional log. It exposes internal implementation detail to the slaves, and its flat-file nature makes it inflexible to read (for example finding the last event sequential start from the start). Replacing it with something new has the potential to reap many benefits. - This project proposes to extend the storage engine API to allow an engine to implement binlog storage. It will be implemented in InnoDB so that binlog events are stored in InnoDB tablespaces, and the standard InnoDB transactional and recovery mechanisms are used to provide crash safety and optionally durability. - This mechanism will allow full crash-recovery and consistency without any fsync per commit (--innodb-flush-log-at-trx-commit=0), providing huge improvement in throughput for applications that do not require strict durability requirements. And it will reduce fsync requirement to one per commit for durable config, with greatly improved opportunities for group commit. Below a long and somewhat raw write-up of my current state of design. There was some interest expressed in learning more about this work, an interest that I'm very pleased with ;-). The current state of my prototype implementation is available in the branch knielsen_binlog_in_engine: https://github.com/MariaDB/server/commits/knielsen_binlog_in_engine Architectural considerations. The fundamental problem with the binlog is that it functions as a transactional log, but it is separate from the InnoDB transactional log. This is very unoptimal. To ensure consistency in case of crash, a two-phase commit protocol is needed that requires two fsync per commit, which is very costly in terms of performance. And it causes a lot of complexity in the code as well. We should make it so that there is only a single transactional log ("single source of truth") used, at least for the case where only a single storage engine is used (eg. InnoDB) for all tables. Some different options for doing this can be considered (explained in detail below): 1. Implement a new binlog format that is stored in a new InnoDB tablespace type (or more generally in any storage engine that implements the appropriate new API for binlog). 2. Keep the current binlog format, extend the InnoDB redo log format with write-ahead records for binlog writes. Integrate binlog writes with InnoDB checkpointing and buffer pool flush. 3. Store the binlog events directly inside the InnoDB redo log. Implement log archiving of the InnoDB log to preserve old binlog data as long as desired. 4. (Mentioned only for completeness): Implement a general redo log facility on the server level, and change both InnoDB and binlog to use that facility. I am currently working on option (1), inspired by talks I had with Marko Mäkelä at the MariaDB fest last Autumn. He suggested that a new type of tablespace be added to InnoDB which would hold the binlog, and explained that this would require relative small changes in the InnoDB code. I think this is the best option. The InnoDB log and tablespace implementation is very mature, it is very good to be able to re-use it for the binlog. The integration into InnoDB looks to be very clean, and all the existing infrastructure for log write, recovery, checkpointing, buffer pool can be re-used directly. All the low-level detail of how the binlog is stored on disk can be handled efficiently inside the InnoDB code. This option will require a lot of changes in the binlog and replication code, which is the main cost of choosing this option. I have a _lot_ of experience with the MariaDB replication code, and I think this is feasible. And I think there are another set of huge benefits possible from implementing a new binlog format. The existing binlog format is very naive and neither very flexible, nor well suited as a high-performance transactional log. So changing the binlog format is something that should be done eventually anyway. Option (2) would also be feasible I think, and has the advantage of needing less changes to the replication code. Monty mentioned seeing a patch that sounded something like this. The main disadvantage is that I think this will require complex interaction between low-level InnoDB checkpointing code and low-level binlog writing code, which I fear will eventually lead to complex code that will be hard to maintain and will stiffle further innovation on either side. It also doesn't address the other problems with the current binlog format. To me, letting the storage engine control how it stores data transactionally - including the binlog events - is the "right" solution, and worth persuing over a quicker hack that tries to avoid changing code. Option (3) is also interesting. The idea would be that the binlog becomes identical to the InnoDB redo log, storing the binlog data interleaved with the InnoDB log records. This is conceptually simpler than options (1) and (2), in the sense that we don't have to first write-ahead-log a record for modifying binlog data, and then later write the actual data modification to a separate tablespace/file. But the InnoDB log is cyclic, and old data is overwritten as the log LSN wraps over the end. Thus, this option (3) will require implementing log archiving in InnoDB, which I am not sure how much work will be required. Either the binlog data could be archived by copying out asynchroneously to persistent binlog files, which becomes somewhat similar to option (1) or (2) then perhaps. Or InnoDB could be changed to create a new log file at wrap-over, renaming the old one to archive it, though this will leave the binlog data somewhat "dilluted", mixed up with InnoDB internal redo records. One pragmatic reason I've chosen option (1) over this option (3) for now is that I am more comfortable with doing large changes to replication code than to InnoDB code. Also, option (3) would more drastically change the way users can work with the binlog files, as a large amount of the most recent data would be sitting inside the InnoDB redo log and not in dedicated binlog files, unlike options (1) and (2). Still, option (3) is an interesting idea and has the potential to reduce the write amplification that will occur from (1) and (2). So I'm eager to hear any suggestions around this. Option (4) I mention only for completeness. Somehow, the "clean" way to handle transactional logging would be if the server layer provides a general logging service, and all the components would then use this for logging. InnoDB and the binlog, but also all the other logs or log-like things we have, eg. Aria, DDL log, partitioning, etc. However, I don't think it makes sense to try to discard the whole InnoDB write-ahead logging code and replace it with some new implementation done on the server level. And I'm not even sure it makes sense from a theoretical point of view - somehow, the logging method used is intrinsically a property of the storage engine; it seems dubious if a general server-level log facility could be designed that would be optimal for both InnoDB and RocksDB, for example. High-level design. Introduce a new option --binlog-storage-engine=innodb. Extend the storage engine API with the option for an engine to provide a binlog implementation, supported by InnoDB initially. Initially the option cannot be changed dynamically. When --binlog-storage-engine=innodb, the binlog files will use a different format, stored as a new type of InnoDB tablespace. It should support that the old part of the binlog is the legacy format and the new is the InnoDB format, to facilitate migrations. Maybe also support going back again, though this is less critical initially. A goal is to clean up some of the unnecessary complexity of the legacy binlog format. The old format will be supported for the foreseeable future, so the new format can break backwards compatibility. For example, the name of binlog files will be fixed, binlog-<NNNNNN>.ibb, so each file can be identified solely by its number NNNNNN. The option to decide the directory in which to store the binlog files should be supported though. I think we can require the slaves to use GTID when --binlog-storage-engine=innodb is enabled on the master. This way we can avoid exposing slaves to internal implementation details of the binlog, and the slaves no longer need to update the file relay-log.info at every commit. We should also be able to get rid of the Format_description_log_event and the Rotate_log_event at the start of every binlog file, so that the slave can view the events from the master as one linear stream and not care about how the data is split into separate binlog files on the master. The binlog format will be page based. This will allow pre-allocating the binlog files and efficiently writing them page by page. And it will be possible to access the files without scanning them sequentially from the start; eg. we can find the last event by reading the last page of the file, binary-searching for the last used page if the binlog file is partially written. The GTID indexes can be stored inside the same tablespace as the binlog data (eg. at the end of the tablespace), avoiding the need for a separate index file. GTID index design is still mostly TBD, but I think it can be implemented so that indexes are updated transactionally as part of the InnoDB commit and no separate crash recovery of GTID indexes is needed. With GTID indexes being guaranteed available, we can use them to obtain the GTID state at the start of each binlog file, and avoid the need for the Gtid_list_log_event at the start of the binlog. With a page-based log with extensible format, metadata can be added to the binlog that is only used on the master without having to introduce new replication events that are not relevant to the replication on the slave. This can be used eg. to eliminate the Binlog_checkpoint_log_event, for example. Possibly the binlog checkpoint mechanism can be completely avoided for the --binlog-storage-engine=innodb case, since there is no more 2-phase commit needed in the common case of an InnoDB-only transaction. The existing binlog checksum and encryption will no longer be used, instead the standard InnoDB checksums and encryption will be reused. Implementation plan. The storage engine API must be extended to provide a facilities for writing to the binlog and for reading from the binlog. The design of the API is TBD, should be discussed to try to make it generic and suitable for other engines than InnoDB. When writing to binlog in InnoDB, the central idea is to use the same mini-transaction (mtr) that marks the transaction as committed in InnoDB. This is what makes the binlog data guaranteed consistent with the InnoDB table data without the need for two-phase commit. The initial version I think can use the existing binlog group commit framework; this will simplify the implementation. This will thus keep the LOCK_commit_ordered and the queue_for_group_commit() mechanisms. Later work can then be to see if this can be even further improved in terms of scalability. I want to implement that large event groups can be split into multiple chunks in the binlog that no longer need to be consecutive. The final chunk will be the one that contains the GTID and marks the event group binlogged; this final chunk will then refer back to the other parts. This way, a large transactions can be binlogged without stalling the writing of other (smaller) transactions in parallel, which is a bottleneck in the legacy binlog. And it avoids problems with exceeding the maximum size of an mtr. In the first version, I think the binlog reader in the dump thread will collect and concatenate the different chunks before sending them to the slave, so simplify the initial implementation. But a later change could allow the slave to receive the different chunks interleaved between different event groups. This can even eventually allow the slave to speculatively execute events even before the transaction has committed on the master, to potentially reduce slave lag to less than the time of even a single transaction; this would be a generalisation of the --binlog-alter-two-phase feature. The binlog will be stored in separate tablespace files, each of size --max-binlog-size. The binlog files will be pre-allocated by a background thread. Since event groups can spill over to the next file, each file can be a fixed size, and hopefully also the rotate events will no longer be necessary. A busy server will quickly cycle through tablespace files, and we want to avoid "using up" a new tablespace ID for each file (InnoDB tablespace IDs are limited to 2**32 and are not reused). Two system tablespace IDs will be reserved for the binlog, and new binlog files will alternate between them. This way, the currently written binlog can be active while the previous one is being flushed to disk and the remaining writes checkpointed. Once the previous log has been flushed completely, its tablespace ID can be re-used for the next, pre-allocated binlog file and be ready for becoming active. This way, the switching to the next binlog file should be able to occur seamlessly, without any stalls, as long as page flushing can keep up. The flushing of binlog pages will be prioritised, to avoid stalling binlog writes, to free up buffer pool pages that can be used more efficiently than holding binlog data, and to quickly make the binlog files readable from outside the server (eg. with mysqlbinlog). The binlog dump thread that reads from the binlog and sends the data to slaves will use a binlog reading API implemented in InnoDB. I will prefer to read directly from the binlog files, in order to reduce pressure on the InnoDB buffer pool etc. A slave that connects from an old position back in time may need to read a lot of old data from the binlogs; there is little value in loading this data into the buffer pool, evicting other more useful pages. The reader can lookup in the InnoDB buffer pool with the BUF_GET_IF_IN_POOL flag. This way, the data can be accessed from the buffer pool if it is present. If not present, we can be sure that the data will be in the data file, and can read it from the file directly. If the data is known to be already flushed to disk before the specific binlog position, then the buffer pool lookup can be skipped altogether. The mysqlbinlog program will need to be extended somehow to be able to read from an InnoDB tablespace (or in general other storage engine). I think this means mysqlbinlog needs some kind of engine plugin facility for reading binlog. Note that the -read-from-remote-server option will also be available to read data from a mysqlbinlog that doesn't the new format, or to read the very newest data before it gets written to disk. Final words. This is still an early draft of the feature, as it is being worked on and refined; things are subject to change. And this also means that things are *open* to change, according to any suggestions with a good rationale.

2 7

Review of MDEV-6017 Add support for Indexes on Expressions
by Michael Widenius 03 Dec '24

03 Dec '24

Hi! Here is the review for MDEV-6017 Add support for Indexes on Expressions commit c1e7574c799163f5d8a26ca8e0f91af6c91f7077 Instead of adding bool function virtual bool vcol_subst_analyzer(uchar **) { return false; } Why not add Item_base_t flag: IS_SIMPLE_BOOL_FUNC This takes one bit instead of increasing ALL created item base structures with 8 bytes. As we have 100+ items, adding a virtual function increases code space with 1K or more + the functions. You can also replace vcol_subst_analyzer() with a top level function: bool Item::vcol_subst_analyzer/( { return (type() == Item::FUNC_ITEM && (((Item_func*) this)->bitmap_bit() & (BITMAP_EQ | BITMAP_EQUAL | BITMAP_LT | BITMAP_GT | BITMAP_LE | BITMAP_GE | BITMAP_BETWEEN | BITMAP_IN | BITMAP_ISNULL | BITMAP_ISNOTNULL ) } For this to work, you would have to add the last two bitmaps to enum Bitmap in item_func.h @@ -3272,6 +3292,7 @@ class Item_cond :public Item_bool_func bool excl_dep_on_table(table_map tab_map) override; bool excl_dep_on_grouping_fields(st_select_lex *sel) override; + bool vcol_subst_analyzer(uchar **) override { return true; } I assume you need the above to be able to traverse AND and OR expressions for WHERE. Please add a comment that this is the intent. If you decide to use the bitmaps, add an explanation why AND and OR is in the vcol_subst_analyzer list. ---- +void collect_indexed_vcols_for_table(TABLE *tbl, List<Field> *vcol_fields) +{ Please use table instead of tbl. We use table as a varible much more than tbl: ((/my/maria-10.5/sql)) grep tbl *.* | wc 1143 5371 73297 ((/my/maria-10.5/sql)) grep table *.* | wc 40886 236754 2832202 + for (uint i=0; i < tbl->s->keys; i++) + { + // note: we could also support histograms here + if (!tbl->keys_in_use_for_query.is_set(i)) + continue; I wonder if it would be a good idea to extend bitmap to have an iterator like we have for tables where we only loop over the set bits. From sql_bitmap.h: class Table_map_iterator { ulonglong bmp; public: Table_map_iterator(ulonglong t): bmp(t){} uint next_bit() { if (!bmp) return BITMAP_END; uint bit= my_find_first_bit(bmp); bmp &= ~(1ULL << bit); return bit; } int operator++(int) { return next_bit(); } enum { BITMAP_END= 64 }; }; Would be quite trival to do also for keys. This would help tables with many keys and also would allows us to remove the 'call' to if (!tbl->keys_in_use_for_query.is_set(i)) We could also do an optimized version where we implement bitmap_get_next_set() where we do not have to reset the previous bits for each call. --------- + if (field->vcol_info) + vcol_fields->push_back(field); You have to take care of error handling here. Please change the functions collect_indexed_vcols_for_table(), collect_indexed_vcols_for_join() and other functions that calls push_back() to bool and return 1 in case of error. + item->top_level_compile(ctx->thd, + &Item::vcol_subst_analyzer, &yes, + &Item::vcol_subst_transformer, (uchar*)ctx); You also have to check the return of the above for errors. Another option is to just check thd->fatal_error() as this will be set in case of OOM. .... + { + if (auto nested_join= table->nested_join) + subst_vcols_in_join_list(ctx, &nested_join->join_list); PLEASE do not use auto. I want to know the type while reading and debugging the code! auto can also introduce bugs if a type changes. + @todo: this cannot handle VIEW columns currently. View columns are wrapped + in Item_direct_view_ref objects which do not comsider themselves equal to + Item_field objects they're wrapping. Please add this also to the commit comment. + /* + Do not do the substitution if the datatypes do not match. + Virtual column's datatype can be different from the expression, for + example: + + col3 INT AS (CONCAT(col1, col2)) + + Do not allow substitutions that change the semantics of comparison. + At the moment, we require that datatypes are the same. + This probably could be relaxed. + + For strings: we allow two cases: + - vcol_expr and vcol_field have the same collation. + - vcol_field has the same collation as the comparison collation. + + (Note: MySQL calls resolve_type() after it has done the substitution. + This can potentially update the comparator. The idea is that this + shouldn't be necessary as we do not want to change the comparator. + Changing the comparator will change the semantics of the condition, + our point is that this must not happen) + */ Move to a function comment. It is hard to read read the function code with this long comment in the middle. + if (vcol_expr->type_handler_for_comparison() != + vcol_field->type_handler_for_comparison() || + vcol_expr->maybe_null() != vcol_field->maybe_null()) + fail_cause="type mismatch"; Why cannot we handle this case if vcol_expr->maybe_null is 0 and vcol_field->maybe_null() is set? In this case we know that the vcol_field's we are going to encounter can never be null, which should satisfy our requirements. + Item_field *itf= new (thd->mem_root) Item_field(thd, vcol_field); + if (!itf) + return false; Better to return true in case of errors, like we do in most of the other code. Remove and a blank line instead + //Item *const_expr; ... remove + //const_expr= args[0]; ----- +Item* Item_func_in::vcol_subst_transformer(THD *thd, uchar *arg) +{ + Vcol_subst_context *ctx= (Vcol_subst_context*)arg; + if (!compatible_types_scalar_bisection_possible()) Add comment /* Check that all arguments inside IN() are constants */ Regards, Monty

1 0

Re: 5b6b23563bf: MDEV-34979 generate SBOM from server builds
by Sergei Golubchik 02 Dec '24

02 Dec '24

Hi, Vladislav, Thanks! See comments below On Dec 02, Vladislav Vaintroub wrote: > revision-id: 5b6b23563bf (mariadb-11.7.1-5-g5b6b23563bf) > parent(s): eff9c198e32 > author: Vladislav Vaintroub > committer: Vladislav Vaintroub > timestamp: 2024-11-25 19:04:18 +0100 > message: > > MDEV-34979 generate SBOM from server builds > > This commit adds the capability to generate a Software Bill of Materials > (SBOM) from server builds. > > It introduces a new WITH_SBOM variable, which defaults to ON for package > builds (i.e if BUILD_CONFIG is used) and to OFF otherwise. > > When enabled, the build process will produce an sbom.json document in > CycloneDX format, capturing information about various dependencies, > which is gathered from various sources. > > We use git submodule information and CMake external projects properties > to gather version information for 3rd party code, but also handle > dependencies if external code is part of our repository > (zlib, or Connect storage engine's minizip) > > The SBOM document is stored in the root build directory in sbom.json file, > but is not currently installed. > > diff --git a/CMakeLists.txt b/CMakeLists.txt > --- a/CMakeLists.txt > +++ b/CMakeLists.txt > @@ -581,6 +581,13 @@ ENDIF() > > INCLUDE(build_depends) > > +INCLUDE(CMakeDependentOption) > +CMAKE_DEPENDENT_OPTION(WITH_SBOM "Generate Software Bill of Materials (SBOM)" OFF "BUILD_CONFIG" ON) > +IF(WITH_SBOM) > + INCLUDE(generate_sbom) > + GENERATE_SBOM() > +ENDIF() This doesn't work for me, I mean, cmake -DBUILD_CONFIG=mysql_release . does not generate the file, but cmake -DBUILD_CONFIG=mysql_release . -DWITH_SBOM=ON does. May be it'd be more robust to use OPTION() and set it from cmake/build_configuration/mysql_release.cmake? Like it's done for other options. > + > INCLUDE(CPack) > > IF(WIN32 AND SIGNCODE) > diff --git a/cmake/generate_submodule_info.cmake b/cmake/generate_submodule_info.cmake > --- /dev/null > +++ b/cmake/generate_submodule_info.cmake > @@ -0,0 +1,73 @@ ... > + FOREACH(s ${out_list}) > + IF(${counter} EQUAL 0) > + IF (NOT("${s}" MATCHES "Entering '")) does it depend on the locale? > + MESSAGE(FATAL "Unexpected output ${outvar}") > + ENDIF() > + STRING(LENGTH "${s}" slen) > + MATH(EXPR substr_len "${slen} - 11") ... > diff --git a/cmake/generate_sbom.cmake b/cmake/generate_sbom.cmake > --- /dev/null > +++ b/cmake/generate_sbom.cmake > @@ -0,0 +1,240 @@ > +INCLUDE(generate_submodule_info) > +INCLUDE(ExternalProject) > + > +# Remove all matches from a cmake list > +FUNCTION (REMOVE_ALL_MATCHES list_var_name match_str) > + SET(new_list) > + FOREACH(el ${${list_var_name}}) > + IF(NOT("${el}" MATCHES "${match_str}")) > + LIST(APPEND new_list ${el}) > + ENDIF() > + ENDFOREACH() > + SET(${list_var_name} ${new_list} PARENT_SCOPE) > +ENDFUNCTION() you could've used LIST(FILTER ...) instead. (but not LIST(REMOVE_ITEM ...) which is 3.16+) > + > +# Extract user name and repository name from a github URL. > +FUNCTION (EXTRACT_REPO_NAME_AND_USER repo_url repo_name_var repo_user_var) > + IF(repo_url MATCHES "^git@") > + # normalize to https-style URLs > + STRING(REGEX REPLACE "^git@([^:]+):(.*)$" "https://\\1/\\2" repo_url "${repo_url}") I don't think we have that, is it even tested? :) note, I'm not saying it isn't needed, better to have it than not. > + ENDIF() > + # Extract the repository user > + STRING(REGEX REPLACE "https://([^/]+)/([^/]+)/.*" "\\2" repo_user "${repo_url}") > + > + STRING(REGEX REPLACE ".*/([^/]*)$" "\\1" repo_name "${repo_url}") > + STRING(REGEX REPLACE "\\.git$" "" repo_name "${repo_name}") > + > + SET(${repo_name_var} ${repo_name} PARENT_SCOPE) > + SET(${repo_user_var} ${repo_user} PARENT_SCOPE) > +ENDFUNCTION() > + > +# Add a known 3rd party dependency for SBOM generation > +# Currently used for "vendored" (part of our repository) source code we know about > +# such as zlib, as well ExternalProject_Add() projects > +MACRO(ADD_THIRD_PARTY_DEPENDENCY name url tag rev version description) > + LIST(FIND ALL_THIRD_PARTY ${name} idx) > + IF (idx GREATER -1) > + MESSAGE(FATAL_ERROR "${name} is already in ALL_THIRD_PARTY") > + ENDIF() > + SET(${name}_URL ${url}) > + SET(${name}_TAG ${tag}) > + SET(${name}_REVISION ${rev}) > + SET(${name}_DESCRIPTION "${description}") > + SET(${name}_VERSION "${version}") > + LIST(APPEND ALL_THIRD_PARTY ${name}) > +ENDMACRO() > + > +# Match third party component with supplier > +# CyclonDX documentation says it is > +# "The organization that supplied the component. > +# The supplier may often be the manufacturer, but may also be a distributor or repackager." > +# > +# Perhaps it can always be "MariaDB", but security team recommendation is different > +# more towards "author" > +FUNCTION (sbom_get_supplier repo_name repo_user varname) why a function? just curious > + IF("${repo_name_SUPPLIER}") > + SET(${varname} "${repo_name_SUPPLIER}" PARENT_SCOPE) ... > + # It is possible to provide EXTRA_SBOM_DEPENDENCIES > + # and accompanying per-dependency data, to extend generared sbom > + # document. > + # Example below injects an extra "ncurses" dependency using several > + # command line parameters for CMake. > + # -DEXTRA_SBOM_DEPENDENCIES=ncurses > + # -Dncurses_URL=https://github.com/mirror/ncurses > + # -Dncurses_TAG=v6.4 > + # -Dncurses_VERSION=6.4 > + # -Dncurses_DESCRIPTION="A fake extra dependency" > + SET(ALL_THIRD_PARTY ${ALL_SUBMODULES} ${EXTRA_SBOM_DEPENDENCIES}) okay. Though I personally wouldn't use ALL_SUBMODULES and EXTRA_SBOM_DEPENDENCIES and instead would scan GET_CMAKE_PROPERTY(VARS VARIABLES) for *_TAG or something > + > + IF(TARGET ha_connect OR TARGET connect) > + ADD_THIRD_PARTY_DEPENDENCY(minizip > + "https://github.com/zlib-ng/minizip-ng" > + "" 252588f "" > + "Vendored minizip, inside connect storage engine, storage/connect/zip.c et al") did you verify that our file matches the content of the file from that repo at that commit? also, a though - how can we prevent our minigzip.c from being modified without updaing the dependency? Suggestion, check the commit hash, like git log -n1 --format=%h storage/connect/zip.c > + ENDIF() > + > + IF(TARGET libfmt) > + ExternalProject_GET_PROPERTY(libfmt URL) > + STRING(REGEX MATCH "[0-9]+\\.[0-9]+\\.[0-9]+" libmt_TAG "${URL}") > + ADD_THIRD_PARTY_DEPENDENCY(libfmt > + "https://github.com/fmtlib/fmt" > + "${libmt_TAG}" "${libmt_TAG}" "${libmt_TAG}" > + "header only library, used in server") > + ENDIF() I wonder, wouldn't it be simpler and more robust to overwrite or wrap ExternalProject() to do the above automatically? (using the same name or a different one like MyExternalProject) ... Also, I noticed that the server entry gets "purl": "pkg:github//@5b6b23563bf" perhaps it should have a repo name here? Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

1 0

Re: 534dda1234b: MDEV-35522: MariaDB Audit does not detect all DCLs forms when masking password
by Sergei Golubchik 02 Dec '24

02 Dec '24

Hi, Oleksandr, Ok to push. But do you no longer delete server_audit.log after the test? On Dec 02, Oleksandr Byelkin wrote: > revision-id: 534dda1234b (mariadb-10.5.27-64-g534dda1234b) > parent(s): fdb6db6b47f > author: Oleksandr Byelkin > committer: Oleksandr Byelkin > timestamp: 2024-11-29 11:45:01 +0100 > message: > > MDEV-35522: MariaDB Audit does not detect all DCLs forms when masking password > > 1. skip OR REPLACE > (to make it filter_query_type made recursive) > > 2. skip SET STATEMENT ... FOR before checking > statements with passwords > > diff --git a/mysql-test/suite/plugins/t/server_audit_pwd_mask.test b/mysql-test/suite/plugins/t/server_audit_pwd_mask.test > --- a/mysql-test/suite/plugins/t/server_audit_pwd_mask.test > +++ b/mysql-test/suite/plugins/t/server_audit_pwd_mask.test > --echo # cleaunup > +DROP SERVER s1; > DROP USER u1; > DROP USER u2; > set global server_audit_logging=off; > ---remove_file $SEARCH_FILE > +##--remove_file $SEARCH_FILE why? > --disable_warnings > UNINSTALL PLUGIN ed25519; > UNINSTALL PLUGIN server_audit; > --enable_warnings Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 1

Re: ff3d06f6601: MDEV-22217 Make OS character sets "utf8" and "utf-8" map to MariaDB character set "utf8mb4"
by Sergei Golubchik 16 Nov '24

16 Nov '24

Hi, Alexander, Of course, there can be no comments about the code change here. But the question is, is it something we want to do? 1. Is utf8/utf-8 locale indeed 4-byte utf8? 2. Why is hard-coding to utf8mb4 better than having it dependent on UTF8_IS_UTF8MB3 ? On Nov 16, Alexander Barkov wrote: > revision-id: ff3d06f6601 (mariadb-11.6.1-44-gff3d06f6601) > parent(s): 4b6922a315f > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2024-10-28 14:11:40 +0400 > message: > > MDEV-22217 Make OS character sets "utf8" and "utf-8" map to MariaDB character set "utf8mb4" > > diff --git a/mysql-test/main/mysql_locale_posix.test b/mysql-test/main/mysql_locale_posix.test > index c047b7b1ae9..b37d04baca2 100644 > --- a/mysql-test/main/mysql_locale_posix.test > +++ b/mysql-test/main/mysql_locale_posix.test > @@ -203,3 +203,20 @@ DROP DATABASE `Ã³`; > + > +--exec LC_ALL=en_US.utf8 $MYSQL --default-character-set=auto test -e "SELECT @@character_set_client" > +--exec LC_ALL=en_US.utf8 $MYSQL --default-character-set=auto test -e "SELECT @@character_set_connection" > +--exec LC_ALL=en_US.utf8 $MYSQL --default-character-set=auto test -e "SELECT @@character_set_results" may be do it in one SELECT? or in SHOW VARIABLES LIKE 'character_set%' ? to have it a tiny bit faster? Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

1 0

2024 Q4 release part 2 (11.6.2, 11.7,1)
by Daniel Black 12 Nov '24

12 Nov '24

All, The 11.6.2 and 11.7.1 releases are now built: * 11.6.2 d8dad8c3b54cd09fefce7bc3b9749f427eed9709 (GA) * 11.7.1 74331a48dd3dadfe5041f3440b81a7183e9fd4a8 release notes still work in progress: * https://mariadb.com/kb/en/mariadb-11-6-2-release-notes/ * 11.7.1 still being prepared but will be 11.6.2 like + the features of 11.7.0 release notes + Vector search, + bugs fixes since 11.7.0. Both are rolling releases and once GA, like 11.6.2, will be the last of the release series. containers for testing are available: * quay.io/mariadb-foundation/mariadb-devel:11.6 * quay.io/mariadb-foundation/mariadb-devel:11.7 Build packages for RPM/Deb distros repo/source files: https://ci.mariadb.org/ * 11.6.2 - bb-11.6-release-latest* * 11.7.1 - bb-11.7-release-latest* Sources: * https://ci.mariadb.org/51755/ (11.6.2) * https://ci.mariadb.org/51770/ (11.7.1) Now begins the process of loading packages to mirrors and Docker Official Images releases.

1 0

Contribute to MariaDB compatibility in open source projects?
by Robert Silén 06 Nov '24

06 Nov '24

All, We have been working on community outreach as described in an earlier blog Improving MariaDB support in open source projects <https://mariadb.org/improving-mariadb-support-in-open-source-projects/>. Many projects have made efforts to improve their MariaDB compatibility with or without our help, and also documented compatibility more visibly in READMEs and documentation. However, we at the Foundation don't have the knowhow or resources to help out everybody, so I thought I'd share a list of some issues. Would somebody like to contribute to any of them? Any comment on the issues, or code contributions to them would be much appreciated. Thank you. Do share any thoughts or actions on this. Robert Silén, community advocacy at MariaDB Foundation robert.silen(a)mariadb.org - https://github.com/SOCI/soci/issues/1150: add MariaDB to README - https://github.com/tinode/chat/pull/924: Help set up CI for MariaDB (and MySQL) - https://github.com/inspircd/inspircd-docs/issues/239: tls and srv not available in MariaDB - https://github.com/umami-software/umami/discussions/2943: BIN_TO_UUID() not available in MariaDB (MDEV-15854 <https://jira.mariadb.org/browse/MDEV-15854>) - https://github.com/diesel-rs/diesel/issues/1882: Create a distinct backend for MariaDB - https://github.com/grafana/grafana: Create a dedicated DataStore for MariaDB - https://github.com/drizzle-team/drizzle-orm/pull/1692: WIP: Add support for Maria DB (core API implementation) - https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1079753: Create mariadb specific dbconfig-mariadb - https://core.trac.wordpress.org/ticket/60037: Fix version checking in next release - https://gitlab.com/zoph-project/zoph/-/issues/267: give a thumbs up to their own addition of MariaDB to documentation (preferably recommend MariaDB)

2 1

Re: e8021aaf28e: auth: refactor MYSQL_AUTHENTICATION_INTERFACE_VERSION naming
by Sergei Golubchik 28 Oct '24

28 Oct '24

Hi, Nikita, This looks very confusing, this is not how versions work. What are you trying to achieve by that? On Oct 07, Nikita Malyavin wrote: > revision-id: e8021aaf28e (mariadb-11.6.1-9-ge8021aaf28e) > parent(s): a5b80531fbb > author: Nikita Malyavin > committer: Nikita Malyavin > timestamp: 2024-10-04 00:16:17 +0200 > message: > > auth: refactor MYSQL_AUTHENTICATION_INTERFACE_VERSION naming > > Rename MYSQL_AUTHENTICATION_INTERFACE_VERSION to > MYSQL_AUTH_INTERFACE_VERSION_2_02. > This is the current latest version, and it will be fixed to all auth plugins > for current implementation. > MYSQL_AUTHENTICATION_INTERFACE_VERSION will stay for compatibility with 3-rd > party auth plugins and will be fixed to V. 2.02. > Add MYSQL_AUTHENTICATION_INTERFACE_VERSION_LAST that will be updated as needed. > Add MYSQL_AUTH_INTERFACE_MAJOR_MASK to identify major version. > Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 5

Re: b17308e254a: MDEV-34854 Parsec sends garbage when using an empty password
by Sergei Golubchik 28 Oct '24

28 Oct '24

Hi, Nikita, On Oct 07, Nikita Malyavin wrote: > revision-id: b17308e254a (mariadb-11.6.1-10-gb17308e254a) > parent(s): e8021aaf28e > author: Nikita Malyavin > committer: Nikita Malyavin > timestamp: 2024-10-04 01:33:47 +0200 > message: > > MDEV-34854 Parsec sends garbage when using an empty password > > When an empty password is set, the server doesn't call > st_mysql_auth::hash_password and leaves > MYSQL_SERVER_AUTH_INFO::auth_string empty. > > Fix: > generate hashes for empty passwords as well. This breaks some auth > plugins, so we increment interface version and do it only from Auth V. > 2.03. > > Some empty passwords could be already stored with no though. The user "with no though" ? > will have to call SET PASSWORD once again, anyway the authentication > wouldn't have worked for such password. ok, I presume you mean ed25519 only, because mysql_native_password worked with an empty password and it generates an empty hash for it. > diff --git a/mysql-test/suite/plugins/r/parsec.result b/mysql-test/suite/plugins/r/parsec.result > index 512c066e2d7..b7e3537af29 100644 > --- a/mysql-test/suite/plugins/r/parsec.result > +++ b/mysql-test/suite/plugins/r/parsec.result ... let's add ed25519 test too, for completeness. > diff --git a/sql/sql_acl.cc b/sql/sql_acl.cc > index 2722ea2ea19..ba05a5656c5 100644 > --- a/sql/sql_acl.cc > +++ b/sql/sql_acl.cc > @@ -2402,7 +2402,10 @@ static int set_user_auth(THD *thd, const LEX_CSTRING &user, > res= ER_NOT_VALID_PASSWORD; > goto end; > } > - if (pwtext.length) > + > + // Starting from version 2.03 we also generate hash for empty passwords. > + if ((info->interface_version >= MYSQL_AUTH_INTERFACE_VERSION_2_03 I don't understand this MYSQL_AUTH_INTERFACE_VERSION_2_03 thing. First, again, that's not how a version is supposed to work. Second, this empty-password change isn't a change in the API. You can simply start calling hash_password() for empty passwords and it'll just work. I've tried :) > + && pwtext.str) || pwtext.length) > { > if (info->hash_password) > { Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 3

2024 Q4 release
by Daniel Black 27 Oct '24

27 Oct '24

All, Just letting you know that we're in the progress of getting the releases ready. The issue where this is tracked is: * https://jira.mariadb.org/browse/MDEV-35030 Branches being released are at the tips of these branches: bb-10.5-release (LTS) bb-10.6-release (LTS) bb-10.11-release (LTS) bb-11.2-release bb-11.4-release (LTS) bb-11.6-release bb-11.7-release So far the code is merged up to bb-11.4-release. Later branches will be updated, check history before using. There are two blocking bugs on bb-11.2-release onwards affecting compilation of the columnstore and connect storage engine. In addition to compiling the source yourselves; Containers: quay.ion/mariadb-foundation/mariadb-devel:{release} list: https://quay.io/repository/mariadb-foundation/mariadb-devel?tab=tags note: * 11.4 is from the 11.4 branch rather than bb-11.4-release, * 11.2 didn't build due to blocker bugs). Repos for testing: https://ci.mariadb.org/ contains for deb and rpm distros a repo file in the form: bb-{version}-release-latest-{arch}-{distro}-{version}-{deb|rpm}-autobake.{sources|repo} Please report any serious regressions with a Blocker priority bug, and mark this issue blocking MDEV-35030.

1 0

Re: 692ca003f8c: MDEV-33658 2/2 Cannot add a foreign key on a table with a matching long UNIQUE
by Sergei Golubchik 23 Oct '24

23 Oct '24

Hi, Nikita, On Oct 18, Nikita Malyavin wrote: > revision-id: 692ca003f8c (mariadb-10.5.26-169-g692ca003f8c) > parent(s): ba988513ec7 > author: Nikita Malyavin > committer: Nikita Malyavin > timestamp: 2024-10-13 19:33:14 +0200 > message: > > MDEV-33658 2/2 Cannot add a foreign key on a table with a matching long UNIQUE > > Cannot add a foreign key on a table with a long UNIQUE multi-column index, that > contains a foreign key as a prefix. > > Check that index algorithms match during the "generated" keys duplicate removal. > > diff --git a/mysql-test/main/long_unique_bugs.test b/mysql-test/main/long_unique_bugs.test > index 2f85328bda3..08b49b6c5da 100644 > --- a/mysql-test/main/long_unique_bugs.test > +++ b/mysql-test/main/long_unique_bugs.test > @@ -706,4 +706,19 @@ alter table t1 enable keys; > insert into t1 values (2); > drop table t1; > > +--echo # > +--echo # Cannot add a foreign key on a table with a long UNIQUE multi-column > +--echo # index, that contains a foreign key as a prefix. > +--echo # > +create table a (id int(19) primary key) engine = innodb; > +create table b (id int primary key, > + long_text varchar(1000), > + constraint unique_b unique key (id, long_text) > + ) engine = innodb default charset utf8mb4; > + > +alter table b add constraint b_fk_ida foreign key (id) references a (id); > +show create table b; > +drop table b; > +drop table a; Two problems with this test case. A minor one - it doesn't start with the --echo # MDEV-33658 Cannot add a foreign key on a table with a matching long UNIQUE line. Please add it. A more serious problem - it doesn't fail on a vanilla 10.5. The original set of SQL commands from MDEV-33658 fails with "Foreign key constraint is incorrectly formed", but your test doesn't. You might've oversimplified it. Please, fix the test case to fail on vanilla 10.5 with the above error. > + > --echo # End of 10.5 tests > diff --git a/sql/sql_class.cc b/sql/sql_class.cc > index 99ba9e3587b..3d01b9d5584 100644 > --- a/sql/sql_class.cc > +++ b/sql/sql_class.cc > @@ -203,7 +203,7 @@ Foreign_key::Foreign_key(const Foreign_key &rhs, MEM_ROOT *mem_root) > > /* > Test if a foreign key (= generated key) is a prefix of the given key > - (ignoring key name, key type and order of columns) > + (ignoring key name, key type and order of columns, but minding the algorithm) it doesn't ignore the order of columns and "key type" is basically algorithm, so the above can simply be "(ignoring key name)" > > NOTES: > This is only used to test if an index for a FOREIGN KEY exists > @@ -218,6 +218,17 @@ Foreign_key::Foreign_key(const Foreign_key &rhs, MEM_ROOT *mem_root) > > bool is_foreign_key_prefix(Key *a, Key *b) > { > + ha_key_alg a_alg= a->key_create_info.algorithm; > + ha_key_alg b_alg= b->key_create_info.algorithm; > + > + bool is_a_btree= a_alg == HA_KEY_ALG_BTREE || a_alg == HA_KEY_ALG_UNDEF; > + bool is_b_btree= b_alg == HA_KEY_ALG_BTREE || b_alg == HA_KEY_ALG_UNDEF; > + > + // Either the algorithms are total match, or they both are btree > + // in order to be able to prefix > + if (a_alg != b_alg && !(is_a_btree && is_b_btree)) > + return false; I don't see why algorithms can be not b-tree, if they match. What use case did you have in mind? > + > /* Ensure that 'a' is the generated key */ > if (a->generated) > { > Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 1

Update documentation for non-blocking client library
by Kristian Nielsen 22 Oct '24

22 Oct '24

Hi Georg, Following MDEV-34859, I wrote a small update for the documentation on the new cmake option -DWITH_BOOST_CONTEXT=ON cmake option, patch included below. However, I am not sure how I can include it in the documentation? Feel free to just take it yourself, or let me know a preferred way to submit it. - Kristian. diff --git a/configuration_options.md b/configuration_options.md index 6e8da57..726b4f8 100644 --- a/configuration_options.md +++ b/configuration_options.md @@ -27,6 +27,11 @@ If you want to use a different generator, e.g. for nmake on Windows, you need to |WITH_OPENSSL|ON| Possible values are ON or OFF. Not supported anymore since Connector/C 3.0| |WITH_SSL|SCHANNEL (windows), otherwise OPENSSL|Specifies type of TLS/SSL library. E.g. GNUTLS, OPENSSL or SCHANNEL (Windows only). OFF disables TLS/SSL functionality| +### Non-blocking client library options +| Option | Default | Description | +|-|-|- +|WITH_BOOST_CONTEXT|OFF| Use `boost::context` instead of `ucontext` for the non-blocking client API. Can be used to build the non-blocking API on platforms that do not have `ucontext`. Note that on x86_64 (aka amd64), i386, and aarch64 (aka arm64), a native implementation is always used over `ucontext` or `boost::context`. (Added in 3.3.12)| + ### Client plugins Client plugins can be configured as dynamic plugins (DYNAMIC) or built-in plugins (STATIC) by specifying the plugin name followed by suffix `_PLUGIN_TYPE` as key, and `DYNAMIC` or `STATIC` as value.

2 1

Community request: audit the PARSEC plugin
by Andrew Hutchings 16 Oct '24

16 Oct '24

Hi all, There is a new authentication plugin called PARSEC that is set to become the default authentication plugin in MariaDB soon (MDEV-32618 and MDEV-12320). The current documentation for it can be found here: https://mariadb.com/kb/en/authentication-plugin-parsec/ I personally think that using a single salt and sending it to the client could be a problem. I do not work in the security industry and the documentation has a few errors in it and pieces missing, so I'm unclear how certain things work. I think the ext-salt requirements will likely add complications to load balancing scenarios and replication. This is a request for anyone in the community to please audit this plugin before it becomes the default. Once it is the default, it will be difficult to change it. Kind Regards -- Andrew (LinuxJedi) Hutchings Chief Contributions Officer MariaDB Foundation

2 2

Re: 778c67f7af0: MDEV-18151 Skipped error returning for GRANT/SET PASSWORD
by Sergei Golubchik 09 Oct '24

09 Oct '24

Hi, Oleksandr, On Oct 09, Oleksandr Byelkin wrote: > revision-id: 778c67f7af0 (mariadb-10.5.26-154-g778c67f7af0) > parent(s): e9c999caf48 > author: Oleksandr Byelkin > committer: Oleksandr Byelkin > timestamp: 2024-10-04 09:28:46 +0200 > message: > > MDEV-18151 Skipped error returning for GRANT/SET PASSWORD > > Make error issueing for GRANT and SET PASSWORD the same. typo. "issuing" > Report errors wich were skipped before. > > diff --git a/sql/sql_acl.cc b/sql/sql_acl.cc > --- a/sql/sql_acl.cc > +++ b/sql/sql_acl.cc > @@ -2370,6 +2369,21 @@ static int set_user_auth(THD *thd, const LEX_CSTRING &user, > > res= 0; > end: > + switch(res) > + { > + case ER_OUTOFMEMORY: // should be reported by my_malloc > + case ER_NOT_VALID_PASSWORD: // should be reported by plugin > + case 0: > + break; > + case ER_SET_PASSWORD_AUTH_PLUGIN: > + my_error(res, MYF(0), plugin_name); > + break; > + case ER_PASSWD_LENGTH: > + my_error(ER_PASSWD_LENGTH, MYF(0), SCRAMBLED_PASSWORD_CHAR_LENGTH); this is also reported by the plugin. The caller doesn't know what the correct length is, it's not SCRAMBLED_PASSWORD_CHAR_LENGTH, this value is different for different plugins. > + break; > + default: > + DBUG_ASSERT(0); > + } > if (unlock_plugin) > plugin_unlock(thd, plugin); > return res; > Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

1 0

PRs with one approval pending the second
by Otto Kekäläinen 04 Oct '24

04 Oct '24

Hi *all* developers, Hopefully someone else than Andrew or Sergei replies as a project of the size of MariaDB should have multiple developers who are capable of reviewing PRs, and not overload a few. These PRs already have one approval, but would need a second to be merged. If you feel you are knowledgeable in the domain of the PRs, please help out by reviewing them. Thanks! https://github.com/MariaDB/server/pull/3225: MDEV-29955: add ${PCRE_LIBRARY_PATH} to link directories https://github.com/MariaDB/server/pull/3540: Add MTRs to further test coverage around TLS functionality

1 0

PRs with two approvals but not merged
by Otto Kekäläinen 04 Oct '24

04 Oct '24

Hi, Here are two PRs that have already two approvals, yet they have not been merged for many weeks. Can you please state explicitly on the PR as a comment what the submitter needs to do, or if there is nothing to do, just merge it? * https://github.com/MariaDB/server/pull/3208 * https://github.com/MariaDB/server/pull/2556 Thanks

3 3

More extensive major version upgrade testing
by Otto Kekäläinen 04 Oct '24

04 Oct '24

Hi, If any dev here is interested in the topic of ensuring the major version upgrades (and essentially running mariadb-upgrade) continues to work flawlessly and regressions are less likely to surface unnoticed, please review and consider adding your approvals to https://github.com/MariaDB/server/pull/3503 and share feedback on WIP https://github.com/MariaDB/server/pull/3563. Thanks!

2 1

Re: [PATCH] MDEV-28413 System versioning is incorrect in Spider
by Yuchen Pei 03 Oct '24

03 Oct '24

Hi Aleksey, Thanks for the patch. Please see below for some initial review comments. There are some todos for myself noted below which I will look into tomorrow. > From b4465fc6b51dfc4e645569201e6026ec6405367b Mon Sep 17 00:00:00 2001 > From: Aleksey Midenkov <midenok(a)gmail.com> > Date: Mon, 30 Sep 2024 22:26:51 +0300 > Subject: [PATCH] MDEV-28413 System versioning is incorrect in Spider > Basic functioning of system versioning in Spider: UPDATE, DELETE, > DELETE HISTORY, AS OF, FROM .. TO, BETWEEN .. AND. > More testing should be done for: > LOAD DATA, joins, derived tables, multi-update/delete and what else? > Note that we are still using HA_CAN_DIRECT_UPDATE_AND_DELETE, but not > for periods in DELETE (fwiw there is condition > !table_list->has_period(), but it is missing in UDPATE). > --- > .../mysql-test/spider/r/versioning.result | 142 ++++++++++++++++ > .../mysql-test/spider/t/versioning.test | 151 ++++++++++++++++++ > storage/spider/spd_db_conn.cc | 19 ++- > storage/spider/spd_db_include.h | 1 + > storage/spider/spd_db_mysql.cc | 25 ++- > 5 files changed, 335 insertions(+), 3 deletions(-) > create mode 100644 storage/spider/mysql-test/spider/r/versioning.result > create mode 100644 storage/spider/mysql-test/spider/t/versioning.test > [... 149 lines elided] > diff --git a/storage/spider/mysql-test/spider/t/versioning.test b/storage/spider/mysql-test/spider/t/versioning.test > new file mode 100644 > index 00000000000..6093169bff4 > --- /dev/null > +++ b/storage/spider/mysql-test/spider/t/versioning.test > @@ -0,0 +1,151 @@ > +--source include/have_innodb.inc > +--source suite/versioning/common.inc > + > +install plugin spider soname 'ha_spider'; For init the spider plugin in spider tests we commonly do --disable_query_log --disable_result_log --source ../t/test_init.inc --enable_result_log --enable_query_log > +set spider_same_server_link= on; > + > +--replace_result $MASTER_1_MYPORT MASTER_1_MYPORT > +eval create server s foreign data wrapper mysql options > +(host '127.0.0.1', database 'test', user 'root', port $MASTER_1_MYPORT); Use evalp instead of eval and you can save the --replace_result line > + > + > +create table t1 ( > + x int, y int > +) with system versioning; > + > +create table t2 ( > + x int, y int, > + row_start timestamp(6) as row start invisible, > + row_end timestamp(6) as row end invisible, > + period for system_time (row_start, row_end) > +) with system versioning; > + > +create or replace table t1_sp ( > + x int, y int) > +engine=spider comment='wrapper "mysql", srv "s", table "t1"' > +with system versioning; > + > +create or replace table t2_sp ( > + x int, y int, > + row_start timestamp(6) as row start invisible, > + row_end timestamp(6) as row end invisible, > + period for system_time (row_start, row_end)) > +engine=spider comment='wrapper "mysql", srv "s", table "t2"' > +with system versioning; > + > +--echo # Timestamps are not propagated (subject of MDEV-16546, but not for implicit system fields) > +set timestamp= unix_timestamp('2000-01-01 00:00:00'); I tried executing directly using select spider_direct_sql('set timestamp= unix_timestamp("2000-01-01 00:00:00");', '', 'srv "s"'); But then I get extra rows in subsequent select queries and I think it is caused by the timezones. It could be a separate task to fix this, but we should warn users about it. > +insert into t1_sp values (1, 1), (2, 2); It may be obvious to you, but I find the timestamps at times hard to keep track of, so I suggest adding a comment to each DML statement indicating the time, as well as the select row_start / row_end statements. For example, change the line above to insert into t1_sp values (1, 1), (2, 2); # T1 > +insert into t2_sp values (1, 1), (2, 2); and change this line to insert into t2_sp values (1, 1), (2, 2); # T2 > +--sleep 0.01 > + > +set timestamp= unix_timestamp('2000-01-01 00:11:11'); > +--echo # check dml generating history > +delete from t1_sp where x = 1; add # T3 > +--sleep 0.01 > +delete from t2_sp where x = 1; # T4 > +--sleep 0.01 > + > +select row_end into @t1 from t2 for system_time all where x = 1; And so we can annotate this line with the timestamp above, like so: select row_end into @t1 from t2 for system_time all where x = 1; # @t1 = T4 > + > +update t1_sp set y= y + 1; # T5 > +select row_start into @t2 from t1 where x = 2; # @t2 = T5 > +--sleep 0.01 > +update t2_sp set y= y + 1; # T6 With the annotations I suggested, results from the following block of selects becomes easier to understand. > + > +select * from t1_sp; > +select * from t2_sp; > +select * from t1_sp for system_time as of timestamp @t1; > +select * from t2_sp for system_time as of timestamp @t1; > +select * from t1_sp for system_time as of timestamp @t2; > +select * from t2_sp for system_time as of timestamp @t2; > +select * from t1_sp for system_time from timestamp @t1 to timestamp @t2; > +select * from t2_sp for system_time from timestamp @t1 to timestamp @t2; > +select * from t1_sp for system_time between timestamp @t1 and timestamp @t2; > +select * from t2_sp for system_time between timestamp @t1 and timestamp @t2; > + > +--echo # spider cannot call functions, it fails in spider_create_group_by_handler() > + # at: > + # > + # 1555 if (spider_db_print_item_type(item, null, spider, null, null, 0, > + # 1556 roop_count, true, fields_arg)) > + # 1557 { > + # 1558 dbug_print("info",("spider dbton_id=%d can't create select", roop_count)); > + # 1559 spider_clear_bit(dbton_bitmap, roop_count); > + # 1560 keep_going = false; > + # 1561 break; > + # 1562 } > + # This is fine. The Spider group-by-handler (GBH) is not meant to handle every situation. Either remove this block starting from the --echo, or make it more succinct by replacing it with something like (Note that GBH supports certain functions like trim): # The spider group-by-handler does not support the check_row function > +--echo # but somehow it doesn't fail the query and continues without FOR SYSTEM_TIME ALL clause > +select *, check_row(row_start, row_end) from t1_sp for system_time all order by row_end, y, x; > +select *, check_row(row_start, row_end) from t2_sp for system_time all order by row_end, y, x; I will check why the execution without the group-by-handler results in wrong results. > +--echo # SELECT from base table works as usual: > +select *, check_row(row_start, row_end) from t1 for system_time all order by row_end, y, x; > +select *, check_row(row_start, row_end) from t2 for system_time all order by row_end, y, x; > + > +--echo # here it works nice (append_table_list() has the chance to add FOR SYSTEM_TIME ALL) > + > +# Usage: SHOW_TIMESTAMPS=1 mtr versioning.spider Shouldn't the test name be spider.versioning? > +if ($SHOW_TIMESTAMPS) > +{ > + select @t1, @t2; > +} > +if (!$SHOW_TIMESTAMPS) > +{ > +--replace_regex /\d\d\d\d-\d\d-\d\d \d\d:\d\d:\d\d.\d\d\d\d\d\d/TIMESTAMP/ > +} > +select *, row_start, row_end from t1_sp for system_time all order by row_end, y, x; > + > +if (!$SHOW_TIMESTAMPS) > +{ > +--replace_regex /\d\d\d\d-\d\d-\d\d \d\d:\d\d:\d\d.\d\d\d\d\d\d/TIMESTAMP/ > +} > +select *, row_start, row_end from t2_sp for system_time all order by row_end, y, x; > + > +delete history from t1_sp before system_time @t1; > +delete history from t2_sp before system_time @t1; > + > +if (!$SHOW_TIMESTAMPS) > +{ > +--replace_regex /\d\d\d\d-\d\d-\d\d \d\d:\d\d:\d\d.\d\d\d\d\d\d/TIMESTAMP/ > +} > +select *, row_start, row_end from t1_sp for system_time all order by row_end, y, x; > + > +if (!$SHOW_TIMESTAMPS) > +{ > +--replace_regex /\d\d\d\d-\d\d-\d\d \d\d:\d\d:\d\d.\d\d\d\d\d\d/TIMESTAMP/ > +} > +select *, row_start, row_end from t2_sp for system_time all order by row_end, y, x; > + > +select * from t1 for system_time all; > +select * from t2 for system_time all; > + > +delete history from t1_sp; > +delete history from t2_sp; > + > +if (!$SHOW_TIMESTAMPS) > +{ > +--replace_regex /\d\d\d\d-\d\d-\d\d \d\d:\d\d:\d\d.\d\d\d\d\d\d/TIMESTAMP/ > +} > +select *, row_start, row_end from t1_sp for system_time all order by row_end, y, x; > + > +if (!$SHOW_TIMESTAMPS) > +{ > +--replace_regex /\d\d\d\d-\d\d-\d\d \d\d:\d\d:\d\d.\d\d\d\d\d\d/TIMESTAMP/ > +} > +select *, row_start, row_end from t2_sp for system_time all order by row_end, y, x; > + > +select * from t1 for system_time all; > +select * from t2 for system_time all; > + > +drop tables t1, t2, t1_sp, t2_sp; > +set timestamp= default; > +--source suite/versioning/common_finish.inc > + > +uninstall plugin spider; > +drop table mysql.spider_link_failed_log, mysql.spider_link_mon_servers, mysql.spider_tables, mysql.spider_table_crd, mysql.spider_table_position_for_recovery, mysql.spider_table_sts, mysql.spider_xa, mysql.spider_xa_failed_log, mysql.spider_xa_member; > +drop function spider_direct_sql; > +drop function spider_bg_direct_sql; > +drop function spider_ping_table; > +drop function spider_copy_tables; > +drop function spider_flush_table_mon_cache; For deinit the spider plugin in spider tests we commonly do --disable_query_log --disable_result_log --source ../t/test_deinit.inc --enable_result_log --enable_query_log Please replace the lines from "uninstall plugin spider;" to the end of the test with this block. > diff --git a/storage/spider/spd_db_conn.cc b/storage/spider/spd_db_conn.cc > index 8750e87237e..896fce8adf5 100644 > --- a/storage/spider/spd_db_conn.cc > +++ b/storage/spider/spd_db_conn.cc > @@ -1515,9 +1515,8 @@ int spider_db_append_key_where_internal( > ) { > SPIDER_RESULT_LIST *result_list = &spider->result_list; > SPIDER_SHARE *share = spider->share; > -#ifndef DBUG_OFF > TABLE *table = spider->get_table(); > -#endif Why remove the #ifndef and #endif? > [... 99 lines elided] Best, Yuchen

1 1

[Maria-developers] MariaDB // Temporal tables
by Robert Palm 02 Oct '24

02 Oct '24

2 2

Help wanted: Encryption Plugin
by ssss.aaa2002＠gmail.com 20 Sep '24

20 Sep '24

Hello I wonder if anyone can help me out for this case I'm trying to build an encryption plugin for mariaDB and I built this https://gist.github.com/Mahmoudgalalz/ae1de7a57f8b9f37056efc2018575ce4 I managed to compile it, but I'm running into an issue when I'm trying to use encrypted option MariaDB [test]> CREATE TABLE test_encrypted (id INT PRIMARY KEY, sensitive_data VARCHAR(255)) ENGINE=InnoDB ENCRYPTED=YES; ERROR 2013 (HY000): Lost connection to server during query MariaDB [test]> CREATE TABLE t1 (a VARCHAR(8)) ENGINE=InnoDB ENCRYPTED=YES ENCRYPTION_KEY_ID=1; ERROR 2006 (HY000): Server has gone away No connection. Trying to reconnect... Connection id: 31 Current database: test ERROR 2013 (HY000): Lost connection to server during query so if anyone can help me out to figure out what is wrong and should I correct or consider would be very much appreciated.

2 4

Help wanted: Encryption Plugin
by ssss.aaa2002＠gmail.com 19 Sep '24

19 Sep '24

Hello I wonder if anyone can help me out for this case I'm trying to build an encryption plugin for mariaDB and I built this https://gist.github.com/Mahmoudgalalz/ae1de7a57f8b9f37056efc2018575ce4 I managed to compile it, but I'm running into an issue when I'm trying to use encrypted option MariaDB [test]> CREATE TABLE test_encrypted (id INT PRIMARY KEY, sensitive_data VARCHAR(255)) ENGINE=InnoDB ENCRYPTED=YES; ERROR 2013 (HY000): Lost connection to server during query MariaDB [test]> CREATE TABLE t1 (a VARCHAR(8)) ENGINE=InnoDB ENCRYPTED=YES ENCRYPTION_KEY_ID=1; ERROR 2006 (HY000): Server has gone away No connection. Trying to reconnect... Connection id: 31 Current database: test ERROR 2013 (HY000): Lost connection to server during query so if anyone can help me out to figure out what is wrong and should I correct or consider would be very much appreciated.

1 0

Re: [PATCH] MDEV-32014 Rename binlog cache temporary file to binlog file for large transaction
by Kristian Nielsen 10 Sep '24

10 Sep '24

> From: Libing Song <anders.slb(a)alibaba-inc.com> > > Description > =========== > When a transaction commits, it copies the binlog events from > binlog cache to binlog file. Very large transactions > (eg. gigabytes) can stall other transactions for a long time > because the data is copied while holding LOCK_log, which blocks > other commits from binlogging. Hi Libing Song, Here is my full review of the patch for MDEV-32014 Rename binlog cache temporary file to binlog file for large transaction. First let me say that thanks, reviewing this patch was a pleasant surprise. The binlog code that you are integrating this new feature in is complex to work with, but overall I think you have managed to do it a pretty clean way. That is great work, and I think a lot of effort must have gone into getting it this way. I have a number of comments and requests for changes below, so I wanted to give this overall good impression up-front. Also thanks to Brandon for doing first review. My apologies if I am repeating something already discussed, but Github pull request reviews of non-trivial patches are almost impossible to read, with reviews being split in fragments, patches disappearing after force-pushes, etc. The root cause for the problem being addressed by the patch is a fundamental limitation in the current binlog format. It requires all transactions to be binlogged as a single event group, using consecutive bytes in a single binlog file. Thus, while writing the large transaction to the binlog during commit, all other transaction commits will have to wait before they can commit. Eventually, we should move towards an enhanced binlog format that doesn't have this limitation, so that transactions can be partially binlogged during execution, avoiding this bottlenect. There are a number of other problems caused by this limitation that are not addressed by this patch. Changing the binlog format is a complex task not easily done, so this patch should be seen as a temporary solution to one specific bottleneck, until if/when this underlying limitation can hopefully be removed. First some overall comments: As Brandon also remarked, the name "binlog_free_flush" does not describe the feature very clearly. I would suggest to use the term "rotate" in the name, as the concept of binlog rotation is well known in MariaDB replication, and the patch effectively performs large commits by doing it as a binlog rotation instead of a simple write to the existing binlog file. So I suggest to name the option --binlog-commit-by-rotate-threshold. And I suggest in the code (names and comments) to use the phrase "rotate_by_commit" instead of "free_flush". The patch enables the feature by default, with a threshold of 128MB. We need to consider if that is appropriate, or if the feature should be disabled by default. Reasons for disabling by default: - The feature causes some binlog files to be cut short of the configured --max-binlog-size, which will be a surprise to users. - The feature adds an overhead of 4096 pre-allocated bytes at the start of every trx cache (though the overhead is minimal). - A default of 128MB is somewhat arbitrary. - Releasing this feature disabled by default is somewhat less risky wrt. regressions for users that upgrade. Reasons to enable by default: - Most users will otherwise not know the feature exists, and will thus not benefit from this work. - It will get much less testing "in the wild". So the decision will be a compromise between these two. If I had to make the choice, I would default to this feature being disabled. But I'm open to hear opinions the other way. The patch calls the trx_group_commit_leader() function. That is ackward, as commit-by-rotate does not actually participate in group commit. But the problem is with the existing function trx_group_commit_leader(), which does too much in a single function. So please do a small refactoring of trx_group_commit_leader() and keep only the first part of that function that fetches the group commit queue. Move the second part to a new function trx_group_commit_with_engines() that is called by trx_group_commit_leader() passing the "queue" and "last_in_queue" values. And assert in trx_group_commit_with_engines() that LOCK_log is held on entry and comment that this function will release the lock. Then you can from your Binlog_free_flush::commit() function call into trx_group_commit_with_engines() directly, avoiding the need for adding a true/false flag, and also avoiding that the trx_group_commit_leader() function is sometimes called with LOCK_log held and sometimes not. Same situation with MYSQL_BIN_LOG::write_transaction_to_binlog_events(), this function is also doing too much and makes integrating your code ackward. Please refactor the MYSQL_BIN_LOG::write_transaction_to_binlog_events() function something like this: bool MYSQL_BIN_LOG::write_transaction_to_binlog_events(group_commit_entry *entry) { if (binlog_commit_by_rotate.should_commit_by_rotate(entry) && binlog_commit_by_rotate.commit(entry)) { /* Commit done by renaming the trx/stmt cache. */ } else { if (write_transaction_with_group_commit(entry)) return true; /* Error */ } if (unlikely(entry->error)) { write_transaction_handle_error(entry); return 1; } return 0; } Introducing two new functions: bool write_transaction_with_group_commit(group_commit_entry *entry) This function contains the code from write_transaction_to_binlog_events() up to and including this: if (likely(!entry->error)) return entry->thd->wait_for_prior_commit(); You don't need the wait_for_prior_commit() in the commit-by-rotate case, as you already did this in Binlog_free_flush::commit()). You also don't need to handle the (!opt_optimize_thread_scheduling) case for commit-by-rotate, this is old code that is being deprecated. Just add a condition in should_free_flush() to skip if opt_optimize_thread_scheduling is set. Other new function: void write_transaction_handle_error(group_commit_entry *entry) This function contains the remainder of the write_transaction_to_binlog_events() function starting from "switch (entry->error) ...". This way, the commit-by-rotate case is cleanly separated from the normal group commit case and the code of write_transaction_to_binlog_events() becomes much cleaner. Now follows more detailed comments on specific parts of the patch: > diff --git a/sql/log.cc b/sql/log.cc > index 34f9ad745fc..3dc57b21c05 100644 > --- a/sql/log.cc > +++ b/sql/log.cc > @@ -163,6 +163,111 @@ static SHOW_VAR binlog_status_vars_detail[]= > + void set_reserved_bytes(uint32 header_len) > + { > + // Gtid event length > + header_len+= LOG_EVENT_HEADER_LEN + Gtid_log_event::max_data_length + > + BINLOG_CHECKSUM_LEN; > + header_len= header_len - (header_len % IO_SIZE) + IO_SIZE; This has an off-by-one error, it will round up IO_SIZE to 2*IO_SIZE. Use eg. header_len = (header_len + (IO_SIZE - 1)) & ~(IO_SIZE - 1) You can use static_assert((IO_SIZE & (IO_SIZE - 1)) == 0, "power of two") if you want to catch if anyone would ever change IO_SIZE to be not a power of two. +private: + Binlog_free_flush &operator=(const Binlog_free_flush &); + Binlog_free_flush(const Binlog_free_flush &); Add before this a comment "singleton object, prevent copying by mistake". > + char m_cache_dir[FN_REFLEN]; This member appears to be unused? > + /** The cache_data which will be renamed to binlog. */ > + binlog_cache_data *m_cache_data{nullptr}; Also appears unused. Maybe add a comment that m_entry and m_replaced are protected by LOCK_log? > +static Binlog_free_flush binlog_free_flush; > +ulonglong opt_binlog_free_flush_threshold= 10 * 1024 * 1024; This initializes the value to 10 MB, but the default value in sys_vars.cc is 128MB, so that seems inconsistent. > @@ -4126,8 +4238,7 @@ bool MYSQL_BIN_LOG::open(const char *log_name, > /* Notify the io thread that binlog is rotated to a new file */ > if (is_relay_log) > signal_relay_log_update(); > - else > - update_binlog_end_pos(); > + Why do you remove update_binlog_end_pos() here? Doesn't the binlog dump threads need this update? > @@ -8703,7 +8821,16 @@ MYSQL_BIN_LOG::queue_for_group_commit(group_commit_entry *orig_entry) > bool > MYSQL_BIN_LOG::write_transaction_to_binlog_events(group_commit_entry *entry) > { > - int is_leader= queue_for_group_commit(entry); > + int is_leader; > + > + if (binlog_free_flush.should_free_flush(entry) && > + binlog_free_flush.commit(entry)) > + { > + is_leader= 1; > + goto commit; > + } > + > + is_leader= queue_for_group_commit(entry); As described above, re-factor the write_transaction_to_binlog_events() function to avoid this assignment of is_leader and the goto. > @@ -8863,6 +8992,16 @@ MYSQL_BIN_LOG::trx_group_commit_leader(group_commit_entry *leader) > uint64 commit_id; > DBUG_ENTER("MYSQL_BIN_LOG::trx_group_commit_leader"); > > + /* > + When move a binlog cache to a binlog file, the transaction itself is > + a group. > + */ > + if (unlikely(is_free_flush)) > + { > + last_in_queue= leader; > + queue= leader; > + } > + else As described above, re-factor trx_group_commit_leader() to avoid this check in trx_group_commit_leader(), instead calling directly into the relevant code now in trx_group_commit_with_engines(). > +inline bool Binlog_free_flush::should_free_flush( > + const MYSQL_BIN_LOG::group_commit_entry *entry) const > +{ > + /* Do not free flush if total_bytes smaller than limit size. */ > + if (likely(cache_data->get_byte_position() <= > + opt_binlog_free_flush_threshold)) > + return false; Put this check as the first check in the function. So the common case of a small transaction does not need to run any of the other checks. And as suggested above, put a check: if (unlikely(!opt_optimize_thread_scheduling)) return false; (This is not critical, but just to make things a bit easier; there is no reason for you/us to worry about making commit-by-rotate work with disabled optimize_thread_scheduling, this option isn't used any more. Otherwise you'd need to test that your patch works correctly with optimize_thread_scheduling disabled, which would just be a waste of time.) > +bool Binlog_free_flush::commit(MYSQL_BIN_LOG::group_commit_entry *entry) > +{ > + // It will be released by trx_group_commit_leader > + mysql_mutex_lock(&mysql_bin_log.LOCK_log); After suggested changes, comment should read "It will be released by trx_group_commit_with_engines()." > + mysql_bin_log.trx_group_commit_leader(entry, true /* is_free_flush */); As discussed above, change this to call a new function mysql_bin_log.trx_group_commit_with_engines(), which contains the second part of trx_group_commit_leader() that you need to run. Then there is no longer a need to pass a `true` flag here to skip the first part. Also, before the call, this is the place to set entry->next to nullptr, to convert the stand-alone entry into a (singleton) list (instead of initializing it in MYSQL_BIN_LOG::write_transaction_to_binlog().) > @@ -8301,6 +8418,7 @@ MYSQL_BIN_LOG::write_transaction_to_binlog(THD *thd, > DBUG_RETURN(0); > } > > + entry.next= nullptr; > entry.thd= thd; > entry.cache_mngr= cache_mngr; > entry.error= 0; Do you need to initialize entry->next here, and if so why? At this point, the entry is not a list, it is a stand-alone struct that is not yet linked into any list, the next pointer should be set appropriately when/if the struct is later put into a list. More important than the minor cost of initializing, by redundantly initializing here we are hiding bugs (eg. Valgrind/msan/asan) if some code would incorrectly read the next pointer. > +bool Binlog_free_flush::replace_binlog_file() > +{ > + size_t binlog_size= my_b_tell(&mysql_bin_log.log_file); > + size_t required_size= binlog_size; > + // space for Gtid_log_event > + required_size+= LOG_EVENT_HEADER_LEN + Gtid_log_event::max_data_length + > + BINLOG_CHECKSUM_LEN; Just a remark, here you actually know the real required size of the GTID event, so you don't need to conservatively estimate Gtid_log_event::max_data_length. But I think doing it this way is fine, it is simpler and in most cases we are rounding up to IO_SIZE anyway, so I'm fine with keeping it this way. > + sql_print_information("Could not rename binlog cache to binlog, " > + "require %llu bytes but only %llu bytes reserved.", > + required_size, m_cache_data->file_reserved_bytes()); This log message will perhaps appear a bit out-of-context in the error log. Suggested clarification: "Could not rename binlog cache to binlog (as requested by --binlog-commit-by-rotate-threshold). Required ..." > + // Set the cache file as binlog file. > + mysql_file_close(mysql_bin_log.log_file.file, MYF(MY_WME)); Do not to pass MY_WME here (instead pass MYF(0)). Since you are ignoring the return of the function, so we don't want mysql_file_close to call my_error() leaving an error code set even though the operation succeeds. > +size_t Binlog_free_flush::get_gtid_event_pad_size() > +{ > + size_t begin_pos= my_b_tell(&mysql_bin_log.log_file); > + size_t pad_to_size= > + m_cache_data->file_reserved_bytes() - begin_pos - LOG_EVENT_HEADER_LEN; > + > + if (binlog_checksum_options != BINLOG_CHECKSUM_ALG_OFF) > + pad_to_size-= BINLOG_CHECKSUM_LEN; Adjusting with BINLOG_CHECKSUM_LEN is to account for the 4 bytes of checksum at the end of the GTID_EVENT, right? But why do you subtract LOG_EVENT_HEADER_LEN when calculating pad_to_size? (I mean, it is probably correct, otherwise nothing would work I think. I just did not understand it, maybe you can explain in a short comment. Or maybe Gtid_log_event::write() could be modified to do the adjusting for LOG_EVENT_HEADER_LEN and BINLOG_CHECKSUM_LEN). > + // offset must be saved before replace_binlog_file(), it will update the pos > + my_off_t offset= my_b_tell(&log_file); Suggest to clarify comment to "..., it will update the file position". > @@ -756,18 +759,20 @@ class MYSQL_BIN_LOG: public TC_LOG, private Event_log > new_file() is locking. new_file_without_locking() does not acquire > LOCK_log. > */ > - int new_file_impl(); > + int new_file_impl(bool is_free_flush= false); > void do_checkpoint_request(ulong binlog_id); > - int write_transaction_or_stmt(group_commit_entry *entry, uint64 commit_id); > + int write_transaction_or_stmt(group_commit_entry *entry, uint64 commit_id, > + bool is_free_flush= false); > int queue_for_group_commit(group_commit_entry *entry); > bool write_transaction_to_binlog_events(group_commit_entry *entry); > - void trx_group_commit_leader(group_commit_entry *leader); > + void trx_group_commit_leader(group_commit_entry *leader, > + bool is_free_flush= false); > bool is_xidlist_idle_nolock(); > void update_gtid_index(uint32 offset, rpl_gtid gtid); > > public: > void purge(bool all); > - int new_file_without_locking(); > + int new_file_without_locking(bool is_free_flush= false); > /* > A list of struct xid_count_per_binlog is used to keep track of how many > XIDs are in prepared, but not committed, state in each binlog. And how > @@ -997,7 +1002,8 @@ class MYSQL_BIN_LOG: public TC_LOG, private Event_log > enum cache_type io_cache_type_arg, > ulong max_size, > bool null_created, > - bool need_mutex); > + bool need_mutex, > + bool is_free_flush = false); > bool open_index_file(const char *index_file_name_arg, > const char *log_name, bool need_mutex); > /* Use this to start writing a new log file */ > @@ -1037,7 +1043,7 @@ class MYSQL_BIN_LOG: public TC_LOG, private Event_log > bool is_active(const char* log_file_name); > bool can_purge_log(const char *log_file_name, bool interactive); > int update_log_index(LOG_INFO* linfo, bool need_update_threads); > - int rotate(bool force_rotate, bool* check_purge); > + int rotate(bool force_rotate, bool* check_purge, bool is_free_flush= false); > void checkpoint_and_purge(ulong binlog_id); > int rotate_and_purge(bool force_rotate, DYNAMIC_ARRAY* drop_gtid_domain= NULL); > /** Style preference: please add the these new boolean parameters without default value, and instead change existing callers to pass false explicitly. (And the parameter will be "commit_by_rotate" after name change.) > + > +extern ulonglong opt_binlog_free_flush_threshold; > +static Sys_var_ulonglong Sys_binlog_free_flush_threshold( > + "binlog_free_flush_threshold", > + "Try to rename the binlog cache temporary file of the commiting " > + "transaction to a binlog file when its binlog cache size " > + "is bigger than the value of this variable", > + GLOBAL_VAR(opt_binlog_free_flush_threshold), > + CMD_LINE(REQUIRED_ARG), VALID_RANGE(10 * 1024 * 1024, ULLONG_MAX), > + DEFAULT(128 * 1024 * 1024), BLOCK_SIZE(1)); As described above: - Suggested name: "binlog_commit_by_rotate_threshold". - Suggested text: "For transactions that exceed this size in the binlog, commit by a more efficient method that forces a binlog rotation and re-uses the pre-existing temporary file as the next binlog file." - Let me hear opinions on whether this should default to 128 MB or should default to eg. ULONGLONG_MAX to disable the feature by default. > diff --git a/mysql-test/main/tmp_space_usage.test b/mysql-test/main/tmp_space_usage.test > index af7b295f343..1685dbbc450 100644 > --- a/mysql-test/main/tmp_space_usage.test > +++ b/mysql-test/main/tmp_space_usage.test > @@ -215,7 +215,8 @@ select count(distinct concat(seq,repeat('x',1000))) from seq_1_to_1000; > > set @save_max_tmp_total_space_usage=@@global.max_tmp_total_space_usage; > set @@global.max_tmp_total_space_usage=64*1024*1024; > -set @@max_tmp_session_space_usage=1179648; > +# Binlog cache reserve 4096 bytes at the begin of the temporary file. > +set @@max_tmp_session_space_usage=1179648+65536; > select @@max_tmp_session_space_usage; > set @save_aria_repair_threads=@@aria_repair_threads; > set @@aria_repair_threads=2; > @@ -224,6 +225,7 @@ set @@max_heap_table_size=16777216; > > CREATE TABLE t1 (a CHAR(255),b INT,INDEX (b)); > INSERT INTO t1 SELECT SEQ,SEQ FROM seq_1_to_100000; > +set @@max_tmp_session_space_usage=1179648; > --error 200 > SELECT * FROM t1 UNION SELECT * FROM t1; > DROP TABLE t1; > @@ -266,11 +268,15 @@ SELECT MIN(VARIABLE_VALUE) OVER (), NTILE(1) OVER (), MAX(VARIABLE_NAME) OVER () > > connect(c1, localhost, root,,); > set @@binlog_format=row; > -CREATE OR REPLACE TABLE t1 (a DATETIME) ENGINE=MyISAM; > +CREATE OR REPLACE TABLE t1 (a DATETIME) ENGINE=InnoDB; > +# Use the transaction to keep binlog cache temporary file large enough > +BEGIN; > INSERT INTO t1 SELECT NOW() FROM seq_1_to_6000; > + > SET max_tmp_session_space_usage = 64*1024; > --error 200 > SELECT * FROM information_schema.ALL_PLUGINS LIMIT 2; > +ROLLBACK; > drop table t1; Can you explain why you need these changes to the test case? I was not able to understand that from the patch. With these changes, I approve of this patch / pull-request to merge into the next MariaDB release. - Kristian.

3 3

Re: bd616a3733c: Auth: set thd->scramble in all cases during writing Initial Handshake Packet
by Sergei Golubchik 10 Sep '24

10 Sep '24

Hi, Nikita, On Sep 09, Nikita Malyavin wrote: > revision-id: bd616a3733c (mariadb-11.6.1-13-gbd616a3733c) > parent(s): bbbb429a1eb > author: Nikita Malyavin > committer: Nikita Malyavin > timestamp: 2024-09-07 20:46:19 +0200 > message: > > Auth: set thd->scramble in all cases during writing Initial Handshake Packet > > --- > sql/sql_acl.cc | 19 +++++++++++++------ > 1 file changed, 13 insertions(+), 6 deletions(-) > > diff --git a/sql/sql_acl.cc b/sql/sql_acl.cc > index 93efc5806cc..7845af55413 100644 > --- a/sql/sql_acl.cc > +++ b/sql/sql_acl.cc > @@ -13504,17 +13504,21 @@ static bool send_server_handshake_packet(MPVIO_EXT *mpvio, > mpvio->cached_server_packet.pkt_len= data_len; > } > > - if (data_len < SCRAMBLE_LENGTH) > + if (thd->scramble[SCRAMBLE_LENGTH] != 0) old code was obviously wrong but the new one makes no sense either. How can thd->scramble[SCRAMBLE_LENGTH] be not zero at this point? I'd think you need to execute the code below unconditionally. > { > + DBUG_ASSERT(thd->scramble[SCRAMBLE_LENGTH] == 1); // Sanity > + > if (data_len) > { > /* > the first packet *must* have at least 20 bytes of a scramble. > - if a plugin provided less, we pad it to 20 with zeros > + if a plugin provided less, we pad it to 20 with zeros, > + plus extra zero termination sign is put in thd->scramble. > + If more is provided, we'll use only 20 bytes as a handshake scramble. Not sure it's a good idea. Why not to send the complete scramble? True, it won't fit on the client side into mysql->scramble[]. Two ways to fix it: * Write the first 20 bytes. The server will only store the first 20 bytes too, and for COM_CHANGE_USER they'll use the first 20 bytes only (for the login auth they can use the complete scramble). That's a rather hackish fix, a proper fix would be * store the complete scramble in MYSQL. The first 20 bytes in MYSQL::scramble_buff, the rest in MySQL::extension.scramble_suffix > */ > - memcpy(scramble_buf, data, data_len); > - bzero(scramble_buf + data_len, SCRAMBLE_LENGTH - data_len); > - data= scramble_buf; > + size_t fill_size= MY_MIN(SCRAMBLE_LENGTH, data_len); > + memcpy(thd->scramble, data, fill_size); > + bzero(thd->scramble + fill_size, SCRAMBLE_LENGTH - fill_size + 1); > } Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 3

Re: 982bf06d560: MDEV-12320 configurable default authentication plugin for the server
by Sergei Golubchik 09 Sep '24

09 Sep '24

Hi, Nikita, This looks good. Minor comments below. On Sep 09, Nikita Malyavin wrote: > revision-id: 982bf06d560 (mariadb-11.6.1-14-g982bf06d560) > parent(s): bd616a3733c > author: Nikita Malyavin > committer: Nikita Malyavin > timestamp: 2024-09-07 21:44:13 +0200 > message: > > MDEV-12320 configurable default authentication plugin for the server > > * Add a new cmdline-only variable "default_auth_plugin". > * A default plugin is locked at the server init and unlocked at the deinit > stages. This means that mysql_native_password and old_password_plugin, when > default, are locked/unlocked twice. doesn't matter, compiled-in plugins are only locked in debug builds, otherwise it's a no-op. > > --- a/sql/sys_vars.cc > +++ b/sql/sys_vars.cc > @@ -4538,6 +4538,14 @@ static Sys_var_plugin Sys_enforce_storage_engine( > DEFAULT(&enforced_storage_engine), NO_MUTEX_GUARD, NOT_IN_BINLOG, > ON_CHECK(check_has_super)); > > +extern const char *default_auth_plugin_name; > +extern LEX_CSTRING native_password_plugin_name; Is it ok? The correct type is Lex_ident_plugin. > +static Sys_var_charptr Sys_default_auth_plugin( there's also Sys_var_lexstring, if you prefer that. > + "default_auth_plugin", "Default plugin, that will be tried first when authenticating new connections", reformat the long line, please > + READ_ONLY GLOBAL_VAR(default_auth_plugin_name), CMD_LINE(OPT_ARG), > + DEFAULT(native_password_plugin_name.str), > + NO_MUTEX_GUARD, NOT_IN_BINLOG); > --- a/sql/sql_acl.cc > +++ b/sql/sql_acl.cc > @@ -2543,9 +2542,20 @@ bool acl_init(bool dont_read_acl_tables) > old_password_plugin= my_plugin_lock_by_name(0, > &old_password_plugin_name, MYSQL_AUTHENTICATION_PLUGIN); > > + Lex_cstring_strlen def_plugin_name(default_auth_plugin_name); > + default_auth_plugin= my_plugin_lock_by_name(NULL, &def_plugin_name, > + MYSQL_AUTHENTICATION_PLUGIN); > + > if (!native_password_plugin || !old_password_plugin) > DBUG_RETURN(1); > > + if (!default_auth_plugin) > + { > + sql_print_error("Default plugin %s could not be loaded", > + default_auth_plugin_name); see init_default_storage_engine() in mysqld.cc - it's for --default-storage-engine option. let's use similar wording for consistency: sql_print_error("Unknown/unsupported authentication plugin: %s", default_auth_plugin_name); > + DBUG_RETURN(1); > + } > + > if (dont_read_acl_tables) > { > DBUG_RETURN(0); /* purecov: tested */ Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

1 0

Re: bbbb429a1eb: Auth: release the plugins from responsibility to fill scramble_buff.
by Sergei Golubchik 09 Sep '24

09 Sep '24

Hi, Nikita, Conceptually ok, see one omission below. On Sep 09, Nikita Malyavin wrote: > revision-id: bbbb429a1eb (mariadb-11.6.1-12-gbbbb429a1eb) > parent(s): db5d1cde450 > author: Nikita Malyavin > committer: Nikita Malyavin > timestamp: 2024-09-07 20:44:02 +0200 > message: > > Auth: release the plugins from responsibility to fill scramble_buff. > > Fill it once during the Initial Handshake Packet parsing. This uses the fact > that the server guarantees first 20 bytes to be the scramble. > > Submodule libmariadb de6305915f8..12d78187061: > diff --git a/libmariadb/libmariadb/mariadb_lib.c b/libmariadb/libmariadb/mariadb_lib.c > index 78195d44..07953ea4 100644 > --- a/libmariadb/libmariadb/mariadb_lib.c > +++ b/libmariadb/libmariadb/mariadb_lib.c > @@ -1954,6 +1954,8 @@ restart: > goto error; > } > } > + memmove(mysql->scramble_buff, scramble_data, SCRAMBLE_LENGTH); > + mysql->scramble_buff[SCRAMBLE_LENGTH]= 0; > /* Set character set */ > if (mysql->options.charset_name) > diff --git a/libmariadb/plugins/auth/my_auth.c b/libmariadb/plugins/auth/my_auth.c > index a2fd519d..47d26150 100644 > --- a/libmariadb/plugins/auth/my_auth.c > +++ b/libmariadb/plugins/auth/my_auth.c > @@ -126,10 +126,6 @@ static int native_password_auth_client(MYSQL_PLUGIN_VIO *vio, MYSQL *mysql) > > if (pkt_len != SCRAMBLE_LENGTH + 1) > return CR_SERVER_HANDSHAKE_ERR; > - > - /* save it in MYSQL */ > - memmove(mysql->scramble_buff, pkt, SCRAMBLE_LENGTH); > - mysql->scramble_buff[SCRAMBLE_LENGTH] = 0; This looks ok > } > > if (mysql && mysql->passwd[0]) > diff --git a/sql-common/client.c b/sql-common/client.c > index 6d030ce0a17..28b477c1e9a 100644 > --- a/sql-common/client.c > +++ b/sql-common/client.c > @@ -4178,10 +4178,6 @@ static int native_password_auth_client(MYSQL_PLUGIN_VIO *vio, MYSQL *mysql) > > if (pkt_len != SCRAMBLE_LENGTH + 1) > DBUG_RETURN(CR_SERVER_HANDSHAKE_ERR); > - > - /* save it in MYSQL */ > - memcpy(mysql->scramble, pkt, SCRAMBLE_LENGTH); > - mysql->scramble[SCRAMBLE_LENGTH] = 0; this isn't, you removed memcpy from the plugin, but didn't add it to mysql_real_connect in sql-common/client.c. > } > > if (mysql->passwd[0]) Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 1

Re: b96afca238a: MDEV-16440 merge 5.7 P_S sysvars instrumentation and tables
by Sergei Golubchik 08 Sep '24

08 Sep '24

Hi, Nikita, Despite the email subject, it's a review of everything in bb-10.10-MDEV-16440, *except* 9c57bbfef92. That is 37e1ccbf678..c16b2429f02 without notifiable_work_zone.h I'll send a separate email about notifiable_work_zone.h On Sep 08, Nikita Malyavin wrote: > revision-id: b96afca238a (mariadb-10.6.1-472-gb96afca238a) > parent(s): 37e1ccbf678 > author: Nikita Malyavin > committer: Nikita Malyavin > timestamp: 2022-06-19 14:52:30 +0300 > message: > > MDEV-16440 merge 5.7 P_S sysvars instrumentation and tables > diff --git a/mysql-test/suite/perfschema/t/variables_stress.test b/mysql-test/suite/perfschema/t/variables_stress.test > new file mode 100644 > index 00000000000..be3012bf8af > --- /dev/null > +++ b/mysql-test/suite/perfschema/t/variables_stress.test > @@ -0,0 +1,71 @@ stress tests generally don't belong in mtr and aren't reliable enough for the mtr. Also, you forgot to check in the result file for it. on the other hand, looking at the patch I got the impression that this stress test helped to find a lot of bugs. > +let $i = 64; > +enable_connect_log; > +while ($i) { > + echo i = $i; > + connect(con_$i, localhost, root); > +# set debug_sync = "tp_hanger wait_for banger"; > + send set debug_dbug = "+d,tp_wanger"; > + dec $i; > +} > +connection default; > + > +SELECT THREAD_ID, PROCESSLIST_ID FROM performance_schema.threads; > +set debug_sync = "now signal banger"; > +connect(banger_1, localhost, root); > +let $banger1= `SELECT THREAD_ID FROM performance_schema.threads WHERE PROCESSLIST_ID = CONNECTION_ID()`; > +send select count(*) from performance_schema.variables_by_thread where variable_name="time_zone"; > +connect(banger_2, localhost, root); > +let $banger2= `SELECT THREAD_ID FROM performance_schema.threads WHERE PROCESSLIST_ID = CONNECTION_ID()`; > +send select count(*) from performance_schema.variables_by_thread where variable_name="time_zone"; > +connect(banger_3, localhost, root); > +let $banger3= `SELECT THREAD_ID FROM performance_schema.threads WHERE PROCESSLIST_ID = CONNECTION_ID()`; > +send select count(*) from performance_schema.variables_by_thread where variable_name="time_zone"; > +connect(banger_4, localhost, root); > +let $banger4= `SELECT THREAD_ID FROM performance_schema.threads WHERE PROCESSLIST_ID = CONNECTION_ID()`; > +send select count(*) from performance_schema.variables_by_thread where variable_name="time_zone"; > +connect(banger_5, localhost, root); > +let $banger5= `SELECT THREAD_ID FROM performance_schema.threads WHERE PROCESSLIST_ID = CONNECTION_ID()`; > + > +send select count(*) from performance_schema.variables_by_thread where variable_name="time_zone"; > + > +let $i = 2000; > +let $j=65; > +while ($i) { > + echo i = $i; > + connection banger_1; > + reap; > + send select count(*) from performance_schema.variables_by_thread where variable_name="time_zone"; > + connection banger_2; > + reap; > + send select count(*) from performance_schema.variables_by_thread where variable_name="time_zone"; > + connection banger_3; > + reap; > + send select count(*) from performance_schema.variables_by_thread where variable_name="time_zone"; > + connection banger_4; > + reap; > + send select count(*) from performance_schema.variables_by_thread where variable_name="time_zone"; > + connection banger_5; > + reap; > + send select count(*) from performance_schema.variables_by_thread where variable_name="time_zone"; > + dec $i; > + > + let $threads= `show status where variable_name = 'threads_connected'`; > + if ($threads < 63) { > + connect(con_$i, localhost, root); > + send set debug_dbug = "+d,tp_wanger"; > + inc $j; > + } > +} > +reap; > +select * from performance_schema.variables_by_thread where variable_name="time_zone"; > +echo $banger1; > +echo $banger2; > +echo $banger3; > +echo $banger4; > +echo $banger5; > +# eval select * from performance_schema.variables_by_thread where variable_name like "%id" and thread_id=197; > +connection default; > +# set global debug_dbug = ""; > +set debug_sync = reset; > + > +show status where variable_name = 'threads_connected'; > diff --git a/sql/sql_plugin.h b/sql/sql_plugin.h > index d4df8c6468f..3d02d1c7509 100644 > --- a/sql/sql_plugin.h > +++ b/sql/sql_plugin.h > @@ -186,6 +186,10 @@ extern SHOW_COMP_OPTION plugin_status(const char *name, size_t len, int type); > extern bool check_valid_path(const char *path, size_t length); > extern void plugin_mutex_init(); > > +template <class T> class Dynamic_array; you can simply include sql_array.h, can't you? > + > +void plugin_lock_by_sys_var_array(THD *thd, Dynamic_array<SHOW_VAR> *vars); > + > typedef my_bool (plugin_foreach_func)(THD *thd, > plugin_ref plugin, > void *arg); > diff --git a/sql/sql_parse.cc b/sql/sql_parse.cc > index 0597b086b7b..ec86389d08f 100644 > --- a/sql/sql_parse.cc > +++ b/sql/sql_parse.cc > @@ -1296,6 +1296,9 @@ dispatch_command_return do_command(THD *thd, bool blocking) > goto out; > } > > + if (unlikely(thd->apc_target.have_apc_requests())) > + thd->apc_target.process_apc_requests(); 1. why here? 2. there generally shouldn't be direct checks of apc requests, the code should do check_killed() often enough. So if you really need a check here (see question 1), add if (thd->check_killed())... > + > packet= (char*) net->read_pos; > /* > 'packet_length' contains length of data, as it was stored in packet > diff --git a/vio/viopipe.c b/vio/viopipe.c > index aeaad311b7e..5106c6d6513 100644 > --- a/vio/viopipe.c > +++ b/vio/viopipe.c > @@ -53,8 +53,9 @@ static size_t wait_overlapped_result(Vio *vio, int timeout) > } > else > { > + DWORD last_error= GetLastError(); > /* Error or timeout, cancel the pending I/O operation. */ > - CancelIo(vio->hPipe); > + CancelIoEx(vio->hPipe, &vio->overlapped); why, CancelIoEx resets last error, but CancelIo doesn't? > /* > If the wait timed out, set error code to indicate a > diff --git a/sql/threadpool_win.cc b/sql/threadpool_win.cc > index ed68e31c755..25844d1f3ef 100644 > --- a/sql/threadpool_win.cc > +++ b/sql/threadpool_win.cc > @@ -131,6 +131,11 @@ void TP_pool_win::resume(TP_connection* c) > SubmitThreadpoolWork(((TP_connection_win*)c)->work); > } > > +int TP_pool_win::wake(TP_connection *c) > +{ > + return 0; > +} How is that supposed to work? How do you wake up threadpool threads on Windows? > + > #define CHECK_ALLOC_ERROR(op) \ > do \ > { \ > diff --git a/mysys/my_thr_init.c b/mysys/my_thr_init.c > index cde34e5bdb9..89f190ee989 100644 > --- a/mysys/my_thr_init.c > +++ b/mysys/my_thr_init.c > @@ -432,9 +432,21 @@ const char *my_thread_name(void) > extern void **my_thread_var_dbug() > { > struct st_my_thread_var *tmp; > + int last_error; > if (!my_thread_global_init_done) > return NULL; > +#ifdef WIN32 > + last_error= GetLastError(); > +#else > + last_error= errno; > +#endif > + > tmp= my_thread_var; > +#ifdef WIN32 > + SetLastError(last_error); > +#else > + errno= last_error; > +#endif huh? my_thread_var can change errno? > return tmp && tmp->init ? &tmp->dbug : 0; > } > #endif /* DBUG_OFF */ > diff --git a/storage/perfschema/pfs_variable.h b/storage/perfschema/pfs_variable.h > index e59b02f2af8..f509e97048c 100644 > --- a/storage/perfschema/pfs_variable.h > +++ b/storage/perfschema/pfs_variable.h > @@ -212,7 +212,7 @@ class Find_THD_variable : public Find_THD_Impl > return false; > > /* Hold this lock to keep THD during materialization. */ > - mysql_mutex_lock(&thd->LOCK_thd_data); > + mysql_mutex_lock(&thd->LOCK_thd_kill); see commits 736a54f49c72 and c6c2a2b8d463 > return true; > } > void set_unsafe_thd(THD *unsafe_thd) { m_unsafe_thd= unsafe_thd; } > diff --git a/sql/threadpool_generic.h b/sql/threadpool_generic.h > index b7a35b7cbf0..e16ec4a4966 100644 > --- a/sql/threadpool_generic.h > +++ b/sql/threadpool_generic.h > @@ -88,9 +89,17 @@ struct TP_connection_generic :public TP_connection > ulonglong abs_wait_timeout; > ulonglong enqueue_time; > TP_file_handle fd; > + /** > + Designates whether fd is currently connected to the poll denoted by > + thread_group->pollfd. See also change_group. > + */ > bool bound_to_poll_descriptor; > int waiting; > bool fix_group; > +#ifndef WIN32 > + Notifiable_work_zone work_zone; > + bool leave_work_zone() final; > +#endif Do you need this #ifndef? WIN32 is *never* defined, it's _WIN32. and _WIN32 is never defined *here*, on Windows we use TP_connection_win. > #ifdef _WIN32 > win_aiosocket win_sock{}; > void init_vio(st_vio *vio) override > diff --git a/sql/sql_class.cc b/sql/sql_class.cc > index a0ba1d7e5a3..bfe49cddbf4 100644 > --- a/sql/sql_class.cc > +++ b/sql/sql_class.cc > @@ -1909,13 +1910,13 @@ void THD::awake_no_mutex(killed_state state_to_set) > } > > /* Broadcast a condition to kick the target if it is waiting on it. */ > -void THD::abort_current_cond_wait(bool force) > +void THD::abort_current_cond_wait(bool force, bool mark_abort) a bit illogical, these values aren't orthogonal. you want a 3-state enum, like ABORT_NONE, ABORT_CONNECTIONS_BUT_NOT_SYSTEM_THREADS, ABORT_ALL > { > mysql_mutex_assert_owner(&LOCK_thd_kill); > if (mysys_var) > { > mysql_mutex_lock(&mysys_var->mutex); > - if (!system_thread || force) // Don't abort locks > + if (mark_abort && (!system_thread || force)) // Don't abort locks > mysys_var->abort=1; > > /* > @@ -4373,6 +4374,14 @@ my_bool thd_net_is_killed(THD *thd) > return thd && thd->killed ? 1 : 0; > } > > +/* returns true if any APC was processed */ > +void thd_net_process_apc_requests(THD *thd) even though it's called from net_serv.cc, there's nothing about NET here, so calling it thd_net_process_apc_requests is misleading > +{ > + if (unlikely(thd->apc_target.have_apc_requests())) > + thd->apc_target.process_apc_requests(); you also call it from the threadpool_common and a couple of times you have these two lines verbatim in other places (e.g. sql_parse.cc) why couldn't you call thd->check_killed() instead? > + DEBUG_SYNC(thd, "net_after_apc"); > +} > + > > void thd_increment_bytes_received(void *thd, size_t length) > { > diff --git a/sql/my_apc.h b/sql/my_apc.h > index cc98e36bbe4..ae60bdc2763 100644 > --- a/sql/my_apc.h > +++ b/sql/my_apc.h > @@ -81,7 +83,7 @@ class Apc_target > */ > inline bool have_apc_requests() > { > - return MY_TEST(apc_calls); > + return MY_TEST(apc_calls.load(std::memory_order_acquire)); why? > } > > inline bool is_enabled() { return enabled; } > @@ -95,14 +97,19 @@ class Apc_target > virtual ~Apc_call() {} > }; > > + class Call_request; > /* Make a call in the target thread (see function definition for details) */ > - bool make_apc_call(THD *caller_thd, Apc_call *call, int timeout_sec, bool *timed_out); > + bool make_apc_call(THD *caller_thd, Apc_call *call, > + int timeout_sec, bool *timed_out); > + > + void enqueue_request(Call_request *request_buff, Apc_call *call); > + int wait_for_completion(THD *caller_thd, Call_request *request, > + int timeout_sec); You wrote > In pfs_variable.cc we will have to make an additional step between > enqueue_request and wait_for_completion. These two functions will be called > directly and therefore both should have a public interface and I don't think I quite agree with that. Your "additional step" seems to be abort_current_cond_wait() and notify_apc(). It's not an exotic additional step that only pfs_variable.cc needs, anyone, who makes an apc call needs the target thread to wake up and process it. It's something that should be part of apc code, not done only in pfs_variable.cc And this case you likely won't need enqueue_request and wait_for_completion as a public api > > #ifndef DBUG_OFF > int n_calls_processed; /* Number of calls served by this target */ > #endif > private: > - class Call_request; > > /* > Non-zero value means we're enabled. It's an int, not bool, because one can > diff --git a/sql/threadpool.h b/sql/threadpool.h > index 7737d056b4a..08e85d362e7 100644 > --- a/sql/threadpool.h > +++ b/sql/threadpool.h > @@ -112,6 +119,8 @@ struct TP_connection > > /* Read for the next client command (async) with specified timeout */ > virtual int start_io() = 0; > + IF_WIN(,virtual) bool leave_work_zone(){ return true; } > + IF_WIN(Notifiable_work_zone work_zone;,) // Dummy object. Ugh, this is unreadable. IF_WIN was supposed to be used in expressions. And why do you need virtual methods here, again? > > virtual void wait_begin(int type)= 0; > virtual void wait_end() = 0; > diff --git a/sql/threadpool_generic.cc b/sql/threadpool_generic.cc > index eb08441a4d5..f61dd90606b 100644 > --- a/sql/threadpool_generic.cc > +++ b/sql/threadpool_generic.cc > @@ -114,7 +114,8 @@ struct pool_timer_t > > static pool_timer_t pool_timer; > > -static void queue_put(thread_group_t *thread_group, TP_connection_generic *connection); > +IF_DBUG(,static) you can remove this IF_DBUG, I don't see you use queue_put outside of this file. > +void queue_put(thread_group_t *thread_group, TP_connection_generic *connection); > static void queue_put(thread_group_t *thread_group, native_event *ev, int cnt); > static int wake_thread(thread_group_t *thread_group,bool due_to_stall); > static int wake_or_create_thread(thread_group_t *thread_group, bool due_to_stall=false); > @@ -1485,8 +1520,41 @@ static int change_group(TP_connection_generic *c, > } > > > +#ifndef WIN32 > +bool TP_connection_generic::leave_work_zone() > +{ > + auto leave_state= work_zone.try_leave(); > + while (unlikely(leave_state == Notifiable_work_zone::SIGNAL)) > + { > + if (thd->apc_target.have_apc_requests()) > + thd->apc_target.process_apc_requests(); > + leave_state= work_zone.try_leave(); > + } > + return leave_state == Notifiable_work_zone::OWNER; > +} > +#endif > + > +#ifndef DBUG_OFF > +#include <random> > +static thread_local std::mt19937 mt_random{std::random_device()()}; > +static thread_local std::uniform_int_distribution<int> kill_dist{1, 100000}; > +void queue_put(thread_group_t *thread_group, TP_connection_generic *connection); > +void tp_send(TP_connection_type* c); > +#endif > + > int TP_connection_generic::start_io() > { > + if(DBUG_IF("tp_wanger")) > + { > + bool survive= kill_dist(mt_random) != 555; does that compile? you define kill_dist under #ifndef DBUG_OFF. You can fix this with, like #ifndef DBUG_OFF if (DBUG_IF("tp_wanger")) { static thread_local std::mt19937 ... etc } #endif > + if (survive) > + { > + state= TP_STATE_INTERRUPTED; > + tp_send(this); > + } > + return survive ? 0 : 1; > + } > + > /* > Usually, connection will stay in the same group for the entire > connection's life. However, we do allow group_count to > diff --git a/vio/viosocket.c b/vio/viosocket.c > index 002ff274b74..45c3d80a33d 100644 > --- a/vio/viosocket.c > +++ b/vio/viosocket.c > @@ -909,20 +916,53 @@ static my_bool socket_peek_read(Vio *vio, uint *bytes) > */ > > #ifndef _WIN32 > +#ifdef _GNU_SOURCE > +#define PFD_SIZE 1 > +static inline int vio_poll(struct pollfd pfd[], nfds_t nr, int timeout) > +{ > + struct timespec tm, *tm_arg= NULL; > + sigset_t signals; > + /* Convert the timeout, in milliseconds, to seconds and microseconds. */ > + if (timeout >= 0) > + { > + tm.tv_sec= timeout / 1000; > + tm.tv_nsec= (timeout % 1000) * 1000 * 1000; > + tm_arg= &tm; > + } > + sigemptyset(&signals); > + > + return ppoll(pfd, 1, tm_arg, &signals); > +} > +#else > +#define PFD_SIZE 2 > +#define vio_poll poll > +#endif why do you need that? How is your vio_poll() different from poll() ? > + > int vio_io_wait(Vio *vio, enum enum_vio_io_event event, int timeout) > { > int ret; > short revents __attribute__((unused)) = 0; > - struct pollfd pfd; > + const short revents_read = MY_POLL_SET_IN | MY_POLL_SET_ERR | POLLRDHUP; > + struct pollfd pfd[PFD_SIZE]; > my_socket sd= mysql_socket_getfd(vio->mysql_socket); > + int pfd_size; > MYSQL_SOCKET_WAIT_VARIABLES(locker, state) /* no ';' */ > DBUG_ENTER("vio_io_wait"); > DBUG_PRINT("enter", ("timeout: %d", timeout)); > > - memset(&pfd, 0, sizeof(pfd)); > + memset(pfd, 0, sizeof(pfd)); > > - pfd.fd= sd; > + pfd[0].fd= sd; > > +#ifndef _GNU_SOURCE > + my_socket self_pipe= threadlocal_get_self_pipe(); > + if (self_pipe) > + { > + pfd[1].fd= self_pipe; > + pfd[1].events= MY_POLL_SET_IN; > + } > +#endif > + pfd_size= PFD_SIZE; really? I'd expect it to be essentially pfd_size= self_pipe ? 2 : 1; > /* > Set the poll bitmask describing the type of events. > The error flags are only valid in the revents bitmask. > @@ -970,16 +1023,16 @@ int vio_io_wait(Vio *vio, enum enum_vio_io_event event, int timeout) > } > > #else > - > int vio_io_wait(Vio *vio, enum enum_vio_io_event event, int timeout) > { > int ret; > - struct timeval tm; > - my_socket fd= mysql_socket_getfd(vio->mysql_socket); > - fd_set readfds, writefds, exceptfds; > MYSQL_SOCKET_WAIT_VARIABLES(locker, state) /* no ';' */ > DBUG_ENTER("vio_io_wait"); > > + /* Not read , e.g connect or write - use select() based wait */ > + struct timeval tm; > + my_socket fd= mysql_socket_getfd(vio->mysql_socket); > + fd_set readfds, writefds, exceptfds; it's C, you cannot mix declarations and code > /* Convert the timeout, in milliseconds, to seconds and microseconds. */ > if (timeout >= 0) > { > @@ -1013,14 +1066,17 @@ int vio_io_wait(Vio *vio, enum enum_vio_io_event event, int timeout) > ret= select(0, &readfds, &writefds, &exceptfds, (timeout >= 0) ? &tm : NULL); > > END_SOCKET_WAIT(locker, timeout); > /* Set error code to indicate a timeout error. */ > if (ret == 0) > WSASetLastError(SOCKET_ETIMEDOUT); > > /* Error or timeout? */ > - if (ret <= 0) > + if (ret <= 0){ indentation > + DBUG_PRINT("vio", ("vio_io_wait: error return: %d. Last error: %d", > + ret, socket_errno)); > DBUG_RETURN(ret); > + } > > /* The requested I/O event is ready? */ > switch (event) > diff --git a/sql/sql_connect.cc b/sql/sql_connect.cc > index 9beb3cca8f8..1c8975cfb23 100644 > --- a/sql/sql_connect.cc > +++ b/sql/sql_connect.cc > @@ -23,7 +23,53 @@ > #include "mariadb.h" > #include "mysqld.h" > #include "sql_priv.h" > -#ifndef _WIN32 > +#ifdef _WIN32 > +#ifdef _WIN32_WINNT > +#undef _WIN32_WINNT > +#endif > + > +#define _WIN32_WINNT 0x0500 > +#include "windows.h" // QueueUserAPC2 > +#define WINDOWS_KERNEL32_DLLNAME_W "kernel32" > + > +// https://github.com/dotnet/runtime/pull/55649/files > + > +// These declarations are for a new special user-mode APC feature introduced in Windows. These are not yet available in Windows > +// SDK headers, so some names below are prefixed with "CLONE_" to avoid conflicts in the future. Once the prefixed declarations > +// become available in the Windows SDK headers, the prefixed declarations below can be removed in favor of the SDK ones. > + > +//enum CLONE_QUEUE_USER_APC_FLAGS > +//{ > +// CLONE_QUEUE_USER_APC_FLAGS_NONE = 0x0, > +// CLONE_QUEUE_USER_APC_FLAGS_SPECIAL_USER_APC = 0x1, > +// > +// CLONE_QUEUE_USER_APC_CALLBACK_DATA_CONTEXT = 0x10000 > +//}; > +//typedef BOOL (WINAPI *QueueUserAPC2Proc)(PAPCFUNC ApcRoutine, HANDLE Thread, ULONG_PTR Data, CLONE_QUEUE_USER_APC_FLAGS Flags); > +//void c() > +//{ > +// HMODULE hKernel32 = LoadLibraryExA(WINDOWS_KERNEL32_DLLNAME_W, NULL, LOAD_LIBRARY_SEARCH_SYSTEM32); > +// > +// // See if QueueUserAPC2 exists > +// QueueUserAPC2Proc pfnQueueUserAPC2Proc = (QueueUserAPC2Proc)GetProcAddress(hKernel32, "QueueUserAPC2"); > +// if (pfnQueueUserAPC2Proc == nullptr) > +// { > +// abort(); > +// } > +// > +// // See if QueueUserAPC2 supports the special user-mode APC with a callback that includes the interrupted CONTEXT. A special > +// // user-mode APC can interrupt a thread that is in user mode and not in a non-alertable wait. > +// if (!(*pfnQueueUserAPC2Proc)(EmptyApcCallback, GetCurrentThread(), 0, SpecialUserModeApcWithContextFlags)) > +// { > +// return; > +// } > +// > +// return pfnQueueUserAPC2Proc; > +//} > +// > +//static QueueUserAPC2Proc pfnQueueUserAPC2 = InitializeSpecialUserModeApc(); what's that big commented block for? and win-something defines/includes - still needed? > + > +#else > #include <netdb.h> // getservbyname, servent > #endif > #include "sql_audit.h" > @@ -1353,6 +1399,107 @@ bool thd_is_connection_alive(THD *thd) > return FALSE; > } > > +#if !defined(_WIN32) > +static void self_pipe_write(); > +#if !defined(_GNU_SOURCE) > +class Thread_apc_context; > +static thread_local Thread_apc_context *_THR_APC_CTX= NULL; may be it should be part of THD (rather, Apc_target) or mysys_var? that is, accessible as current_thd->...->self_pipe or my_thread_var->self_pipe the first one (in Apc_target) seems quite logical > +#endif > + > +class Thread_apc_context > +{ > +public: > +#ifndef _GNU_SOURCE > + my_socket self_pipe[2]{}; > +#endif > + > + bool setup_thread_apc() > + { > +#if defined(_GNU_SOURCE) > + struct sigaction act {}; > + act.sa_handler= [](int) -> void {self_pipe_write();}; why? it doesn't do anything, as far as I can see > + act.sa_flags= 0; > + int ret= sigaction(SIG_APC_NOTIFY, &act, NULL); > + DBUG_ASSERT(ret == 0); > + > + sigset_t signals; > + ret|= sigemptyset(&signals); > + DBUG_ASSERT(ret == 0); > + ret|= sigaddset(&signals, SIG_APC_NOTIFY); > + DBUG_ASSERT(ret == 0); > + > + ret|= pthread_sigmask(SIG_BLOCK, &signals, NULL); > + DBUG_ASSERT(ret == 0); > +#else > + // Self-pipe trick. Create a new pipe and store it thread-locally > + // It can be accessed from Vio later. See also vio_io_wait() > + // From the other end, it is accessed through a threadlocal > + // _THR_APC_CTX, from a SIG_APC_NOTIFY signal handler. > + int ret= pipe(self_pipe); > + > + struct sigaction act {}; > + act.sa_handler= [](int) -> void { self_pipe_write(); }; > + act.sa_flags= 0; > + ret|= sigaction(SIG_APC_NOTIFY, &act, NULL); why? you send a signal and convert it to a self-pipe write. why not to write to the pipe in the first place? like Apc_target::wake() which will send a signal or write to a pipe, as appropriate. And TP_pool::wake() will call thd->apc_target->wake(); > + return ret == 0; > +#endif > + return true; > + } > + bool inited; > + Thread_apc_context() > + { > + inited = setup_thread_apc(); > +#ifndef _GNU_SOURCE > + if (inited) > + _THR_APC_CTX= this; > +#endif > + } > +#ifndef _GNU_SOURCE > + ~Thread_apc_context() > + { > + _THR_APC_CTX= NULL; > + closesocket(self_pipe[0]); > + closesocket(self_pipe[1]); > + } > +#endif > +}; > + > +#ifdef _GNU_SOURCE > +static void self_pipe_write() > +{ > + // No self-pipe is actually used. Instead, ppoll is used to wake up on signal. > +} > +#else > + > +my_socket threadlocal_get_self_pipe() > +{ > + return _THR_APC_CTX ? _THR_APC_CTX->self_pipe[0] : 0; > +} > + > +static void self_pipe_write() > +{ > + DBUG_ASSERT(IF_WIN(0, pthread_self() == select_thread) || _THR_APC_CTX); > + if (!_THR_APC_CTX) > + return; // SIGUSR1 is used in thr_alarm, which can wake up main thread. > + char buf[4]{}; > + send(_THR_APC_CTX->self_pipe[1], buf, sizeof buf, 0); > +} > +#endif > +#endif /* !_WIN32 */ > + > +bool thread_per_connection_notify_apc(THD *thd) > +{ > +#ifdef WIN32 > + auto vio= thd->net.vio; > + HANDLE h= > + (vio->type == VIO_TYPE_NAMEDPIPE) ? vio->hPipe : (HANDLE)vio->mysql_socket.fd; > + return CancelIoEx(h, NULL); > +#else > + bool ret = pthread_kill(thd->mysys_var->pthread_self, SIG_APC_NOTIFY) == 0; > + DBUG_ASSERT(ret || errno == EAGAIN); > + return ret; > +#endif > +} 1. I'd prefer APC framework implementation to stay in my_apc.cc. not being spread over three (?) different files. 2. how is it diferent from ::wake() ? > > void do_handle_one_connection(CONNECT *connect, bool put_in_cache) > { > diff --git a/sql/threadpool_common.cc b/sql/threadpool_common.cc > index f72f46b3a6b..8f24bc9e4f7 100644 > --- a/sql/threadpool_common.cc > +++ b/sql/threadpool_common.cc > @@ -320,6 +346,10 @@ static void threadpool_remove_connection(THD *thd) > end_connection(thd); > close_connection(thd, 0); > unlink_thd(thd); > + // The rest of APC requests should be processed after unlinking. > + // This guarantees that no new APC requests can be added. > + // TODO: better notify the requestor with some KILLED state here. > + thd->apc_target.process_apc_requests(); not sure it guarantees much. One can find the thd before the unlink and add a new request after the unlink. or is it impossible? > PSI_CALL_delete_current_thread(); // before THD is destroyed > delete thd; > > diff --git a/sql/my_apc.cc b/sql/my_apc.cc > index 9777deb399a..a89d9c8bc19 100644 > --- a/sql/my_apc.cc > +++ b/sql/my_apc.cc > @@ -47,11 +47,12 @@ void Apc_target::init(mysql_mutex_t *target_mutex) > void Apc_target::enqueue_request(Call_request *qe) > { > mysql_mutex_assert_owner(LOCK_thd_kill_ptr); > - if (apc_calls) > + Call_request *old_apc_calls= apc_calls; eh, what? you modify apc_calls without a lock somewhere? > + if (old_apc_calls) > { > - Call_request *after= apc_calls->prev; > - qe->next= apc_calls; > - apc_calls->prev= qe; > + Call_request *after= old_apc_calls->prev; > + qe->next= old_apc_calls; > + old_apc_calls->prev= qe; > > qe->prev= after; > after->next= qe; > @@ -100,89 +109,118 @@ void init_show_explain_psi_keys(void) > if (PSI_server == NULL) > return; > > - PSI_server->register_cond("sql", show_explain_psi_conds, > - array_elements(show_explain_psi_conds)); > + PSI_server->register_cond("sql", apc_request_psi_conds, > + array_elements(apc_request_psi_conds)); > + PSI_server->register_mutex("sql", apc_request_psi_locks, > + array_elements(apc_request_psi_locks)); > } > #endif > > - > -/* > - Make an APC (Async Procedure Call) to another thread. > - > - @detail > - Make an APC call: schedule it for execution and wait until the target > - thread has executed it. > - > - - The caller is responsible for making sure he's not posting request > - to the thread he's calling this function from. > - > - - The caller must have locked target_mutex. The function will release it. > - > - @retval FALSE - Ok, the call has been made > - @retval TRUE - Call wasnt made (either the target is in disabled state or > - timeout occurred) > -*/ > - > -bool Apc_target::make_apc_call(THD *caller_thd, Apc_call *call, > - int timeout_sec, bool *timed_out) > +/** > + Wait gracefully until the request is completed. > + @retval 0 -- Success > + @retval 1 -- Timeout > + */ > +int Apc_target::wait_for_completion(THD *caller_thd, Call_request *apc_request, > + int timeout_sec) > { > - bool res= TRUE; > - *timed_out= FALSE; > - > - if (enabled) > - { > - /* Create and post the request */ > - Call_request apc_request; > - apc_request.call= call; > - apc_request.processed= FALSE; > - mysql_cond_init(key_show_explain_request_COND, &apc_request.COND_request, > - NULL); > - enqueue_request(&apc_request); > - apc_request.what="enqueued by make_apc_call"; > - > struct timespec abstime; > const int timeout= timeout_sec; > set_timespec(abstime, timeout); > > + DBUG_EXECUTE_IF("apc_timeout", set_timespec_nsec(abstime, 1000000);); > + int res = 1; > int wait_res= 0; > PSI_stage_info old_stage; > - caller_thd->ENTER_COND(&apc_request.COND_request, LOCK_thd_kill_ptr, > + > + mysql_mutex_unlock(LOCK_thd_kill_ptr); I don't understand how this works. After you release LOCK_thd_kill_ptr, the THD might be destroyed. How do you detect that? By the wait timing out? Who will destroy the Call_request? > + > + mysql_mutex_lock(&apc_request->LOCK_request); > + caller_thd->ENTER_COND(&apc_request->COND_request, &apc_request->LOCK_request, > &stage_show_explain, &old_stage); > /* todo: how about processing other errors here? */ > - while (!apc_request.processed && (wait_res != ETIMEDOUT)) > + while (!apc_request->processed && (wait_res != ETIMEDOUT)) > { > - /* We own LOCK_thd_kill_ptr */ > - wait_res= mysql_cond_timedwait(&apc_request.COND_request, > - LOCK_thd_kill_ptr, &abstime); > - // &apc_request.LOCK_request, &abstime); > + wait_res= mysql_cond_timedwait(&apc_request->COND_request, > + &apc_request->LOCK_request, &abstime); > if (caller_thd->killed) > break; > + > + if (caller_thd->apc_target.have_apc_requests()) > + { > + mysql_mutex_unlock(&apc_request->LOCK_request); > + caller_thd->apc_target.process_apc_requests(); > + mysql_mutex_lock(&apc_request->LOCK_request); > + } > } > > - if (!apc_request.processed) > + if (!apc_request->processed) > { > /* > The wait has timed out, or this thread was KILLed. > - Remove the request from the queue (ok to do because we own > - LOCK_thd_kill_ptr) > + We can't delete it from the queue, because LOCK_thd_kill_ptr is already > + released. It can't be reacquired because of the ordering with > + apc_request->LOCK_request. > + However, apc_request->processed is guarded by this lock. > + Set processed= TRUE and transfer the ownership to the processor thread. > + It should free the resources itself. > + apc_request cannot be referred after unlock anymore in this case. > */ > - apc_request.processed= TRUE; > - dequeue_request(&apc_request); > - *timed_out= TRUE; > - res= TRUE; > + apc_request->processed= TRUE; > + res= 1; > } > else > { > /* Request was successfully executed and dequeued by the target thread */ > - res= FALSE; > + res= 0; > } > - /* > - exit_cond() will call mysql_mutex_unlock(LOCK_thd_kill_ptr) for us: > - */ > + > + /* EXIT_COND() will call mysql_mutex_unlock(LOCK_request) for us */ > caller_thd->EXIT_COND(&old_stage); > > - /* Destroy all APC request data */ > - mysql_cond_destroy(&apc_request.COND_request); > + return res; > +} > + > +/** Create and post the request */ > +void Apc_target::enqueue_request(Call_request *request_buff, Apc_call *call) > +{ > + request_buff->call= call; > + request_buff->processed= FALSE; > + enqueue_request(request_buff); > + request_buff->what="enqueued by make_apc_call"; > +} > + > +/** > + Make an APC (Async Procedure Call) to another thread. > + > + @detail > + Make an APC call: schedule it for execution and wait until the target > + thread has executed it. > + > + - The caller is responsible for making sure he's not posting request > + to the thread he's calling this function from. > + > + - The caller must have locked target_mutex. The function will release it. > + > + @retval FALSE - Ok, the call has been made > + @retval TRUE - Call wasnt made (either the target is in disabled state or > + timeout occurred) > +*/ > + > +bool Apc_target::make_apc_call(THD *caller_thd, Apc_call *call, > + int timeout_sec, bool *timed_out) > +{ > + bool res= TRUE; > + *timed_out= FALSE; > + > + if (enabled) > + { > + Call_request *apc_request= new Call_request; > + enqueue_request(apc_request, call); > + res= wait_for_completion(caller_thd, apc_request, timeout_sec); > + *timed_out= res; > + if (!*timed_out) > + delete apc_request; > } > else > { > diff --git a/storage/perfschema/pfs_variable.cc b/storage/perfschema/pfs_variable.cc > index 17b7dfc200c..698ff1eca6c 100644 > --- a/storage/perfschema/pfs_variable.cc > +++ b/storage/perfschema/pfs_variable.cc > @@ -164,7 +167,11 @@ int PFS_system_variable_cache::do_materialize_global(void) > during materialization. > */ > if (!m_external_init) > + { > + mysql_mutex_lock(&LOCK_plugin); > init_show_var_array(OPT_GLOBAL, true); > + mysql_mutex_unlock(&LOCK_plugin); 1. why not move the locking inside init_show_var_array()? 2. init_show_var_array() locks LOCK_system_variables_hash - is that not enough? > + } > > /* Resolve the value for each SHOW_VAR in the array, add to cache. */ > for (SHOW_VAR *show_var= m_show_var_array.front(); > @@ -230,37 +236,26 @@ int PFS_system_variable_cache::do_materialize_all(THD *unsafe_thd) > m_materialized= false; > m_cache.clear(); > > - /* Block plugins from unloading. */ > - mysql_mutex_lock(&LOCK_plugin_delete); also please remove all references of LOCK_plugin_delete > - > /* > Build array of SHOW_VARs from system variable hash. Do this within > LOCK_plugin_delete to ensure that the hash table remains unchanged > while this thread is materialized. > */ > if (!m_external_init) > + { > + mysql_mutex_lock(&LOCK_plugin); > init_show_var_array(OPT_SESSION, false); > + mysql_mutex_unlock(&LOCK_plugin); > + } > > /* Get and lock a validated THD from the thread manager. */ > if ((m_safe_thd= get_THD(unsafe_thd)) != NULL) > { > DEBUG_SYNC(m_current_thd, "materialize_session_variable_array_THD_locked"); > - for (SHOW_VAR *show_var= m_show_var_array.front(); > - show_var->value && (show_var != m_show_var_array.end()); show_var++) > - { > - /* Resolve value, convert to text, add to cache. */ > - System_variable system_var(m_safe_thd, show_var, m_query_scope, false); > - m_cache.push(system_var); > - } > - > - /* Release lock taken in get_THD(). */ > - mysql_mutex_unlock(&m_safe_thd->LOCK_thd_data); > + ret= make_call(&PFS_system_variable_cache::refresh_vars, 1); > > m_materialized= true; > - ret= 0; > } > - > - mysql_mutex_unlock(&LOCK_plugin_delete); > return ret; > } > > @@ -312,6 +307,105 @@ void PFS_system_variable_cache::free_mem_root(void) > } > } > } > +class PFS_system_variable_cache_apc: public Apc_target::Apc_call > +{ > +public: > + typedef PFS_system_variable_cache::Request_func Request; > + PFS_system_variable_cache_apc(PFS_system_variable_cache *pfs, Request func, > + uint param) > + : m_pfs(pfs), m_func(func), m_param(param) {} > +private: > + PFS_system_variable_cache *m_pfs; > + Request m_func; > + uint m_param; > + > + void call_in_target_thread() override > + { > + call(m_pfs, m_func, m_param); > + } > +public: > + static void call(PFS_system_variable_cache *pfs, Request func, uint param) > + { > + THD *safe_thd= pfs->safe_thd(); > + > + DBUG_ASSERT(pfs->query_scope() == OPT_SESSION); > + > + mysql_mutex_lock(&LOCK_global_system_variables); > + if (!safe_thd->variables.dynamic_variables_ptr || > + global_system_variables.dynamic_variables_head > > + safe_thd->variables.dynamic_variables_head) > + { > + mysql_prlock_rdlock(&LOCK_system_variables_hash); > + sync_dynamic_session_variables(safe_thd, false); > + mysql_prlock_unlock(&LOCK_system_variables_hash); > + } > + mysql_mutex_unlock(&LOCK_global_system_variables); > + > + (pfs->*func)(param); > + } > +}; > + > +void PFS_system_variable_cache::refresh_vars(uint all) > +{ > + for (SHOW_VAR *show_var= m_show_var_array.front(); > + show_var->value && (show_var != m_show_var_array.end()); show_var++) > + { > + sys_var *value= (sys_var *)show_var->value; > + > + /* Match the system variable scope to the target scope. */ > + if (all || match_scope(value->scope())) > + { > + /* Resolve value, convert to text, add to cache. */ > + System_variable system_var(m_safe_thd, show_var, m_query_scope, false); > + m_cache.push(system_var); > + } > + } > +} > +void PFS_system_variable_cache::refresh_one_var(uint index) > +{ > + SHOW_VAR *show_var= &m_show_var_array.at(index); > + > + if (show_var && show_var->value && > + (show_var != m_show_var_array.end())) > + { > + sys_var *value= (sys_var *)show_var->value; > + > + /* Match the system variable scope to the target scope. */ > + if (match_scope(value->scope())) > + { > + /* Resolve value, convert to text, add to cache. */ > + System_variable system_var(m_safe_thd, show_var, m_query_scope, false); > + m_cache.push(system_var); > + } > + } > +} > + > + > +#define MAKE_CALL_MAX_RETRIES 3 unused > + > +int PFS_system_variable_cache::make_call(Request_func func, uint param) > +{ > + int ret= 0; > + THD *requestor_thd= m_current_thd; > + if (requestor_thd == m_safe_thd) > + { > + mysql_mutex_unlock(&m_safe_thd->LOCK_thd_kill); > + PFS_system_variable_cache_apc::call(this, func, param); > + } > + else > + { > + PFS_system_variable_cache_apc apc_call(this, func, param); > + auto *request= new Apc_target::Call_request; > + m_safe_thd->apc_target.enqueue_request(request, &apc_call); > + m_safe_thd->abort_current_cond_wait(false, false); > + m_safe_thd->scheduler->notify_apc(m_safe_thd); > + DEBUG_SYNC(requestor_thd, "apc_after_notify"); > + ret= m_safe_thd->apc_target.wait_for_completion(requestor_thd, request, 10); > + if (ret == 0) > + delete request; > + } > + return ret; > +} > > /** > Build a SESSION system variable cache for a pfs_thread. Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

1 0

Re: c46e7c5cb64: MDEV-12252 ROW data type for stored function return values
by Sergei Golubchik 05 Sep '24

05 Sep '24

Hi, Alexander, Looks good, thanks. Great commit comment too. Just a few questions and test suggestions - see below. Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org On Aug 23, Alexander Barkov wrote: > revision-id: c46e7c5cb64 (mariadb-11.4.1-5-gc46e7c5cb64) > parent(s): ae6684d79f4 > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2024-02-20 13:49:54 +0400 > message: > > MDEV-12252 ROW data type for stored function return values > diff --git a/mysql-test/main/sp-anchor-row-type-table.test b/mysql-test/main/sp-anchor-row-type-table.test > index 7123f9160db..2fbcfbffc1b 100644 > --- a/mysql-test/main/sp-anchor-row-type-table.test > +++ b/mysql-test/main/sp-anchor-row-type-table.test > @@ -939,3 +939,70 @@ DROP TABLE t1; > --echo # > --echo # End of 10.4 tests > --echo # > + > +--echo # > +--echo # Start of 11.3 tests > +--echo # > + > +--echo # > +--echo # MDEV-12252 ROW data type for stored function return values > +--echo # > + > +--error ER_PARSE_ERROR > +CREATE FUNCTION f1() RETURNS ROW TYPE OF step1.step2.step3 RETURN ROW(1,2); > + > +CREATE FUNCTION f1() RETURNS ROW TYPE OF t1 RETURN ROW(1,2); you don't resolve t1 at CREATE FUNCTION time. why? is it how TYPE OF works in DECLARE too? > +SHOW CREATE FUNCTION f1; > +--error ER_NO_SUCH_TABLE > +SELECT f1(); > +DROP FUNCTION f1; > + > +CREATE FUNCTION f1() RETURNS ROW TYPE OF test.t1 RETURN ROW(1,2); > +SHOW CREATE FUNCTION f1; > +--error ER_NO_SUCH_TABLE > +SELECT f1(); > +DROP FUNCTION f1; > + > + > +CREATE TABLE t1 (a INT, b VARCHAR(32)); > +CREATE FUNCTION f1() RETURNS ROW TYPE OF test.t1 RETURN (SELECT * FROM t1); > +SHOW CREATE FUNCTION f1; > +DELIMITER $$; > +CREATE PROCEDURE p1() > +BEGIN > + DECLARE r ROW TYPE OF t1 DEFAULT f1(); > + SELECT r.a, r.b; > +END; > +$$ > +DELIMITER ;$$ > + > +# Testing with no rows > +CALL p1(); > +--error ER_OPERAND_COLUMNS > +SELECT f1(); > +SELECT f1()=ROW(1,'b1') AS c; > +SELECT f1()=ROW(NULL,NULL) AS c; > + > +# Testing with one row > +INSERT INTO t1 VALUES (1,'b1'); > +CALL p1(); > +--error ER_OPERAND_COLUMNS > +SELECT f1(); > +SELECT f1()=ROW(1,'b1') AS c; > +SELECT f1()=ROW(1,'') AS c; > +SELECT f1()=ROW(2,'b1') AS c; > +SELECT f1()=ROW(1,NULL) AS c; > +SELECT f1()=ROW(NULL,'b1') AS c; do one more SHOW CREATE FUNCTION here > + > +# Testing with two rows > +INSERT INTO t1 VALUES (2,'b2'); > +--error ER_SUBQUERY_NO_1_ROW > +CALL p1(); > +--error ER_OPERAND_COLUMNS > +SELECT f1(); > +--error ER_SUBQUERY_NO_1_ROW > +SELECT f1()=ROW(1,'b1') AS c; > + > +DROP PROCEDURE p1; > +DROP FUNCTION f1; > +DROP TABLE t1; > diff --git a/mysql-test/main/sp-anchor-type.test b/mysql-test/main/sp-anchor-type.test > index 0e24ef900d8..fed53fd011a 100644 > --- a/mysql-test/main/sp-anchor-type.test > +++ b/mysql-test/main/sp-anchor-type.test > @@ -767,3 +767,78 @@ BEGIN NOT ATOMIC > END; > $$ > DELIMITER ;$$ > + > + > +--echo # > +--echo # Start of 11.3 tests > +--echo # > + > +--echo # > +--echo # MDEV-12252 ROW data type for stored function return values > +--echo # > + > +--error ER_PARSE_ERROR > +CREATE FUNCTION f1() RETURNS TYPE OF a RETURN 1; > + > +CREATE FUNCTION f1() RETURNS TYPE OF t1.a RETURN (SELECT min(a) FROM t1); > +SHOW CREATE FUNCTION f1; > +--error ER_NO_SUCH_TABLE > +SELECT f1(); > +CREATE TABLE t1 (b INT); > +--error ER_BAD_FIELD_ERROR > +SELECT f1(); > +DROP TABLE t1; > +DROP FUNCTION f1; > + > +CREATE DATABASE db1; > +DELIMITER $$; > +CREATE FUNCTION db1.f1() RETURNS TYPE OF db1.t1.a > +BEGIN > + RETURN (SELECT min(a) FROM t1); > +END; > +$$ > +DELIMITER ;$$ > +SHOW CREATE FUNCTION db1.f1; > +--error ER_NO_SUCH_TABLE > +SELECT db1.f1(); > +CREATE TABLE db1.t1 (b TIME); > +--error ER_BAD_FIELD_ERROR > +SELECT db1.f1(); > +DROP TABLE db1.t1; > +CREATE TABLE db1.t1 (a TIME); > +SELECT db1.f1(); > +INSERT INTO db1.t1 VALUES ('10:20:30'); > +SELECT db1.f1(); > +DROP TABLE db1.t1; > +DROP FUNCTION db1.f1; > +DROP DATABASE db1; > + > +CREATE TABLE t1 (a TIME); > +CREATE FUNCTION f1() RETURNS TYPE OF test.t1.a RETURN (SELECT min(a) FROM t1); > +SHOW CREATE FUNCTION f1; > +SELECT f1(); > +DROP FUNCTION f1; > +DROP TABLE t1; > + > +CREATE TABLE t1 (a TIME); > +DELIMITER $$; > +CREATE FUNCTION f1() RETURNS TYPE OF t1.a > +BEGIN > + RETURN (SELECT min(a) FROM t1); > +END; > +$$ > +DELIMITER ;$$ > +SHOW CREATE FUNCTION f1; > +SELECT f1(); > +CREATE TABLE t2 AS SELECT f1(); > +SHOW CREATE TABLE t2; > +SELECT * FROM t2; > +DROP TABLE t2; > +INSERT INTO t1 VALUES ('10:20:30'); > +SELECT f1(); > +CREATE TABLE t2 AS SELECT f1(); > +SHOW CREATE TABLE t2; > +SELECT * FROM t2; > +DROP TABLE t2; please, also try here DROP TABLE t1; CREATE TABLE t1 (a INT); INSERT INTO t1 VALUES (10); CREATE TABLE t2 AS SELECT f1(); the point is to change TYPE OF t1.a after the function has been successfully called, and thus is already in the sp cache. > +DROP FUNCTION f1; > +DROP TABLE t1; > diff --git a/mysql-test/suite/compat/oracle/t/sp-package.test b/mysql-test/suite/compat/oracle/t/sp-package.test > index aefede41c8b..7c231d736c5 100644 > --- a/mysql-test/suite/compat/oracle/t/sp-package.test > +++ b/mysql-test/suite/compat/oracle/t/sp-package.test > @@ -3109,3 +3109,79 @@ DROP TABLE t1; > DROP FUNCTION f1_deterministic; > DROP FUNCTION f2_not_deterministic; > DROP PACKAGE pkg1; > + > +--echo # > +--echo # MDEV-12252 ROW data type for stored function return values > +--echo # > + > +--echo # > +--echo # Testing fixed ROW type with package routines > +--echo # > + > +DELIMITER $$; > +CREATE PACKAGE pkg > +AS > + FUNCTION f1() RETURN ROW(a INT, b VARCHAR(32)); > + PROCEDURE p1(r ROW(a INT, b VARCHAR(32))); > + PROCEDURE p2(); > +END; > +$$ > +CREATE PACKAGE BODY pkg > +AS > + FUNCTION f1() RETURN ROW(a INT, b VARCHAR(32)) AS > + BEGIN > + RETURN ROW(1,'b1'); > + END; > + PROCEDURE p1(r ROW(a INT, b VARCHAR(32))) AS > + BEGIN > + SELECT r.a, r.b; > + END; > + PROCEDURE p2() AS > + BEGIN > + CALL p1(f1()); > + END; > +END; > +$$ > +DELIMITER ;$$ > +CALL pkg.p1(pkg.f1()); > +CALL pkg.p2; > +DROP PACKAGE pkg; > + > + > +--echo # > +--echo # Testing table%ROWTYPE with package routines > +--echo # now you can also add package tests outside of oracle mode > + > +CREATE TABLE t1 (a INT, b VARCHAR(32)); > +INSERT INTO t1 VALUES (1,'b1'); > +DELIMITER /; > +CREATE PACKAGE pkg > +AS > + FUNCTION f1 RETURN t1%ROWTYPE; > + PROCEDURE p1(r t1%ROWTYPE); > + PROCEDURE p2; > +END; > +/ > +CREATE PACKAGE BODY pkg > +AS > + FUNCTION f1 RETURN t1%ROWTYPE AS > + r t1%ROWTYPE; > + BEGIN > + SELECT * INTO r FROM t1; > + RETURN r; > + END; > + PROCEDURE p1(r t1%ROWTYPE) AS > + BEGIN > + SELECT r.a || ' ' || r.b; > + END; > + PROCEDURE p2 AS > + BEGIN > + p1(f1()); > + END; > +END; > +/ > +DELIMITER ;/ > +CALL pkg.p1(pkg.f1()); > +CALL pkg.p2; > +DROP PACKAGE pkg; > +DROP TABLE t1; > diff --git a/sql/field.h b/sql/field.h > index 7f1c243a180..7af069a9735 100644 > --- a/sql/field.h > +++ b/sql/field.h > @@ -1373,6 +1374,20 @@ class Field: public Value_source > in str and restore it with set() if needed > */ > virtual void sql_type(String &str) const =0; > + void sql_type_for_sp_returns(String &str) const This is oddly specific. Strange that we don't need to print the field type with charset/collation anywhere else, but for sp returns we do. > + { > + sql_type(str); > + if (has_charset()) > + { > + str.append(STRING_WITH_LEN(" CHARSET ")); > + str.append(charset()->cs_name); > + if (Charset(charset()).can_have_collate_clause()) > + { > + str.append(STRING_WITH_LEN(" COLLATE ")); > + str.append(charset()->coll_name); > + } > + } > + } > virtual void sql_rpl_type(String *str) const { sql_type(*str); } > virtual uint size_of() const =0; // For new field > inline bool is_null(my_ptrdiff_t row_offset= 0) const > diff --git a/sql/field.cc b/sql/field.cc > index 35be045620d..23b050de221 100644 > --- a/sql/field.cc > +++ b/sql/field.cc > @@ -2768,6 +2768,32 @@ bool Field_row::sp_prepare_and_store_item(THD *thd, Item **value) > } > > > +uint Field_row::cols() const > +{ > + // The table with ROW members must be already instantiated > + DBUG_ASSERT(m_table); > + return m_table->s->fields; > +} > + > + > +void Field_row::sql_type(String &res) const would be more logical to call this Field_row::sql_type_for_sp_returns() > +{ > + res.set_ascii(STRING_WITH_LEN("row(")); > + for (uint i= 0; i < m_table->s->fields; i++) > + { > + if (i > 0) > + res.append(','); > + Field *field= m_table->field[i]; > + res.append(field->field_name); > + res.append(' '); > + StringBuffer<64> col; > + field->sql_type_for_sp_returns(col); > + res.append(col.to_lex_cstring()); > + } > + res.append(')'); > +} > + > + > /**************************************************************************** > Functions for the Field_decimal class > This is an number stored as a pre-space (or pre-zero) string > diff --git a/sql/item.cc b/sql/item.cc > index 54ad511cb6b..0a125752979 100644 > --- a/sql/item.cc > +++ b/sql/item.cc > @@ -3001,10 +3003,31 @@ Item_sp::init_result_field(THD *thd, uint max_length, uint maybe_null, > dummy_table->s->table_name= empty_clex_str; > dummy_table->maybe_null= maybe_null; > > + if (m_sp->m_return_field_def.is_column_type_ref() && > + m_sp->m_return_field_def.column_type_ref()-> > + resolve_type_ref(thd, &m_sp->m_return_field_def)) > + DBUG_RETURN(TRUE); Here you overwrite old m_return_field_def value which was a column_type_ref with its actual type, don't you? What will happen if you change the column's type after the sp was already executed once? > + > if (!(sp_result_field= m_sp->create_result_field(max_length, name, > dummy_table))) > DBUG_RETURN(TRUE); > > + /* > + In case of a ROW return type we need to create Item_field_row > + on top of Field_row, and remember it in sp_result_item_field_row. > + ROW members are later accessed using sp_result_item_field_row->addr(i), > + e.g. when copying the function return value to a local variable. > + For scalar return types no Item_field is needed around sp_result_field, > + as the value is fetched directly from sp_result_field, > + inside Item_func_sp::val_xxx() methods. Sorry, I don't understand that. Why cannot you use Field_row here? > + */ > + if (Field_row *field_row= dynamic_cast<Field_row*>(sp_result_field)) > + { > + if (!(sp_result_item_field_row= > + m_sp->m_return_field_def.make_item_field_row(thd, field_row))) > + DBUG_RETURN(true); > + } > + > if (sp_result_field->pack_length() > sizeof(result_buf)) > { > void *tmp; > diff --git a/sql/sp.cc b/sql/sp.cc > index ddaeeb8cb7a..7eab8304c31 100644 > --- a/sql/sp.cc > +++ b/sql/sp.cc > @@ -1091,18 +1136,19 @@ sp_returns_type(THD *thd, String &result, const sp_head *sp) > bzero((char*) &share, sizeof(share)); > table.in_use= thd; > table.s = &share; > - field= sp->create_result_field(0, 0, &table); > - field->sql_type(result); > + field= create_result_field(0, 0, &table); > > - if (field->has_charset()) > + if (m_return_field_def.is_row()) > { > - result.append(STRING_WITH_LEN(" CHARSET ")); > - result.append(field->charset()->cs_name); > - if (Charset(field->charset()).can_have_collate_clause()) > - { > - result.append(STRING_WITH_LEN(" COLLATE ")); > - result.append(field->charset()->coll_name); > - } > + Field_row *field_row= dynamic_cast<Field_row*>(field); > + if (!field_row->row_create_fields( > + thd, m_return_field_def.row_field_definitions())) > + field->sql_type(result); when is this used? you're creating a TABLE and Field's - only to print types and charsets. This might be tolerable, but only if it's not needed often > + } > + else > + { > + DBUG_ASSERT(m_return_field_def.type_handler()->is_scalar_type()); > + field->sql_type_for_sp_returns(result); > } > > delete field; > diff --git a/sql/sql_yacc.yy b/sql/sql_yacc.yy > index 5977e309ae4..f11e4f89466 100644 > --- a/sql/sql_yacc.yy > +++ b/sql/sql_yacc.yy > @@ -18347,7 +18338,50 @@ sp_parameters: > ; > > sf_returned_type_clause: > - RETURNS_SYM sf_return_type > + RETURNS_SYM > + { > + LEX *lex= Lex; you still do this lex=Lex? :) it became obsolete, like, 15 years ago. If not more. > + lex->init_last_field(&lex->sphead->m_return_field_def, > + &empty_clex_str); > + } > + sf_return_type > + ; > + > +sf_return_type: > + field_type > + { > + if (unlikely(Lex->sf_return_fill_definition($1))) > + MYSQL_YYABORT; > + } > + | ROW_SYM row_type_body > + { > + if (Lex->sf_return_fill_definition_row($2)) > + MYSQL_YYABORT; > + } > + | ROW_SYM TYPE_SYM OF_SYM ident > + { > + if (Lex->sf_return_fill_definition_rowtype_of( > + Qualified_column_ident(&$4))) > + MYSQL_YYABORT; > + } > + | ROW_SYM TYPE_SYM OF_SYM ident '.' ident > + { > + if (Lex->sf_return_fill_definition_rowtype_of( > + Qualified_column_ident(&$4, &$6))) > + MYSQL_YYABORT; > + } > + | TYPE_SYM OF_SYM ident '.' ident > + { > + if (Lex->sf_return_fill_definition_type_of( > + Qualified_column_ident(&$3, &$5))) > + MYSQL_YYABORT; > + } > + | TYPE_SYM OF_SYM ident '.' ident '.' ident > + { > + if (Lex->sf_return_fill_definition_type_of( > + Qualified_column_ident(thd, &$3, &$5, &$7))) > + MYSQL_YYABORT; > + } > ; > > package_implementation_item_declaration:

2 1

Clean up executable test result files:
by Otto Kekäläinen 04 Sep '24

04 Sep '24

Hi! In https://github.com/MariaDB/server/commit/9cb75f333fa2ae816909f95a4e19e6ddac… Alexander made some test result files executable (look at 100644 → 100755 in git diff). I sent twice a PR to fix this: - Oct 2023: https://github.com/MariaDB/server/pull/2779 - Feb 2024: https://github.com/MariaDB/server/pull/3042 Unfortunately, I can see that on the main branch this is executable again: mariadb-server main+ ± find mysql-test -executable -name *.test -or -executable -name *.result mysql-test/main/lowercase_table2.result Could the next person doing merges fix this tiny annoyance once and for all? Thanks :)

1 0

Re: [PATCH] MDEV-32014 Rename binlog cache temporary file to binlog file for large transaction
by Kristian Nielsen 03 Sep '24

03 Sep '24

Andrei Elkin <andrei.elkin(a)mariadb.com> writes: >> So please do a small refactoring of trx_group_commit_leader() and keep >> only ... > > and no need for refactoring really arises. Well, the refactoring is good, even without the MDEV-32014 patch. The function trx_group_commit_leader() is already too long and complicated, and does too much in a single function. And the first part (obtaining the queue) is cleanly separated from the second part (committing each entry in the queue in-order in binlog and participating engines). It's just nice that this improvement of the existing code also makes the MDEV-32014 patch integrate cleaner. > I noted earlier on this part on the PR page and LiBing seems to have > practically proven (by the most recent commit that you must've missed) that > conducting the rename inside the flush stage of > trx_group_commit_leader() is viable: > > https://github.com/MariaDB/server/pull/3402/commits/287511a703f7dff47939633… > > By that measure the flushing is expanded onto possibly multiple "implicitly" > rotating log files Hm, I don't like the change 287511a703f7dff47939633475edd436a542cf6f. In fact, that is one of the things I like the most about Libing Song's patch, that it so cleanly separates the commit-by-rotate logic from the complex group commit logic in queue_for_group_commit() and trx_group_commit_leader(), which is already too complex. The original patch without 287511a703f7dff47939633475edd436a542cf6f seems much cleaner. It is better to do as in the original patch, and treat commit-by-rotate like a normal binlog rotate, not try to participate that transaction in binlog group commit. We cannot share the fsync anyway, as the files are different. And it introduces a lot of complexities to suddenly have binlog rotations in the middle of group commit - what about the complex accounting around xid_count and binlog checkpoints? There will be a lot of corner cases eg. if _two_ commit-by-rotate happens in the same group commit, with lots of potential for subtle bugs that will occur only very rarely in production and be almost impossible to track down. I strongly prefer the original patch I reviewed (with my suggested changes to write_transaction_to_binlog_events() and trx_group_commit_leader()), where the commit-by-rotate logic is cleanly separated from the group commit logic and implemented mostly using the existing binlog rotate logic. - Kristian.

2 2

Re: [PATCH] MDEV-34857: Implement --slave-abort-blocking-timeout
by Michael Widenius 03 Sep '24

03 Sep '24

Review of MDEV-34857: Implement --slave-abort-blocking-timeout If a slave replicating an event has waited for more than @@slave_abort_blocking_timeout for a conflicting metadata lock held by a non-replication thread, the blocking query is killed to allow replication to proceed and not be blocked indefinitely by a user query. <cut> diff --git a/sql/mdl.cc b/sql/mdl.cc index faccd1c9476..9845718e165 100644 --- a/sql/mdl.cc +++ b/sql/mdl.cc <cut> @@ -2397,14 +2398,39 @@ MDL_context::acquire_lock(MDL_request *mdl_request, double lock_wait_timeout) find_deadlock(); - struct timespec abs_timeout, abs_shortwait; + struct timespec abs_timeout, abs_shortwait, abs_abort_blocking_timeout; + bool abort_blocking_enabled= false; + double abort_blocking_timeout= slave_abort_blocking_timeout; Add a comment here. Something like /* This code implements aborting of queries on the slaves that stops replication from continuing. */ + if (abort_blocking_timeout < lock_wait_timeout && + m_owner->get_thd()->rgi_slave) + { + set_timespec_nsec(abs_abort_blocking_timeout, + (ulonglong)(abort_blocking_timeout * 1000000000ULL)); + abort_blocking_enabled= true; + } set_timespec_nsec(abs_timeout, (ulonglong)(lock_wait_timeout * 1000000000ULL)); - set_timespec(abs_shortwait, 1); wait_status= MDL_wait::EMPTY; - while (cmp_timespec(abs_shortwait, abs_timeout) <= 0) + for (;;) { + bool abort_blocking= false; + set_timespec(abs_shortwait, 1); + if (abort_blocking_enabled && + cmp_timespec(abs_shortwait, abs_abort_blocking_timeout) >= 0) + { + /* + If a slave DDL has waited for --slave-abort-select-timeout, then notify + any blocking SELECT once before continuing to wait until the full + timeout. + */ + abs_shortwait= abs_abort_blocking_timeout; + abort_blocking= true; + abort_blocking_enabled= false; + } + if (cmp_timespec(abs_shortwait, abs_timeout) > 0) + break; Is the above right last compare right? Assuming that cmp_timespec(abs_shortwait, abs_abort_blocking_timeout) > 0 then the above will not break and we will not signal other threads to break even if the previous code expectrs us to abort blocking. (I think the cmp_timespec() return test should be the same in both cases) Shouldn't the code be: for (;;) { bool abort_blocking= false; set_timespec(abs_shortwait, 1); if (abort_blocking_enabled && cmp_timespec(abs_shortwait, abs_abort_blocking_timeout) >= 0) { /* If a slave DDL has waited for --slave-abort-select-timeout, then notify any blocking SELECT once before continuing to wait until the full timeout. */ abort_blocking= true; abort_blocking_enabled= false; } else if (cmp_timespec(abs_shortwait, abs_timeout) > 0) break; Or have I missed something? /* abs_timeout is far away. Wait a short while and notify locks. */ wait_status= m_wait.timed_wait(m_owner, &abs_shortwait, FALSE, mdl_request->key.get_wait_state_name()); @@ -2425,9 +2451,8 @@ MDL_context::acquire_lock(MDL_request *mdl_request, double lock_wait_timeout) mysql_prlock_wrlock(&lock->m_rwlock); if (lock->needs_notification(ticket)) - lock->notify_conflicting_locks(this); + lock->notify_conflicting_locks(this, abort_blocking); mysql_prlock_unlock(&lock->m_rwlock); - set_timespec(abs_shortwait, 1); I like the new code in that we are not setting abs_shortwait two times anymore. (Using a while loop in the is case was sub optimal. Ok to push after the above change or if you have a good argument why the current code is correct

1 0

Re: [PATCH] MDEV-32014 Rename binlog cache temporary file to binlog file for large transaction
by andrei.elkin＠pp.inet.fi 02 Sep '24

02 Sep '24

(Re-sending to the list) Howdy Kristian. > > The patch calls the trx_group_commit_leader() function. That is ackward, as > commit-by-rotate does not actually participate in group commit. But the > problem is with the existing function trx_group_commit_leader(), which does > too much in a single function. I noted earlier on this part on the PR page and LiBing seems to have practically proven (by the most recent commit that you must've missed) that conducting the rename inside the flush stage of trx_group_commit_leader() is viable: https://github.com/MariaDB/server/pull/3402/commits/287511a703f7dff47939633… By that measure the flushing is expanded onto possibly multiple "implicitly" rotating log files > > So please do a small refactoring of trx_group_commit_leader() and keep > only ... and no need for refactoring really arises. Could you please assess that approach, cheers, Andrei

1 0

Next LTS version in February 2025?
by Otto Kekäläinen 30 Aug '24

30 Aug '24

Hi! Traditionally there has been a new major MariaDB version in February of the year Debian has a new major version. What is the plan for February 2025? I was reading https://mariadb.org/about/#maintenance-policy and old mariadb.orb blog posts but none was explicit about this.

2 2

Re: fec3696fef6: MDEV-15751 CURRENT_TIMESTAMP should return a TIMESTAMP [WITH TIME ZONE?]
by Sergei Golubchik 29 Aug '24

29 Aug '24

Hi, Alexander, This has waited far too long :( > commit fec3696fef6 > Author: Alexander Barkov <bar(a)mariadb.com> > Date: Tue Feb 14 13:27:46 2023 +0400 > > diff --git a/sql/sql_class.h b/sql/sql_class.h > index f0bf695a1c1..7378225eaa2 100644 > --- a/sql/sql_class.h > +++ b/sql/sql_class.h > @@ -3900,7 +3900,8 @@ class THD: public THD_count, /* this must be first */ > } > const Type_handler *type_handler_for_datetime() const; > bool timestamp_to_TIME(MYSQL_TIME *ltime, my_time_t ts, > - ulong sec_part, date_mode_t fuzzydate); > + ulong sec_part, date_mode_t fuzzydate, > + bool zero_timesteamp_means_zero_datetime); I don't understand it. NOW() cannot return a zero timestamp, how this feature can possibly need changes to zero timestamp treatment? > inline my_time_t query_start() { return start_time; } > inline ulong query_start_sec_part() > { used|= QUERY_START_SEC_PART_USED; return start_time_sec_part; } > diff --git a/sql/item_timefunc.h b/sql/item_timefunc.h > index a5f6d9307c6..1c463c8a332 100644 > --- a/sql/item_timefunc.h > +++ b/sql/item_timefunc.h > @@ -707,6 +707,49 @@ class Item_datetimefunc :public Item_func > }; > > > +class Item_timestampfunc: public Item_func > +{ > +protected: > + Datetime to_datetime(THD *thd) > + { > + return Timestamp_or_zero_datetime_native_null(thd, this).to_datetime(thd); how does it differ from Datetime(this) ? > + } > +public: > + Item_timestampfunc(THD *thd): Item_func(thd) {} > + Item_timestampfunc(THD *thd, Item *a): Item_func(thd, a) {} > + const Type_handler *type_handler() const override > + { return &type_handler_timestamp2; } > + bool get_date(THD *thd, MYSQL_TIME *ltime, date_mode_t fuzzydate) override > + { > + return null_value= to_datetime(thd).copy_to_mysql_time(ltime); > + } > + double val_real() override > + { > + Datetime dt= to_datetime(current_thd); > + null_value= !dt.is_valid_datetime(); > + return dt.to_double(); may be `return null_value ? 0 : dt.to_double()` ? why to call to_double if it isn't a valid datetime? (same everywhere below) > + } > + longlong val_int() override > + { > + Datetime dt= to_datetime(current_thd); > + null_value= !dt.is_valid_datetime(); > + return dt.to_longlong(); > + } > + my_decimal *val_decimal(my_decimal *to) override > + { > + Datetime dt= to_datetime(current_thd); > + null_value= !dt.is_valid_datetime(); > + return dt.to_decimal(to); > + } > + String *val_str(String *to) override > + { > + Datetime dt= to_datetime(current_thd); > + null_value= !dt.is_valid_datetime(); > + return dt.to_string(to, decimals); > + } > +}; > + > + > /* Abstract CURTIME function. Children should define what time zone is used */ > > class Item_func_curtime :public Item_timefunc > @@ -837,20 +880,48 @@ class Item_func_now :public Item_datetimefunc > }; > > > -class Item_func_now_local :public Item_func_now > +/* > + This is a replacement for Item_func_now_local, > + returning TIMESTAMP instead of DATETIME. why did you rename it? why didn't you change Item_func_now_utc? > +*/ > +class Item_func_current_timestamp: public Item_timestampfunc > { > public: > - Item_func_now_local(THD *thd, uint dec): Item_func_now(thd, dec) {} > + Item_func_current_timestamp(THD *thd, uint dec) > + :Item_timestampfunc(thd) > + { decimals= dec; } why not Item_timestampfunc(thd, dec) ? > LEX_CSTRING func_name_cstring() const override > { > static LEX_CSTRING name= {STRING_WITH_LEN("current_timestamp") }; > return name; > } > - int save_in_field(Field *field, bool no_conversions) override; > - void store_now_in_TIME(THD *thd, MYSQL_TIME *now_time) override; > + void print(String *str, enum_query_type query_type) override > + { > + str->append(func_name_cstring()); > + str->append('('); > + if (decimals) > + str->append_ulonglong(decimals); > + str->append(')'); > + } > + bool fix_length_and_dec(THD *thd) override > + { > + if (check_decimal_scale_or_error(TIME_SECOND_PART_DIGITS)) > + return true; > + fix_attributes_datetime(decimals); > + return false; > + } > + bool val_native(THD *thd, Native *to) override; > + bool check_vcol_func_processor(void *arg) override > + { > + /* > + NOW is safe for replication as slaves will run with same time as > + master > + */ what does this replication comment do in check_vcol_func_processor? (I know it was in the old code too, but "old code did it" never stopped your cleanups before :) > + return mark_unsupported_function(func_name(), "()", arg, VCOL_TIME_FUNC); > + } > enum Functype functype() const override { return NOW_FUNC; } > Item *get_copy(THD *thd) override > - { return get_item_copy<Item_func_now_local>(thd, this); } > + { return get_item_copy<Item_func_current_timestamp>(thd, this); } > }; > > > diff --git a/sql/item_timefunc.cc b/sql/item_timefunc.cc > index 52b9ae7a682..3e224e3e6f7 100644 > --- a/sql/item_timefunc.cc > +++ b/sql/item_timefunc.cc > @@ -1642,32 +1633,10 @@ void Item_func_now::print(String *str, enum_query_type query_type) > } > > > -int Item_func_now_local::save_in_field(Field *field, bool no_conversions) > +bool Item_func_current_timestamp::val_native(THD *thd, Native *to) > { > - if (field->type() == MYSQL_TYPE_TIMESTAMP) > - { > - THD *thd= field->get_thd(); > - my_time_t ts= thd->query_start(); > - ulong sec_part= decimals ? thd->query_start_sec_part() : 0; > - sec_part-= my_time_fraction_remainder(sec_part, decimals); > - field->set_notnull(); > - field->store_timestamp(ts, sec_part); > - return 0; > - } > - else > - return Item_datetimefunc::save_in_field(field, no_conversions); > -} > - > - > -/** > - Converts current time in my_time_t to MYSQL_TIME representation for local > - time zone. Defines time zone (local) used for whole NOW function. > -*/ > -void Item_func_now_local::store_now_in_TIME(THD *thd, MYSQL_TIME *now_time) > -{ > - thd->variables.time_zone->gmt_sec_to_TIME(now_time, thd->query_start()); > - set_sec_part(thd->query_start_sec_part(), now_time, this); > - thd->used|= THD::TIME_ZONE_USED; > + Timestamp ts(Timeval(thd->query_start(), thd->query_start_sec_part())); > + return null_value= ts.trunc(decimals).to_native(to, decimals); can this possibly fail? > } > > > @@ -2732,7 +2693,6 @@ String *Item_func_tochar::val_str(String* str) > > bool Item_func_from_unixtime::fix_length_and_dec(THD *thd) > { > - thd->used|= THD::TIME_ZONE_USED; would be nice to have a test case for this. Like direct insert into a timestamp column no longer uses timezone - e.g. timestamp within dst switch time is inserted correctly. and vice versa, when from_unixtime is converted to a string, like `insert ... values(concat(from_unixtime(...)))` then TIME_ZONE_USED is still set and show binlog events (or mysqlbinlog) should show that the timezone was correctly binlogged, as before. > tz= thd->variables.time_zone; > Type_std_attributes::set( > Type_temporal_attributes_not_fixed_dec(MAX_DATETIME_WIDTH, Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 1

MariaDB patches in Debian
by Otto Kekäläinen 27 Aug '24

27 Aug '24

Hi! Could perhaps Tuukka or Daniel review the patches currently carried in Debian if some of them would be applicable to be imported or re-implemented directly upstream so they can be dropped in Debian? https://udd.debian.org/patches.cgi?src=mariadb&version=1%3A11.4.3-1 Thanks

1 0

Re: [MariaDB commits] [PATCH 1/6] MDEV-34696: do_gco_wait() completes too early on InnoDB dict stats updates
by andrei.elkin＠pp.inet.fi 23 Aug '24

23 Aug '24

Howdy Kristian. > Before doing mark_start_commit(), check that there is no pending deadlock > kill. If there is a pending kill, we won't commit (we will abort, roll back, > and retry). Then we should not mark the commit as started, since that could > potentially make the following GCO start too early, before we completed the > commit after the retry. > > This condition could trigger in some corner cases, where InnoDB would take > temporarily table/row locks that are released again immediately, not held > until the transaction commits. This happens with dict_stats updates and > possibly auto-increment locks. > > Such locks can be passed to thd_rpl_deadlock_check() and cause a deadlock > kill to be scheduled in the background. But since the blocking locks are > held only temporarily, they can be released before the background kill > happens. This way, the kill can be delayed until after mark_start_commit() > has been called. Thus we need to check the synchronous indication > rgi->killed_for_retry, not just the asynchroneous thd->killed. I think I understood what's going on, also thanks to a verbose Jira ticket description. > > Signed-off-by: Kristian Nielsen <knielsen(a)knielsen-hq.org> > --- > sql/rpl_parallel.cc | 20 ++++++++++++++++---- > sql/rpl_rli.cc | 17 +++++++++++++++++ > 2 files changed, 33 insertions(+), 4 deletions(-) > > diff --git a/sql/rpl_parallel.cc b/sql/rpl_parallel.cc > index 1cfdf96ee3b..9c4222d7817 100644 > --- a/sql/rpl_parallel.cc > +++ b/sql/rpl_parallel.cc > @@ -1450,11 +1450,23 @@ handle_rpl_parallel_thread(void *arg) > after mark_start_commit(), we have to unmark, which has at least a > theoretical possibility of leaving a window where it looks like all > transactions in a GCO have started committing, while in fact one > - will need to rollback and retry. This is not supposed to be possible > - (since there is a deadlock, at least one transaction should be > - blocked from reaching commit), but this seems a fragile ensurance, > - and there were historically a number of subtle bugs in this area. > + will need to rollback and retry. > + > + Normally this will not happen, since the kill is there to resolve a > + deadlock that is preventing at least one transaction from proceeding. > + One case it can happen is with InnoDB dict stats update, which can > + temporarily cause transactions to block each other, but locks are > + released immediately, they don't linger until commit. There could be > + other similar cases, there were historically a number of subtle bugs > + in this area. > + > + But once we start the commit, we can expect that no new lock > + conflicts will be introduced. So by handling any lingering deadlock > + kill at this point just before mark_start_commit(), we should be > + robust even towards spurious deadlock kills. > */ > + if (rgi->killed_for_retry != rpl_group_info::RETRY_KILL_NONE) > + wait_for_pending_deadlock_kill(thd, rgi); Assuming my understanding, please correct me if anything, I left a question on this block in https://github.com/MariaDB/server/commit/df0c36a354ffde2766ebed2615642c74b7… quoted further. ... this guard would not let a victim of an optimistic conflict proceed until it clears out itself it is such a victim. For instance in the binlog sequence like T_1, T_2, D_3 where T_2 is the victim, D_3 is a DDL, the victim T_2 won't anymore release the next gco member D_3 into its execution. But what if at this point T_2 is only going to be marked in rgi->killed_for_retry by T_1? That is it is apparently able, with this patch as well, having ...NONE killed status go through. T_1 would reach this very point later and when that is done before T_2 finds itself killed, T_1 would release D_3, and afterward T_2 would finally see itself KILLed into retry. Cheers, Andrei

2 4

Open PRs pending second approver (has already one)
by Otto Kekäläinen 23 Aug '24

23 Aug '24

Hi! There are couple PRs that have 1 approval, and are pending the second one. Would somebody have time to review them? In order of importance: https://github.com/MariaDB/server/pull/3401 "MDEV-34680 Asynchronous and Buffered Logging in MySQL Audit Plugin" https://github.com/MariaDB/server/pull/3406 (related but zero approvals: https://github.com/MariaDB/server/pull/3324 "MDEV-12182 Add client tcp port to audit plugin logs") https://github.com/MariaDB/server/pull/3208 "Refine OpenSSL test names and allow more complete OpenSSL version coverage" https://github.com/MariaDB/server/pull/3225 "MDEV-29955: add ${PCRE_LIBRARY_PATH} to link directories" Thanks, Otto

1 0

Re: 186aa192d1f: MDEV-34704 Quick mode produces the bug for mariadb client
by Sergei Golubchik 22 Aug '24

22 Aug '24

Hi, Sanja, ok to push, but see one comment below On Aug 22, Oleksandr Byelkin wrote: > revision-id: 186aa192d1f (mariadb-11.5.1-34-g186aa192d1f) > parent(s): 9e1923ca238 > author: Oleksandr Byelkin > committer: Oleksandr Byelkin > timestamp: 2024-08-05 16:16:09 +0200 > message: > > MDEV-34704 Quick mode produces the bug for mariadb client > > --quick-max-column-width parameter added to limit field > width in --quick mode. > > diff --git a/mysql-test/main/client.test b/mysql-test/main/client.test > new file mode 100644 > index 00000000000..59e687ff8b9 > --- /dev/null > +++ b/mysql-test/main/client.test > @@ -0,0 +1,33 @@ > +--echo # > +--echo # MDEV-34704: Quick mode produces the bug for mariadb client > +--echo # > + > + > +create table t1 (aaaaaaaaa char (5), aaaaa char (10), a char (127), b char(1)); > +insert into t1 values ("X", "X", "X", "X"); > + to make the result file more readable, I'd suggest to add here, like --echo mysql --table --quick ... --echo mysql --table --quick --quick-max-column-width=0 ... etc before every corresponding --exec > +--exec echo "select * from test.t1;" | $MYSQL --table --quick 2>&1 > + > +--exec echo "select * from test.t1;" | $MYSQL --table --quick --quick-max-column-width=0 2>&1 > + > +--exec echo "select * from test.t1;" | $MYSQL --table --quick --quick-max-column-width=10 2>&1 > + > +--exec echo "select * from test.t1;" | $MYSQL --table --quick --quick-max-column-width=20 2>&1 > + > +insert into t1 values ("01234", "0123456789", "01234567890123456789", "1"); > + > +--exec echo "select * from test.t1;" | $MYSQL --table --quick 2>&1 > + > + > +--exec echo "select * from test.t1;" | $MYSQL --table --quick --quick-max-column-width=0 2>&1 > + > +--exec echo "select * from test.t1;" | $MYSQL --table --quick --quick-max-column-width=10 2>&1 > + > +--exec echo "select * from test.t1;" | $MYSQL --table --quick --quick-max-column-width=20 2>&1 > + > +drop table t1; > + > +--echo # > +--echo # End of 10.7 tests > +--echo # As Marko said, use here only one line --echo # End of 10.7 tests otherwise merge mixes it up with --echo # --echo # MDEV-34704: Quick mode produces the bug for mariadb client --echo # Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

1 0

Suggestion: use 'main' as the development branch instead of an ever changing 11.x
by Otto Kekäläinen 22 Aug '24

22 Aug '24

Hi! Has the core developers had discussions about using branch name 'main' for development instead of switching to a new branch every 3 months? If all new features and other additions would always target the branch 'main' it would lower the barrier of entry for new contributors, and decrease the number of unnecessary rebases done by contributors. If the permanent development target was 'main', you would need to create new 11.x branches only at "freeze" when a new major version branch is cut for release. MariaDB is currently the only open source project I know that does *not* have a main/master/trunk branch and instead does these constant new branch names. I imagine this is not only annoying for contributors, but also the core developers who waste time asking contributors to rebase PRs every 3 months (see https://github.com/MariaDB/server/pull/2556 and https://github.com/MariaDB/server/pull/2671 as examples). - Otto

7 11

Re: [MariaDB commits] [PATCH 3/6] Fix sporadic test failure in rpl.rpl_create_drop_event
by Kristian Nielsen 22 Aug '24

22 Aug '24

Andrei Elkin <andrei.elkin(a)mariadb.com> writes: > Kristian Nielsen via developers <developers(a)lists.mariadb.org> writes: >> The already-started instance of the event happily continues to run after >> being dropped. I wonder if that's a bug, or if the documentation is >> wrong? > > To this question, I vaguely remember discussions about this matter which > must've concuded in that it's not a bug when the scheduler races ahead > of DROP EVENT to launch a piece of job, like INSERT in this case. Aha, I see. Following up on this, I found this comment in the code Stops the scheduler (again). Waits for acknowledgement from the scheduler that it has stopped - synchronous stopping. Already running events will not be stopped. If the user needs them stopped manual intervention is needed. So then the behaviour seen in the test case is expected, and just fixing the test to handle the possible race should be correct. I also see in the code that the thread running the events starts by loading the event definition from the system tables (mysql.event ...), db_repository->load_named_event(). So there should be no memory issues even if the event is dropped while an instance of it is running. Thanks! - Kristian.

1 0

Re: [MariaDB commits] [PATCH 3/6] Fix sporadic test failure in rpl.rpl_create_drop_event
by Kristian Nielsen 21 Aug '24

21 Aug '24

Andrei Elkin <andrei.elkin(a)mariadb.com> writes: >> My question is if it is ok and by design that the event can continue running >> after the scheduler is stopped and even after the event is dropped? That > Should not be ok. The docs https://mariadb.com/kb/en/drop-event/ > > are explicit > > Description > This statement drops the event named event_name. The event immediately ceases being active, and is deleted completely from the server. Thanks for the link, Andrei! Interesting, this doesn't seem to be how it works. This small test starts an event that runs for 3 seconds, but drops the event after 1 second: ----------------------------------------------------------------------- SET GLOBAL event_scheduler= OFF; CREATE TABLE t1(a INT); CREATE EVENT ev1 ON SCHEDULE EVERY 5 SECOND DO INSERT INTO t1 VALUES(SLEEP(3)); SET GLOBAL event_scheduler= ON; --sleep 1 SHOW FULL PROCESSLIST; SET GLOBAL event_scheduler= OFF; DROP EVENT ev1; SHOW FULL PROCESSLIST; --sleep 1 SHOW FULL PROCESSLIST; DROP TABLE t1; ----------------------------------------------------------------------- The already-started instance of the event happily continues to run after being dropped. I wonder if that's a bug, or if the documentation is wrong? And if it is a bug, if there are any use-after-free or similar memory issues with the event continuing to run after being dropped? Valgrind doesn't complain... ----------------------------------------------------------------------- SET GLOBAL event_scheduler= OFF; CREATE TABLE t1(a INT); CREATE EVENT ev1 ON SCHEDULE EVERY 5 SECOND DO INSERT INTO t1 VALUES(SLEEP(3)); Warnings: Warning 1105 Event scheduler is switched off, use SET GLOBAL event_scheduler=ON to enable it. SET GLOBAL event_scheduler= ON; SHOW FULL PROCESSLIST; Id User Host db Command Time State Info Progress 4 root localhost test Query 0 starting SHOW FULL PROCESSLIST 0.000 5 event_scheduler localhost NULL Daemon 1 Waiting for next activation NULL 0.000 6 root localhost test Connect 1 User sleep INSERT INTO t1 VALUES(SLEEP(3)) 0.000 SET GLOBAL event_scheduler= OFF; DROP EVENT ev1; SHOW FULL PROCESSLIST; Id User Host db Command Time State Info Progress 4 root localhost test Query 0 starting SHOW FULL PROCESSLIST 0.000 6 root localhost test Connect 1 User sleep INSERT INTO t1 VALUES(SLEEP(3)) 0.000 SHOW FULL PROCESSLIST; Id User Host db Command Time State Info Progress 4 root localhost test Query 0 starting SHOW FULL PROCESSLIST 0.000 6 root localhost test Connect 2 User sleep INSERT INTO t1 VALUES(SLEEP(3)) 0.000 DROP TABLE t1; ----------------------------------------------------------------------- - Kristian.

1 0

Re: [PATCH 1/6] MDEV-34696: do_gco_wait() completes too early on InnoDB dict stats updates
by Kristian Nielsen 20 Aug '24

20 Aug '24

Andrei Elkin <andrei.elkin(a)mariadb.com> writes: > Howdy Kristian. > I was going to review. Thanks Andrei! > Is the patch is any github branch? Looks so by the MDEV. It'd be good to > have it there for reviewing either. Yes, the entire patch series is in bb-10.5-knielsen: https://github.com/MariaDB/server/commits/bb-10.5-knielsen This particular issue is the commit: MDEV-34696: do_gco_wait() completes too early on InnoDB dict stats updates - Kristian.

1 0

Re: [MariaDB commits] [PATCH 3/6] Fix sporadic test failure in rpl.rpl_create_drop_event
by Kristian Nielsen 20 Aug '24

20 Aug '24

Hi Bar, I want to ask you about this fix of a sporadic failure of the test rpl.rpl_create_drop_event, which is authored by you. Not so much the test case fix, which should be good, but I want to ask someone who knows the event scheduler code better than me to be sure I am not merely hiding a real bug with the test case fix. What happens in the test case when it failed is that a new run of an event is started, and this run can continue while the event scheduler is stopped and even after the event itself is dropped: CREATE OR REPLACE EVENT ev1 ON SCHEDULE EVERY 1 SECOND DO INSERT INTO t1 VALUES (11); ... SET GLOBAL event_scheduler=off; DROP EVENT ev1; My question is if it is ok and by design that the event can continue running after the scheduler is stopped and even after the event is dropped? That this doesn't cause any problems like accessing freed memory or such? Or if you do not know, if you know who else I could ask? Thanks, - Kristian. Kristian Nielsen via commits <commits(a)lists.mariadb.org> writes: > Depending on timing, an extra event run could start just when the event > scheduler is shut down and delay running until after the table has been > dropped; this would cause the test to fail with a "table does not exist" > error in the log. > > Signed-off-by: Kristian Nielsen <knielsen(a)knielsen-hq.org> > --- > mysql-test/suite/rpl/t/rpl_create_drop_event.test | 6 ++++++ > 1 file changed, 6 insertions(+) > > diff --git a/mysql-test/suite/rpl/t/rpl_create_drop_event.test b/mysql-test/suite/rpl/t/rpl_create_drop_event.test > index 96a7e82d6f7..79bb0ffec90 100644 > --- a/mysql-test/suite/rpl/t/rpl_create_drop_event.test > +++ b/mysql-test/suite/rpl/t/rpl_create_drop_event.test > @@ -14,6 +14,12 @@ SET GLOBAL event_scheduler=on; > let $wait_condition= SELECT count(*)>0 FROM t1; > --source include/wait_condition.inc > SET GLOBAL event_scheduler=off; > +# If the time rolls to the next whole second just at this point, a new event > +# run may be scheduled. Wait for this to disappear, otherwise we see occasional > +# test failures if the table gets dropped before the extra event run completes. > +# Expect 5 connections: default, master, master1, server_1, binlog dump thread > +--let $wait_condition= SELECT COUNT(*) = 5 FROM INFORMATION_SCHEMA.PROCESSLIST; > +--source include/wait_condition.inc > SELECT DISTINCT a FROM t1; > DELETE FROM t1;

1 0

Re: [PATCH 1/6] MDEV-34696: do_gco_wait() completes too early on InnoDB dict stats updates
by Kristian Nielsen 20 Aug '24

20 Aug '24

Hi Brandon, Andrei, Does one of you want to review this patch for 10.5? It fixes a real (and serious) issue in parallel replication that is seen as sporadic failure of rpl.rpl_start_alter_4 in Buildbot. This is yet another occurrence of the problem where mark_start_commit() is done too early, this code unfortunately is fragile and has had a number of problems in the past. But I am quite confident about the patch, calling wait_for_pending_deadlock_kill() should be very safe, and should be correct here. The added assertion helps detect the problem more reliably and could also help catch any other remaining issues around mark_start_commit (though hopefully there are no more left now). - Kristian. Kristian Nielsen via commits <commits(a)lists.mariadb.org> writes: > Before doing mark_start_commit(), check that there is no pending deadlock > kill. If there is a pending kill, we won't commit (we will abort, roll back, > and retry). Then we should not mark the commit as started, since that could > potentially make the following GCO start too early, before we completed the > commit after the retry. > > This condition could trigger in some corner cases, where InnoDB would take > temporarily table/row locks that are released again immediately, not held > until the transaction commits. This happens with dict_stats updates and > possibly auto-increment locks. > > Such locks can be passed to thd_rpl_deadlock_check() and cause a deadlock > kill to be scheduled in the background. But since the blocking locks are > held only temporarily, they can be released before the background kill > happens. This way, the kill can be delayed until after mark_start_commit() > has been called. Thus we need to check the synchronous indication > rgi->killed_for_retry, not just the asynchroneous thd->killed. > > Signed-off-by: Kristian Nielsen <knielsen(a)knielsen-hq.org> > --- > sql/rpl_parallel.cc | 20 ++++++++++++++++---- > sql/rpl_rli.cc | 17 +++++++++++++++++ > 2 files changed, 33 insertions(+), 4 deletions(-) > > diff --git a/sql/rpl_parallel.cc b/sql/rpl_parallel.cc > index 1cfdf96ee3b..9c4222d7817 100644 > --- a/sql/rpl_parallel.cc > +++ b/sql/rpl_parallel.cc > @@ -1450,11 +1450,23 @@ handle_rpl_parallel_thread(void *arg) > after mark_start_commit(), we have to unmark, which has at least a > theoretical possibility of leaving a window where it looks like all > transactions in a GCO have started committing, while in fact one > - will need to rollback and retry. This is not supposed to be possible > - (since there is a deadlock, at least one transaction should be > - blocked from reaching commit), but this seems a fragile ensurance, > - and there were historically a number of subtle bugs in this area. > + will need to rollback and retry. > + > + Normally this will not happen, since the kill is there to resolve a > + deadlock that is preventing at least one transaction from proceeding. > + One case it can happen is with InnoDB dict stats update, which can > + temporarily cause transactions to block each other, but locks are > + released immediately, they don't linger until commit. There could be > + other similar cases, there were historically a number of subtle bugs > + in this area. > + > + But once we start the commit, we can expect that no new lock > + conflicts will be introduced. So by handling any lingering deadlock > + kill at this point just before mark_start_commit(), we should be > + robust even towards spurious deadlock kills. > */ > + if (rgi->killed_for_retry != rpl_group_info::RETRY_KILL_NONE) > + wait_for_pending_deadlock_kill(thd, rgi); > if (!thd->killed) > { > DEBUG_SYNC(thd, "rpl_parallel_before_mark_start_commit"); > diff --git a/sql/rpl_rli.cc b/sql/rpl_rli.cc > index 9d4e09a362c..8284b07f7ff 100644 > --- a/sql/rpl_rli.cc > +++ b/sql/rpl_rli.cc > @@ -2518,6 +2518,23 @@ rpl_group_info::unmark_start_commit() > > e= this->parallel_entry; > mysql_mutex_lock(&e->LOCK_parallel_entry); > + /* > + Assert that we have not already wrongly completed this GCO and signalled > + the next one to start, only to now unmark and make the signal invalid. > + This is to catch problems like MDEV-34696. > + > + The error inject rpl_parallel_simulate_temp_err_xid is used to test this > + precise situation, that we handle it gracefully if it somehow occurs in a > + release build. So disable the assert in this case. > + */ > +#ifndef DBUG_OFF > + bool allow_unmark_after_complete= false; > + DBUG_EXECUTE_IF("rpl_parallel_simulate_temp_err_xid", > + allow_unmark_after_complete= true;); > + DBUG_ASSERT(!gco->next_gco || > + gco->next_gco->wait_count > e->count_committing_event_groups || > + allow_unmark_after_complete); > +#endif > --e->count_committing_event_groups; > mysql_mutex_unlock(&e->LOCK_parallel_entry); > }

1 0

Re: [PATCH] MDEV-34481 optimize away waiting for owned by prepared xa non-unique index record
by Kristian Nielsen 15 Aug '24

15 Aug '24

Andrei Elkin <andrei.elkin(a)mariadb.com> writes: >> This is not the way to communicate back from engine to server. Instead, >> simply return a different error code than HA_ERR_LOCK_WAIT_TIMEOUT from >> InnoDB that you can check for here. You can maybe use one of the existing >> HA_ERR_* from include/my_base.h, or add a new one if necessary. > However, adding a new error is clearly more intrusive. The new error is completely handled inside the code you add for signalling the condition inside InnoDB locking code and handling it in replication row event apply. It will not be visible to users. So it is not intrusive to add a new one. Just be careful to not pick a number that conflicts with another entry in a later branch, if you do this in a GA release. Alternatively, you can use the existing HA_ERR_ROW_NOT_VISIBLE error, which is used in a somewhat similar way in storage/maria/ma_blockrec.c and nowhere else. But I would just add a new error for clarity. > And at the same time I thought a some sort of runtime state the slave > applier had always missed. So the intent of this struct is future-minded > as well. This is not the point. Even if the rgi->exec_flags already existed, it would still be wrong to use it for this. Here is the code where you are handling the condition: > + if (error == HA_ERR_LOCK_WAIT_TIMEOUT && !is_index_unique && > + (rgi->exec_flags & (1 << rpl_group_info::HIT_BUSY_INDEX))) > + { > + rgi->exec_flags &= ~(1 << rpl_group_info::HIT_BUSY_INDEX); > + skip_first_compare= true; So you _already_ are using a specific error code to signal the condition from InnoDB to replication, the error HA_ERR_LOCK_WAIT_TIMEOUT. But this error can also be thrown in other situations, so you need to add these extra conditions to distinguish. So essentially, you are using the wrong error code to signal the condition. Use a correct error code that uniquely identifies the condition that occured, and then you can avoid these extra conditions and simplify the code. > I would be fine with either method. Good, then please change it to return a proper error code. The only reason that you are using HA_ERR_LOCK_WAIT_TIMEOUT is that then you could do the quick change inside InnoDB to set its timeout value to 0, and this happens to then propagate the HA_ERR_LOCK_WAIT_TIMEOUT up to the callers without waiting. But that's not an appropriate way to design new code and APIs. Yes, it is frustrating when conceptually simple changes need extra effort, due to the need to work in an existing old and complex code base like MariaDB. But that is just the way it is, it is the same for all of us. - Kristian.

1 0

Can we get MariaDB GitHub CI to consistently be green?
by Otto Kekäläinen 10 Aug '24

10 Aug '24

Hi! Thanks to everyone who has worked on polishing the CI tests and integrations. Looking at e.g. 10.11 branch[1] I see that commit ccb7a1e[2] has a green checkmark next to it and all 15 CI jobs passed. I just wanted to check if everyone developing the MariaDB Server is committed to getting the CI to be consistently green? This means that GitHub rules need to be a bit more strict, not allowing any failing tests jobs at all, and developers and managers need to agree to stop adding new commits on any release branch if it can't be done without a fully passing CI run. Thanks to Daniel's review on latest failures we know these are at the moment recurring: * MDEV-25614 Galera test failure on GCF-354 included in MDEV-33073 always green buildbot * MDEV-33785 aarch64 macos encryption.create_or_replace * MDEV-33601 galera_3nodes.galera_safe_to_bootstrap test failing * MDEV-33786 galera_3nodes.galera_vote_rejoin_mysqldump mysql_shutdown failed at line 85: I see two approaches to get to consistently green CI: 1) Stop all development and focus on just fixing these, don't continue until CI is fully green, and once it is fully green make the GitHub branch protection settings one notch stricter to not allow any new commits unless the CI is fully green so it never regresses again. 2) Disable these tests and make the rules in GitHub branch protection one notch stricter right away, and not allow any new commits unless the CI is fully green ensuring no new recurring failures are introduced. - Otto [1] https://github.com/MariaDB/server/commits/10.11 [2] https://github.com/MariaDB/server/commit/ccb7a1e9a15e6a47aba97f9bdbfab2e4bf…

7 12

Re: [PATCH] MDEV-34481 optimize away waiting for owned by prepared xa non-unique index record
by Kristian Nielsen 01 Aug '24

01 Aug '24

Andrei Elkin <andrei.elkin(a)mariadb.com> writes: > Indeed a combination of the engine locking protocol and the slave retry > applies correctly to IDEMPOTENT mode. >> The optimistic parallel replication should work correctly with >> IDEMPOTENT, > Indeed. Ok good to know. It's definitely a tricky issue, but apparently no known bugs, at least to the two of us. As you saw, I did have to try a test case to feel confident it works ;-) - Kristian.

1 0

Re: [PATCH] MDEV-34481 optimize away waiting for owned by prepared xa non-unique index record
by Kristian Nielsen 31 Jul '24

31 Jul '24

Andrei Elkin <andrei.elkin(a)mariadb.com> writes: > Kristian Nielsen <knielsen(a)knielsen-hq.org> writes: >> You need to make sure that --slave_exec_mode=IDEMPOTENT is not used. >> Otherwise you can get silent data corruption. > > IDEMPOTENT is always risky. Here it would add up another bit, in the > parallel mode. What do you mean "IDEMPOTENT is always risky"? As far as I know, if the slave would replicate without error in STRICT, it will replicate exactly the same way in IDEMPOTENT. Only if there is an error in STRICT mode will the IDEMPOTENT mode do something different (ignore the error in some cases). So the risk is that we might fail to detect that the slave is diverged from the master, right? Note that diverged slave can be undetected even in STRICT mode in many cases. > Btw should IDEMPOTENT be disallowed in the parallel mode actually? No, why? IDEMPOTENT should work just as documented in all parallel replication modes. Or do you know of any bug with IDEMPOTENT and optimistic parallel replication that I don't know of? If so do you have a test case? The optimistic parallel replication should work correctly with IDEMPOTENT, since the second query that's ignoring the error should still be taking InnoDB row locks, which will conflict as normal with the first transaction and cause the second to be retried, this time without (ignored) error. I just tried with the below testcase, it seems to work. Only when you fiddle with the InnoDB locking like in your patch and ignore certain locks, do you get problems, and then you can end up with a later transaction completely ignoring a lock that is temporarily held by an earlier XA PREPARE (even though the corresponding XA COMMIT is also earlier). And this will result in incorrect replication and diverged slave, I think (I did not test). This is not "risky", this is incorrect behaviour and a serious bug. So it seems you will need to maybe give an error when replicating any XA transaction if slave_exec_mode=IDEMPOTENT? Or handle it some other way to avoid silently replicating incorrectly. As you know, my opinion is that this is just yet another symptom of the real issue of MDEV-32020, but I know you're going to ignore that. So I've given up caring about what happens to XA replication, as long as it doesn't negatively affect non-XA stuff. > I just don't see any its usefullness except creating a chaotic state. Why do you think it creates a "chaotic state"? - Kristian. ----------------------------------------------------------------------- --source include/have_innodb.inc --source include/have_binlog_format_row.inc --source include/master-slave.inc --connection master CALL mtr.add_suppression("Can't find record in 't1'"); ALTER TABLE mysql.gtid_slave_pos ENGINE=InnoDB; CREATE TABLE t1 (a INT PRIMARY KEY, b INT) ENGINE=InnoDB; CREATE TABLE t2 (a INT PRIMARY KEY) ENGINE=InnoDB; BEGIN; INSERT INTO t1 VALUES (1, 1); INSERT INTO t1 VALUES (2, 1); INSERT INTO t1 VALUES (3, 1); INSERT INTO t1 VALUES (4, 1); INSERT INTO t1 VALUES (5, 1); COMMIT; --sync_slave_with_master --source include/stop_slave.inc SET @old_exec_mode= @@GLOBAL.slave_exec_mode; SET GLOBAL slave_exec_mode=IDEMPOTENT; SET @old_parallel_mode= @@GLOBAL.slave_parallel_mode; SET GLOBAL slave_parallel_mode= optimistic; SET @old_threads= @@GLOBAL.slave_parallel_threads; SET GLOBAL slave_parallel_threads= 8; --source include/start_slave.inc # Block certain queries on the slave temporarily. BEGIN; INSERT INTO t2 VALUES (1), (2), (3); --connection master UPDATE t1 SET b=10 WHERE a=1; BEGIN; INSERT INTO t2 VALUES (1); # A DELETE to make room for a later INSERT, will be delayed a bit on slave. DELETE FROM t1 WHERE a=2; COMMIT; UPDATE t1 SET b=20 WHERE a=3; BEGIN; INSERT INTO t2 VALUES (2); # An INSERT to be deleted again shortly, will be delayed a bit on slave. INSERT INTO t1 VALUES (6,2); COMMIT; UPDATE t1 SET b=30 WHERE a=4; # This INSERT cannot run until after the earlier DELETE. INSERT INTO t1 VALUES (2, 1); # This DELETE cannot run until after the earlier INSERT. DELETE FROM t1 WHERE a=6; SELECT * FROM t1 ORDER BY a; --save_master_pos --connection slave --sleep 1 ROLLBACK; --sync_with_master SELECT * FROM t1 ORDER BY a; --source include/stop_slave.inc SET GLOBAL slave_exec_mode= @old_exec_mode; SET GLOBAL slave_parallel_mode= @old_parallel_mode; SET GLOBAL slave_parallel_threads= @old_threads; --source include/start_slave.inc --connection master DROP TABLE t1, t2; --source include/rpl_end.inc

1 0

Re: 1287c901: TLS/SSL changes (major rework)
by Sergei Golubchik 31 Jul '24

31 Jul '24

Hi, Georg, On Jul 31, Georg Richter wrote: > revision-id: 1287c901 (v3.4.0-5-g1287c901) > parent(s): 5386f1a3 > author: Georg Richter > committer: Georg Richter > timestamp: 2024-07-16 13:12:26 +0200 > message: > > TLS/SSL changes (major rework) > diff --git a/include/ma_common.h b/include/ma_common.h > index dfa96621..dc900b0d 100644 > --- a/include/ma_common.h > +++ b/include/ma_common.h > @@ -131,15 +131,9 @@ typedef struct st_mariadb_field_extension > MARIADB_CONST_STRING metadata[MARIADB_FIELD_ATTR_LAST+1]; /* 10.5 */ > } MA_FIELD_EXTENSION; > > -#if defined(HAVE_SCHANNEL) || defined(HAVE_GNUTLS) > -#define reset_tls_self_signed_error(mysql) \ > - do { \ > - free((char*)mysql->net.tls_self_signed_error); \ > - mysql->net.tls_self_signed_error= 0; \ > - } while(0) > -#else > -#define reset_tls_self_signed_error(mysql) \ > - do { \ > - mysql->net.tls_self_signed_error= 0; \ > +#ifdef HAVE_TLS > +#define reset_tls_error(mysql) \ > + do { \ > + mysql->net.tls_verify_status= 0; \ you've lost the original error message when you've changed char* pointer to the message to my_bool flag. > } while(0) > #endif > diff --git a/include/ma_tls.h b/include/ma_tls.h > index 23f53560..444ea4aa 100644 > --- a/include/ma_tls.h > +++ b/include/ma_tls.h > @@ -134,6 +141,7 @@ const char *ma_tls_get_cipher(MARIADB_TLS *ssl); > hash_type hash_type as defined in ma_hash.h > fp buffer for fingerprint > fp_len buffer length > + my_bool verify_period I don't see any 'my bool verify_period' in the list of arguments > > Returns: > actual size of finger print > @@ -150,7 +158,7 @@ unsigned int ma_tls_get_finger_print(MARIADB_TLS *ctls, uint hash_type, char *fp > int ma_tls_get_protocol_version(MARIADB_TLS *ctls); > const char *ma_pvio_tls_get_protocol_version(MARIADB_TLS *ctls); > int ma_pvio_tls_get_protocol_version_id(MARIADB_TLS *ctls); > -unsigned int ma_tls_get_peer_cert_info(MARIADB_TLS *ctls); > +unsigned int ma_tls_get_peer_cert_info(MARIADB_TLS *ctls, unsigned int size); is this part of the public API? > void ma_tls_set_connection(MYSQL *mysql); > > /* Function prototypes */ > diff --git a/include/mysql.h b/include/mysql.h > index 6c84a462..ba92527d 100644 > --- a/include/mysql.h > +++ b/include/mysql.h > @@ -448,6 +449,16 @@ typedef struct st_mysql_time > #define MYSQL_WAIT_EXCEPT 4 > #define MYSQL_WAIT_TIMEOUT 8 > > +#define MARIADB_TLS_VERIFY_OK 0 > +#define MARIADB_TLS_VERIFY_TRUST 1 > +#define MARIADB_TLS_VERIFY_HOST 2 we've agreed to swap two previous lines > +#define MARIADB_TLS_VERIFY_PERIOD 4 > +#define MARIADB_TLS_VERIFY_FINGERPRINT 8 is that right? you want to allow fingerprints even if the cert is expired? > +#define MARIADB_TLS_VERIFY_REVOKED 16 > +#define MARIADB_TLS_VERIFY_UNKNOWN 32 > +#define MARIADB_TLS_VERIFY_ERROR 128 /* last */ > + > + > typedef struct character_set > { > unsigned int number; /* character set number */ > diff --git a/libmariadb/ma_tls.c b/libmariadb/ma_tls.c > index 1bda7dcf..f6ea6661 100644 > --- a/libmariadb/ma_tls.c > +++ b/libmariadb/ma_tls.c > @@ -103,9 +103,54 @@ my_bool ma_pvio_tls_close(MARIADB_TLS *ctls) > return ma_tls_close(ctls); > } > > -int ma_pvio_tls_verify_server_cert(MARIADB_TLS *ctls) > +int ma_pvio_tls_verify_server_cert(MARIADB_TLS *ctls, unsigned int flags) > { > - return ma_tls_verify_server_cert(ctls); > + MYSQL *mysql; > + int rc; > + > + if (!ctls || !ctls->pvio || !ctls->pvio->mysql) > + return 0; > + > + mysql= ctls->pvio->mysql; > + > + /* Skip peer certificate verification */ > + if (ctls->pvio->mysql->options.extension->tls_allow_invalid_server_cert) > + { > + return 0; > + } > + > + rc= ma_tls_verify_server_cert(ctls, flags); > + > + /* Set error messages */ > + if (!mysql->net.last_errno) > + { > + if (mysql->net.tls_verify_status & MARIADB_TLS_VERIFY_PERIOD) > + my_set_error(mysql, CR_SSL_CONNECTION_ERROR, SQLSTATE_UNKNOWN, > + ER(CR_SSL_CONNECTION_ERROR), > + "Certificate not yet valid or expired"); > + else if (mysql->net.tls_verify_status & MARIADB_TLS_VERIFY_FINGERPRINT) > + my_set_error(mysql, CR_SSL_CONNECTION_ERROR, SQLSTATE_UNKNOWN, > + ER(CR_SSL_CONNECTION_ERROR), > + "Fingerprint validation of peer certificate failed"); > + else if (mysql->net.tls_verify_status & MARIADB_TLS_VERIFY_REVOKED) > + my_set_error(mysql, CR_SSL_CONNECTION_ERROR, SQLSTATE_UNKNOWN, > + ER(CR_SSL_CONNECTION_ERROR), > + "Certificate revoked"); > + else if (mysql->net.tls_verify_status & MARIADB_TLS_VERIFY_HOST) > + my_set_error(mysql, CR_SSL_CONNECTION_ERROR, SQLSTATE_UNKNOWN, > + ER(CR_SSL_CONNECTION_ERROR), > + "Hostname verification failed"); > + else if (mysql->net.tls_verify_status & MARIADB_TLS_VERIFY_UNKNOWN) > + my_set_error(mysql, CR_SSL_CONNECTION_ERROR, SQLSTATE_UNKNOWN, > + ER(CR_SSL_CONNECTION_ERROR), > + "Peer certificate verification failed"); > + else if (mysql->net.tls_verify_status & MARIADB_TLS_VERIFY_TRUST) > + my_set_error(mysql, CR_SSL_CONNECTION_ERROR, SQLSTATE_UNKNOWN, > + ER(CR_SSL_CONNECTION_ERROR), > + "Peer certificate is not trusted"); let's try to preserve the error as set by the ssl library here > + } > + > + return rc; > } > > const char *ma_pvio_tls_cipher(MARIADB_TLS *ctls) > diff --git a/plugins/auth/my_auth.c b/plugins/auth/my_auth.c > index faea968c..e7429fae 100644 > --- a/plugins/auth/my_auth.c > +++ b/plugins/auth/my_auth.c > @@ -382,14 +419,30 @@ static int send_client_reply_packet(MCPVIO_EXT *mpvio, > errno); > goto error; > } > + mysql->net.tls_verify_status = 0; > if (ma_pvio_start_ssl(mysql->net.pvio)) > goto error; > - if (mysql->net.tls_self_signed_error && > - (!mysql->passwd || !mysql->passwd[0] || !hashing(mpvio->plugin))) > - { > - /* cannot use auth to validate the cert */ > - set_error_from_tls_self_signed_error(mysql); > - goto error; > + > + verify_flags= MARIADB_TLS_VERIFY_PERIOD | MARIADB_TLS_VERIFY_REVOKED; > + if (have_fingerprint(mysql)) > + { > + verify_flags|= MARIADB_TLS_VERIFY_FINGERPRINT; > + } else { > + verify_flags|= MARIADB_TLS_VERIFY_TRUST | MARIADB_TLS_VERIFY_HOST; > + } > + > + if (ma_pvio_tls_verify_server_cert(mysql->net.pvio->ctls, verify_flags) > MARIADB_TLS_VERIFY_OK) > + { > + if (mysql->net.tls_verify_status > MARIADB_TLS_VERIFY_TRUST || > + (mysql->options.ssl_ca || mysql->options.ssl_capath)) > + goto error; > + > + if (is_local_connection(mysql->net.pvio)) I'd say this should be earlier. Like if (!is_local_connection(mysql->net.pvio)) if (have_fingerprint(mysql)) { verify_flags|= MARIADB_TLS_VERIFY_FINGERPRINT; } else { verify_flags|= MARIADB_TLS_VERIFY_TRUST | MARIADB_TLS_VERIFY_HOST; } > + { > + CLEAR_CLIENT_ERROR(mysql); > + } > + else if (!password_and_hashing(mysql, mpvio->plugin)) > + goto error; > } > } > #endif /* HAVE_TLS */ > @@ -769,8 +822,14 @@ retry: > auth_plugin= &dummy_fallback_client_plugin; > > /* can we use this plugin with this tls server cert ? */ > - if (mysql->net.tls_self_signed_error && !hashing(auth_plugin)) > - return set_error_from_tls_self_signed_error(mysql); > + if ((mysql->net.tls_verify_status & MARIADB_TLS_VERIFY_TRUST) && > + !password_and_hashing(mysql, auth_plugin)) > + { > + my_set_error(mysql, CR_SSL_CONNECTION_ERROR, SQLSTATE_UNKNOWN, > + ER(CR_SSL_CONNECTION_ERROR), > + "Certificate verification failure: The certificate is NOT trusted."); already commented about using library error message > + return 1; > + } > goto retry; > } > /* > @@ -782,7 +841,9 @@ retry: > if (ma_read_ok_packet(mysql, mysql->net.read_pos + 1, pkt_length)) > return -1; > > - if (!mysql->net.tls_self_signed_error) > + if (!mysql->net.tls_verify_status || > + ((mysql->net.tls_verify_status & MARIADB_TLS_VERIFY_TRUST) && > + is_local_connection(mysql->net.pvio))) Not needed, see above > return 0; > > assert(mysql->options.use_ssl); > diff --git a/libmariadb/secure/openssl.c b/libmariadb/secure/openssl.c > index 8231a244..b5b573a9 100644 > --- a/libmariadb/secure/openssl.c > +++ b/libmariadb/secure/openssl.c > @@ -459,24 +461,38 @@ error: > return NULL; > } > > -unsigned int ma_tls_get_peer_cert_info(MARIADB_TLS *ctls) > +unsigned int ma_tls_get_peer_cert_info(MARIADB_TLS *ctls, uint hash_size) > { > X509 *cert; > + unsigned int hash_alg; > SSL *ssl; > + char fp[129]; > > + switch (hash_size) { > + case 0: > + case 256: > + hash_alg= MA_HASH_SHA256; > + break; > + case 384: > + hash_alg= MA_HASH_SHA384; > + break; > + case 512: > + hash_alg= MA_HASH_SHA512; > + break; > + default: > + return 1; > + } > + > if (!ctls || !ctls->ssl) > return 1; > > - /* Did we already read peer cert information ? */ > - if (ctls->cert_info.version) > - return 0; > - > ssl= (SSL *)ctls->ssl; > > /* Store peer certificate information */ > + if (!ctls->cert_info.version) > + { when can it be already set? > if ((cert= SSL_get_peer_certificate(ssl))) > { > - char fp[33]; > #if OPENSSL_VERSION_NUMBER >= 0x10101000L > const ASN1_TIME *not_before= X509_get0_notBefore(cert), > *not_after= X509_get0_notAfter(cert); > @@ -714,9 +714,40 @@ my_bool ma_tls_close(MARIADB_TLS *ctls) > return rc; > } > > -int ma_tls_verify_server_cert(MARIADB_TLS *ctls) > +/** Check for possible errors, and store the result in net.tls_verify_status. > + verification will happen after handshake by ma_tls_verify_server_cert(). > + To retrieve all errors, this callback function returns always true. > + (By default OpenSSL stops verification after first error > +*/ > +static int ma_verification_callback(int preverify_ok __attribute__((unused)), X509_STORE_CTX *ctx) > { > - X509 *cert; > + SSL *ssl; > + > + int x509_err= X509_STORE_CTX_get_error(ctx); > + > + if ((ssl = X509_STORE_CTX_get_ex_data(ctx, SSL_get_ex_data_X509_STORE_CTX_idx()))) > + { > + MYSQL *mysql= (MYSQL *)SSL_get_app_data(ssl); > + > + if ((x509_err == X509_V_ERR_DEPTH_ZERO_SELF_SIGNED_CERT || > + x509_err == X509_V_ERR_SELF_SIGNED_CERT_IN_CHAIN)) > + mysql->net.tls_verify_status|= MARIADB_TLS_VERIFY_TRUST; > + else if (x509_err == X509_V_ERR_CERT_REVOKED) > + mysql->net.tls_verify_status|= MARIADB_TLS_VERIFY_REVOKED; > + else if (x509_err == X509_V_ERR_CERT_NOT_YET_VALID || > + x509_err == X509_V_ERR_CERT_HAS_EXPIRED) > + mysql->net.tls_verify_status|= MARIADB_TLS_VERIFY_PERIOD; > + else if (x509_err != X509_V_OK) > + mysql->net.tls_verify_status|= MARIADB_TLS_VERIFY_UNKNOWN; and here you need to store the error message in mysql->net. for example, else if (x509_err == X509_V_ERR_CERT_NOT_YET_VALID || x509_err == X509_V_ERR_CERT_HAS_EXPIRED) { if (mysql->net.tls_verify_status < MARIADB_TLS_VERIFY_PERIOD) mysql->net.tls_verify_error= X509_verify_cert_error_string(x509_err); mysql->net.tls_verify_status|= MARIADB_TLS_VERIFY_PERIOD; } or else if (x509_err == X509_V_ERR_CERT_NOT_YET_VALID || x509_err == X509_V_ERR_CERT_HAS_EXPIRED) { if (mysql->net.tls_verify_status < MARIADB_TLS_VERIFY_PERIOD) my_set_error(mysql, CR_SSL_CONNECTION_ERROR, SQLSTATE_UNKNOWN, ER(CR_SSL_CONNECTION_ERROR), X509_verify_cert_error_string(x509_err)); mysql->net.tls_verify_status|= MARIADB_TLS_VERIFY_PERIOD; } > + } > + > + /* continue verification */ > + return 1; > +} > + > +int ma_tls_verify_server_cert(MARIADB_TLS *ctls, unsigned int verify_flags) > +{ > + X509 *cert= NULL; > MYSQL *mysql; > SSL *ssl; > MARIADB_PVIO *pvio; > @@ -734,52 +765,97 @@ int ma_tls_verify_server_cert(MARIADB_TLS *ctls) > mysql= (MYSQL *)SSL_get_app_data(ssl); > pvio= mysql->net.pvio; > > + if (verify_flags & MARIADB_TLS_VERIFY_FINGERPRINT) > + { > + if (ma_pvio_tls_check_fp(ctls, mysql->options.extension->tls_fp, mysql->options.extension->tls_fp_list)) > + { > + mysql->net.tls_verify_status |= MARIADB_TLS_VERIFY_FINGERPRINT; > + return 1; > + } > + > + /* if certificates are valid and no revocation error occured, > + we can return */ > + if (!(mysql->net.tls_verify_status & MARIADB_TLS_VERIFY_PERIOD) && > + !(mysql->net.tls_verify_status & MARIADB_TLS_VERIFY_REVOKED)) you can check this before trying the fingerprint > + { > + mysql->net.tls_verify_status= MARIADB_TLS_VERIFY_OK; > + return 0; > + } > + } > + > + if (mysql->net.tls_verify_status & verify_flags) > + { > + return 1; > + } > + > + if (verify_flags & MARIADB_TLS_VERIFY_HOST) > + { > if (!mysql->host) > { > pvio->set_error(mysql, CR_SSL_CONNECTION_ERROR, SQLSTATE_UNKNOWN, > ER(CR_SSL_CONNECTION_ERROR), "Invalid (empty) hostname"); > - return 1; > + mysql->net.tls_verify_status|= MARIADB_TLS_VERIFY_HOST; > + return MARIADB_TLS_VERIFY_ERROR; > } > > if (!(cert= SSL_get_peer_certificate(ssl))) > { > pvio->set_error(mysql, CR_SSL_CONNECTION_ERROR, SQLSTATE_UNKNOWN, > ER(CR_SSL_CONNECTION_ERROR), "Unable to get server certificate"); > - return 1; > + mysql->net.tls_verify_status|= MARIADB_TLS_VERIFY_HOST; not sure it's MARIADB_TLS_VERIFY_HOST, may be MARIADB_TLS_VERIFY_UNKNOWN ? > + return MARIADB_TLS_VERIFY_ERROR; > } > + > #ifdef HAVE_OPENSSL_CHECK_HOST > if (X509_check_host(cert, mysql->host, strlen(mysql->host), 0, 0) != 1 > && X509_check_ip_asc(cert, mysql->host, 0) != 1) > + { > + mysql->net.tls_verify_status|= MARIADB_TLS_VERIFY_HOST; > goto error; > + } > #else > x509sn= X509_get_subject_name(cert); > > if ((cn_pos= X509_NAME_get_index_by_NID(x509sn, NID_commonName, -1)) < 0) > + { > + mysql->net.tls_verify_status|= MARIADB_TLS_VERIFY_HOST; > goto error; > + } > > if (!(cn_entry= X509_NAME_get_entry(x509sn, cn_pos))) > + { > + mysql->net.tls_verify_status|= MARIADB_TLS_VERIFY_HOST; > goto error; > + } > > if (!(cn_asn1 = X509_NAME_ENTRY_get_data(cn_entry))) > + { > + mysql->net.tls_verify_status|= MARIADB_TLS_VERIFY_HOST; > goto error; > + } > > cn_str = (char *)ASN1_STRING_data(cn_asn1); > > /* Make sure there is no embedded \0 in the CN */ > if ((size_t)ASN1_STRING_length(cn_asn1) != strlen(cn_str)) > + { > + mysql->net.tls_verify_status|= MARIADB_TLS_VERIFY_HOST; > goto error; > + } > > if (strcmp(cn_str, mysql->host)) > + { > + mysql->net.tls_verify_status|= MARIADB_TLS_VERIFY_HOST; > goto error; > + } > #endif > X509_free(cert); > - > + } > return 0; > error: > + if (cert) > X509_free(cert); > > - pvio->set_error(mysql, CR_SSL_CONNECTION_ERROR, SQLSTATE_UNKNOWN, > - ER(CR_SSL_CONNECTION_ERROR), "Validation of SSL server certificate failed"); > return 1; > } > Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

1 0

Re: [PATCH] MDEV-34481 optimize away waiting for owned by prepared xa non-unique index record
by Kristian Nielsen 31 Jul '24

31 Jul '24

Andrei Elkin <andrei.elkin(a)mariadb.com> writes: >> Also, are you aware that when you skip a record like this, you may be >> skipping the very record that needs to be changed? > Yet luckily this is covered by the BASE (of the patch) logics. > After(needless) scan the rest of duplicate key record set the third group > of events would have to retry, to success. (This is optimizable, but I You need to make sure that --slave_exec_mode=IDEMPOTENT is not used. Otherwise you can get silent data corruption. - Kristian.

1 0

Re: [PATCH] MDEV-34481 optimize away waiting for owned by prepared xa non-unique index record
by Kristian Nielsen 31 Jul '24

31 Jul '24

> From: Andrei <andrei.elkin(a)mariadb.com> > > This work partly implements a ROW binlog format MDEV-33455 part > that is makes non-unique-index-only table XA replication safe in RBR. > Existence of at least one non-null unique index has always guaranteed > safety (no hang to error). Hi Andrei, A couple early comments on your MDEV-34481 patch, about things that affect non-XA parts of the code and will need to be changed: > diff --git a/sql/sql_class.cc b/sql/sql_class.cc > index 6419a58fbe4..b2321cfd865 100644 > --- a/sql/sql_class.cc > +++ b/sql/sql_class.cc > @@ -5444,20 +5449,33 @@ thd_need_wait_reports(const MYSQL_THD thd) > transaction. > */ > extern "C" int > -thd_rpl_deadlock_check(MYSQL_THD thd, MYSQL_THD other_thd) > +thd_rpl_deadlock_check(MYSQL_THD thd, MYSQL_THD other_thd, > + bool is_other_prepared, bool is_index_unique, > + uint *flagged) > + > + /* > + Return to the caller the fact of the non-unique index wait lock > + conflicts with one of a prepared state transaction. > + */ > + if (!is_index_unique && rgi && rgi->is_row_event_execution()) { > + if (is_other_prepared && !other_thd) > + { > + rgi->exec_flags |= 1 << rpl_group_info::HIT_BUSY_INDEX; > + (*flagged)++; > + } > + } > + No, thd_rpl_deadlock_check() has a specific purpose related to handling deadlock between active transactions. Don't add this XA-specific and unrelated logic to this function. Instead, add a separate function that InnoDB can call in the specific case you need. That function then only needs the parameters `thd` and `flagged`, the others are not needed: extern "C" bool thd_xa_skip_lock_check(MYSQL_THD thd, uint *flagged) { return (thd->rgi_slave && thd->rgi_slave->us_row_event_execution()); } Then in InnoDB, call something like this: if (lock->trx->state == TRX_STATE_PREPARED && (!lock->index->is_unique() || lock->index->n_nullable)) count_non_unique_index_hit++; About the HIT_BUSY_INDEX, see here: > diff --git a/sql/log_event_server.cc b/sql/log_event_server.cc > index f3d01b7d9ed..166295ed263 100644 > --- a/sql/log_event_server.cc > +++ b/sql/log_event_server.cc > @@ -8149,11 +8154,20 @@ int Rows_log_event::find_row(rpl_group_info *rgi) > HA_READ_KEY_EXACT)))) > { > DBUG_PRINT("info",("no record matching the key found in the table")); > - if (error == HA_ERR_KEY_NOT_FOUND) > - error= row_not_found_error(rgi); > - table->file->print_error(error, MYF(0)); > - table->file->ha_index_end(); > - goto end; > + if (error == HA_ERR_LOCK_WAIT_TIMEOUT && !is_index_unique && > + (rgi->exec_flags & (1 << rpl_group_info::HIT_BUSY_INDEX))) > + { > + rgi->exec_flags &= ~(1 << rpl_group_info::HIT_BUSY_INDEX); > + skip_first_compare= true; > + } Here, you are effectively making a decision in the storage engine to skip this record which is locked by an XA PREPAREd transaction. But you are communicating this back to the server in a very convoluted way, by introducing the rgi->exec_flags and setting HIT_BUSY_INDEX. This is not the way to communicate back from engine to server. Instead, simply return a different error code than HA_ERR_LOCK_WAIT_TIMEOUT from InnoDB that you can check for here. You can maybe use one of the existing HA_ERR_* from include/my_base.h, or add a new one if necessary. if (error == HA_ERR_<something>) skip_first_compare= true; > diff --git a/sql/rpl_rli.h b/sql/rpl_rli.h > index 3f24481839b..a9ec44b58cb 100644 > --- a/sql/rpl_rli.h > +++ b/sql/rpl_rli.h > @@ -833,6 +833,15 @@ struct rpl_group_info > RETRY_KILL_KILLED > }; > uchar killed_for_retry; > + enum enum_exec_flags { > + HIT_BUSY_INDEX= 0 > + }; > + /* > + Being executed replication event's context. It could contain > + notification from engine such as attempts to lock already acquired > + index record. > + */ > + uint32 exec_flags; ... and then you don't need to add this. Also, are you aware that when you skip a record like this, you may be skipping the very record that needs to be changed? For example a case like this, with three transactions in order: XA START 'T1'; UPDATE my_table SET value=10 WHERE key=2 XA END 'T1'; XA PREPARE 'T1'; XA COMMIT 'T1'; XA START 'T2'; UPDATE my_table SET value=20 WHERE key=2; Transaction 'T2' may run speculatively in optimistic parallel replication after prepare of T1, but before commit of T1. Then it will skip the record key=2, IIUC. I didn't see any explanation of this case in the patch commit message or comments. Also I don't see this case tested in the test case, that needs to be added I believe. - Kristian.

1 0

Re: [MariaDB commits] [PATCH] MDEV-33856 New definition for Seconds_Behind_Master
by Kristian Nielsen 25 Jul '24

25 Jul '24

Hi Monty, Here is my review of the MDEV-33856 patch: > commit cc516f4d9913d732065c7abef983df42e5a8bab3 (origin/bb-11.6-MDEV-33856-seconds_behind_master) > Author: Monty <monty(a)mariadb.org> > Date: Tue May 14 23:47:59 2024 +0300 > > MDEV-33856 New definition for Seconds_Behind_Master If I understand the patch correctly, in introduces a new `Master_slave_time_diff` value as an alternative to the traditional Seconds_behind_master, with a clearer semantics. And does not change how Seconds_behind_master works. I think this is fine, and can help to preserve backwards compatibility. Just maybe mention this in the commit message, and/or change the bug title to "Alternative definition for Seconds_Behind_Master". > diff --git a/mysql-test/suite/perfschema/t/relaylog.test b/mysql-test/suite/perfschema/t/relaylog.test > index 12fc96a8b27..16f72efae98 100644 > --- a/mysql-test/suite/perfschema/t/relaylog.test > +++ b/mysql-test/suite/perfschema/t/relaylog.test > @@ -43,7 +43,7 @@ select > if (sum_number_of_bytes_read > 0, "MANY", "NONE") as SUM_NUMBER_OF_BYTES_READ, > if (sum_number_of_bytes_write > 0, "MANY", "NONE") as SUM_NUMBER_OF_BYTES_WRITE > from performance_schema.file_summary_by_instance > - where file_name like "%master-%" order by file_name; > + where file_name like "%master-%" and file_name not like "%sql/share%" order by file_name; > > select * from performance_schema.file_summary_by_instance > where file_name like "%slave-%" order by file_name; > @@ -96,7 +96,7 @@ sync_slave_with_master; > -- echo "============ Performance schema on slave ============" > > select * from performance_schema.file_summary_by_instance > - where file_name like "%master-%" order by file_name; > + where file_name like "%master-%" and file_name not like "%sql/share%" order by file_name; > > select > substring(file_name, locate("slave-", file_name)) as FILE_NAME, What is the reason for these changes (adding match on "%sql/share%" ? I don't understand the relation to the subject of the patch. > diff --git a/mysql-test/suite/rpl/t/master_last_event_time.inc b/mysql-test/suite/rpl/t/master_last_event_time.inc > new file mode 100644 > index 00000000000..0790d557a7b > --- /dev/null > +++ b/mysql-test/suite/rpl/t/master_last_event_time.inc > > +--connection server_3 > +--sleep 1 > +--source include/start_slave.inc > +--source include/sync_with_master_gtid.inc What is the purpose of --sleep 1 here? Usually it's wrong to have sleeps in the tests. Sometimes it is correct, but then there should be a comment explaining why a sleep is necessary (and sufficient) here. This sleep looks wrong, since there's a proper wait immediately after for the slave to sync up. > diff --git a/mysql-test/suite/rpl/t/master_last_event_time_stmt.test b/mysql-test/suite/rpl/t/master_last_event_time_stmt.test > new file mode 100644 > index 00000000000..fe0f16a3db3 > --- /dev/null > +++ b/mysql-test/suite/rpl/t/master_last_event_time_stmt.test > @@ -0,0 +1,84 @@ > +# > +# Statement specific tests for master_last_event_time > +# > +--source include/have_binlog_format_statement.inc > > +# Slave will replicate the above with a row event which will be very fast > +# compared to the master event. I don't understand this comment. Why will the slave replicate using a row event, when the test case sets binlog_format=statement? > diff --git a/sql/log.cc b/sql/log.cc > index 1cc0fa77f08..63d9c756911 100644 > --- a/sql/log.cc > +++ b/sql/log.cc > @@ -1898,10 +1898,20 @@ static inline int > binlog_commit_flush_xid_caches(THD *thd, binlog_cache_mngr *cache_mngr, > bool all, my_xid xid) > Xid_log_event end_evt(thd, xid, TRUE); > - return (binlog_flush_cache(thd, cache_mngr, &end_evt, all, TRUE, TRUE)); > + if (!thd->slave_thread && ! thd->user_time.val) > + { > + /* > + Ensure that on the master the event time is the time of commit, > + not the start of statement time. > + */ > + my_hrtime_t hrtime= my_hrtime(); > + end_evt.when= hrtime_to_my_time(hrtime); > + } > + DBUG_RETURN(binlog_flush_cache(thd, cache_mngr, &end_evt, all, TRUE, TRUE)); Ok, so correct me if I'm wrong, but I think the purpose here is: You want to have the time when the transaction ended (in case of long-running query). But for auto-committed query, there is no explicit COMMIT query, so the thd->start_time is the start of the transaction. So here you ensure that the XID_EVENT will have the time at start of commit in either case. If this is correct, probably could use a comment explaining this. Some comments: 1. It is better to check for thd->rgi_slave being non-NULL than for thd->slave_thread. The thd->rgi_slave is associated with the replication of some event or transaction. The thd->slave_thread just identifies the thread/connection. (This is mostly style/code clarity, as in current code the two values would track each other.) 2. Why the condition on thd->user_time.val? What is the situation where we are not replicating an event, but user_time is set and we do not want to put the current time in the XID_EVENT? 3. This handles an XID end event, which is for InnoDB transactions. But MyISAM/Aria use a different end event, a COMMIT query, and there are are few other cases as well. Don't you want consistently to have the end of the query as the time in all end events? Then this code should probably instead be in binlog_flush_cache() or perhaps write_transaction_to_binlog()? 4. Even given (3), there are still cases (eg. DDL) where there is no end event, and thus there is no event containing the time for the end of the operation on the master. Won't this cause some inconsistency in how the slave determines and displays its timings with this patch? If not, why not? > for (; has_xid && !entry.need_unlog && ha_info; ha_info= ha_info->next()) > { > if (ha_info->is_started() && ha_info->ht() != binlog_hton && > !ha_info->ht()->commit_checkpoint_request) > + { > entry.need_unlog= true; > + break; > + } > } Agree, this code always irritated me ;-) You could also move the condition (!entry.need_unlog && ha_info) out of the loop in an outer if () to not have to check it on every loop iteration (just an observation, entirely up to you, it's not something your patch introduced). > diff --git a/sql/log_event.cc b/sql/log_event.cc > index 3802f76efe0..706a4903eb0 100644 > --- a/sql/log_event.cc > +++ b/sql/log_event.cc > @@ -1104,10 +1116,12 @@ Log_event* Log_event::read_log_event(const uchar *buf, uint event_len, > switch(event_type) { > case QUERY_EVENT: > ev= new Query_log_event(buf, event_len, fdle, QUERY_EVENT); > + set_orig_exec_time_in_thd(((Query_log_event*) ev)->exec_time); > break; > case QUERY_COMPRESSED_EVENT: > ev= new Query_compressed_log_event(buf, event_len, fdle, > - QUERY_COMPRESSED_EVENT); > + QUERY_COMPRESSED_EVENT); > + set_orig_exec_time_in_thd(((Query_compressed_log_event*) ev)->exec_time); > @@ -964,6 +964,18 @@ Log_event* Log_event::read_log_event(IO_CACHE* file, > DBUG_RETURN(res); > } > > +/* > + Update thd->orig_exec_time > +*/ > + > +inline void set_orig_exec_time_in_thd(my_time_t exec_time) > +{ > +#if !defined(MYSQL_CLIENT) && defined(HAVE_REPLICATION) > + THD *thd= current_thd; > + if (likely(thd)) > + thd->orig_exec_time= exec_time; > +#endif > +} Using current_thd doesn't seem right here, and in fact it's not, as seen by this part of the patch where thd->orig_exec_time has to be set again for parallel replication: > diff --git a/sql/rpl_parallel.cc b/sql/rpl_parallel.cc > index 51ca3b183ec..d279761767a 100644 > --- a/sql/rpl_parallel.cc > +++ b/sql/rpl_parallel.cc > @@ -60,7 +60,10 @@ rpt_handle_event(rpl_parallel_thread::queued_event *qev, > rgi->future_event_relay_log_pos= qev->future_event_relay_log_pos; > strcpy(rgi->future_event_master_log_name, qev->future_event_master_log_name); > if (event_can_update_last_master_timestamp(ev)) > + { > rgi->last_master_timestamp= ev->when + ev->exec_time; > + thd->orig_exec_time= ev->exec_time; > + } So let's not set thd->orig_exec_time from Log_event::read_log_event(), it's redundant (for parallel replication), and logically setting it should be done when applying the event, not when reading the event. Instead, set the thd->orig_exec_time in exec_relay_log_event(), so we retain similarity with how replication is done in the parallel and non-parallel case. There is already a similar if (event_can_update_last_master_timestamp(ev)) in exec_relay_log_event(). (Also see comments below about moving this from THD to rpl_group_info to not increase THD just for this very specific case). > + > +/* > + Get the time when the event had been executed on the master. > + This works for both query events and load data events. > +*/ > + > +#if Q_EXEC_TIME_OFFSET != L_EXEC_TIME_OFFSET > +#error "Q_EXEC_TIME_OFFSET is not same as L_EXEC_TIME_OFFSET" > +#endif > + > +time_t query_event_get_time(const uchar *buf, > + const Format_description_log_event > + *description_event) > +{ > + time_t when= uint4korr(buf); > + buf+= description_event->common_header_len; > + return when + uint4korr(buf + Q_EXEC_TIME_OFFSET); Suggestion: Add an assert here: DBUG_ASSERT(LOG_EVENT_IS_QUERY(buf[EVENT_TYPE_OFFSET]) || LOG_EVENT_IS_LOAD_DATA(buf[EVENT_TYPE_OFFSET])); just to avoid someone mistakenly adding code later to call this for a wrong event type. > diff --git a/sql/sql_class.h b/sql/sql_class.h > index 2d671139a6c..05c9c4e8012 100644 > --- a/sql/sql_class.h > +++ b/sql/sql_class.h > @@ -3168,6 +3168,7 @@ class THD: public THD_count, /* this must be first */ > ulonglong start_utime, utime_after_lock, utime_after_query; > /* This can be used by handlers to send signals to the SQL level */ > ulonglong replication_flags; > + my_time_t orig_exec_time; // Exec time for last read binlog event > @@ -1348,8 +1346,14 @@ Query_log_event::Query_log_event(THD* thd_arg, const char* query_arg, > memset(&host, 0, sizeof(host)); > error_code= errcode; > > - end_time= my_time(0); > - exec_time = (ulong) (end_time - thd_arg->start_time); > + /* > + For slave threads, remember the original master exec time. > + This is needed to be able to calculate the master commit time. > + */ > + exec_time= ((thd->slave_thread) ? > + thd->orig_exec_time : > + (my_time(0) - thd_arg->start_time)); > + I think we can here avoid increasing the THD size, which is sad for a special case like this. Instead put the orig_exec_time into THD::rgi_slave, extending the struct rpl_group_info. Then the assignment above will be to rgi->orig_exec_time instead of thd->orig_exec_time, and the check here will be for thd->rgi_slave instead of thd->slave_thread. The rgi is the appropriate place for orig_exec_time, as it is a property of the transaction/event group being replicated. @@ -3171,146 +2856,170 @@ static const LEX_CSTRING msg_ignored= { STRING_WITH_LEN("Ignored") }; #endif > -static bool send_show_master_info_data(THD *thd, Master_info *mi, bool full, > - String *gtid_pos) > +inline void store_string_or_null(Field **field, const char *str) > +{ > > - mysql_mutex_unlock(&mi->run_lock); > +void store_master_info(THD *thd, Master_info *mi, TABLE *table, > > - protocol->store_string_or_null(mi->host, &my_charset_bin); > - protocol->store_string_or_null(mi->user, &my_charset_bin); > - protocol->store((uint32) mi->port); > - protocol->store((uint32) mi->connect_retry); > - protocol->store(mi->master_log_name, strlen(mi->master_log_name), > + store_string_or_null(field++, mi->host); > + store_string_or_null(field++, mi->user); > + (*field++)->store((uint32) mi->port); > + (*field++)->store((uint32) mi->connect_retry); > + (*field++)->store(mi->master_log_name, strlen(mi->master_log_name), So this patch refactors the code for SHOW SLAVE STATUS to use a shared code path with information_schema tables? That sounds like a good change. But for the future, please do such refactoring as a separate patch, you can see my patch series of pre-computing binlog checksums (MDEV-31273) for an example. Doing it together makes it very hard to review the actual code changes and see what is refactoring and what is new logic for master_slave_time_diff. > + if (unlikely(!rli->slave_timestamp) && Log_event::is_group_event(typ)) > + { > + /* > + First event for this slave. Assume that all the slave was up to date > + with the master just before the current event. > + */ > + rli->slave_timestamp= (time_t) ev->when + (time_t) ev->exec_time-1; I think the comment is meant to explain why there is the "-1" / minus-one in this expression, but it's not really clear exactly what is meant, so perhaps elaborate it a bit. Does it mean that master_slave_time_diff will start out as "1 second" just after the slave restarts? One way or the other, I do not have a strong opinion. But I want to give a general remark, from my experience with the troubles over the years with Seconds_Behind_Master. The problem with Seconds_Behind_Master was always that it didn't quite do what someone expected in some particular case, and then a new special case was added to try to accomodate. And after a number of iterations of this, we ended up with a semantics for Seconds_Behind_Master that no-one _really_ understands, and which still doesn't quite always do what people expect (which is in any case impossible if different people have different expectations). This "-1" here is clearly on the same road towards adding special cases. So my strong advice would be that, going forward, be _extremely_ conservative about modifying the semantics of master_slave_time_diff. The one good thing about the new status is that it has a (hopefully) clear semantics that can be documented (now including this "-1") and completely understood by users. Let's stick to that no matter who comes and wants just another small tweak, unless such tweaks actually make the semantics and documentation _simpler_. > + /* > + replay_log.description_event_for_exec can be null if the slave thread > + is getting killed > + */ > + if (LOG_EVENT_IS_QUERY((Log_event_type) buf[EVENT_TYPE_OFFSET]) || > + LOG_EVENT_IS_LOAD_DATA((Log_event_type) buf[EVENT_TYPE_OFFSET])) > + { > + time_t exec_time= query_event_get_time(buf, rli->relay_log. > + description_event_for_queue); > + set_if_bigger(rli->newest_master_timestamp, exec_time); > + } This confused me, because in Query_log_event, the exec_time is the duration of the query (a time interval), not an absolute time. So please use the name "end_time" for the variable instead of "exec_time", and consider renaming query_event_get_time() to query_event_get_end_time(). > > +static int make_slave_status_old_format(THD *thd, ST_SCHEMA_TABLE *schema_table) > +{ > + ST_FIELD_INFO *field_info= schema_table->fields_info; > + Name_resolution_context *context= &thd->lex->first_select_lex()->context; > + DBUG_ASSERT(thd->lex->sql_command == SQLCOM_SHOW_SLAVE_STAT); > + bool all_slaves= thd->lex->mi.show_all_slaves; > + ulonglong used_fields= ~0ULL; > + > + if (!all_slaves) > + { > + /* Remove 2 first fields and all fields above and including field 55 */ > + used_fields&= ~((1ULL << 0) | (1ULL << 1)); > + used_fields&= ((1ULL << 56)-1); Should there be some constant used for the magic numbers 55/56? And/or an assertion that this value is correct? Also, is the intention here that we will from here on never add more fields to legacy SHOW SLAVE STATUS? If so, there should probably be a comment about this; also see the test case mysql-test/suite/rpl/t/rpl_show_slave_status.test which has comments that might need updating. - Kristian

2 2