- developers - lists.mariadb.org

pthread_yield() vs std::this_thread::yield()
by Sergey Vojtovich 17 May '25

17 May '25

Monty and I had an argument recently regarding pthread_yield() and std::this_thread::yield() usage. I do want to make this discussion public, so that we can pick either approach, get it documented and make sure everybody is aware of reasoning and decision. Probably even add a bb check that wouldn't allow unwanted function calls to pass. Lets try to keep this discussion focused on std::this_thread::yield() and avoid discussing other C++11 features. Since I'm biased towards std::this_thread::yield() I'd let Monty elaborate his concerns and explain why he prefers pthread_yield(). But generally it comes to compatibility and portability. My position is: pthread_yield() should be removed in favour of std::this_thread::yield(). Benefits: 1. Must be available to all current MariaDB versions. C++11 is a must since 2018. 2. Compatibility and portability, which we don't have to care about. 3. I have no performance concerns after analysing how it is implemented by clang and gcc. 4. Goes inline with "MDEV-35461 - Remove redundant checks for standard library functions" 5. Standard things looks more community-friendly, e.g. why should MSVC developers care about POSIX threads? Currently pthread_yield() can be defined to: 1. SwitchToThread() - Win32 (Wlad, 2009) MSVC seems to support std::this_thread::yield() since 2015: https://learn.microsoft.com/en-us/cpp/standard-library/thread-functions?vie… 2. thr_yield() - Solaris (Serg, 2009) Oracle Developer Studio 12.6 supports std::this_thread::yield() since 2017: https://docs.oracle.com/cd/E77782_01/html/E77789/bkabe.html#OSSCPgnyio 3. pthread_yield_np() - OS X (Serg, 2009) XCode is based on clang, which supports std::this_thread::yield() since 2013. sched_yield() - default (Serg, 2009) pthread_yield() - deprecated since 2021 (Serg, 2009) Possible concerns: 1. Compatibility with recent platforms. C++11 is widely available, thread models seem to be well supported. InnoDB uses std::this_thread_yield() since 2021. 2. Compatibility with older (ancient?) platforms. We're way off into C++11 since 2018. Making recent MariaDB compile on such platforms is probably going to be a heavy effort. If we do really want to have these platforms supported, they must be present in buildbot. 3. Performance impact of std::this_thread::yield(). See below. In many cases it is just a wrapper around functions like sched_yield. clang: libcxx/include/__thread/this_thread.h:inline _LIBCPP_HIDE_FROM_ABI void yield() _NOEXCEPT { __libcpp_thread_yield(); } libcxx/src/support/win32/thread_win32.cpp:void __libcpp_thread_yield() { SwitchToThread(); } libcxx/include/__thread/support/pthread.h:inline _LIBCPP_HIDE_FROM_ABI void __libcpp_thread_yield() { sched_yield(); } gcc additionally supports VxWorks: yield() noexcept { #if defined _GLIBCXX_HAS_GTHREADS && defined _GLIBCXX_USE_SCHED_YIELD __gthread_yield(); #endif } int __gthread_yield (void) { return taskDelay (0); } __GTHREAD_INLINE int __gthread_yield (void) { return __gthrw_(sched_yield) (); } Regards, Sergey

5 7

Re: MDEV-34705 benchmarking
by Kristian Nielsen 16 May '25

16 May '25

Steve Shaw <steve.shaw(a)hammerdb.com> writes: > One follow-up question, with the new build if I set log_bin=1 (or > skip-log-bin=0) I get the following: > > 2025-05-16 15:17:43 0 [ERROR] Binlog name can not be set with > --log-bin when --binlog-storage-engine is used. Use --binlog-directory > to specify a separate directory for binlogs Sorry, as I wrote originally, you have to use in the config file: log_bin (without any argument). I also missed the detail that you put the "=1" in there, since it looks so natural. (The reason is historical. In the existing server, --log-bin takes the basename to use for the binlog files, it's not an on/off switch. So log_bin=1 means to call the binlog files "1.000001", "1.000002", ... In the new binlog I removed the ability to specify the basename, hence the error rather than silently ignoring the user's specified name). > How do I check its working as intended rather than not having enabled > the binlog. When the new binlog is enabled, the data directory (or --binlog-directory if specified) will contain binlog files named like this: binlog-000000.ibb binlog-000001.ibb ... - Kristian.

1 0

Re: MDEV-34705 benchmarking
by Kristian Nielsen 16 May '25

16 May '25

Steve Shaw <steve.shaw(a)mariadb.com> writes: > pipeline as other testing is currently underway so will probably be early > next week. Ok, sounds great! > To confirm we will test the mariadb-11.4.6 branch as a baseline and then I would suggest to use the commit e176066a9e02cf2bdedf9e79fb150029dde430f6 (which is the commit that knielsen_binlog_in_engine was last rebased onto) as the base, just to avoid the possibility that the 136 extra commits in 11.4.6 could influence anything. > the knielsen_binlog_in_engine branch for comparison. > we will set these parameters for mariadb-11.4.6 > > log_bin=1 > innodb_flush_log_at_trx_commit=1 > sync_binlog=1 > > and then these for knielsen_binlog_in_engine > > log_bin=1 > innodb_flush_log_at_trx_commit=1 > sync_binlog=1 > binlog_storage_engine=innodb Yes, this is probably the most interesting starting comparison. > Where you would like the results posting to? To the MDEV or reply to this > email thread? Well, I think the developers@ mailing list is a good place for discussions like this, but either works for me. And as I said, be sure to let me know in case of any issues, I'll try to help. - Kristian.

1 0

MDEV-34705 benchmarking
by Kristian Nielsen 16 May '25

16 May '25

Hi Steve, Brandon Nesterenko told me you would be interested in benchmarking the MDEV-34705 binlog-in-engine feature. The issue https://jira.mariadb.org/browse/MDEV-34705 has a design writeup. The feature is a new implementation of the binlog that is integrated with the InnoDB storage engine, and can therefore be more efficient. The code is in the branch: knielsen_binlog_in_engine Performance-wise, this feature has two major goals: 1. Reduce the number of fsync() required in durable configuration (--innodb-flush-log-at-trx-commit=1 --sync-binlog=1) from two to one per (group) commit, giving a large speedup. 2. Make the non-durable configuration (--innodb-flush-log-at-trx-commit=0|2) crash-safe for the binlog also, by avoiding the need for two-phase commit between InnoDB and binlog. Thus the primary goal of benchmarking is to verify the speedup in (1), and to verify that crash-safety in (2) is obtained without performance regression. Also interesting is to see how much faster new (2) is than old (1), since this comparison applies when crash-safety is required but durability is not. The speedup is expected to be higher at lower concurrency (as fsync cost is amortised over fewer commits); it is also interesting to test for scalabitily regressions at higher concurrency. Read-only workloads should be mostly unaffected. To enable the new binlog in the branch, use the following configuration: binlog_storage_engine=innodb log_bin # Optional binlog location, defaults to datadir. binlog_directory=/data/m/my/binlogs/ I'd be very interested in any results of testing the new feature, and happy to hear any comments/suggestions/questions. If you start testing the code, I will advise that this is work-in-progress; so a good idea to communicate any findings early. There are still pieces missing (and surely bugs), and I'm happy to quickly fix anything that blocks testing. In my own preliminary benchmarks, I have successfully run sysbench (using Mark Callaghan's scripts), where I see typically 50%-150% speedup in (1), and 10%-20% in (2). Results depend heavily on transaction load, concurrency, and benchmark machine disk performance. - Kristian.

1 0

Windows line endings in MariaDB test result files?
by Otto Kekäläinen 14 May '25

14 May '25

Hi, While importing 11.4.6 into Debian I ran into issues with git and quilt failing to convert text files properly between Unix and Windows line endings. Thus asking are Windows CRLF necessary in the new mysql-test/main/mysql-interactive.result test? commit e5147c8140450f28505a227159d2feba37dd1c39 Author: Sergei Golubchik <serg(a)mariadb.org> Date: Tue Jan 30 16:39:28 2024 +0100 regression introduced by MDEV-14448 diff --git a/mysql-test/main/mysql-interactive.result b/mysql-test/main/mysql-interactive.result new file mode 100644 index 00000000000..a18c018b932 --- /dev/null +++ b/mysql-test/main/mysql-interactive.result @@ -0,0 +1,24 @@ +# +# regression introduced by MDEV-14448 +# +delimiter $^M +select 1;^M +$^M +Welcome to the MariaDB monitor. Commands end with ; or \g.^M +Your MariaDB connection id is X^M +Server version: Y^M +Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.^M +^M +Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.^M +^M +MariaDB [(none)]> delimiter $^M +MariaDB [(none)]> select 1;^M + -> $^M ++---+^M +| 1 |^M ++---+^M +| 1 |^M ++---+^M +1 row in set^M +^M +MariaDB [(none)]> All the other files with Windows line endings seem to be have them intentionally: ± find . -type f -exec file {} \; | grep -i "CRLF" ./storage/connect/mysql-test/connect/std_data/emp.txt: ASCII text, with CRLF line terminators ./storage/connect/mysql-test/connect/std_data/boyswin.txt: ASCII text, with CRLF line terminators ./storage/connect/mysql-test/connect/std_data/expenses.txt: ASCII text, with CRLF line terminators ./storage/connect/mysql-test/connect/r/xml.result: HTML document, UTF-8 Unicode text, with CRLF, LF line terminators ./mysql-test/std_data/loadxml2.dat: XML 1.0 document, ASCII text, with very long lines, with CRLF, LF line terminators ./mysql-test/main/loadxml.result: ASCII text, with CRLF, LF line terminators ./mysql-test/main/func_regexp_pcre.result: UTF-8 Unicode text, with very long lines, with CRLF, CR, LF line terminators ./mysql-test/main/mysql-interactive.result: ASCII text, with CRLF, LF line terminators ./mysql-test/main/mysql_binary_mode.result: ASCII text, with CRLF, LF line terminators ./extra/wolfssl/wolfssl/certs/test/smime-test-canon.p7s: MIME entity, ASCII text, with CRLF, CR line terminators ./extra/wolfssl/wolfssl/wolfssl.rc: C source, Little-endian UTF-16 Unicode text, with CRLF line terminators

2 4

MariaDB near 50% in WordPress usage stats
by Otto Kekäläinen 13 May '25

13 May '25

Hi all, Glad to see that MariaDB has now surpassed MySQL in WordPress usage stats at https://wordpress.org/about/stats/ Version Usage MySQL 8.0 20.77% MariaDB 10.6 19.88% MariaDB 10.11 18.60% MySQL 5.7 17.78% MySQL 5.5 7.03% MariaDB 10.5 6.77% MariaDB 10.3 2.74% MySQL 5.6 1.99% MariaDB 11.4 1.27% MariaDB 10.4 1.07% MariaDB 5.5 0.46% MariaDB 10.1 0.43% MariaDB 10.2 0.32% MySQL 5.1 0.17% MySQL 8.4 0.14% I haven't verified this myself, but I heard that MySQL 8.4 deprecated SQL_CALC_FOUND_ROWS which is heavily used in the WordPress code base, and thus the MySQL 8.4 adoption is nearly zero. Good reminder to have collaboration and joint testing with some of the largest open source projects to avoid accidentally having a new MariaDB version being incompatible with them..

2 2

Commit ff4209fa0d1c66952bc9f00587f525314464e7be not merged on main?
by Otto Kekäläinen 08 May '25

08 May '25

Hi! I was updating the patch https://salsa.debian.org/mariadb-team/mariadb-server/-/blob/debian/latest/d… in Debian when importing MariaDB 11.4.6, and I noticed that the upstream commit that changes the same lines in https://github.com/MariaDB/server/commit/ff4209fa0d1c66952bc9f00587f5253144… only landed on MariaDB '10.11' and '11.4' branch, but never on 'main' or '11.8'. Could it be that 'main' has had zero merges from older branches since April 3? Or was there maybe some mistake and one merge was overlooked and it wasn't properly applied on all newer maintained branches?

2 2

CI failures visible on GitHub (affecting contributors)
by Otto Kekäläinen 29 Apr '25

29 Apr '25

Hi, I see that on the 11.8 branch (https://github.com/MariaDB/server/commits/11.8/) which is supposed to be fully green and passing CI as it is soon to be released as a new LTS version, actually isn't consistently green. Out of the latest 10 CI runs the jobs passing rates are: 12/!2 11/12 12/!2 12/!2 8/8 10/12 8/12 7/8 8/8 11/12 Questions: - What is the current plan to get CI to be consistently green? - How is it possible that a commit is pushed on the 11.8 branch even though CI is not passing? Isn't branch protection preventing it? - If there is a way to get commits on 11.8 branch with failures bypassing branch protection, are all developers committed to follow-up on their commits and fix them before they do any other work? Or what is the plan for how to ensure post-push test failures are fixed? - The jobs buildbot/amd64-debian-11-msan-clang-16 and continuous-integration/appveyor/branch seem to be the most common failing ones. Can they be removed now to make the CI consistently green? And apply full branch protection so that they, or any other jobs, can't be added back to unless they are actually green? - Why is the CI occasionally tunning 8 jobs instead of 12? I also see plenty of red at https://buildbot.mariadb.org/#/grid?branch=11.8. Hopefully we get as much green as possible for the start of the 11.8 maintenance so that any potential regressions in future stable maintenance releases of 11.8 series are easier to catch. Most of the issues seem to be test issues and not real MariaDB bugs. I have also done a bunch of testing on Debian QA systems for MariaDB 11.8 and I have only found a couple small issues, so overall 11.8 is looking good! - Otto

3 3

MariaDB 11.8 and Debian status - information to fellow Linux distro packagers/maintainers
by Otto Kekäläinen 27 Apr '25

27 Apr '25

Hi! Do we have any Linux distro packagers and maintainers on this mailing list? If you are working on importing MariaDB 11.8 into a Linux distro, you may want to take a peek at what was done in Debian, as it may save you some work in debugging build or test issues: * https://salsa.debian.org/mariadb-team/mariadb-server/-/commits/debian/latest * https://salsa.debian.org/mariadb-team/galera-4/-/commits/debian/latest You might also benefit from looking at work-in-progress and open Merge Requests at : * https://salsa.debian.org/mariadb-team/mariadb-server/-/merge_requests In particular the MR https://salsa.debian.org/mariadb-team/mariadb-server/-/merge_requests/115 about a NEWS item to Debian users is likely to have useful text/summary that others might want to re-use: * https://salsa.debian.org/otto/mariadb-server/-/blob/dev/NEWS-11.8.2/debian/… Overall Debian package statuses are best viewed at: * https://tracker.debian.org/pkg/mariadb * https://tracker.debian.org/pkg/galera-4 ..and respective in Ubuntu: * https://launchpad.net/ubuntu/+source/mariadb * https://launchpad.net/ubuntu/+source/galera-4 I've fixed a lot of bugs in past weeks. The Debian tracker MariaDB bugs are down from 60 to 44, and in Ubuntu down to 3. For Galera the numbers are now down to 1 and 0. Looking forward to collaborate with maintainers of MariaDB and Galera packages in other Linux distributions! - Otto

1 0

Remaining InnoDB parts for MDEV-34705 binlog-in-engine?
by Kristian Nielsen 25 Apr '25

25 Apr '25

Hi Marko, I have been making some good progress on the new binlog implementation, I implemented all of what we discussed regarding the InnoDB side I believe, and also got a lot done on the integration with the server side code. I wanted to ask you what outstanding issues on the InnoDB side remain from your point of view, if any? The page format now is implemented in the new design, as described in a separate mail: https://lists.mariadb.org/hyperkitty/list/developers@lists.mariadb.org/thre… All the InnoDB-specific page overhead is gone, only a 4-byte CRC32 remains at the end of each page. The first page is reserved as a header page. The new binlog code no longer uses the InnoDB buffer pool, nor create any InnoDB tablespace structures (except for the two reserved tablespace IDs for the redo records). The code has its own buffer pool replacement (called the "page fifo") and own background flushing. The binlog hooks into the InnoDB crash recovery code following your suggestions, and just the single "memcpy" record is needed. Recovery is implemented (and tested somewhat), and has been carefully designed to recover in all cases (barring any bugs of course) while requiring only the two reserved tablespace IDs. I have two outstanding InnoDB-related issues I know about: 1. In order for a replication setup to be crash-safe, I must ensure that the slave does not receive an event until it has become durable on the master. Otherwise, when the master crashes and comes back up, an event might be applied on the slave that no (longer) exist on the master. For this, I need something in the InnoDB redo log code that will callback into the binlog after a specific LSN has been durably written to disk. This callback will then notify the dump threads (slave connections) somehow that the corresponding part of the binlog can now be safely read. 2. Integration with mariabackup. With the new binlog, mariabackup can backup the binlog together with the rest of the InnoDB data in a consistent way, something that I think is not possible currently? This will also remove the need for mariabackup to extract the binlog position with SHOW MASTER STATUS or for InnoDB to store the binlog position in the undo segments. When the binlog is backed up, the correct GTID position can be obtained after restoring a master backup simply by checking the binlog (SELECT @@gtid_binlog_pos). (For a slave backup, the position is similarly available from the backup of the mysql.gtid_slave_pos table, as before). I am not yet sure of the details of mariabackup support. On the high level, it will involve copying the binlog-NNNNNN.ibb into the backup along with the other InnoDB tablespace files; and applying redo records to those binlog files as part of --prepare. One important detail is to handle that we have only two reserved tablespace IDs for binlog files. The current recovery code requires that only the two most recent binlog files need recovery. We could try to ensure that the same applies duing mariabackup --prepare. An idea is to use the fact that binlog files are strictly append-only. Around the point where mariabackup decides the LSN where it stops copying the redo log that becomes the backup snapshot, we can determine the last binlog file that exists. We will check this just before and just after determining the LSN, and repeat if a new binlog file was created just at that point (which would otherwise leave doubt if the file was created before or after that LSN). Then after the snapshot LSN is determined, we can just copy all the binlog files up to and including (but not after) the snapshot point. This way mariabackup --prepare should be able to run the same binlog recovery code that runs in crash recovery, only needing to consider the two most recent of the copied binlog files. I can imagine other ways to do mariabackup of binlog files, but this is one idea that could be feasible. What do you think? Anything other outstanding issues with the new binlog implementation regarding InnoDB that you are aware of? - Kristian.

1 1