Why is innodb-log-file-mmap is available only on amd64/arm64/ppc64el but not other Linux architectures?
Hi, The documentation at https://mariadb.com/kb/en/innodb-system-variables/#innodb_log_file_mmap says this is supposed to be enabled by default on Linux, regardless of the architecture. However, in my testing it is not available on all architectures. Can somebody help me read the code path of innodb-log-file-mmap to figure out why the visibility/availability of this option it is currently architecture-dependent? ## Context In Debian I maintain an extra test called "configuration-trace" which basically dumps the output of these four commands into a file: /usr/bin/mariadb --print-defaults /usr/bin/mariadb --verbose --help /usr/sbin/mariadbd --print-defaults /usr/sbin/mariadbd --verbose --help Then, using the static file containing the text output of these files, it compares to the output of these commands on a running system to detect if the MariaDB defaults or features randomly changed (for example due to changes in dependencies or platform). This "configuration-trace" is also a handy way to detect what changed upstream in minor MariaDB releases, which aren't supposed to have much changes. The update of the static text output files for MariaDB 11.4.4 were done in https://salsa.debian.org/mariadb-team/mariadb-server/-/commit/e2851fcfca9e88... and they show among others a new variable innodb-log-file-mmap ## Problem The issues I am running into is that this new innodb-log-file-mmap only seems to exist on amd64 and arm64? I discovered this in the Debian CI system (autpkgtests) at https://ci.debian.net/packages/m/mariadb/ which is passing for amd64/arm64/ppc64el, but failing on other platforms: https://ci.debian.net/packages/m/mariadb/unstable/armel/55869348/ https://ci.debian.net/packages/m/mariadb/unstable/armhf/55869349/ https://ci.debian.net/packages/m/mariadb/unstable/i386/55869350/ https://ci.debian.net/packages/m/mariadb/unstable/loong64/55869351/ https://ci.debian.net/packages/m/mariadb/unstable/s390x/55869354/ (The output also shows that mariadbd --verbose --help prints out wsrep-provider twice, once wsrep-provider=ON and once wsrep-provider=none, but that is a different issue.) On systems where it isn't available, shouldn't the option still show up (but as permanently disabled)? It seems odd to have system variables appear or disappear depending on the Linux architecture.
Hi Otto, On Sun, Dec 29, 2024 at 9:04 PM Otto Kekäläinen via developers <developers@lists.mariadb.org> wrote:
The documentation at https://mariadb.com/kb/en/innodb-system-variables/#innodb_log_file_mmap says this is supposed to be enabled by default on Linux, regardless of the architecture. However, in my testing it is not available on all architectures.
The documentation seems to be inaccurate. It does not make sense to implement memory-mapped access to the log on systems where the virtual address space is already scarce. On 32-bit architectures, the size of the physical RAM tends to be close to the maximum usable virtual address space. For good performance, most of the physical memory would have to be allocated to innodb_buffer_pool_size, and on 32-bit systems there may not be enough virtual address space available for mapping the ib_logfile0.
Can somebody help me read the code path of innodb-log-file-mmap to figure out why the visibility/availability of this option it is currently architecture-dependent?
It is not strictly architecture-dependent, but its usefulness will be limited on 32-bit architectures.
https://ci.debian.net/packages/m/mariadb/unstable/armel/55869348/ https://ci.debian.net/packages/m/mariadb/unstable/armhf/55869349/ https://ci.debian.net/packages/m/mariadb/unstable/i386/55869350/
These are 32-bit. By the way, "i386" is misleading, because MariaDB Server does depend on -march=i686 in order to have 64-bit atomic operations.
https://ci.debian.net/packages/m/mariadb/unstable/loong64/55869351/ https://ci.debian.net/packages/m/mariadb/unstable/s390x/55869354/
These should actually be 64-bit and therefore enable innodb_log_file_mmap. Is your configuration check tripping because of the following? static constexpr const char *innodb_log_file_mmap_description= "Whether ib_logfile0" # ifdef HAVE_PMEM " resides in persistent memory or" # endif " should initially be memory-mapped";
On systems where it isn't available, shouldn't the option still show up (but as permanently disabled)?
In my opinion, a smaller footprint is better. To name another example: On Microsoft Windows, WITH_WSREP will not be defined.
It seems odd to have system variables appear or disappear depending on the Linux architecture.
For persistent memory, which might live on as CXL.mem, the instructions to control the cache flushing are architecture dependent. Best regards, Marko -- Marko Mäkelä, Lead Developer InnoDB MariaDB plc
Hi! ...
These should actually be 64-bit and therefore enable innodb_log_file_mmap. Is your configuration check tripping because of the following?
static constexpr const char *innodb_log_file_mmap_description= "Whether ib_logfile0" # ifdef HAVE_PMEM " resides in persistent memory or" # endif " should initially be memory-mapped";
It is not "my configuration check", it is standard output from mariadbd --verbose --help.
On systems where it isn't available, shouldn't the option still show up (but as permanently disabled)?
In my opinion, a smaller footprint is better. To name another example: On Microsoft Windows, WITH_WSREP will not be defined.
That is not the same situation. It is a completely different scenario if the software is missing some options because a feature was not compiled into it, or the software is too old to have some feature. What I now discovered is the **first and only system variable** that may disappear based on what the architecture is. I agree that most of the time less is more, but in the case of configuration options / system variables, it is extremely confusing to users if an option occasionally exists and occasionally not in the same version of MariaDB build with the same dependencies. Can you please help me write a patch (apparently for a_innodb.cc) to make the variable will always be visible?
Hi Otto, On Thu, Jan 2, 2025 at 9:33 AM Otto Kekäläinen <otto@kekalainen.net> wrote:
These should actually be 64-bit and therefore enable innodb_log_file_mmap. Is your configuration check tripping because of the following?
static constexpr const char *innodb_log_file_mmap_description= "Whether ib_logfile0" # ifdef HAVE_PMEM " resides in persistent memory or" # endif " should initially be memory-mapped";
It is not "my configuration check", it is standard output from mariadbd --verbose --help.
You could compare filtered output, for example like this: sed -e 's/ resides in persistent memory or//' I think that it would be extremely misleading to mention persistent memory on systems where it cannot be supported. That would be all 32-bit ISA as well as a few 64-bit ISA. Actually, there exists an implementation of pmem_persist() for RISC-V and LoongArch, but I did not enable those by default, because we do not have those ISA in our CI. The following code in storage/innobase/CMakeLists.txt would have to be adjusted: IF(CMAKE_SIZEOF_VOID_P EQUAL 8) IF(CMAKE_SYSTEM_PROCESSOR MATCHES "(aarch|AARCH|p(ower)?pc|x86_|amd)64") OPTION(WITH_INNODB_PMEM "Support memory-mapped InnoDB redo log" ON) ELSE() # Disable by default on ISA that are not covered by our CI OPTION(WITH_INNODB_PMEM "Support memory-mapped InnoDB redo log" OFF) ENDIF() ENDIF() If the regression tests run on /dev/shm (such as by invoking ./mtr --mem), the tests should execute the function pmem_persist(). If you could test this on RISC-V and LoongArch (both seem to be supported architectures in Debian), we could enable this code for them. I double-checked that s390x cannot be supported. https://www.ibm.com/support/pages/ibm-system-z-io-feature-reference-table does not mention any CXL, so it is not likely that any hardware exists. Also, https://gcc.gnu.org/onlinedocs/gcc/S_002f390-System-z-Built-in-Functions.htm... only mentions some instructions for memory transactions, which https://jira.mariadb.org/browse/MDEV-26769 is already making use of.
I agree that most of the time less is more, but in the case of configuration options / system variables, it is extremely confusing to users if an option occasionally exists and occasionally not in the same version of MariaDB build with the same dependencies.
Can you please help me write a patch (apparently for a_innodb.cc) to make the variable will always be visible?
I can try to see if the memory-mapped log parsing can be implemented on 32-bit systems. But, some of those systems can't possibly support pmem_persist(), and hence the documentation string would necessarily differ from those 64-bit platforms that can support it. Marko -- Marko Mäkelä, Lead Developer InnoDB MariaDB plc
I think that it would be extremely misleading to mention persistent memory on systems where it cannot be supported. That would be all 32-bit ISA as well as a few 64-bit ISA.
It think it should be fairly clear to users if the value is innodb-log-file-mmap=false or innodb-log-file-mmap=none. Users are already familiar with innodb-use-native-aio=false on all platforms that do not support it. I still feel strongly that having system variables that appear and disappear without external reasons is confusing to users, and this innodb-log-file-mmap is now the first system variable that has that behaviour. Sure, I can help test pmem support on various platforms. But it will never be 100% on all platforms, so this variable visibility needs to be addressed separately. Also, just to confirm, you recent work on pmem features means that pmem is still intended to be in MariaDB and the removal in https://github.com/MariaDB/server/commit/8e663f5e9041e8e0998c792bf8d0c297a8b... has been reverted? I was about to follow suite in Debian in https://salsa.debian.org/mariadb-team/mariadb-server/-/merge_requests/86 but I guess I should not anymore? - Otto
Hi Otto, On Fri, Jan 3, 2025 at 6:19 AM Otto Kekäläinen <otto@kekalainen.net> wrote:
It think it should be fairly clear to users if the value is innodb-log-file-mmap=false or innodb-log-file-mmap=none.
Users are already familiar with innodb-use-native-aio=false on all platforms that do not support it. I still feel strongly that having system variables that appear and disappear without external reasons is confusing to users, and this innodb-log-file-mmap is now the first system variable that has that behaviour.
Sure, I can help test pmem support on various platforms. But it will never be 100% on all platforms, so this variable visibility needs to be addressed separately.
OK. What would be the procedure to get a particular branch or commit tested on Debian Salsa CI?
Also, just to confirm, you recent work on pmem features means that pmem is still intended to be in MariaDB and the removal in https://github.com/MariaDB/server/commit/8e663f5e9041e8e0998c792bf8d0c297a8b... has been reverted?
Like that commit message says, the PMEM support was only removed from the 10.6 branch, where it made practically no performance difference. MDEV-14425 (originally in the 10.8 short-term-support release) introduced a more scalable log file format that is designed for PMEM. When I tested it on a PMEM device, I think that I observed a 15% improvement in throughput in a write heavy benchmark. While both Intel and Micron have discontinued their NVDIMM product lines, I think that the technology makes sense and hope that some options will remain. One option might be https://www.rambus.com/blogs/compute-express-link/; the blog post is a year old.
I was about to follow suite in Debian in https://salsa.debian.org/mariadb-team/mariadb-server/-/merge_requests/86 but I guess I should not anymore?
A dependency on libpmem was removed from all MariaDB Server versions, so anything related to that should also be removed from any Debian downstream packaging. InnoDB only ever needed pmem_persist() from that library, to control the flushing of some cache lines. Marko -- Marko Mäkelä, Lead Developer InnoDB MariaDB plc
Hi!
It think it should be fairly clear to users if the value is innodb-log-file-mmap=false or innodb-log-file-mmap=none.
Users are already familiar with innodb-use-native-aio=false on all platforms that do not support it. I still feel strongly that having system variables that appear and disappear without external reasons is confusing to users, and this innodb-log-file-mmap is now the first system variable that has that behaviour.
Sure, I can help test pmem support on various platforms. But it will never be 100% on all platforms, so this variable visibility needs to be addressed separately.
OK. What would be the procedure to get a particular branch or commit tested on Debian Salsa CI?
salsa.debian.org is open for anyone, so if you want, you can register an account, fork, push changes etc and see the Salsa CI run. In this case however it only runs for amd64. The issue I describe is only visible on other architectures after upload to Debian archives, which triggered ci.debian.net runs (which is separate from Salsa CI). It is probably just easier that you push email me a patch or share me an url where I can download it, and I can do the testing for you.
Also, just to confirm, you recent work on pmem features means that pmem is still intended to be in MariaDB and the removal in https://github.com/MariaDB/server/commit/8e663f5e9041e8e0998c792bf8d0c297a8b... has been reverted?
Like that commit message says, the PMEM support was only removed from the 10.6 branch, where it made practically no performance difference.
Yes it mentions that but the message isn't clear about what it is referring to by "PMEM support", and metadata shows the commit was merged on all branches (as everything in MariaDB is). ..
I was about to follow suite in Debian in https://salsa.debian.org/mariadb-team/mariadb-server/-/merge_requests/86 but I guess I should not anymore?
A dependency on libpmem was removed from all MariaDB Server versions, so anything related to that should also be removed from any Debian downstream packaging.
InnoDB only ever needed pmem_persist() from that library, to control the flushing of some cache lines.
Thanks for the clarification. With this additional context I understand now that https://github.com/MariaDB/server/commit/3f9f5ca48e6be55613926964314f550c8ca... re-implemented that function in MariaDB and thus the pmem library can be removed everywhere, but that MariaDB continues to use pmem kernel API in 10.11+.
Hi Otto, On Sun, Jan 5, 2025 at 1:43 AM Otto Kekäläinen <otto@kekalainen.net> wrote:
It is probably just easier that you push email me a patch or share me an url where I can download it, and I can do the testing for you.
I just created https://github.com/MariaDB/server/pull/3732 with the following of 2 commits, which are currently also available in the 10.11-MDEV-35785 branch: https://github.com/MariaDB/server/pull/3732/commits/10f0f36457852075af5ce784... https://github.com/MariaDB/server/pull/3732/commits/4268d64587b10e4b8dce2b50...
Thanks for the clarification. With this additional context I understand now that https://github.com/MariaDB/server/commit/3f9f5ca48e6be55613926964314f550c8ca... re-implemented that function in MariaDB and thus the pmem library can be removed everywhere, but that MariaDB continues to use pmem kernel API in 10.11+.
The whole point of PMEM is that there is no kernel API, other than the MAP_SYNC flag of the mmap() system call. All that a durable log write takes is writing back some cache lines, which can be initiated with an unprivileged instruction, without any context switch between user and kernel mode being necessary. I'm looking forward to your test results on RISC-V and LoongArch. Marko -- Marko Mäkelä, Lead Developer InnoDB MariaDB plc
It is probably just easier that you push email me a patch or share me an url where I can download it, and I can do the testing for you.
I just created https://github.com/MariaDB/server/pull/3732 with the following of 2 commits, which are currently also available in the 10.11-MDEV-35785 branch: https://github.com/MariaDB/server/pull/3732/commits/10f0f36457852075af5ce784... https://github.com/MariaDB/server/pull/3732/commits/4268d64587b10e4b8dce2b50...
Thanks! Let's continue the discussion in the PR. I will figure out how to best do the testing and post results.
participants (2)
-
Marko Mäkelä -
Otto Kekäläinen