[Maria-developers] Regression in crossbuilding MariaDB
Hello! While working on https://salsa.debian.org/mariadb-team/mariadb-server/-/merge_requests/46 (Import new upstream version 10.11.3 to Debian) I noticed that amd64->arm64 crossbuilding no longer works. The CI shows that Salsa-CI job `test-crossbuild-arm64` no longer passes. Based on Salsa-CI runs I can see that it [still worked in upstream commit e55e761e](https://salsa.debian.org/otto/mariadb-server/-/jobs/4217220) but [failed in upstream commit 06b443be](https://salsa.debian.org/otto/mariadb-server/-/jobs/4217664), thus some of these commits in between regressed it: ``` 06b443be34e Nes CC 3.3 4a668c18926 MDEV-29401 InnoDB history list length increased in 10.6 compared to 10.5 f272463b028 Cleanup of MDEV-14974: --port ignored for --host=localhost bb1d1dc8465 Merge 10.5 into 10.6 c5e50c48bba MDEV-30837: Remove usage of AWK in autobake-debs.sh f99a891858a MDEV-30837: Remove usage of AWK from Debian init and postinst scripts 5740638c4c3 MDEV-31132 Deadlock between DDL and purge of InnoDB history d4265fbde58 MDEV-26055: Correct the formula for adaptive flushing 898320b5f8a MDEV-30804 addendum for 10.6+ branches c22ab93f8af MDEV-26827 fixup: Prevent a hang in LRU eviction 818d5e48146 Merge 10.5 into 10.6 0976afec889 MDEV-31114 Assertion !...is_waiting() failed in os_aio_wait_until_no_pending_writes() 2c567b2fa37 MDEV-30996 insert.. select in presence of full text index freezes all other commits at commit time 51e62cb3b3b MDEV-26782 InnoDB temporary tablespace: reclaiming of free space does not work 204e7225dce Cleanup: MONITOR_EXISTING trx_undo_slots_used, trx_undo_slots_cached 86767bcc0f1 MDEV-29593 Purge misses a chance to free not-yet-reused undo pages 40eff3f8681 MDEV-26827 fixup: hangs and !os_aio_pending_writes() assertion failures ``` Example of error in (cross)build log: ``` Building CXX object CMakeFiles/cmTC_9b8d6.dir/src.cxx.o /usr/lib/ccache/c++ -DHAVE_INCONSISTENT_MISSING_DESTRUCTOR_OVERRIDE -g -O2 -ffile-prefix-map=/builds/otto/mariadb-server/debian/output/source_dir=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -Wdate-time -D_FORTIFY_SOURCE=2 -pie -fPIC -fstack-protector --param=ssp-buffer-size=4 -Wall -Wextra -Woverloaded-virtual -Wconversion -g -Wsuggest-override -Winconsistent-missing-destructor-override -o CMakeFiles/cmTC_9b8d6.dir/src.cxx.o -c /builds/otto/mariadb-server/debian/output/source_dir/builddir-native/CMakeFiles/CMakeScratch/TryCompile-Nn98w6/src.cxx c++: error: unrecognized command-line option '-Winconsistent-missing-destructor-override' gmake[3]: *** [CMakeFiles/cmTC_9b8d6.dir/build.make:78: CMakeFiles/cmTC_9b8d6.dir/src.cxx.o] Error 1 gmake[3]: Leaving directory '/builds/otto/mariadb-server/debian/output/source_dir/builddir-native/CMakeFiles/CMakeScratch/TryCompile-Nn98w6' gmake[2]: *** [Makefile:127: cmTC_9b8d6/fast] Error 2 ``` Is this enough information for somebody to guess what regressed? - Otto
Hi Otto, git grep --recurse inconsistent-missing-destructor-override tells me that the option is present in wsrep-lib/CMakeLists.txt and had been added there about 2 months ago: https://github.com/codership/wsrep-lib/commit/66a8fa9b13447b50f8b5e8bb8d2504... The wsrep-lib submodule was updated a couple of weeks ago in the 10.4 branch, in https://github.com/MariaDB/server/commit/c6ef9b1c1a59dd5dae81888d47f1749e1a4... I hope that this helps. Marko On Tue, May 16, 2023 at 9:20 AM Otto Kekäläinen <otto@kekalainen.net> wrote:
Hello!
While working on https://salsa.debian.org/mariadb-team/mariadb-server/-/merge_requests/46 (Import new upstream version 10.11.3 to Debian) I noticed that amd64->arm64 crossbuilding no longer works.
The CI shows that Salsa-CI job `test-crossbuild-arm64` no longer passes. Based on Salsa-CI runs I can see that it [still worked in upstream commit e55e761e](https://salsa.debian.org/otto/mariadb-server/-/jobs/4217220) but [failed in upstream commit 06b443be](https://salsa.debian.org/otto/mariadb-server/-/jobs/4217664), thus some of these commits in between regressed it:
``` 06b443be34e Nes CC 3.3 4a668c18926 MDEV-29401 InnoDB history list length increased in 10.6 compared to 10.5 f272463b028 Cleanup of MDEV-14974: --port ignored for --host=localhost bb1d1dc8465 Merge 10.5 into 10.6 c5e50c48bba MDEV-30837: Remove usage of AWK in autobake-debs.sh f99a891858a MDEV-30837: Remove usage of AWK from Debian init and postinst scripts 5740638c4c3 MDEV-31132 Deadlock between DDL and purge of InnoDB history d4265fbde58 MDEV-26055: Correct the formula for adaptive flushing 898320b5f8a MDEV-30804 addendum for 10.6+ branches c22ab93f8af MDEV-26827 fixup: Prevent a hang in LRU eviction 818d5e48146 Merge 10.5 into 10.6 0976afec889 MDEV-31114 Assertion !...is_waiting() failed in os_aio_wait_until_no_pending_writes() 2c567b2fa37 MDEV-30996 insert.. select in presence of full text index freezes all other commits at commit time 51e62cb3b3b MDEV-26782 InnoDB temporary tablespace: reclaiming of free space does not work 204e7225dce Cleanup: MONITOR_EXISTING trx_undo_slots_used, trx_undo_slots_cached 86767bcc0f1 MDEV-29593 Purge misses a chance to free not-yet-reused undo pages 40eff3f8681 MDEV-26827 fixup: hangs and !os_aio_pending_writes() assertion failures ```
Example of error in (cross)build log: ``` Building CXX object CMakeFiles/cmTC_9b8d6.dir/src.cxx.o /usr/lib/ccache/c++ -DHAVE_INCONSISTENT_MISSING_DESTRUCTOR_OVERRIDE -g -O2 -ffile-prefix-map=/builds/otto/mariadb-server/debian/output/source_dir=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -Wdate-time -D_FORTIFY_SOURCE=2 -pie -fPIC -fstack-protector --param=ssp-buffer-size=4 -Wall -Wextra -Woverloaded-virtual -Wconversion -g -Wsuggest-override -Winconsistent-missing-destructor-override -o CMakeFiles/cmTC_9b8d6.dir/src.cxx.o -c /builds/otto/mariadb-server/debian/output/source_dir/builddir-native/CMakeFiles/CMakeScratch/TryCompile-Nn98w6/src.cxx c++: error: unrecognized command-line option '-Winconsistent-missing-destructor-override' gmake[3]: *** [CMakeFiles/cmTC_9b8d6.dir/build.make:78: CMakeFiles/cmTC_9b8d6.dir/src.cxx.o] Error 1 gmake[3]: Leaving directory '/builds/otto/mariadb-server/debian/output/source_dir/builddir-native/CMakeFiles/CMakeScratch/TryCompile-Nn98w6' gmake[2]: *** [Makefile:127: cmTC_9b8d6/fast] Error 2 ```
Is this enough information for somebody to guess what regressed?
- Otto
_______________________________________________ Mailing list: https://launchpad.net/~maria-developers Post to : maria-developers@lists.launchpad.net Unsubscribe : https://launchpad.net/~maria-developers More help : https://help.launchpad.net/ListHelp
-- Marko Mäkelä, Lead Developer InnoDB MariaDB plc
Hi, Otto, On May 15, Otto Kekäläinen wrote:
Hello!
While working on https://salsa.debian.org/mariadb-team/mariadb-server/-/merge_requests/46 (Import new upstream version 10.11.3 to Debian) I noticed that amd64->arm64 crossbuilding no longer works.
The CI shows that Salsa-CI job `test-crossbuild-arm64` no longer passes. Based on Salsa-CI runs I can see that it [still worked in upstream commit e55e761e](https://salsa.debian.org/otto/mariadb-server/-/jobs/4217220) but [failed in upstream commit 06b443be](https://salsa.debian.org/otto/mariadb-server/-/jobs/4217664), thus some of these commits in between regressed it:
Here're relevant lines from the log: ... -- Could NOT find ZLIB (missing: ZLIB_LIBRARY) (found version "1.2.11") ... CMake Error: The following variables are used in this project, but they are set to NOTFOUND. Please set them or make sure they are set and tested correctly in the CMake files: ZLIB_LIBRARY ... //Path to a library. ZLIB_LIBRARY_DEBUG:FILEPATH=ZLIB_LIBRARY_DEBUG-NOTFOUND //Path to a library. ZLIB_LIBRARY_RELEASE:FILEPATH=ZLIB_LIBRARY_RELEASE-NOTFOUND ... note that zlib detection is part of cmake, not something we've changed, so it's could be caused by a change in your build environment. Regards, Sergei VP of MariaDB Server Engineering and security@mariadb.org
Hi!
Here're relevant lines from the log: ... -- Could NOT find ZLIB (missing: ZLIB_LIBRARY) (found version "1.2.11") ... CMake Error: The following variables are used in this project, but they are set to NOTFOUND. Please set them or make sure they are set and tested correctly in the CMake files: ZLIB_LIBRARY
Indeed, the configure stage already has errors. I was thrown off by the fact that the build proceeds and has errors a long section without errors and then starts erroring later on.
note that zlib detection is part of cmake, not something we've changed, so it's could be caused by a change in your build environment.
The build environment in the CI is identical for builds done on the same day, so the regression must be in MariaDB 10.11.3. Your pointer was enough to find out the issue, thanks for it! Seems the MariaDB source code has two copies of zlib embedded, and the new one in libmariadb/external/zlib/CMakeLists.txt somehow bleeded into the main MariaDB build and made ZLIB_LIBRARY fail. I already confirmed this in https://salsa.debian.org/otto/mariadb-server/-/pipelines/528997 where I reverted libmariadb to older version. Now I am working on the final fix, which might even be as simple as just deleting both copies of the embedded zlib in MariaDB and MariaDB Connector C sources, as embedding source code of external projects is bad practice anyway. I also noticed that apparently the MySQl build was broken for the same reason and they did this: https://github.com/mysql/mysql-server/commit/1f2b9d62#diff-9c0008f51a59d8b1a... I won't do the same as we really should not be using embedded copies of zlib, but the system zlib, always and everywhere. My fix when I have the final version will be posted as a commit in https://salsa.debian.org/mariadb-team/mariadb-server/-/merge_requests/46. - Otto
Hi Otto, On Thu, May 18, 2023 at 12:14:43AM -0700, Otto Kekäläinen wrote:
Here're relevant lines from the log: ... -- Could NOT find ZLIB (missing: ZLIB_LIBRARY) (found version "1.2.11") ... CMake Error: The following variables are used in this project, but they are set to NOTFOUND. Please set them or make sure they are set and tested correctly in the CMake files: ZLIB_LIBRARY
Indeed, the configure stage already has errors. I was thrown off by the fact that the build proceeds and has errors a long section without errors and then starts erroring later on.
When I looked into the pipeline (thanks for linking it), this also jumped into my eye. Note that this happens during a native build pass and then it really makes sense: There only is a dependency for the host architecture zlib, but none for the build architecture one. If the native build pass needs zlib, you need to duplicate the zlib dependency with a ":native" annotation. Whether that need is real is beyond my understanding though. Hope this helps Helmut
architecture zlib, but none for the build architecture one. If the native build pass needs zlib, you need to duplicate the zlib dependency with a ":native" annotation. Whether that need is real is beyond my understanding though.
Thanks Helmut, I tested this and it worked. I had one build [1] where simply reverting libmariadb to the old version made the crossbuild work but it wasn't consistent. Using :native seems to be the final solution now. This is in 1:10.11.3-1~exp1 and uploaded in experimental. [1] https://salsa.debian.org/otto/mariadb-server/-/jobs/4226182
Hi, Otto, On May 18, Otto Kekäläinen wrote:
Now I am working on the final fix, which might even be as simple as just deleting both copies of the embedded zlib in MariaDB and MariaDB Connector C sources, as embedding source code of external projects is bad practice anyway.
Right, it's also future proof, no more zlib issues ever. Users will get a smaller source download too. bindled zlib is only needed for our bintar builds, never for debs. Regards, Sergei VP of MariaDB Server Engineering and security@mariadb.org
participants (4)
-
Helmut Grohne -
Marko Mäkelä -
Otto Kekäläinen -
Sergei Golubchik