[Maria-developers] CI failing on almost all branches - time to adopt protected branches?
Hello! Has there recently been any discussions about enabling the Github 'protected branch' feature? Review of current mainlines shows that ALL but 10.4 are broken and most of them on different tests. I am afraid that quality and test fixes will never catch up if people are allowed to push code that makes testable things fail. Using protected branches would prevent people from being able to push commits on mainline that fail in CI. https://github.com/MariaDB/server/tree/10.3 / 4c4939b NOT OK: - amd64-debian-10-debug-embedded fails on sql_sequence.kill - amd64-fedora-36 fails on main.func_math https://github.com/MariaDB/server/tree/10.4 / 1767390 OK https://github.com/MariaDB/server/tree/10.5 / ac5a534 NOT OK: - amd64-ubuntu-2004-msan fails on main.selectivity_innodb - amd64-windows fails to compile https://github.com/MariaDB/server/tree/10.6 / dc1d621 NOT OK: - buildbot/amd64-ubuntu-2204-msan fails on innodb.gap_locks https://github.com/MariaDB/server/tree/10.7 / bc656c4 NOT OK: - buildbot/amd64-debian-10 fails to start MTR https://github.com/MariaDB/server/tree/10.8 / dd2fe81 NOT OK: - amd64-ubuntu-2204-msan fails on MTR timeout (crash?) https://github.com/MariaDB/server/tree/10.9 / 55e78eb NOT OK: - buildbot/amd64-ubuntu-2204-msan fails on mariabackup.incremental_page_compressed innodb.innodb_page_compressed https://github.com/MariaDB/server/tree/10.10 / 191821f NOT OK: - amd64-ubuntu-2004-debug fails on binlog.binlog_truncate_multi_engine - amd64-ubuntu-2204-debug-ps fails on main.bad_startup_options https://github.com/MariaDB/server/tree/10.11 / d84a282 NOT OK: - buildbot/amd64-ubuntu-2204-msan fails on mariabackup.incremental_page_compressed innodb.innodb_page_compressed https://github.com/MariaDB/server/tree/11.0 / 8e55d7e NOT OK: - buildbot/amd64-ubuntu-2004-msan fails on innodb.table_definition_cache_debug main.lock_sync - continuous-integration/appveyor/branch fails on main.innodb_ext_key
Hi Otto, all, For what it is worth, the amd64-ubuntu-2204-msan builder is only being set up. Unlike the older clang-13 based builder, it revealed a few surprises, as you can read in https://jira.mariadb.org/browse/MDEV-30936. I helped with fixing trouble related to that, because I heard that there is a plan to make it a mandatory builder for branch protection. I took a deeper look at one of the failures that you pointed to: https://buildbot.mariadb.org/#/builders/561/builds/119 (amd64-ubuntu-2204-msan failure for 10.8). The client-side log ends like this: stress.wrapper w4 [ skipped ] Only to be run with mtr --stress stress.deadlock_drop_table w9 [ pass ] 64 sys_vars.alter_algorithm_basic w4 [ pass ] 18 stress.misc w10 [ pass ] 3252 innodb.innodb_bug52663 'innodb' w5 [ pass ] 50025 innodb_gis.rtree_search 'innodb' w3 [ pass ] 7166 innodb_fts.sync_ddl 'innodb,orig' w2 [ pass ] 15247 innodb.log_corruption 'innodb' w1 [ pass ] 44124 stress.ddl_memory w6 [ pass ] 159732 stress.ddl_myisam w7 [ pass ] 162212 command timed out: 600 seconds without output running [b'bash', b'-xc', b'cd mysql-test && LD_LIBRARY_PATH=/msan-libs MSAN_OPTIONS=abort_on_error=1:poison_in_dtor=0 ./mtr --mem --big-test --force --retry=0 --skip-test=.*compression.* --max-test-fail=100 --parallel=$(expr 5 \\* 2)'], attempting to kill process killed by signal 9 program finished with exit code -1 elapsedTime=2771.840262 After some analysis, I found the log of the worker that is apparently hung: https://ci.mariadb.org/33865/logs/amd64-ubuntu-2204-msan/mysqld.1.err.8 The last test in that log corresponds to the following in the client-side log: innodb.innodb_page_compressed 'innodb,lzma' w8 [ fail ] This test fails because the LZMA library is available in the test environment, but not in MSAN-instrumented form. In MemorySanitizer, any data returned by uninstrumented code will be assumed to be uninitialized. I hope that the CI maintainers will address this failure soon, if it was not done already. After that test failure, that server was restarted for running the test versioning.alter, but there is no mention of that test in the client log, and no sign of an attempted server shutdown either. For this type of test driver timeout, the ticket https://jira.mariadb.org/browse/MDEV-30836 has been filed, because the claimed fix in https://jira.mariadb.org/browse/MDEV-29023 only appears to actually work in the 10.3 branch.
https://github.com/MariaDB/server/tree/10.3 / 4c4939b NOT OK: - amd64-debian-10-debug-embedded fails on sql_sequence.kill - amd64-fedora-36 fails on main.func_math
The undefined behaviour in main.func_math was only fixed starting with the 10.5 branch: https://jira.mariadb.org/browse/MDEV-26645 Maybe we should just disable building 10.3 and 10.4 on Fedora 36 or later?
https://github.com/MariaDB/server/tree/10.5 / ac5a534 NOT OK: - amd64-ubuntu-2004-msan fails on main.selectivity_innodb - amd64-windows fails to compile
Apparently the Windows build has been rerun. The failed run is https://buildbot.mariadb.org/#/builders/234/builds/17303 and it failed due to the following: 195>CUSTOMBUILD : CMake error : Cannot restore timestamp "D:/Buildbot/amd64-windows/build/storage/perfschema/unittest/CMakeFiles/generate.stamp": Access is denied. [D:\Buildbot\amd64-windows\build\storage\perfschema\unittest\pfs_instr-t.vcxproj] This is a mystery error that occurs from time to time.
The 10.7 branch has reached its end of life. The commit that you mention is for the final mariadb-10.7.8 release, February 1. Best regards, Marko
participants (2)
-
Marko Mäkelä
-
Otto Kekäläinen