Hi!
CI bugs are being treated very seriously at the moment via MDEV-33073 always green buildbot, being a Blocker bug that includes all the CI failures that we notice.
Thanks for the reply Daniel, you have always been one of those taking the CI very seriously. The reason I wrote to the developers mailing list is that I wish to raise this for a wider audience and get input from both core contributors and other contributors. For example Trevor (CC'd as I am not sure if he is on this list) filed https://github.com/MariaDB/server/pull/2958 which failed in CI. Since the mainline was already failing ("red") and the PR submission showed lots of failing tests, Trevor had to do a lot of extra work figuring out which tests failed due to his changes, and which ones were already broken (which led to 3 separate PRs now in #3075, #3076 and #3077). I suspect core developers don't suffer from failing CI to the same extent as they simply bypass it, or have much more time on their hands and can spend time learning what failures can be ignored which week and month. The fact that the CI is not green seems to be a topic where the core developers are perhaps a bit blind to the bigger picture, while non-core contributors struggle with the extra work it incurs. Also in the eyes of the wider public, a constantly failing CI erodes trust in quality. While I understand that the natural reply is "we will get to green soon" and it makes a lot of sense, I am afraid it might be a overly optimistic. We've had in the past recurring the situation that Daniel, Sergei and Monty all say the same week they want to fix all failing tests, but it only lasts for a short while and then we are back to failures on mainline CI. Thus, to permanently enforce have CI green on mainline branches I proposed:
I see two approaches to get to consistently green CI:
1) Stop all development and focus on just fixing these, don't continue until CI is fully green, and once it is fully green make the GitHub branch protection settings one notch stricter to not allow any new commits unless the CI is fully green so it never regresses again.
2) Disable these tests and make the rules in GitHub branch protection one notch stricter right away, and not allow any new commits unless the CI is fully green ensuring no new recurring failures are introduced.
What do other developers think about this? - Otto