[Maria-developers] Old buildbot 10.5 branch failure review
Hello! I noticed there are quite a lot of upgrade test failures on the 10.5 branch, which is a stable release and should definitely be all green all the time. Here is quick review of those issues with suggested fixes. Maybe Daniel or Elena can do them? They are all easy ones and would make 7 currently red CI steps turn into green. ## Overview: http://buildbot.askmonty.org/buildbot/grid?branch=10.5&category=main&category=experimental ## kvm-deb-buster-amd6 minor-upgrade-all (http://buildbot.askmonty.org/buildbot/builders/kvm-deb-buster-amd64/builds/3...): ++ diff -U1000 /home/buildbot/mariadb.org-tools-master/buildbot/baselines/ldd.10.5.buster.amd64 /home/buildbot/ldd.new --- /home/buildbot/mariadb.org-tools-master/buildbot/baselines/ldd.10.5.buster.amd64 2021-03-16 21:03:04.000000000 -0400 +++ /home/buildbot/ldd.new 2021-03-17 17:23:42.066634818 -0400 - liblz4.so.1 => /usr/lib/x86_64-linux-gnu/liblz4.so.1 Seems some dependency stopped including liblz4. This didn't affect MariaDB. This is a valid change and the fix is to "reset" the ldd.10.5.buster.amd64 file on buildbot.askmonty.org so that there would be no diff no more. ## kvm-deb-buster-amd64 minor-upgrade-all - liblz4.so.1 => /usr/lib/x86_64-linux-gnu/liblz4.so.1 Same as above. ## kvm-deb-sid-x86 - minor-upgrade-all and ## kvm-deb-sid-amd64 - minor-upgrade-all: - libhogweed.so.5 => /lib/x86_64-linux-gnu/libhogweed.so.5 + libhogweed.so.6 => /lib/x86_64-linux-gnu/libhogweed.so.6 - libnettle.so.7 => /lib/x86_64-linux-gnu/libnettle.so.7 + libnettle.so.8 => /lib/x86_64-linux-gnu/libnettle.so.8 Similar issue to buster minor-upgrade-all failure. This is a valid dependency change and the fix is to "reset" the ldd.10.5.sid.amd64 and ldd.10.5.sid.x86 filse on buildbot.askmonty.org so that there would be no diff no more. ## kvm-zyp-opensuse150-amd64 minor-upgrade-all: --- /tmp/ldd.baseline 2021-03-17 19:42:17.468328791 +0000 +++ /home/buildbot/ldd.new 2021-03-17 19:42:16.928328910 +0000 === /usr/sbin/mysqld === /usr/bin/mysql + libedit.so.0 => /usr/lib64/libedit.so.0 === /usr/bin/mysqlaccess === /usr/bin/mysqladmin === /usr/bin/mysqlbinlog === /usr/bin/mysqlcheck === /usr/bin/mysql_client_test === /usr/bin/mysql_client_test_embedded === /usr/bin/mysql_config === /usr/bin/mysql_convert_table_format === /usr/bin/mysqld_multi === /usr/bin/mysqld_safe === /usr/bin/mysqld_safe_helper === /usr/bin/mysqldump === /usr/bin/mysqldumpslow === /usr/bin/mysql_embedded + libedit.so.0 => /usr/lib64/libedit.so.0 Seems legit, library has correct path now. Fix by "resetting" the ldd.baseline. # kvm-zyp-sles150-amd64 minor-upgrade-all: --- /tmp/ldd.baseline 2021-03-17 20:59:56.412361298 +0000 +++ /home/buildbot/ldd.new 2021-03-17 20:59:56.024361282 +0000 === /usr/sbin/mysqld - libpcre2-8.so.0 => /usr/lib64/libpcre2-8.so.0 === /usr/bin/mysql === /usr/bin/mysqlaccess === /usr/bin/mysqladmin === /usr/bin/mysqlbinlog === /usr/bin/mysqlcheck === /usr/bin/mysql_client_test === /usr/bin/mysql_client_test_embedded - libpcre2-8.so.0 => /usr/lib64/libpcre2-8.so.0 === /usr/bin/mysql_config === /usr/bin/mysql_convert_table_format === /usr/bin/mysqld_multi === /usr/bin/mysqld_safe === /usr/bin/mysqld_safe_helper === /usr/bin/mysqldump === /usr/bin/mysqldumpslow === /usr/bin/mysql_embedded - libpcre2-8.so.0 => /usr/lib64/libpcre2-8.so.0 === /usr/bin/mysql_find_rows === /usr/bin/mysql_fix_extensions === /usr/bin/mysqlhotcopy === /usr/bin/mysqlimport === /usr/bin/mysql_install_db === /usr/bin/mysql_ldb === /usr/bin/mysql_plugin === /usr/bin/mysql_secure_installation === /usr/bin/mysql_setpermission === /usr/bin/mysqlshow === /usr/bin/mysqlslap === /usr/bin/mysqltest - libpcre2-8.so.0 => /usr/lib64/libpcre2-8.so.0 - libpcre2-posix.so.2 => /usr/lib64/libpcre2-posix.so.2 === /usr/bin/mysqltest_embedded - libpcre2-8.so.0 => /usr/lib64/libpcre2-8.so.0 - libpcre2-posix.so.2 => /usr/lib64/libpcre2-posix.so.2 Seems legit, library has correct path now. Fix by "resetting" the ldd.baseline. ## kvm-deb-sid-amd64 minor-upgrade-columnstore: The following packages have unmet dependencies: mariadb-plugin-columnstore : Depends: mariadb-server-10.5 (= 1:10.5.8+maria~sid) but 1:10.5.9+maria~sid is to be installed E: Unable to correct problems, you have held broken packages. The mariadb-plugin-columnstore binary packages don't exist for kvm-deb-sid-amd64, as they were skipped when making the 10.5.9 release. Either build them or disable the test until 10.5.10 is out and they are again built. By disabling the test there would be less "red" and less time wasted by developers checking the red ones on why their commits (potentially) fail on CI. Thanks!
Hi Otto, On 3/18/21 8:50 AM, Otto Kekäläinen wrote:
Hello!
I noticed there are quite a lot of upgrade test failures on the 10.5 branch, which is a stable release and should definitely be all green all the time.
Here is quick review of those issues with suggested fixes. Maybe Daniel or Elena can do them?
They are all easy ones and would make 7 currently red CI steps turn into green.
## Overview: http://buildbot.askmonty.org/buildbot/grid?branch=10.5&category=main&category=experimental
## kvm-deb-buster-amd6 minor-upgrade-all (http://buildbot.askmonty.org/buildbot/builders/kvm-deb-buster-amd64/builds/3...):
++ diff -U1000 /home/buildbot/mariadb.org-tools-master/buildbot/baselines/ldd.10.5.buster.amd64 /home/buildbot/ldd.new --- /home/buildbot/mariadb.org-tools-master/buildbot/baselines/ldd.10.5.buster.amd64 2021-03-16 21:03:04.000000000 -0400 +++ /home/buildbot/ldd.new 2021-03-17 17:23:42.066634818 -0400 - liblz4.so.1 => /usr/lib/x86_64-linux-gnu/liblz4.so.1
Seems some dependency stopped including liblz4. This didn't affect MariaDB. This is a valid change and the fix is to "reset" the ldd.10.5.buster.amd64 file on buildbot.askmonty.org so that there would be no diff no more.
It says that ha_mroonga.so which was linking with liblz4 isn't anymore. I don't know why it happened, and maybe it's legit this time, but this is exactly the situation for which this whole check was added, and mroonga seems to have a lot of code related to lz4; so I will need someone's (probably Sergei's) confirmation that it's expected before I change the baseline. Or maybe Daniel knows why it has happened, that will also be sufficient.
## kvm-deb-buster-amd64 minor-upgrade-all - liblz4.so.1 => /usr/lib/x86_64-linux-gnu/liblz4.so.1
Same as above.
Same as above.
## kvm-deb-sid-x86 - minor-upgrade-all and ## kvm-deb-sid-amd64 - minor-upgrade-all:
- libhogweed.so.5 => /lib/x86_64-linux-gnu/libhogweed.so.5 + libhogweed.so.6 => /lib/x86_64-linux-gnu/libhogweed.so.6 - libnettle.so.7 => /lib/x86_64-linux-gnu/libnettle.so.7 + libnettle.so.8 => /lib/x86_64-linux-gnu/libnettle.so.8
Similar issue to buster minor-upgrade-all failure. This is a valid dependency change and the fix is to "reset" the ldd.10.5.sid.amd64 and ldd.10.5.sid.x86 filse on buildbot.askmonty.org so that there would be no diff no more.
I have now disabled a failure upon this check for sid. There is no point doing it on a system where libraries change easily by design. For curious minds, the diff will still be in the output, and a warning will occur.
## kvm-zyp-opensuse150-amd64 minor-upgrade-all:
--- /tmp/ldd.baseline 2021-03-17 19:42:17.468328791 +0000 +++ /home/buildbot/ldd.new 2021-03-17 19:42:16.928328910 +0000 === /usr/sbin/mysqld === /usr/bin/mysql + libedit.so.0 => /usr/lib64/libedit.so.0 === /usr/bin/mysqlaccess === /usr/bin/mysqladmin === /usr/bin/mysqlbinlog === /usr/bin/mysqlcheck === /usr/bin/mysql_client_test === /usr/bin/mysql_client_test_embedded === /usr/bin/mysql_config === /usr/bin/mysql_convert_table_format === /usr/bin/mysqld_multi === /usr/bin/mysqld_safe === /usr/bin/mysqld_safe_helper === /usr/bin/mysqldump === /usr/bin/mysqldumpslow === /usr/bin/mysql_embedded + libedit.so.0 => /usr/lib64/libedit.so.0
Seems legit, library has correct path now. Fix by "resetting" the ldd.baseline.
It is not about changing the path, it says that previously mysql and mysql_embedded weren't linked with libedit and now they are. Same note about Sergei's or Daniel's confirmation applies.
# kvm-zyp-sles150-amd64 minor-upgrade-all:
--- /tmp/ldd.baseline 2021-03-17 20:59:56.412361298 +0000 +++ /home/buildbot/ldd.new 2021-03-17 20:59:56.024361282 +0000 === /usr/sbin/mysqld - libpcre2-8.so.0 => /usr/lib64/libpcre2-8.so.0 === /usr/bin/mysql === /usr/bin/mysqlaccess === /usr/bin/mysqladmin === /usr/bin/mysqlbinlog === /usr/bin/mysqlcheck === /usr/bin/mysql_client_test === /usr/bin/mysql_client_test_embedded - libpcre2-8.so.0 => /usr/lib64/libpcre2-8.so.0 === /usr/bin/mysql_config === /usr/bin/mysql_convert_table_format === /usr/bin/mysqld_multi === /usr/bin/mysqld_safe === /usr/bin/mysqld_safe_helper === /usr/bin/mysqldump === /usr/bin/mysqldumpslow === /usr/bin/mysql_embedded - libpcre2-8.so.0 => /usr/lib64/libpcre2-8.so.0 === /usr/bin/mysql_find_rows === /usr/bin/mysql_fix_extensions === /usr/bin/mysqlhotcopy === /usr/bin/mysqlimport === /usr/bin/mysql_install_db === /usr/bin/mysql_ldb === /usr/bin/mysql_plugin === /usr/bin/mysql_secure_installation === /usr/bin/mysql_setpermission === /usr/bin/mysqlshow === /usr/bin/mysqlslap === /usr/bin/mysqltest - libpcre2-8.so.0 => /usr/lib64/libpcre2-8.so.0 - libpcre2-posix.so.2 => /usr/lib64/libpcre2-posix.so.2 === /usr/bin/mysqltest_embedded - libpcre2-8.so.0 => /usr/lib64/libpcre2-8.so.0 - libpcre2-posix.so.2 => /usr/lib64/libpcre2-posix.so.2
Seems legit, library has correct path now. Fix by "resetting" the ldd.baseline.
It is not about a different path, it's about losing libpcre from the list. Same note about needing a confirmation applies.
## kvm-deb-sid-amd64 minor-upgrade-columnstore:
The following packages have unmet dependencies: mariadb-plugin-columnstore : Depends: mariadb-server-10.5 (= 1:10.5.8+maria~sid) but 1:10.5.9+maria~sid is to be installed E: Unable to correct problems, you have held broken packages.
The mariadb-plugin-columnstore binary packages don't exist for kvm-deb-sid-amd64, as they were skipped when making the 10.5.9 release. Either build them or disable the test until 10.5.10 is out and they are again built. By disabling the test there would be less "red" and less time wasted by developers checking the red ones on why their commits (potentially) fail on CI.
Disabled together with the ldd check/fail on Sid. I have restarted one of previously failed builds to make sure the change works as expected, will see the result tomorrow. Regards, Elena
Thanks!
On Fri, Mar 19, 2021 at 11:24 AM Elena Stepanova <elenst@mariadb.com> wrote:
Hi Otto,
On 3/18/21 8:50 AM, Otto Kekäläinen wrote:
Hello!
I noticed there are quite a lot of upgrade test failures on the 10.5 branch, which is a stable release and should definitely be all green all the time.
Here is quick review of those issues with suggested fixes. Maybe Daniel or Elena can do them?
They are all easy ones and would make 7 currently red CI steps turn into green.
## Overview:
http://buildbot.askmonty.org/buildbot/grid?branch=10.5&category=main&category=experimental
## kvm-deb-buster-amd6 minor-upgrade-all (
http://buildbot.askmonty.org/buildbot/builders/kvm-deb-buster-amd64/builds/3... ):
++ diff -U1000
/home/buildbot/mariadb.org-tools-master/buildbot/baselines/ldd.10.5.buster.amd64
/home/buildbot/ldd.new --- /home/buildbot/mariadb.org-tools-master/buildbot/baselines/ldd.10.5.buster.amd64 2021-03-16 21:03:04.000000000 -0400 +++ /home/buildbot/ldd.new 2021-03-17 17:23:42.066634818 -0400 - liblz4.so.1 => /usr/lib/x86_64-linux-gnu/liblz4.so.1
Seems some dependency stopped including liblz4. This didn't affect MariaDB. This is a valid change and the fix is to "reset" the ldd.10.5.buster.amd64 file on buildbot.askmonty.org so that there would be no diff no more.
It says that ha_mroonga.so which was linking with liblz4 isn't anymore. I don't know why it happened, and maybe it's legit this time, but this is exactly the situation for which this whole check was added, and mroonga seems to have a lot of code related to lz4; so I will need someone's (probably Sergei's) confirmation that it's expected before I change the baseline. Or maybe Daniel knows why it has happened, that will also be sufficient.
I did a brief look yesterday. It seems parts of mroonga are configured to use lz4. So it seems like a legitimate failure, however I haven't looked at why it stopped finding /linking against it.
Hi, Daniel! On Mar 19, Elena Stepanova wrote:
## kvm-deb-buster-amd6 minor-upgrade-all (http://buildbot.askmonty.org/buildbot/builders/kvm-deb-buster-amd64/builds/3...):
++ diff -U1000 /home/buildbot/mariadb.org-tools-master/buildbot/baselines/ldd.10.5.buster.amd64 /home/buildbot/ldd.new --- /home/buildbot/mariadb.org-tools-master/buildbot/baselines/ldd.10.5.buster.amd64 2021-03-16 21:03:04.000000000 -0400 +++ /home/buildbot/ldd.new 2021-03-17 17:23:42.066634818 -0400 - liblz4.so.1 => /usr/lib/x86_64-linux-gnu/liblz4.so.1
It says that ha_mroonga.so which was linking with liblz4 isn't anymore. I don't know why it happened, and maybe it's legit this time, but this is exactly the situation for which this whole check was added, and mroonga seems to have a lot of code related to lz4; so I will need someone's (probably Sergei's) confirmation that it's expected before I change the baseline. Or maybe Daniel knows why it has happened, that will also be sufficient.
In the build from Fri Mar 12 09:53:25 2021 mroonga uses liblz4, in the next build from Tue Mar 16 15:41:50 2021 it no longer does. The commit changes are unrelated, but the timestamp on the vm-buster-amd64-build.qcow2 is Mar 12 21:39. There must've been some change in the VM. And as a side effect, pkg-config was removed.
## kvm-zyp-opensuse150-amd64 minor-upgrade-all:
--- /tmp/ldd.baseline 2021-03-17 19:42:17.468328791 +0000 +++ /home/buildbot/ldd.new 2021-03-17 19:42:16.928328910 +0000 === /usr/sbin/mysqld === /usr/bin/mysql + libedit.so.0 => /usr/lib64/libedit.so.0 === /usr/bin/mysqlaccess === /usr/bin/mysqladmin === /usr/bin/mysqldumpslow === /usr/bin/mysql_embedded + libedit.so.0 => /usr/lib64/libedit.so.0
It is not about changing the path, it says that previously mysql and mysql_embedded weren't linked with libedit and now they are. Same note about Sergei's or Daniel's confirmation applies.
The difference has happened between the same two commits. libedit-devel was installed, for some reason.
# kvm-zyp-sles150-amd64 minor-upgrade-all:
--- /tmp/ldd.baseline 2021-03-17 20:59:56.412361298 +0000 +++ /home/buildbot/ldd.new 2021-03-17 20:59:56.024361282 +0000 === /usr/sbin/mysqld - libpcre2-8.so.0 => /usr/lib64/libpcre2-8.so.0 === /usr/bin/mysql === /usr/bin/mysqlaccess === /usr/bin/mysqladmin === /usr/bin/mysqlbinlog === /usr/bin/mysqlcheck === /usr/bin/mysql_client_test === /usr/bin/mysql_client_test_embedded - libpcre2-8.so.0 => /usr/lib64/libpcre2-8.so.0 === /usr/bin/mysql_config === /usr/bin/mysql_convert_table_format === /usr/bin/mysqld_multi === /usr/bin/mysqld_safe === /usr/bin/mysqld_safe_helper === /usr/bin/mysqldump === /usr/bin/mysqldumpslow === /usr/bin/mysql_embedded - libpcre2-8.so.0 => /usr/lib64/libpcre2-8.so.0 === /usr/bin/mysql_find_rows === /usr/bin/mysql_fix_extensions === /usr/bin/mysqlhotcopy === /usr/bin/mysqlimport === /usr/bin/mysql_install_db === /usr/bin/mysql_ldb === /usr/bin/mysql_plugin === /usr/bin/mysql_secure_installation === /usr/bin/mysql_setpermission === /usr/bin/mysqlshow === /usr/bin/mysqlslap === /usr/bin/mysqltest - libpcre2-8.so.0 => /usr/lib64/libpcre2-8.so.0 - libpcre2-posix.so.2 => /usr/lib64/libpcre2-posix.so.2 === /usr/bin/mysqltest_embedded - libpcre2-8.so.0 => /usr/lib64/libpcre2-8.so.0 - libpcre2-posix.so.2 => /usr/lib64/libpcre2-posix.so.2
It is not about a different path, it's about losing libpcre from the list. Same note about needing a confirmation applies.
And this too, same two commits. This time pcre2-devel has disappeared. Regards, Sergei VP of MariaDB Server Engineering and security@mariadb.org
participants (4)
-
Daniel Black
-
Elena Stepanova
-
Otto Kekäläinen
-
Sergei Golubchik