MTR failures on official Debian buildd after MariaDB 1:10.11.6-1 upload
Hi! I just wanted to let you know there are a bunch of MTR failures/regressions at https://buildd.debian.org/status/package.php?p=mariadb which I need to fix or add to skip-lists. The previous upload of MariaDB 1:10.11.5-3 had more 'green' than ever before, and now there are regressions on 6 archs. [image: image.png]
On Wed, Nov 29, 2023 at 7:40 PM Otto Kekäläinen via developers <developers@lists.mariadb.org> wrote:
Hi!
I just wanted to let you know there are a bunch of MTR failures/regressions at https://buildd.debian.org/status/package.php?p=mariadb which I need to fix or add to skip-lists.
The previous upload of MariaDB 1:10.11.5-3 had more 'green' than ever before, and now there are regressions on 6 archs.
obviously untested arches regress. armhf/powerpc/sparc So the problem is more than slow query in the table mysq.slow_log. Patches welcome. CURRENT_TEST: main.subselect /<<PKGBUILDDIR>>/builddir/client//mariadb-test: Error on delete of '/<<PKGBUILDDIR>>/builddir/mysql-test/var/1/tmp/subselect.out.file.1' (Errcode: 2 "No such file or directory") mysqltest: At line 6003: query 'Select (Select Sum(`TestCase`.Revenue) From mysql.slow_log E Where TestCase.TemplateID not in (Select 1 from mysql.slow_log where 2=2) ) As `ControlRev` From (Select 3 as Revenue, 4 as TemplateID) As `TestCase` Group By TestCase.Revenue, TestCase.TemplateID' failed: ER_SUBQUERY_NO_1_ROW (1242): Subquery returns more than 1 row ppc64le MDEV-30728) Frequent 'InnoDB: Database page corruption on disk' on MariaDB 10.11 in Debian autopktest on arch ppc64el - kernel fix being considered\https://patchwork.ozlabs.org/project/linuxppc-dev/patch/1921539696.48534988.... and is being eagerly watched by Debian's kernel maintainer. main.analyze_format_json hppa: no gethrtime or cycles implementation so no output measures in main.analyze_format_json hurd-i386 same reason it has never worked, and will never work, as detailed last time. sh4 dependency problem sparc something threadpool related. Need arch advice.
_______________________________________________ developers mailing list -- developers@lists.mariadb.org To unsubscribe send an email to developers-leave@lists.mariadb.org
armhf/powerpc/sparc
So the problem is more than slow query in the table mysq.slow_log. Patches welcome.
CURRENT_TEST: main.subselect /<<PKGBUILDDIR>>/builddir/client//mariadb-test: Error on delete of '/<<PKGBUILDDIR>>/builddir/mysql-test/var/1/tmp/subselect.out.file.1' (Errcode: 2 "No such file or directory") mysqltest: At line 6003: query 'Select (Select Sum(`TestCase`.Revenue) From mysql.slow_log E Where TestCase.TemplateID not in (Select 1 from mysql.slow_log where 2=2) ) As `ControlRev` From (Select 3 as Revenue, 4 as TemplateID) As `TestCase` Group By TestCase.Revenue, TestCase.TemplateID' failed: ER_SUBQUERY_NO_1_ROW (1242): Subquery returns more than 1 row
Indeed, these archs show regression in the various subselect tests. I will dig more into this one. Potentially related: https://jira.mariadb.org/browse/MDEV-32514 / https://jira.mariadb.org/browse/MDEV-32518 (which was on 10.11.6 as https://github.com/MariaDB/server/commit/7533062f17912424e1f859c5becaf3bf36b... and https://github.com/MariaDB/server/commit/699cfee595b917ab05a99426bc6e9adbaa3...).
sh4
dependency problem
Comparing latest build log to the previous that passed in 10.11.5-3 I see this line is now present that is different: CMake Error: Generator: execution of make failed. Make command was: /usr/bin/cmake -E env VERBOSE=1 /usr/bin/gmake -f Makefile cmTC_cc6d8/fast -- Looking for sigset - not found https://buildd.debian.org/status/fetch.php?pkg=mariadb&arch=sh4&ver=1%3A10.11.5-3&stamp=1697366222&raw=0 vs https://buildd.debian.org/status/fetch.php?pkg=mariadb&arch=sh4&ver=1%3A10.11.6-1&stamp=1701153270&raw=0
sparc
something threadpool related. Need arch advice.
In https://buildd.debian.org/status/fetch.php?pkg=mariadb&arch=sparc64&ver=1%3A10.11.6-1&stamp=1701155365&raw=0 we see errors like: main.index_merge_innodb 'innodb' w16 [ retry-fail ] Found warnings/errors in server log file! Test ended at 2023-11-28 07:04:20 line Warning : threadpool thread could not be created :Bad address,current number of threads in pool 5 I suspect it is most likely related to https://jira.mariadb.org/browse/MDEV-30728 as that same test has been failing on multiple architectures randomly for a long time.
On Thu, Nov 30, 2023 at 5:29 PM Otto Kekäläinen <otto@kekalainen.net> wrote:
armhf/powerpc/sparc
So the problem is more than slow query in the table mysq.slow_log. Patches welcome.
CURRENT_TEST: main.subselect /<<PKGBUILDDIR>>/builddir/client//mariadb-test: Error on delete of '/<<PKGBUILDDIR>>/builddir/mysql-test/var/1/tmp/subselect.out.file.1' (Errcode: 2 "No such file or directory") mysqltest: At line 6003: query 'Select (Select Sum(`TestCase`.Revenue) From mysql.slow_log E Where TestCase.TemplateID not in (Select 1 from mysql.slow_log where 2=2) ) As `ControlRev` From (Select 3 as Revenue, 4 as TemplateID) As `TestCase` Group By TestCase.Revenue, TestCase.TemplateID' failed: ER_SUBQUERY_NO_1_ROW (1242): Subquery returns more than 1 row
Indeed, these archs show regression in the various subselect tests. I will dig more into this one.
Potentially related: https://jira.mariadb.org/browse/MDEV-32514 / https://jira.mariadb.org/browse/MDEV-32518 (which was on 10.11.6 as https://github.com/MariaDB/server/commit/7533062f17912424e1f859c5becaf3bf36b... and https://github.com/MariaDB/server/commit/699cfee595b917ab05a99426bc6e9adbaa3...).
possibly fixed, yes it was a good one to raise. https://github.com/MariaDB/server/pull/2883 MDEV-32843) main.subselect, main.subselect_no_exists_to_in often fails on CI with ER_SUBQUERY_NO_1_ROW
sh4
dependency problem
Comparing latest build log to the previous that passed in 10.11.5-3 I see this line is now present that is different:
CMake Error: Generator: execution of make failed. Make command was: /usr/bin/cmake -E env VERBOSE=1 /usr/bin/gmake -f Makefile cmTC_cc6d8/fast -- Looking for sigset - not found
sounds like an environmental problem.
https://buildd.debian.org/status/fetch.php?pkg=mariadb&arch=sh4&ver=1%3A10.11.5-3&stamp=1697366222&raw=0 vs https://buildd.debian.org/status/fetch.php?pkg=mariadb&arch=sh4&ver=1%3A10.11.6-1&stamp=1701153270&raw=0
sparc
something threadpool related. Need arch advice.
In https://buildd.debian.org/status/fetch.php?pkg=mariadb&arch=sparc64&ver=1%3A10.11.6-1&stamp=1701155365&raw=0 we see errors like:
main.index_merge_innodb 'innodb' w16 [ retry-fail ] Found warnings/errors in server log file! Test ended at 2023-11-28 07:04:20 line Warning : threadpool thread could not be created :Bad address,current number of threads in pool 5
I suspect it is most likely related to https://jira.mariadb.org/browse/MDEV-30728 as that same test has been failing on multiple architectures randomly for a long time.
It hasn't failed on non-ppc64le for quite a while https://buildbot.mariadb.org/cr/?branch=10.6&revision=&platform=&dt=&bbnum=&typ=&info=&test_name=main.index_merge_innodb&test_variant=&info_text=&failure_text=&limit=100#
Oddly, 4 out of these 6 that regressed passed now in a rebuild, so they were sporadic: https://buildd.debian.org/status/package.php?p=mariadb
Hi! Currently MariaDB 10.11.6 is in Debian unstable/testing and the Debian CI at https://ci.debian.net/packages/m/mariadb/ is red mostly due to this which has a known patch: https://jira.mariadb.org/browse/MDEV-32843
https://github.com/MariaDB/server/pull/2883 MDEV-32843) main.subselect, main.subselect_no_exists_to_in often fails on CI with ER_SUBQUERY_NO_1_ROW
Additionally many archs (but not all) seem to fail on: main.ssl_crl 'path' w7 [ fail ] ... -ERROR 2026 (HY000): TLS/SSL error: sslv3 alert certificate revoked +ERROR 2026 (HY000): TLS/SSL error: ssl/tls alert certificate revoked Full example: https://ci.debian.net/packages/m/mariadb/unstable/amd64/41032421/ Does anyone know what this is off the top of their head?
Hi, Otto, On Dec 17, Otto Kekäläinen via developers wrote:
Additionally many archs (but not all) seem to fail on:
main.ssl_crl 'path' w7 [ fail ] ... -ERROR 2026 (HY000): TLS/SSL error: sslv3 alert certificate revoked +ERROR 2026 (HY000): TLS/SSL error: ssl/tls alert certificate revoked
Does anyone know what this is off the top of their head?
It's the commit https://github.com/openssl/openssl/commit/81b741f68984 in the openssl sources. We need to adjust the test, apparently. Regards, Sergei Chief Architect, MariaDB Server and security@mariadb.org
participants (3)
-
Daniel Black -
Otto Kekäläinen -
Sergei Golubchik