[Maria-discuss] How can I separate one test from mysql_test
Hi guys,
I've found few particular tests from the test suite that is failing due to the OpenSSL-3 version and I want to separate them and give them to the OpenSSL team in Red Hat.
The idea behind this is to give them a set of commands/config options (reproducer) for each test and they will try to find out what's the exact problem there and help us develop the patch.
The tests that I'm interested in are (there are more of them, I just need to see how to reproduce few of them and I'll do the same for the other ones as well): encryption.corrupted_during_recovery encryption.innodb-checksum-algorithm encryption.innodb-discard-import-change encryption.innodb_page_encryption_key_change binlog_encryption.rpl_loaddata_local binlog_encryption.rpl_parallel_free_deferred_event
IMPORTANT: These tests are failing only with the 'cbc' cipher in use, the 'ctr' is passing.
Could you please help me with how to make a reproducer for each test, so I can create them and give them to the OpenSSL team?
Thank you for any help you can provide Lukas
Hi, Lukas!
Do you use some patches to make mariadb compile with OpenSSL-3? I thought it won't work out of the box.
/Sergei
On Oct 05, Lukas Javorsky wrote:
Hi guys,
I've found few particular tests from the test suite that is failing due to the OpenSSL-3 version and I want to separate them and give them to the OpenSSL team in Red Hat.
The idea behind this is to give them a set of commands/config options (reproducer) for each test and they will try to find out what's the exact problem there and help us develop the patch.
The tests that I'm interested in are (there are more of them, I just need to see how to reproduce few of them and I'll do the same for the other ones as well): encryption.corrupted_during_recovery encryption.innodb-checksum-algorithm encryption.innodb-discard-import-change encryption.innodb_page_encryption_key_change binlog_encryption.rpl_loaddata_local binlog_encryption.rpl_parallel_free_deferred_event
IMPORTANT: These tests are failing only with the 'cbc' cipher in use, the 'ctr' is passing.
Could you please help me with how to make a reproducer for each test, so I can create them and give them to the OpenSSL team?
Thank you for any help you can provide Lukas
-- S pozdravom/ Best regards
Lukáš Javorský
Associate Software Engineer, Core service - Databases
Red Hat https://www.redhat.com
Purkyňova 115 (TPB-C)
612 00 Brno - Královo Pole
ljavorsk@redhat.com https://www.redhat.com
Regards, Sergei VP of MariaDB Server Engineering and security@mariadb.org
Hi Sergei,
We have some very simple patch which helps MariaDB to build with the openssl-3 version, however, the test suite is failing (Failed 56/3396 tests). So that raises a flag, that something may not work as expected, and we want to figure it out.
Speaking of, what's your status on the openssl-3 support in MariaDB Server? Do you have some WIP or estimate in which major version this could be expected?
Thank you Lukas
On Tue, Oct 5, 2021 at 4:41 PM Sergei Golubchik serg@mariadb.org wrote:
Hi, Lukas!
Do you use some patches to make mariadb compile with OpenSSL-3? I thought it won't work out of the box.
/Sergei
On Oct 05, Lukas Javorsky wrote:
Hi guys,
I've found few particular tests from the test suite that is failing due
to
the OpenSSL-3 version and I want to separate them and give them to the OpenSSL team in Red Hat.
The idea behind this is to give them a set of commands/config options (reproducer) for each test and they will try to find out what's the exact problem there and help us develop the patch.
The tests that I'm interested in are (there are more of them, I just need to see how to reproduce few of them and I'll do the same for the other
ones
as well): encryption.corrupted_during_recovery encryption.innodb-checksum-algorithm encryption.innodb-discard-import-change encryption.innodb_page_encryption_key_change binlog_encryption.rpl_loaddata_local binlog_encryption.rpl_parallel_free_deferred_event
IMPORTANT: These tests are failing only with the 'cbc' cipher in use, the 'ctr' is passing.
Could you please help me with how to make a reproducer for each test, so
I
can create them and give them to the OpenSSL team?
Thank you for any help you can provide Lukas
-- S pozdravom/ Best regards
Lukáš Javorský
Associate Software Engineer, Core service - Databases
Red Hat https://www.redhat.com
Purkyňova 115 (TPB-C)
612 00 Brno - Královo Pole
ljavorsk@redhat.com https://www.redhat.com
Regards, Sergei VP of MariaDB Server Engineering and security@mariadb.org
We have some very simple patch which helps MariaDB to build with the
openssl-3 version, however, the test suite is failing (Failed 56/3396 tests). The patch is linked in the JIRA ticket as well: https://jira.mariadb.org/browse/MDEV-25785
On Wed, Oct 6, 2021 at 12:21 PM Lukas Javorsky ljavorsk@redhat.com wrote:
Hi Sergei,
We have some very simple patch which helps MariaDB to build with the openssl-3 version, however, the test suite is failing (Failed 56/3396 tests). So that raises a flag, that something may not work as expected, and we want to figure it out.
Speaking of, what's your status on the openssl-3 support in MariaDB Server? Do you have some WIP or estimate in which major version this could be expected?
Thank you Lukas
On Tue, Oct 5, 2021 at 4:41 PM Sergei Golubchik serg@mariadb.org wrote:
Hi, Lukas!
Do you use some patches to make mariadb compile with OpenSSL-3? I thought it won't work out of the box.
/Sergei
On Oct 05, Lukas Javorsky wrote:
Hi guys,
I've found few particular tests from the test suite that is failing due
to
the OpenSSL-3 version and I want to separate them and give them to the OpenSSL team in Red Hat.
The idea behind this is to give them a set of commands/config options (reproducer) for each test and they will try to find out what's the
exact
problem there and help us develop the patch.
The tests that I'm interested in are (there are more of them, I just
need
to see how to reproduce few of them and I'll do the same for the other
ones
as well): encryption.corrupted_during_recovery encryption.innodb-checksum-algorithm encryption.innodb-discard-import-change encryption.innodb_page_encryption_key_change binlog_encryption.rpl_loaddata_local binlog_encryption.rpl_parallel_free_deferred_event
IMPORTANT: These tests are failing only with the 'cbc' cipher in use,
the
'ctr' is passing.
Could you please help me with how to make a reproducer for each test,
so I
can create them and give them to the OpenSSL team?
Thank you for any help you can provide Lukas
-- S pozdravom/ Best regards
Lukáš Javorský
Associate Software Engineer, Core service - Databases
Red Hat https://www.redhat.com
Purkyňova 115 (TPB-C)
612 00 Brno - Královo Pole
ljavorsk@redhat.com https://www.redhat.com
Regards, Sergei VP of MariaDB Server Engineering and security@mariadb.org
-- S pozdravom/ Best regards
Lukáš Javorský
Associate Software Engineer, Core service - Databases
Red Hat https://www.redhat.com
Purkyňova 115 (TPB-C)
612 00 Brno - Královo Pole
ljavorsk@redhat.com https://www.redhat.com
Hi, Lukas!
Sorry, for not replying earlier.
There is no WIP or estimate yes. Only an understanding that we have to do it, it's unavoidable, and highly likely we'll need to do it in existing versions, not only in a new development branch.
And Connector/C already supports OpenSSL 3.0 (CONC-503), so we have a reasonably good idea of what needs to be done.
On Oct 06, Lukas Javorsky wrote:
Hi Sergei,
We have some very simple patch which helps MariaDB to build with the openssl-3 version, however, the test suite is failing (Failed 56/3396 tests). So that raises a flag, that something may not work as expected, and we want to figure it out.
Speaking of, what's your status on the openssl-3 support in MariaDB Server? Do you have some WIP or estimate in which major version this could be expected?
Thank you Lukas
Regards, Sergei VP of MariaDB Server Engineering and security@mariadb.org
Hi Sergei,
Thank you for letting me know. Yes, I have noticed that Connector/C is already supporting OpenSSL-3.0, so I was waiting for Server 10.6 (as compiled against Connector/C 3.2) will be supporting it as well.
And back to my original question, could you help me (maybe where to look) with the reproducer for the tests mentioned above?
Lukas
On Mon, Oct 11, 2021 at 11:53 PM Sergei Golubchik serg@mariadb.org wrote:
Hi, Lukas!
Sorry, for not replying earlier.
There is no WIP or estimate yes. Only an understanding that we have to do it, it's unavoidable, and highly likely we'll need to do it in existing versions, not only in a new development branch.
And Connector/C already supports OpenSSL 3.0 (CONC-503), so we have a reasonably good idea of what needs to be done.
On Oct 06, Lukas Javorsky wrote:
Hi Sergei,
We have some very simple patch which helps MariaDB to build with the openssl-3 version, however, the test suite is failing (Failed 56/3396 tests). So that raises a flag, that something may not work as expected, and we want to figure it out.
Speaking of, what's your status on the openssl-3 support in MariaDB Server? Do you have some WIP or estimate in which major version this could be expected?
Thank you Lukas
Regards, Sergei VP of MariaDB Server Engineering and security@mariadb.org
Hi, Lukas!
On Oct 05, Lukas Javorsky wrote:
Hi guys,
I've found few particular tests from the test suite that is failing due to the OpenSSL-3 version and I want to separate them and give them to the OpenSSL team in Red Hat.
The idea behind this is to give them a set of commands/config options (reproducer) for each test and they will try to find out what's the exact problem there and help us develop the patch.
The tests that I'm interested in are (there are more of them, I just need to see how to reproduce few of them and I'll do the same for the other ones as well): encryption.corrupted_during_recovery encryption.innodb-checksum-algorithm encryption.innodb-discard-import-change encryption.innodb_page_encryption_key_change binlog_encryption.rpl_loaddata_local binlog_encryption.rpl_parallel_free_deferred_event
IMPORTANT: These tests are failing only with the 'cbc' cipher in use, the 'ctr' is passing.
Could you please help me with how to make a reproducer for each test, so I can create them and give them to the OpenSSL team?
How do these tests fail? What do you need as a reproducer, a small standalone program?
Regards, Sergei VP of MariaDB Server Engineering and security@mariadb.org
Hi Sergei,
How do these tests fail?
For example: Test = encryption.corrupted_during_recovery Error: "mysqltest: At line 25: query 'INSERT INTO t1 VALUES(2)' failed: 1932: Table 'test.t1' doesn't exist in engine"
For the full log you can view it here: https://kojihub.stream.rdu2.redhat.com/kojifiles/work/tasks/7208/367208/buil...
What do you need as a reproducer, a small standalone program?
The best way is to create some kind of script where you can see all of the options that are passed to the server when the service is initialized. And the following SQL commands that are executed to the server (or if the test is not executing any commands in the server, then whatever it's executed after the server is started).
On Tue, Oct 12, 2021 at 10:47 PM Sergei Golubchik serg@mariadb.org wrote:
Hi, Lukas!
On Oct 05, Lukas Javorsky wrote:
Hi guys,
I've found few particular tests from the test suite that is failing due
to
the OpenSSL-3 version and I want to separate them and give them to the OpenSSL team in Red Hat.
The idea behind this is to give them a set of commands/config options (reproducer) for each test and they will try to find out what's the exact problem there and help us develop the patch.
The tests that I'm interested in are (there are more of them, I just need to see how to reproduce few of them and I'll do the same for the other
ones
as well): encryption.corrupted_during_recovery encryption.innodb-checksum-algorithm encryption.innodb-discard-import-change encryption.innodb_page_encryption_key_change binlog_encryption.rpl_loaddata_local binlog_encryption.rpl_parallel_free_deferred_event
IMPORTANT: These tests are failing only with the 'cbc' cipher in use, the 'ctr' is passing.
Could you please help me with how to make a reproducer for each test, so I can create them and give them to the OpenSSL team?
How do these tests fail? What do you need as a reproducer, a small standalone program?
Regards, Sergei VP of MariaDB Server Engineering and security@mariadb.org
Hi, Lukas!
On Oct 13, Lukas Javorsky wrote:
Hi Sergei,
How do these tests fail?
For example: Test = encryption.corrupted_during_recovery Error: "mysqltest: At line 25: query 'INSERT INTO t1 VALUES(2)' failed: 1932: Table 'test.t1' doesn't exist in engine"
For the full log you can view it here: https://kojihub.stream.rdu2.redhat.com/kojifiles/work/tasks/7208/367208/buil...
I cannot see it, the host doesn't even resolve.
Can you share the error log? it's somewhere in var/*/mysqld.1.err and var is normally in mysql-test/ unless you used mysql-test-run.pl --vardir=xxx
What do you need as a reproducer, a small standalone program?
The best way is to create some kind of script where you can see all of the options that are passed to the server when the service is initialized. And the following SQL commands that are executed to the server (or if the test is not executing any commands in the server, then whatever it's executed after the server is started).
But that's just mysql-test-run, you have it. --verbose will show all options that are passed to the server. var/my.cnf is the config file.
One can probably even use ltrace to get all calls into libssl.so, but I personally haven't tried that.
Regards, Sergei VP of MariaDB Server Engineering and security@mariadb.org
Hi Sergei,
I'm sorry for the late reply.
I've attached the mysqld.1.err from '/home/user/encryption/1/log/mysqld.1.err'. I'm not sure how to read those test results, could you please give me more insight into these logs and files? E.g. what are those 1-7 directories?
I'm testing the 'suite/encryption/t/innodb-checksum-algorithm.test'. I'm using the '--verbose' and '--vardir=/home/user/encryption/' options.
The structure is quite unique, and I'm not sure which files could be helpful to me. I've attached the structure to this email (structure.txt) - command "tree /home/user/encryption"
On Wed, Oct 13, 2021 at 3:58 PM Sergei Golubchik serg@mariadb.org wrote:
Hi, Lukas!
On Oct 13, Lukas Javorsky wrote:
Hi Sergei,
How do these tests fail?
For example: Test = encryption.corrupted_during_recovery Error: "mysqltest: At line 25: query 'INSERT INTO t1 VALUES(2)' failed: 1932: Table 'test.t1' doesn't exist in engine"
For the full log you can view it here:
https://kojihub.stream.rdu2.redhat.com/kojifiles/work/tasks/7208/367208/buil...
I cannot see it, the host doesn't even resolve.
Can you share the error log? it's somewhere in var/*/mysqld.1.err and var is normally in mysql-test/ unless you used mysql-test-run.pl --vardir=xxx
What do you need as a reproducer, a small standalone program?
The best way is to create some kind of script where you can see all of
the
options that are passed to the server when the service is initialized. And the following SQL commands that are executed to the server (or if the test is not executing any commands in the server, then whatever it's executed after the server is started).
But that's just mysql-test-run, you have it. --verbose will show all options that are passed to the server. var/my.cnf is the config file.
One can probably even use ltrace to get all calls into libssl.so, but I personally haven't tried that.
Regards, Sergei VP of MariaDB Server Engineering and security@mariadb.org
Hi Sergei,
I've somehow managed to get onto another problem and that is the start of the mariadb.service.
I'm trying to reproduce one of the tests " encryption.innodb-checksum-algorithm"
The error I'm getting (cat /var/log/mariadb/mysqld.1.err) is this: 2021-11-08 4:06:46 0 [Warning] /usr/libexec/mariadbd: unknown variable 'loose-feedback-debug-startup-interval=20' 2021-11-08 4:06:46 0 [Warning] /usr/libexec/mariadbd: unknown variable 'loose-feedback-debug-first-interval=60' 2021-11-08 4:06:46 0 [Warning] /usr/libexec/mariadbd: unknown variable 'loose-feedback-debug-interval=60' 2021-11-08 4:06:46 0 [Warning] /usr/libexec/mariadbd: unknown option '--loose-pam-debug' 2021-11-08 4:06:46 0 [Warning] /usr/libexec/mariadbd: unknown option '--loose-aria' 2021-11-08 4:06:46 0 [Warning] Failed to setup SSL 2021-11-08 4:06:46 0 [Warning] SSL error: SSL_CTX_set_default_verify_paths failed 2021-11-08 4:06:46 0 [Warning] SSL error: error:80000002:system library::No such file or directory 2021-11-08 4:06:46 0 [Warning] SSL error: error:10000080:BIO routines::no such file 2021-11-08 4:06:46 0 [Warning] SSL error: error:05880002:x509 certificate routines::system lib 2021-11-08 4:06:46 0 [Note] Server socket created on IP: '127.0.0.1'. 2021-11-08 4:06:46 0 [ERROR] Can't start server: Bind on TCP/IP port. Got error: 13: Permission denied 2021-11-08 4:06:46 0 [ERROR] Do you already have another mysqld server running on port: 16020 ? 2021-11-08 4:06:46 0 [ERROR] Aborting
Do you have any idea what could cause this? I'm not sure how to go further with this reproducer unless I fix this error.
I would be so grateful for even multiple ideas where the problem could be, so I can check them.
Thank you for your response. Lukas
On Tue, Nov 2, 2021 at 11:08 AM Lukas Javorsky ljavorsk@redhat.com wrote:
Hi Sergei,
I'm sorry for the late reply.
I've attached the mysqld.1.err from '/home/user/encryption/1/log/mysqld.1.err'. I'm not sure how to read those test results, could you please give me more insight into these logs and files? E.g. what are those 1-7 directories?
I'm testing the 'suite/encryption/t/innodb-checksum-algorithm.test'. I'm using the '--verbose' and '--vardir=/home/user/encryption/' options.
The structure is quite unique, and I'm not sure which files could be helpful to me. I've attached the structure to this email (structure.txt) - command "tree /home/user/encryption"
On Wed, Oct 13, 2021 at 3:58 PM Sergei Golubchik serg@mariadb.org wrote:
Hi, Lukas!
On Oct 13, Lukas Javorsky wrote:
Hi Sergei,
How do these tests fail?
For example: Test = encryption.corrupted_during_recovery Error: "mysqltest: At line 25: query 'INSERT INTO t1 VALUES(2)' failed: 1932: Table 'test.t1' doesn't exist in engine"
For the full log you can view it here:
https://kojihub.stream.rdu2.redhat.com/kojifiles/work/tasks/7208/367208/buil...
I cannot see it, the host doesn't even resolve.
Can you share the error log? it's somewhere in var/*/mysqld.1.err and var is normally in mysql-test/ unless you used mysql-test-run.pl --vardir=xxx
What do you need as a reproducer, a small standalone program?
The best way is to create some kind of script where you can see all of
the
options that are passed to the server when the service is initialized. And the following SQL commands that are executed to the server (or if
the
test is not executing any commands in the server, then whatever it's executed after the server is started).
But that's just mysql-test-run, you have it. --verbose will show all options that are passed to the server. var/my.cnf is the config file.
One can probably even use ltrace to get all calls into libssl.so, but I personally haven't tried that.
Regards, Sergei VP of MariaDB Server Engineering and security@mariadb.org
-- S pozdravom/ Best regards
Lukáš Javorský
Associate Software Engineer, Core service - Databases
Red Hat https://www.redhat.com
Purkyňova 115 (TPB-C)
612 00 Brno - Královo Pole
ljavorsk@redhat.com https://www.redhat.com
Hi, Lukas!
This looks very strange. Perhaps, indeed, the previous server process didn't die and is still bound to the port 16020?
This is not very likely, mtr kills all processes with -9, there shouldn't be anything hanging around. But how otherwise 13, "Permission denied" can happen?
By the way, we're working on OpenSSL 3.0 support right now, MDEV-25785. There should be a patch ready very soon.
Regards, Sergei VP of MariaDB Server Engineering and security@mariadb.org
On Nov 18, Lukas Javorsky wrote:
Hi Sergei,
I've somehow managed to get onto another problem and that is the start of the mariadb.service.
I'm trying to reproduce one of the tests "encryption.innodb-checksum-algorithm"
The error I'm getting (cat /var/log/mariadb/mysqld.1.err) is this: 2021-11-08 4:06:46 0 [Warning] Failed to setup SSL 2021-11-08 4:06:46 0 [Warning] SSL error: SSL_CTX_set_default_verify_paths failed 2021-11-08 4:06:46 0 [Warning] SSL error: error:80000002:system library::No such file or directory 2021-11-08 4:06:46 0 [Warning] SSL error: error:10000080:BIO routines::no such file 2021-11-08 4:06:46 0 [Warning] SSL error: error:05880002:x509 certificate routines::system lib 2021-11-08 4:06:46 0 [Note] Server socket created on IP: '127.0.0.1'. 2021-11-08 4:06:46 0 [ERROR] Can't start server: Bind on TCP/IP port. Got error: 13: Permission denied 2021-11-08 4:06:46 0 [ERROR] Do you already have another mysqld server running on port: 16020 ? 2021-11-08 4:06:46 0 [ERROR] Aborting
Do you have any idea what could cause this? I'm not sure how to go further with this reproducer unless I fix this error.
I would be so grateful for even multiple ideas where the problem could be, so I can check them.
Thank you for your response. Lukas
On Fri, Nov 19, 2021 at 8:13 AM Sergei Golubchik serg@mariadb.org wrote:
Hi, Lukas!
This looks very strange. Perhaps, indeed, the previous server process didn't die and is still bound to the port 16020?
This is not very likely, mtr kills all processes with -9, there shouldn't be anything hanging around. But how otherwise 13, "Permission denied" can happen?
Permission denied on a tcp bind looks like a selinux thing. The bind ports are minimal and have a mysqld_port_t type. Given the /var/log/mariadb/ path of the log it looks like an installed thing so it looks more likely to be covered by the selinux policy.
semanage port -a -t mysqld_port_t -p tcp 16000-16040
By the way, we're working on OpenSSL 3.0 support right now, MDEV-25785. There should be a patch ready very soon.
:-)
Thank you for your responses.
By the way, we're working on OpenSSL 3.0 support right now, MDEV-25785. There should be a patch ready very soon.
Will you update the existing versions as well? For example MariaDB-Server-10.5? Or at least will the patch be compatible with the older versions? We could downstream import the patch in the worst case.
On Thu, Nov 18, 2021 at 11:47 PM Daniel Black daniel@mariadb.org wrote:
On Fri, Nov 19, 2021 at 8:13 AM Sergei Golubchik serg@mariadb.org wrote:
Hi, Lukas!
This looks very strange. Perhaps, indeed, the previous server process didn't die and is still bound to the port 16020?
This is not very likely, mtr kills all processes with -9, there shouldn't be anything hanging around. But how otherwise 13, "Permission denied" can happen?
Permission denied on a tcp bind looks like a selinux thing. The bind ports are minimal and have a mysqld_port_t type. Given the /var/log/mariadb/ path of the log it looks like an installed thing so it looks more likely to be covered by the selinux policy.
semanage port -a -t mysqld_port_t -p tcp 16000-16040
By the way, we're working on OpenSSL 3.0 support right now, MDEV-25785. There should be a patch ready very soon.
:-)
Hi, Lukas!
On Nov 19, Lukas Javorsky wrote:
Thank you for your responses.
By the way, we're working on OpenSSL 3.0 support right now, MDEV-25785. There should be a patch ready very soon.
Will you update the existing versions as well? For example MariaDB-Server-10.5?
This is the plan. Same as we did with OpenSSL 1.1 - first introduce it in the trunk, see how it works, fix bugs, then backport.
Or at least will the patch be compatible with the older versions? We could downstream import the patch in the worst case.
I believe it won't be needed and we'll backport the patch.
Regards, Sergei VP of MariaDB Server Engineering and security@mariadb.org
participants (3)
-
Daniel Black -
Lukas Javorsky -
Sergei Golubchik