How do I get detailed TLS debugging information?
Replication master: Server version: 10.5.21-MariaDB-0+deb11u1-log Debian 11
Replication slave: Server version: 10.5.21-MariaDB-0+deb11u1 Debian 11
When `require_secure_transport = on` is enabled on the master, I get:
"error reconnecting to master 'user@domain.com:3306' - retry-time: 10 maximum-retries: 86400 message: Access denied for user 'user'@'domain.com' (using password: YES)."
If I turn `require_secure_transport` off, replication works.
So the error in the log *looks* like an authentication failure message, but it's clearly a TLS issue.
journalctl -fu mariadb.service just shows the same: "Access denied for user 'user'@'domain' (using password: YES)"
How can I get verbose messages about what is happening with TLS (why is it being rejected)?
On 31.10.23 22:45, Nick Lockheart via discuss wrote:
How can I get verbose messages about what is happening with TLS (why is it being rejected)?
unfortunately your best options are either to capture the initial TLS dialog packages and to analyze them with Wireshark, or to use the OpenSSL s_client tool to emulate a mysql protocol connection trying to switch to TLS:
see e.g.: https://serverfault.com/a/931652
On Tue, 2023-10-31 at 23:09 +0100, Hartmut Holzgraefe via discuss wrote:
On 31.10.23 22:45, Nick Lockheart via discuss wrote:
How can I get verbose messages about what is happening with TLS (why is it being rejected)?
unfortunately your best options are either to capture the initial TLS dialog packages and to analyze them with Wireshark, or to use the OpenSSL s_client tool to emulate a mysql protocol connection trying to switch to TLS:
see e.g.: https://serverfault.com/a/931652
I just tried using openssl s_client -starttls mysql -connect from the replication slave server, connecting to the master. In the output, I see:
--- SSL handshake has read 4228 bytes and written 461 bytes Verification: OK ---
Does that mean there is no issue with the certificates?
On Tue, 2023-10-31 at 23:09 +0100, Hartmut Holzgraefe via discuss wrote:
On 31.10.23 22:45, Nick Lockheart via discuss wrote:
How can I get verbose messages about what is happening with TLS (why is it being rejected)?
unfortunately your best options are either to capture the initial TLS dialog packages and to analyze them with Wireshark, or to use the OpenSSL s_client tool to emulate a mysql protocol connection trying to switch to TLS:
see e.g.: https://serverfault.com/a/931652
One thing that makes this particularly frustrating is that when `require_secure_transport = on` is set on the master, I can still login remotely to the master computer from the slave computer manually, using the same replication user:
mariadb -h masterdomain.com -P 3306 -p'secret' -u rep_slave -v Welcome to the MariaDB monitor. Commands end with ; or \g. Your MariaDB connection id is 191 Server version: 10.5.21-MariaDB-0+deb11u1-log Debian 11
And it appears that TLS is working:
MariaDB [(none)]> SHOW SESSION STATUS LIKE 'Ssl_cipher';
+---------------+------------------------+ | Variable_name | Value | +---------------+------------------------+ | Ssl_cipher | TLS_AES_256_GCM_SHA384 | +---------------+------------------------+ 1 row in set (0.023 sec)
But meanwhile, in the logs, I'm still seeing:
"Access denied for user 'rep_slave'@'domain.com' (using password: YES)"
So it seems like TLS does work for the replication user, too, if I log in manually from the command line (on the remote machine) but the same user is failing (when TLS is enforced) as the automated replication slave user.
On Tue, 31 Oct 2023 17:45:47 -0400 Nick Lockheart via discuss discuss@lists.mariadb.org wrote:
Replication master: Server version: 10.5.21-MariaDB-0+deb11u1-log Debian 11
Replication slave: Server version: 10.5.21-MariaDB-0+deb11u1 Debian 11
When `require_secure_transport = on` is enabled on the master, I get:
"error reconnecting to master 'user@domain.com:3306' - retry-time: 10 maximum-retries: 86400 message: Access denied for user 'user'@'domain.com' (using password: YES)."
Could it be that you need to manually enable TLS on the replica using CHANGE MASTER TO MASTER_SSL=1; https://mariadb.com/kb/en/change-master-to/#tls-options? Setting up the path to the certificate authority and enabling verification is also a good idea (otherwise it's possible for the ISP to MITM the connection by posing as a MariaDB instance with a different host key).
The "Connections using insecure transport are prohibited" error message only has been fixed in MariaDB 11.2; in 10.5.21 it would still be "Access denied for user": https://github.com/MariaDB/server/pull/2581
On Wed, 2023-11-01 at 10:26 +0300, Ivan Krylov via discuss wrote:
On Tue, 31 Oct 2023 17:45:47 -0400 Nick Lockheart via discuss discuss@lists.mariadb.org wrote:
Replication master: Server version: 10.5.21-MariaDB-0+deb11u1-log Debian 11
Replication slave: Server version: 10.5.21-MariaDB-0+deb11u1 Debian 11
When `require_secure_transport = on` is enabled on the master, I get:
"error reconnecting to master 'user@domain.com:3306' - retry-time: 10 maximum-retries: 86400 message: Access denied for user 'user'@'domain.com' (using password: YES)."
Could it be that you need to manually enable TLS on the replica using CHANGE MASTER TO MASTER_SSL=1; https://mariadb.com/kb/en/change-master-to/#tls-options? Setting up the path to the certificate authority and enabling verification is also a good idea (otherwise it's possible for the ISP to MITM the connection by posing as a MariaDB instance with a different host key).
The "Connections using insecure transport are prohibited" error message only has been fixed in MariaDB 11.2; in 10.5.21 it would still be "Access denied for user": https://github.com/MariaDB/server/pull/2581
Thank you, I had thought that TLS settings were taken from the [client- mariadb] block in the configuration file, but it seems that you have to specify the TLS settings in the CHANGE MASTER command.
It is working now with TLS.
participants (3)
-
Hartmut Holzgraefe -
Ivan Krylov -
Nick Lockheart