[Maria-discuss] Trouble getting MariaDB to start up
Hi folks, I'm reasonably experienced with MySQL and MMM for cluster management but now I am working on a project to replace this with a MariaDB Galera cluster. Basic system info: [jg4461@db3 ~]$ uname -a Linux db3.resnet.bris.ac.uk 2.6.32-431.11.2.el6.x86_64 #1 SMP Tue Mar 25 19:59:55 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux [jg4461@db3 ~]$ cat /etc/redhat-release CentOS release 6.5 (Final) [jg4461@db3 ~]$ mysql -V mysql Ver 15.1 Distrib 5.5.36-MariaDB, for Linux (x86_64) using readline 5.1 I've set up a Galera prototype with two nodes and I got the clustering working. Then I decided to add a load of config options from my old MySQL install to the MariaDB install, and now it fails to start up, but doesn't give an obvious error message. The options were mostly to do with InnoDB buffers etc. This is the output in the log file /var/lib/mysql/db3.resnet.bris.ac.uk.err, the init script hangs for ages after this without giving any further output. 140331 15:07:49 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql 140331 15:07:49 mysqld_safe WSREP: Running position recovery with --log_error='/var/lib/mysql/wsrep_recovery.mlbT89' --pid-file='/var/lib/mysql/db3.resnet.bris.ac.uk-recover.pid' 140331 15:07:51 mysqld_safe WSREP: Failed to recover position: '140331 15:07:49 InnoDB: The InnoDB memory heap is disabled 140331 15:07:49 InnoDB: Mutexes and rw_locks use GCC atomic builtins 140331 15:07:49 InnoDB: Compressed tables use zlib 1.2.3 140331 15:07:49 InnoDB: Using Linux native AIO 140331 15:07:49 InnoDB: Initializing buffer pool, size = 1.5G 140331 15:07:49 InnoDB: Completed initialization of buffer pool 140331 15:07:49 InnoDB: highest supported file format is Barracuda. InnoDB: The log sequence number in ibdata files does not match InnoDB: the log sequence number in the ib_logfiles! 140331 15:07:49 InnoDB: Database was not shut down normally! InnoDB: Starting crash recovery. InnoDB: Reading tablespace information from the .ibd files... InnoDB: Restoring possible half-written data pages from the doublewrite InnoDB: buffer... 140331 15:07:49 InnoDB: Waiting for the background threads to start 140331 15:07:50 Percona XtraDB (http://www.percona.com) 5.5.36-MariaDB-33.0 started; log sequence number 1607692 140331 15:07:50 [Note] Plugin 'FEEDBACK' is disabled.' My MariaDB config file is attached. Please could someone point me in the right direction of where to go with debugging this. I can't find any other error message so I'm struggling to track down why the server can't start. Any advice gratefully accepted! Thanks, Jonathan
As a quick look I'd start with wsrep_cluster_address containing the full addresses and pc.wait_prim=no. wsrep_cluster_address = gcomm://192.168.131.94,192.168.186.35,192.168.129.129?pc.wait_prim=no having all the addresses and remove wsrep_provider_options="pc.bootstrap=1" (this is runtime only and not config file from memory). Start one of the nodes with service mysql bootstrap (to define the primary node). and the others just with service mysql start (one at a time). There will be a SST. I'd still with the default rsync as it normally presents fewer authentication errors on start up. Apart from that watch out for selinux failures, firewall rules and which interfaces things are listening/connecting on. ----- Original Message -----
Hi folks,
I'm reasonably experienced with MySQL and MMM for cluster management but now I am working on a project to replace this with a MariaDB Galera cluster.
Basic system info:
[jg4461@db3 ~]$ uname -a Linux db3.resnet.bris.ac.uk 2.6.32-431.11.2.el6.x86_64 #1 SMP Tue Mar 25 19:59:55 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux [jg4461@db3 ~]$ cat /etc/redhat-release CentOS release 6.5 (Final) [jg4461@db3 ~]$ mysql -V mysql Ver 15.1 Distrib 5.5.36-MariaDB, for Linux (x86_64) using readline 5.1
I've set up a Galera prototype with two nodes and I got the clustering working. Then I decided to add a load of config options from my old MySQL install to the MariaDB install, and now it fails to start up, but doesn't give an obvious error message. The options were mostly to do with InnoDB buffers etc. This is the output in the log file /var/lib/mysql/db3.resnet.bris.ac.uk.err, the init script hangs for ages after this without giving any further output.
140331 15:07:49 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql 140331 15:07:49 mysqld_safe WSREP: Running position recovery with --log_error='/var/lib/mysql/wsrep_recovery.mlbT89' --pid-file='/var/lib/mysql/db3.resnet.bris.ac.uk-recover.pid' 140331 15:07:51 mysqld_safe WSREP: Failed to recover position: '140331 15:07:49 InnoDB: The InnoDB memory heap is disabled 140331 15:07:49 InnoDB: Mutexes and rw_locks use GCC atomic builtins 140331 15:07:49 InnoDB: Compressed tables use zlib 1.2.3 140331 15:07:49 InnoDB: Using Linux native AIO 140331 15:07:49 InnoDB: Initializing buffer pool, size = 1.5G 140331 15:07:49 InnoDB: Completed initialization of buffer pool 140331 15:07:49 InnoDB: highest supported file format is Barracuda. InnoDB: The log sequence number in ibdata files does not match InnoDB: the log sequence number in the ib_logfiles! 140331 15:07:49 InnoDB: Database was not shut down normally! InnoDB: Starting crash recovery. InnoDB: Reading tablespace information from the .ibd files... InnoDB: Restoring possible half-written data pages from the doublewrite InnoDB: buffer... 140331 15:07:49 InnoDB: Waiting for the background threads to start 140331 15:07:50 Percona XtraDB (http://www.percona.com) 5.5.36-MariaDB-33.0 started; log sequence number 1607692 140331 15:07:50 [Note] Plugin 'FEEDBACK' is disabled.'
My MariaDB config file is attached. Please could someone point me in the right direction of where to go with debugging this. I can't find any other error message so I'm struggling to track down why the server can't start. Any advice gratefully accepted!
Thanks, Jonathan
_______________________________________________ Mailing list: https://launchpad.net/~maria-discuss Post to : maria-discuss@lists.launchpad.net Unsubscribe : https://launchpad.net/~maria-discuss More help : https://help.launchpad.net/ListHelp
-- -- Daniel Black, Engineer @ Open Query (http://openquery.com.au) Remote expertise & maintenance for MySQL/MariaDB server environments.
On 01/04/14 04:52, Daniel Black wrote:
As a quick look I'd start with wsrep_cluster_address containing the full addresses and pc.wait_prim=no.
wsrep_cluster_address = gcomm://192.168.131.94,192.168.186.35,192.168.129.129?pc.wait_prim=no OK, I've changed this.
having all the addresses and remove wsrep_provider_options="pc.bootstrap=1" (this is runtime only and not config file from memory). OK, have removed that and will use the command line to start up instead.
Start one of the nodes with service mysql bootstrap (to define the primary node). This is still failing for me. It's apparently not to do with the clustering, but for some other reason I can't determine.
and the others just with service mysql start (one at a time). There will be a SST.
I'd still with the default rsync as it normally presents fewer authentication errors on start up.
Apart from that watch out for selinux failures, firewall rules and which interfaces things are listening/connecting on. There's nothing in the SELinux audit log and running in Permissive doesn't help. It just seems to fail to start up without printing anything. The config options I've set work fine on MySQL 5.1 so I'm not sure why they're not working on MariaDB 5.5. Any pointers on places to look for more debugging info?
Thanks, Jonathan
----- Original Message -----
Hi folks,
I'm reasonably experienced with MySQL and MMM for cluster management but now I am working on a project to replace this with a MariaDB Galera cluster.
Basic system info:
[jg4461@db3 ~]$ uname -a Linux db3.resnet.bris.ac.uk 2.6.32-431.11.2.el6.x86_64 #1 SMP Tue Mar 25 19:59:55 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux [jg4461@db3 ~]$ cat /etc/redhat-release CentOS release 6.5 (Final) [jg4461@db3 ~]$ mysql -V mysql Ver 15.1 Distrib 5.5.36-MariaDB, for Linux (x86_64) using readline 5.1
I've set up a Galera prototype with two nodes and I got the clustering working. Then I decided to add a load of config options from my old MySQL install to the MariaDB install, and now it fails to start up, but doesn't give an obvious error message. The options were mostly to do with InnoDB buffers etc. This is the output in the log file /var/lib/mysql/db3.resnet.bris.ac.uk.err, the init script hangs for ages after this without giving any further output.
140331 15:07:49 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql 140331 15:07:49 mysqld_safe WSREP: Running position recovery with --log_error='/var/lib/mysql/wsrep_recovery.mlbT89' --pid-file='/var/lib/mysql/db3.resnet.bris.ac.uk-recover.pid' 140331 15:07:51 mysqld_safe WSREP: Failed to recover position: '140331 15:07:49 InnoDB: The InnoDB memory heap is disabled 140331 15:07:49 InnoDB: Mutexes and rw_locks use GCC atomic builtins 140331 15:07:49 InnoDB: Compressed tables use zlib 1.2.3 140331 15:07:49 InnoDB: Using Linux native AIO 140331 15:07:49 InnoDB: Initializing buffer pool, size = 1.5G 140331 15:07:49 InnoDB: Completed initialization of buffer pool 140331 15:07:49 InnoDB: highest supported file format is Barracuda. InnoDB: The log sequence number in ibdata files does not match InnoDB: the log sequence number in the ib_logfiles! 140331 15:07:49 InnoDB: Database was not shut down normally! InnoDB: Starting crash recovery. InnoDB: Reading tablespace information from the .ibd files... InnoDB: Restoring possible half-written data pages from the doublewrite InnoDB: buffer... 140331 15:07:49 InnoDB: Waiting for the background threads to start 140331 15:07:50 Percona XtraDB (http://www.percona.com) 5.5.36-MariaDB-33.0 started; log sequence number 1607692 140331 15:07:50 [Note] Plugin 'FEEDBACK' is disabled.'
My MariaDB config file is attached. Please could someone point me in the right direction of where to go with debugging this. I can't find any other error message so I'm struggling to track down why the server can't start. Any advice gratefully accepted!
Thanks, Jonathan
_______________________________________________ Mailing list: https://launchpad.net/~maria-discuss Post to : maria-discuss@lists.launchpad.net Unsubscribe : https://launchpad.net/~maria-discuss More help : https://help.launchpad.net/ListHelp
----- Original Message -----
On 01/04/14 04:52, Daniel Black wrote:
Start one of the nodes with service mysql bootstrap (to define the primary node).
This is still failing for me. It's apparently not to do with the clustering, but for some other reason I can't determine.
With this description I can't determine it either.
and the others just with service mysql start (one at a time). There will be a SST.
I'd still with the default rsync as it normally presents fewer authentication errors on start up.
Apart from that watch out for selinux failures, firewall rules and which interfaces things are listening/connecting on. There's nothing in the SELinux audit log and running in Permissive doesn't help. It just seems to fail to start up without printing anything. The config options I've set work fine on MySQL 5.1 so I'm not sure why they're not working on MariaDB 5.5. Any pointers on places to look for more debugging info?
mysql error log. show global status like 'wsrep_%' on the nodes you can connect to. sudo netstat -plntu on the nodes. -- Daniel Black, Engineer @ Open Query (http://openquery.com.au) Remote expertise & maintenance for MySQL/MariaDB server environments.
On 02/04/14 00:32, Daniel Black wrote:
----- Original Message -----
On 01/04/14 04:52, Daniel Black wrote:
Start one of the nodes with service mysql bootstrap (to define the primary node). This is still failing for me. It's apparently not to do with the clustering, but for some other reason I can't determine. With this description I can't determine it either.
Indeed - which is why in my first message I also included the error log and my config file.
and the others just with service mysql start (one at a time). There will be a SST.
I'd still with the default rsync as it normally presents fewer authentication errors on start up.
Apart from that watch out for selinux failures, firewall rules and which interfaces things are listening/connecting on. There's nothing in the SELinux audit log and running in Permissive doesn't help. It just seems to fail to start up without printing anything. The config options I've set work fine on MySQL 5.1 so I'm not sure why they're not working on MariaDB 5.5. Any pointers on places to look for more debugging info?
mysql error log. show global status like 'wsrep_%' on the nodes you can connect to. sudo netstat -plntu on the nodes.
I checked these, and they didn't have any hints. I later found the cause of the problem by trial and error - it seems that specifying the "memlock" option causes the server to silently fail to start up. I have no idea why, but I've removed the option and now it works properly. Thanks, Jonathan
Hi list This is just a curiosity, but anyway: Of course I understand the need for the FLUSH ... FOR EXPORT syntax for InnoDB. But why does MyISAM support it? In which way is it better than WITH READ LOCK? Regards, Federico
Hi, Federico! On Apr 01, Federico Razzoli wrote:
This is just a curiosity, but anyway:
Of course I understand the need for the FLUSH ... FOR EXPORT syntax for InnoDB. But why does MyISAM support it? In which way is it better than WITH READ LOCK?
Because we've redefined the meaning of FLUSH TABLES ... FOR EXPORT. In MySQL it is " This FLUSH TABLES variant applies to InnoDB tables. It is available as of MySQL 5.6.6. The statement ensures that changes to the named tables have been flushed to disk so that binary table copies can be made while the server is running. " and we've removed "applied to InnoDB tables" part, in MariaDB FLUSH TABLES ... FOR EXPORT means that one can do a binary copy of table files. Period. And because one can copy MYD/MYI files under FLUSH TABLES (any variant of it) this means that MyISAM suports FLUSH TABLES ... FOR EXPORT. Regards, Sergei
participants (4)
-
Daniel Black -
Federico Razzoli -
Jonathan Gazeley -
Sergei Golubchik