[Maria-discuss] Mariadb 10.1.6, total cluster recovery - gvwstate.dat
Hi, I am using a Mariadb 10.1.6 three node cluster with wsrep. There is a gvwstate.dat file created in /var/lib/mysql, but in the log file when restarting the nodes I see: [Warning] WSREP: access file(gvwstate.dat) failed(No such file or directory). This seems to be the only thing stopping automatic cluster recovery. Where is mariadb looking for the file, or is there a way of configuring the file location? Thanks, M
Hi, Are you sure there are no other errors? I'm not sure that the file being missing is the problem there: When a node stores the Primary Component state to disk, it saves it as the gvwstate.dat file. The node creates and updates this file when the cluster forms or changes the Primary Component. This ensures that the node retains the latest Primary Component state that it was in. If the node loses connectivity, it has the file to reference. If the node shuts down gracefully, it deletes the file. You might want to paste the full log exert into a Pastie style website, give a fuller description about the node restart (for instance, were you expecting an IST? an SST? Do the nodes form a cluster at all? etc etc ). That would probably be helpful to someone willing to assist. Thanks, Jamie. ________________________________________ From: Maria-discuss [maria-discuss-bounces+jamie.gibbard=netnames.com@lists.launchpad.net] on behalf of Max Zulio [zulioexpress@gmail.com] Sent: 13 August 2015 12:17 To: maria-discuss@lists.launchpad.net Subject: [Maria-discuss] Mariadb 10.1.6, total cluster recovery - gvwstate.dat Hi, I am using a Mariadb 10.1.6 three node cluster with wsrep. There is a gvwstate.dat file created in /var/lib/mysql, but in the log file when restarting the nodes I see: [Warning] WSREP: access file(gvwstate.dat) failed(No such file or directory). This seems to be the only thing stopping automatic cluster recovery. Where is mariadb looking for the file, or is there a way of configuring the file location? Thanks, M NetNames, 25 Canada Square, Canary Wharf, London E14 5LQ, UK | Tel: +44 207 015 9200 | NetNames Limited, Registered in England and Wales, Company number: 3169594, VAT Number: GB 739633893
Hi Jamie, If I kill -9 mysqld on all three of my nodes in the cluster, then start the services simultaneously,the cluster comes back online successfully using the gvwstate.dat file. If I gracefully shutdown mysql on all three nodes as you mentioned the gvwstate.dat file is deleted. This means when the cluster needs to be restarted I am forced to edit my.cnf and replace wsrep_cluster_address=gcomm://x.x.x.x,y.y.y.y,z.z.z.z with wsrep_cluster_address=gcomm:// on one of the nodes before I can bring up the cluster. Is there anyway to have the last node that is shutdown gracefully keep it's gvwstate.dat file so that the next time the cluster is started it can come online without editing the config file? Thanks, Max On Thu, Aug 13, 2015 at 1:34 PM, Jamie Gibbard <Jamie.Gibbard@netnames.com> wrote:
Hi,
Are you sure there are no other errors? I'm not sure that the file being missing is the problem there:
When a node stores the Primary Component state to disk, it saves it as the gvwstate.dat file. The node creates and updates this file when the cluster forms or changes the Primary Component. This ensures that the node retains the latest Primary Component state that it was in. If the node loses connectivity, it has the file to reference. If the node shuts down gracefully, it deletes the file.
You might want to paste the full log exert into a Pastie style website, give a fuller description about the node restart (for instance, were you expecting an IST? an SST? Do the nodes form a cluster at all? etc etc ). That would probably be helpful to someone willing to assist.
Thanks,
Jamie.
________________________________________ From: Maria-discuss [maria-discuss-bounces+jamie.gibbard= netnames.com@lists.launchpad.net] on behalf of Max Zulio [ zulioexpress@gmail.com] Sent: 13 August 2015 12:17 To: maria-discuss@lists.launchpad.net Subject: [Maria-discuss] Mariadb 10.1.6, total cluster recovery - gvwstate.dat
Hi,
I am using a Mariadb 10.1.6 three node cluster with wsrep. There is a gvwstate.dat file created in /var/lib/mysql, but in the log file when restarting the nodes I see: [Warning] WSREP: access file(gvwstate.dat) failed(No such file or directory).
This seems to be the only thing stopping automatic cluster recovery. Where is mariadb looking for the file, or is there a way of configuring the file location?
Thanks,
M NetNames, 25 Canada Square, Canary Wharf, London E14 5LQ, UK | Tel: +44 207 015 9200 | NetNames Limited, Registered in England and Wales, Company number: 3169594, VAT Number: GB 739633893
Hi, Not as far as I am aware, and I would assume there are very good reasons for that. (data consistency, preventing split-brain etc.) If all 3 nodes are offline (gracefully), and you want to bring one up without editing my.cnf, you should use mysqld bootstrap or "service mysql bootstrap" depends on what you are using. ..which effectively starts the cluster with wsrep_cluster_address=gcomm://. You will then not need to edit the config file at all. Of course, when you are killing mysqld on all 3 nodes, you are doing all 3 at the same time so there is little risk of problems. If the nodes were network-partitioned for some period of time and had different data, you probably would not want the cluster to automatically be brought back online, instead manually controlled by you depending on which node is "most correct". Jamie. ________________________________ From: Max Zulio [zulioexpress@gmail.com] Sent: 19 August 2015 17:40 To: Jamie Gibbard Cc: maria-discuss@lists.launchpad.net Subject: Re: [Maria-discuss] Mariadb 10.1.6, total cluster recovery - gvwstate.dat Hi Jamie, If I kill -9 mysqld on all three of my nodes in the cluster, then start the services simultaneously,the cluster comes back online successfully using the gvwstate.dat file. If I gracefully shutdown mysql on all three nodes as you mentioned the gvwstate.dat file is deleted. This means when the cluster needs to be restarted I am forced to edit my.cnf and replace wsrep_cluster_address=gcomm://x.x.x.x,y.y.y.y,z.z.z.z with wsrep_cluster_address=gcomm:// on one of the nodes before I can bring up the cluster. Is there anyway to have the last node that is shutdown gracefully keep it's gvwstate.dat file so that the next time the cluster is started it can come online without editing the config file? Thanks, Max On Thu, Aug 13, 2015 at 1:34 PM, Jamie Gibbard <Jamie.Gibbard@netnames.com<mailto:Jamie.Gibbard@netnames.com>> wrote: Hi, Are you sure there are no other errors? I'm not sure that the file being missing is the problem there: When a node stores the Primary Component state to disk, it saves it as the gvwstate.dat file. The node creates and updates this file when the cluster forms or changes the Primary Component. This ensures that the node retains the latest Primary Component state that it was in. If the node loses connectivity, it has the file to reference. If the node shuts down gracefully, it deletes the file. You might want to paste the full log exert into a Pastie style website, give a fuller description about the node restart (for instance, were you expecting an IST? an SST? Do the nodes form a cluster at all? etc etc ). That would probably be helpful to someone willing to assist. Thanks, Jamie. ________________________________________ From: Maria-discuss [maria-discuss-bounces+jamie.gibbard=netnames.com@lists.launchpad.net<mailto:netnames.com@lists.launchpad.net>] on behalf of Max Zulio [zulioexpress@gmail.com<mailto:zulioexpress@gmail.com>] Sent: 13 August 2015 12:17 To: maria-discuss@lists.launchpad.net<mailto:maria-discuss@lists.launchpad.net> Subject: [Maria-discuss] Mariadb 10.1.6, total cluster recovery - gvwstate.dat Hi, I am using a Mariadb 10.1.6 three node cluster with wsrep. There is a gvwstate.dat file created in /var/lib/mysql, but in the log file when restarting the nodes I see: [Warning] WSREP: access file(gvwstate.dat) failed(No such file or directory). This seems to be the only thing stopping automatic cluster recovery. Where is mariadb looking for the file, or is there a way of configuring the file location? Thanks, M NetNames, 25 Canada Square, Canary Wharf, London E14 5LQ, UK | Tel: +44 207 015 9200<tel:%2B44%20207%20015%209200> | NetNames Limited, Registered in England and Wales, Company number: 3169594, VAT Number: GB 739633893 NetNames, 25 Canada Square, Canary Wharf, London E14 5LQ, UK | Tel: +44 207 015 9200 | NetNames Limited, Registered in England and Wales, Company number: 3169594, VAT Number: GB 739633893
Hi Max, On Wed, Aug 19, 2015 at 9:40 AM, Max Zulio <zulioexpress@gmail.com> wrote:
If I kill -9 mysqld on all three of my nodes in the cluster, then start the services simultaneously,the cluster comes back online successfully using the gvwstate.dat file.
If I gracefully shutdown mysql on all three nodes as you mentioned the gvwstate.dat file is deleted. This means when the cluster needs to be restarted I am forced to edit my.cnf and replace wsrep_cluster_address=gcomm://x.x.x.x,y.y.y.y,z.z.z.z with wsrep_cluster_address=gcomm:// on one of the nodes before I can bring up the cluster.
Is there anyway to have the last node that is shutdown gracefully keep it's gvwstate.dat file so that the next time the cluster is started it can come online without editing the config file?
It might not help a whole lot to only have a gvwstate.dat file on the last node that shuts down, since that would only allow for recovering a cluster with that single node. In that case, it would kind of be like a shortcut to bootstrapping. There is a feature request on Codership's Github that requests a new wsrep_provider_option: pc.recover_minimum_weight. If that ever gets added to Galera, that would allow you to automatically recover a cluster after a graceful shutdown: https://github.com/codership/galera/issues/283 Thanks, Geoff
Hi Max, On Thu, Aug 13, 2015 at 7:17 AM, Max Zulio <zulioexpress@gmail.com> wrote:
Hi,
I am using a Mariadb 10.1.6 three node cluster with wsrep. There is a gvwstate.dat file created in /var/lib/mysql, but in the log file when restarting the nodes I see: [Warning] WSREP: access file(gvwstate.dat) failed(No such file or directory).
This seems to be the only thing stopping automatic cluster recovery. Where is mariadb looking for the file, or is there a way of configuring the file location?
Galera removes gvwsatate.dat file on normal shutdown and recreates it on restart. So, I would guess the gvwstate.dat recovery, in its current state, works only for non-graceful shutdowns. Best, Nirbhay
participants (4)
-
Geoff Montee -
Jamie Gibbard -
Max Zulio -
Nirbhay Choubey