[Maria-discuss] rsync failure?
Have the rsync failure (see the trace below): 2016-11-08 15:17:34 139959897945856 [ERROR] WSREP: Failed to read 'ready <addr>' from: wsrep_sst_rsync --role 'joiner' --address 'node3.frame.mesos' --datadir '/var/lib/mysql/' --parent '114' '' Read: '(null)' 2016-11-08 15:17:34 139959897945856 [ERROR] WSREP: Process completed with error: wsrep_sst_rsync --role 'joiner' --address 'node3.frame.mesos' --datadir '/var/lib/mysql/' --parent '114' '' : 1 (Operation not permitted) 2016-11-08 15:17:34 139960239241984 [ERROR] WSREP: Failed to prepare for 'rsync' SST. Unrecoverable. 2016-11-08 15:17:34 139960239241984 [ERROR] Aborting Can someone with better understanding of replication process explain the nature of this problem? Can it be caused by hostname resolution failure, or would it be storage related? Thank you,
On Tue, Nov 8, 2016 at 10:50 AM, l vic
Have the rsync failure (see the trace below):
2016-11-08 15:17:34 139959897945856 [ERROR] WSREP: Failed to read 'ready <addr>' from: wsrep_sst_rsync --role 'joiner' --address 'node3.frame.mesos' --datadir '/var/lib/mysql/' --parent '114' '' Read: '(null)' 2016-11-08 15:17:34 139959897945856 [ERROR] WSREP: Process completed with error: wsrep_sst_rsync --role 'joiner' --address 'node3.frame.mesos' --datadir '/var/lib/mysql/' --parent '114' '' : 1 (Operation not permitted) 2016-11-08 15:17:34 139960239241984 [ERROR] WSREP: Failed to prepare for 'rsync' SST. Unrecoverable. 2016-11-08 15:17:34 139960239241984 [ERROR] Aborting
Can someone with better understanding of replication process explain the nature of this problem? Can it be caused by hostname resolution failure, or would it be storage related?
Therer is a possibility that something could have gone wrong on the donor node. Checkout its error log to see whether that's the case. Best, Nirbhay
Thank you,
_______________________________________________ Mailing list: https://launchpad.net/~maria-discuss Post to : maria-discuss@lists.launchpad.net Unsubscribe : https://launchpad.net/~maria-discuss More help : https://help.launchpad.net/ListHelp
How can i determine the donor node? Would that be seed node?
Thank you
On Tue, Nov 8, 2016 at 11:04 AM, Nirbhay Choubey
On Tue, Nov 8, 2016 at 10:50 AM, l vic
wrote: Have the rsync failure (see the trace below):
2016-11-08 15:17:34 139959897945856 [ERROR] WSREP: Failed to read 'ready <addr>' from: wsrep_sst_rsync --role 'joiner' --address 'node3.frame.mesos' --datadir '/var/lib/mysql/' --parent '114' '' Read: '(null)' 2016-11-08 15:17:34 139959897945856 [ERROR] WSREP: Process completed with error: wsrep_sst_rsync --role 'joiner' --address 'node3.frame.mesos' --datadir '/var/lib/mysql/' --parent '114' '' : 1 (Operation not permitted) 2016-11-08 15:17:34 139960239241984 [ERROR] WSREP: Failed to prepare for 'rsync' SST. Unrecoverable. 2016-11-08 15:17:34 139960239241984 [ERROR] Aborting
Can someone with better understanding of replication process explain the nature of this problem? Can it be caused by hostname resolution failure, or would it be storage related?
Therer is a possibility that something could have gone wrong on the donor node. Checkout its error log to see whether that's the case.
Best, Nirbhay
Thank you,
_______________________________________________ Mailing list: https://launchpad.net/~maria-discuss Post to : maria-discuss@lists.launchpad.net Unsubscribe : https://launchpad.net/~maria-discuss More help : https://help.launchpad.net/ListHelp
On Tue, Nov 8, 2016 at 11:32 AM, l vic
How can i determine the donor node? Would that be seed node?
On the joiner you will see the following: 2016-11-08 12:13:57 140589576218368 [Note] WSREP: Member 1.0 (node2) requested state transfer from '*any*'. Selected 0.0 (node1)(SYNCED) as donor. In my installation, I have used wsrep_node_name to assign names to the nodes. Best, Nirbhay
Thank you
On Tue, Nov 8, 2016 at 11:04 AM, Nirbhay Choubey
wrote: On Tue, Nov 8, 2016 at 10:50 AM, l vic
wrote: Have the rsync failure (see the trace below):
2016-11-08 15:17:34 139959897945856 [ERROR] WSREP: Failed to read 'ready <addr>' from: wsrep_sst_rsync --role 'joiner' --address 'node3.frame.mesos' --datadir '/var/lib/mysql/' --parent '114' '' Read: '(null)' 2016-11-08 15:17:34 139959897945856 [ERROR] WSREP: Process completed with error: wsrep_sst_rsync --role 'joiner' --address 'node3.frame.mesos' --datadir '/var/lib/mysql/' --parent '114' '' : 1 (Operation not permitted) 2016-11-08 15:17:34 139960239241984 [ERROR] WSREP: Failed to prepare for 'rsync' SST. Unrecoverable. 2016-11-08 15:17:34 139960239241984 [ERROR] Aborting
Can someone with better understanding of replication process explain the nature of this problem? Can it be caused by hostname resolution failure, or would it be storage related?
Therer is a possibility that something could have gone wrong on the donor node. Checkout its error log to see whether that's the case.
Best, Nirbhay
Thank you,
_______________________________________________ Mailing list: https://launchpad.net/~maria-discuss Post to : maria-discuss@lists.launchpad.net Unsubscribe : https://launchpad.net/~maria-discuss More help : https://help.launchpad.net/ListHelp
From your previous answer about using node-name vs. node-address to identify the node:
use wsrep-node-address; wsrep-node-name, OTOH, could be used to assign a
name to the node.
It helps identify the node in the error logs and can be used in
wsrep-sst-donor
On Tue, Nov 8, 2016 at 12:54 PM, Nirbhay Choubey
On Tue, Nov 8, 2016 at 11:32 AM, l vic
wrote: How can i determine the donor node? Would that be seed node?
On the joiner you will see the following:
2016-11-08 12:13:57 140589576218368 [Note] WSREP: Member 1.0 (node2) requested state transfer from '*any*'. Selected 0.0 (node1)(SYNCED) as donor.
In my installation, I have used wsrep_node_name to assign names to the nodes.
Best, Nirbhay
Thank you
On Tue, Nov 8, 2016 at 11:04 AM, Nirbhay Choubey
wrote: On Tue, Nov 8, 2016 at 10:50 AM, l vic
wrote: Have the rsync failure (see the trace below):
2016-11-08 15:17:34 139959897945856 [ERROR] WSREP: Failed to read 'ready <addr>' from: wsrep_sst_rsync --role 'joiner' --address 'node3.frame.mesos' --datadir '/var/lib/mysql/' --parent '114' '' Read: '(null)' 2016-11-08 15:17:34 139959897945856 [ERROR] WSREP: Process completed with error: wsrep_sst_rsync --role 'joiner' --address 'node3.frame.mesos' --datadir '/var/lib/mysql/' --parent '114' '' : 1 (Operation not permitted) 2016-11-08 15:17:34 139960239241984 [ERROR] WSREP: Failed to prepare for 'rsync' SST. Unrecoverable. 2016-11-08 15:17:34 139960239241984 [ERROR] Aborting
Can someone with better understanding of replication process explain the nature of this problem? Can it be caused by hostname resolution failure, or would it be storage related?
Therer is a possibility that something could have gone wrong on the donor node. Checkout its error log to see whether that's the case.
Best, Nirbhay
Thank you,
_______________________________________________ Mailing list: https://launchpad.net/~maria-discuss Post to : maria-discuss@lists.launchpad.net Unsubscribe : https://launchpad.net/~maria-discuss More help : https://help.launchpad.net/ListHelp
on the joiner i see:
WSREP: Failed to prepare for incremental state transfer: Local state UUID
(00000000-0000-0000-0000-000000000000) does not match group state UUID
(f74ffa7e-a5f2-11e6-b2a3-1fead9a6118d): 1 (Operation not permitted)
at galera/src/replicator_str.cpp:prepare_for_IST():482. IST will be
unavailable.
2016-11-08 20:38:23 139911323707136 [Note] WSREP: Member 0.0 () requested
state transfer from '*any*'. Selected 1.0 ()(SYNCED) as donor.
2016-11-08 20:38:23 139911323707136 [Note] WSREP: Shifting PRIMARY ->
JOINER (TO: 4)
2016-11-08 20:38:23 139911628454656 [Note] WSREP: Requesting state
transfer: success, donor: 1
2016-11-08 20:38:24 139911323707136 [Warning] WSREP: 1.0 (): State transfer
to 0.0 () failed: -255 (Unknown error 255)
On Tue, Nov 8, 2016 at 12:54 PM, Nirbhay Choubey
On Tue, Nov 8, 2016 at 11:32 AM, l vic
wrote: How can i determine the donor node? Would that be seed node?
On the joiner you will see the following:
2016-11-08 12:13:57 140589576218368 [Note] WSREP: Member 1.0 (node2) requested state transfer from '*any*'. Selected 0.0 (node1)(SYNCED) as donor.
In my installation, I have used wsrep_node_name to assign names to the nodes.
Best, Nirbhay
Thank you
On Tue, Nov 8, 2016 at 11:04 AM, Nirbhay Choubey
wrote: On Tue, Nov 8, 2016 at 10:50 AM, l vic
wrote: Have the rsync failure (see the trace below):
2016-11-08 15:17:34 139959897945856 [ERROR] WSREP: Failed to read 'ready <addr>' from: wsrep_sst_rsync --role 'joiner' --address 'node3.frame.mesos' --datadir '/var/lib/mysql/' --parent '114' '' Read: '(null)' 2016-11-08 15:17:34 139959897945856 [ERROR] WSREP: Process completed with error: wsrep_sst_rsync --role 'joiner' --address 'node3.frame.mesos' --datadir '/var/lib/mysql/' --parent '114' '' : 1 (Operation not permitted) 2016-11-08 15:17:34 139960239241984 [ERROR] WSREP: Failed to prepare for 'rsync' SST. Unrecoverable. 2016-11-08 15:17:34 139960239241984 [ERROR] Aborting
Can someone with better understanding of replication process explain the nature of this problem? Can it be caused by hostname resolution failure, or would it be storage related?
Therer is a possibility that something could have gone wrong on the donor node. Checkout its error log to see whether that's the case.
Best, Nirbhay
Thank you,
_______________________________________________ Mailing list: https://launchpad.net/~maria-discuss Post to : maria-discuss@lists.launchpad.net Unsubscribe : https://launchpad.net/~maria-discuss More help : https://help.launchpad.net/ListHelp
I don't see any log entries pertaining to node3 join on the donor nodes
On Tue, Nov 8, 2016 at 11:04 AM, Nirbhay Choubey
On Tue, Nov 8, 2016 at 10:50 AM, l vic
wrote: Have the rsync failure (see the trace below):
2016-11-08 15:17:34 139959897945856 [ERROR] WSREP: Failed to read 'ready <addr>' from: wsrep_sst_rsync --role 'joiner' --address 'node3.frame.mesos' --datadir '/var/lib/mysql/' --parent '114' '' Read: '(null)' 2016-11-08 15:17:34 139959897945856 [ERROR] WSREP: Process completed with error: wsrep_sst_rsync --role 'joiner' --address 'node3.frame.mesos' --datadir '/var/lib/mysql/' --parent '114' '' : 1 (Operation not permitted) 2016-11-08 15:17:34 139960239241984 [ERROR] WSREP: Failed to prepare for 'rsync' SST. Unrecoverable. 2016-11-08 15:17:34 139960239241984 [ERROR] Aborting
Can someone with better understanding of replication process explain the nature of this problem? Can it be caused by hostname resolution failure, or would it be storage related?
Therer is a possibility that something could have gone wrong on the donor node. Checkout its error log to see whether that's the case.
Best, Nirbhay
Thank you,
_______________________________________________ Mailing list: https://launchpad.net/~maria-discuss Post to : maria-discuss@lists.launchpad.net Unsubscribe : https://launchpad.net/~maria-discuss More help : https://help.launchpad.net/ListHelp
participants (2)
-
l vic
-
Nirbhay Choubey