[Maria-discuss] SST with 10.1.13 + xtrabackup-v2 is failing
I cant seem to get past this problem when using xtrabackup-v2 to preform an SST. Everything works fine when I use rsync method. This is a new cluster. I know the sst user/password is working which I confirmed. 2016-05-06 14:34:31 140258805017344 [Note] WSREP: State transfer required: Group state: b5013090-12e9-11e6-ba99-bf6bb7612fa3:8 Local state: 00000000-0000-0000-0000-000000000000:-1 2016-05-06 14:34:31 140258805017344 [Note] WSREP: New cluster view: global state: b5013090-12e9-11e6-ba99-bf6bb7612fa3:8, view# 43: Primary, number of nodes: 3, my index: 0, protocol version 3 2016-05-06 14:34:31 140258805017344 [Warning] WSREP: Gap in state sequence. Need state transfer. 2016-05-06 14:34:31 140258796631808 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role 'joiner' --address '10.1.5.42' --datadir '/var/lib/mysql/' --parent '28455' --binlog 'mysql-bin' ' WSREP_SST: [INFO] Streaming with xbstream (20160506 14:34:32.597) WSREP_SST: [INFO] Using socat as streamer (20160506 14:34:32.605) WSREP_SST: [INFO] Evaluating timeout -s9 100 socat -u TCP-LISTEN:4444,reuseaddr stdio | xbstream -x; RC=( ${PIPESTATUS[@]} ) (20160506 14:34:32.740) 2016-05-06 14:34:32 140258805017344 [Note] WSREP: Prepared SST request: xtrabackup-v2|10.1.5.42:4444/xtrabackup_sst//1 2016-05-06 14:34:32 140258805017344 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2016-05-06 14:34:32 140258805017344 [Note] WSREP: REPL Protocols: 7 (3, 2) 2016-05-06 14:34:32 140258905679616 [Note] WSREP: Service thread queue flushed. 2016-05-06 14:34:32 140258805017344 [Note] WSREP: Assign initial position for certification: 8, protocol version: 3 2016-05-06 14:34:32 140258905679616 [Note] WSREP: Service thread queue flushed. 2016-05-06 14:34:32 140258805017344 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (b5013090-12e9-11e6-ba99-bf6bb7612fa3): 1 (Operation not permitted) at galera/src/replicator_str.cpp:prepare_for_IST():482. IST will be unavailable. 2016-05-06 14:34:32 140258832279296 [Note] WSREP: Member 0.0 () requested state transfer from '*any*'. Selected 1.0 ()(SYNCED) as donor. 2016-05-06 14:34:32 140258832279296 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 8) 2016-05-06 14:34:32 140258805017344 [Note] WSREP: Requesting state transfer: success, donor: 1 WSREP_SST: [INFO] Proceeding with SST (20160506 14:34:33.153) WSREP_SST: [INFO] Evaluating socat -u TCP-LISTEN:4444,reuseaddr stdio | xbstream -x; RC=( ${PIPESTATUS[@]} ) (20160506 14:34:33.152) WSREP_SST: [INFO] Cleaning the existing datadir and innodb-data/log directories (20160506 14:34:33.163) removed `/var/lib/mysql/ib_logfile0' removed `/var/lib/mysql/aria_log_control' removed `/var/lib/mysql/aria_log.00000001' removed `/var/lib/mysql/mysql-bin.index' removed `/var/lib/mysql/ib_logfile1' removed `/var/lib/mysql/ibdata1' removed `/var/lib/mysql/mysql.sock' removed `/var/lib/mysql/mysql-bin.state' removed `/var/lib/mysql/mysql-bin.000001' WSREP_SST: [INFO] Waiting for SST streaming to complete! (20160506 14:34:33.252) 2016-05-06 14:34:34 140258842769152 [Note] WSREP: (a4dbb726, 'tcp://0.0.0.0:4567') turning message relay requesting off 2016-05-06 14:34:44 140258832279296 [Note] WSREP: 1.0 (): State transfer to 0.0 () complete. 2016-05-06 14:34:44 140258832279296 [Note] WSREP: Member 1.0 () synced with group. ls: cannot access mysql-bin.[0-9]*: No such file or directory WSREP_SST: [INFO] Preparing the backup at /var/lib/mysql//.sst (20160506 14:34:45.012) WSREP_SST: [INFO] Evaluating innobackupex --no-version-check --apply-log $rebuildcmd ${DATA} &>${DATA}/innobackup.prepare.log (20160506 14:34:45.021) rm: cannot remove `/var/lib/mysql//innobackup.prepare.log': No such file or directory rm: cannot remove `/var/lib/mysql//innobackup.move.log': No such file or directory WSREP_SST: [INFO] Moving the backup to /var/lib/mysql/ (20160506 14:34:50.126) WSREP_SST: [INFO] Evaluating innobackupex --no-version-check --move-back --force-non-empty-directories ${DATA} &>${DATA}/innobackup.move.log (20160506 14:34:50.135) WSREP_SST: [ERROR] Cleanup after exit with status:2 (20160506 14:34:50.167) 2016-05-06 14:34:50 140258796631808 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '10.1.5.42' --datadir '/var/lib/mysql/' --parent '28455' --binlog 'mysql-bin' : 2 (No such file or directory) 2016-05-06 14:34:50 140258796631808 [ERROR] WSREP: Failed to read uuid:seqno and wsrep_gtid_domain_id from joiner script. 2016-05-06 14:34:50 140260071184416 [ERROR] WSREP: SST failed: 2 (No such file or directory) 2016-05-06 14:34:50 140260071184416 [ERROR] Aborting Error in my_thread_global_end(): 1 threads didn't exit The Joiner innobackup.move.log has this error innobackupex version 2.4.2 based on MySQL server 5.7.11 Linux (x86_64) (revision id: 8e86a84) Error: Move file ib_logfile0 to ./ib_logfile0 failed: Destination file exists 14:34:50 UTC - xtrabackup got signal 11 ; This could be because you hit a bug or data is corrupted. This error can also be caused by malfunctioning hardware. Attempting to collect some information that could help diagnose the problem. As this is a crash and something is definitely wrong, the information collection process might fail. Thread pointer: 0x0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0 thread_stack 0x10000 innobackupex(my_print_stacktrace+0x35)[0xedd1a5] innobackupex(handle_fatal_signal+0x273)[0xa3fce3] /lib64/libpthread.so.0(+0xf7e0)[0x7f5527d2b7e0] /lib64/libc.so.6(cfree+0x1c)[0x7f552618593c] innobackupex(_Z9copy_backv+0x377)[0x74f437] innobackupex(main+0xc0f)[0x73e00f] /lib64/libc.so.6(__libc_start_main+0xfd)[0x7f5526128d5d] innobackupex[0x72db39]
This looks curiously similar to this xtrabackup bug: https://bugs.launchpad.net/percona-xtrabackup/+bug/1552416 Are you defining a datadir explicitly in your config? You might check if it's being picked up right in "xtrabackup --help | grep ^datadir" output. ~Andrew On Fri, May 6, 2016 at 9:52 AM, Erik Mathis <erik.mathis@tritondigital.com> wrote:
I cant seem to get past this problem when using xtrabackup-v2 to preform an SST. Everything works fine when I use rsync method. This is a new cluster. I know the sst user/password is working which I confirmed.
2016-05-06 14:34:31 140258805017344 [Note] WSREP: State transfer required: Group state: b5013090-12e9-11e6-ba99-bf6bb7612fa3:8 Local state: 00000000-0000-0000-0000-000000000000:-1 2016-05-06 14:34:31 140258805017344 [Note] WSREP: New cluster view: global state: b5013090-12e9-11e6-ba99-bf6bb7612fa3:8, view# 43: Primary, number of nodes: 3, my index: 0, protocol version 3 2016-05-06 14:34:31 140258805017344 [Warning] WSREP: Gap in state sequence. Need state transfer. 2016-05-06 14:34:31 140258796631808 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role 'joiner' --address '10.1.5.42' --datadir '/var/lib/mysql/' --parent '28455' --binlog 'mysql-bin' ' WSREP_SST: [INFO] Streaming with xbstream (20160506 14:34:32.597) WSREP_SST: [INFO] Using socat as streamer (20160506 14:34:32.605) WSREP_SST: [INFO] Evaluating timeout -s9 100 socat -u TCP-LISTEN:4444,reuseaddr stdio | xbstream -x; RC=( ${PIPESTATUS[@]} ) (20160506 14:34:32.740) 2016-05-06 14:34:32 140258805017344 [Note] WSREP: Prepared SST request: xtrabackup-v2|10.1.5.42:4444/xtrabackup_sst//1 2016-05-06 14:34:32 140258805017344 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2016-05-06 14:34:32 140258805017344 [Note] WSREP: REPL Protocols: 7 (3, 2) 2016-05-06 14:34:32 140258905679616 [Note] WSREP: Service thread queue flushed. 2016-05-06 14:34:32 140258805017344 [Note] WSREP: Assign initial position for certification: 8, protocol version: 3 2016-05-06 14:34:32 140258905679616 [Note] WSREP: Service thread queue flushed. 2016-05-06 14:34:32 140258805017344 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (b5013090-12e9-11e6-ba99-bf6bb7612fa3): 1 (Operation not permitted) at galera/src/replicator_str.cpp:prepare_for_IST():482. IST will be unavailable. 2016-05-06 14:34:32 140258832279296 [Note] WSREP: Member 0.0 () requested state transfer from '*any*'. Selected 1.0 ()(SYNCED) as donor. 2016-05-06 14:34:32 140258832279296 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 8) 2016-05-06 14:34:32 140258805017344 [Note] WSREP: Requesting state transfer: success, donor: 1 WSREP_SST: [INFO] Proceeding with SST (20160506 14:34:33.153) WSREP_SST: [INFO] Evaluating socat -u TCP-LISTEN:4444,reuseaddr stdio | xbstream -x; RC=( ${PIPESTATUS[@]} ) (20160506 14:34:33.152) WSREP_SST: [INFO] Cleaning the existing datadir and innodb-data/log directories (20160506 14:34:33.163) removed `/var/lib/mysql/ib_logfile0' removed `/var/lib/mysql/aria_log_control' removed `/var/lib/mysql/aria_log.00000001' removed `/var/lib/mysql/mysql-bin.index' removed `/var/lib/mysql/ib_logfile1' removed `/var/lib/mysql/ibdata1' removed `/var/lib/mysql/mysql.sock' removed `/var/lib/mysql/mysql-bin.state' removed `/var/lib/mysql/mysql-bin.000001' WSREP_SST: [INFO] Waiting for SST streaming to complete! (20160506 14:34:33.252) 2016-05-06 14:34:34 140258842769152 [Note] WSREP: (a4dbb726, 'tcp://0.0.0.0:4567') turning message relay requesting off 2016-05-06 14:34:44 140258832279296 [Note] WSREP: 1.0 (): State transfer to 0.0 () complete. 2016-05-06 14:34:44 140258832279296 [Note] WSREP: Member 1.0 () synced with group. ls: cannot access mysql-bin.[0-9]*: No such file or directory WSREP_SST: [INFO] Preparing the backup at /var/lib/mysql//.sst (20160506 14:34:45.012) WSREP_SST: [INFO] Evaluating innobackupex --no-version-check --apply-log $rebuildcmd ${DATA} &>${DATA}/innobackup.prepare.log (20160506 14:34:45.021) rm: cannot remove `/var/lib/mysql//innobackup.prepare.log': No such file or directory rm: cannot remove `/var/lib/mysql//innobackup.move.log': No such file or directory WSREP_SST: [INFO] Moving the backup to /var/lib/mysql/ (20160506 14:34:50.126) WSREP_SST: [INFO] Evaluating innobackupex --no-version-check --move-back --force-non-empty-directories ${DATA} &>${DATA}/innobackup.move.log (20160506 14:34:50.135) WSREP_SST: [ERROR] Cleanup after exit with status:2 (20160506 14:34:50.167) 2016-05-06 14:34:50 140258796631808 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '10.1.5.42' --datadir '/var/lib/mysql/' --parent '28455' --binlog 'mysql-bin' : 2 (No such file or directory) 2016-05-06 14:34:50 140258796631808 [ERROR] WSREP: Failed to read uuid:seqno and wsrep_gtid_domain_id from joiner script. 2016-05-06 14:34:50 140260071184416 [ERROR] WSREP: SST failed: 2 (No such file or directory) 2016-05-06 14:34:50 140260071184416 [ERROR] Aborting
Error in my_thread_global_end(): 1 threads didn't exit
The Joiner innobackup.move.log has this error
innobackupex version 2.4.2 based on MySQL server 5.7.11 Linux (x86_64) (revision id: 8e86a84) Error: Move file ib_logfile0 to ./ib_logfile0 failed: Destination file exists 14:34:50 UTC - xtrabackup got signal 11 ; This could be because you hit a bug or data is corrupted. This error can also be caused by malfunctioning hardware. Attempting to collect some information that could help diagnose the problem. As this is a crash and something is definitely wrong, the information collection process might fail.
Thread pointer: 0x0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0 thread_stack 0x10000 innobackupex(my_print_stacktrace+0x35)[0xedd1a5] innobackupex(handle_fatal_signal+0x273)[0xa3fce3] /lib64/libpthread.so.0(+0xf7e0)[0x7f5527d2b7e0] /lib64/libc.so.6(cfree+0x1c)[0x7f552618593c] innobackupex(_Z9copy_backv+0x377)[0x74f437] innobackupex(main+0xc0f)[0x73e00f] /lib64/libc.so.6(__libc_start_main+0xfd)[0x7f5526128d5d] innobackupex[0x72db39]
_______________________________________________ Mailing list: https://launchpad.net/~maria-discuss Post to : maria-discuss@lists.launchpad.net Unsubscribe : https://launchpad.net/~maria-discuss More help : https://help.launchpad.net/ListHelp
Yes, the datadir was missing in the config which Guillaume was kind enough to point out to me. -Erik- On 05/06/2016 01:49 PM, Andrew Garner wrote:
This looks curiously similar to this xtrabackup bug:
https://bugs.launchpad.net/percona-xtrabackup/+bug/1552416
Are you defining a datadir explicitly in your config? You might check if it's being picked up right in "xtrabackup --help | grep ^datadir" output.
~Andrew
On Fri, May 6, 2016 at 9:52 AM, Erik Mathis <erik.mathis@tritondigital.com> wrote:
I cant seem to get past this problem when using xtrabackup-v2 to preform an SST. Everything works fine when I use rsync method. This is a new cluster. I know the sst user/password is working which I confirmed.
2016-05-06 14:34:31 140258805017344 [Note] WSREP: State transfer required: Group state: b5013090-12e9-11e6-ba99-bf6bb7612fa3:8 Local state: 00000000-0000-0000-0000-000000000000:-1 2016-05-06 14:34:31 140258805017344 [Note] WSREP: New cluster view: global state: b5013090-12e9-11e6-ba99-bf6bb7612fa3:8, view# 43: Primary, number of nodes: 3, my index: 0, protocol version 3 2016-05-06 14:34:31 140258805017344 [Warning] WSREP: Gap in state sequence. Need state transfer. 2016-05-06 14:34:31 140258796631808 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role 'joiner' --address '10.1.5.42' --datadir '/var/lib/mysql/' --parent '28455' --binlog 'mysql-bin'' WSREP_SST: [INFO] Streaming with xbstream (20160506 14:34:32.597) WSREP_SST: [INFO] Using socat as streamer (20160506 14:34:32.605) WSREP_SST: [INFO] Evaluating timeout -s9 100 socat -u TCP-LISTEN:4444,reuseaddr stdio | xbstream -x; RC=( ${PIPESTATUS[@]} ) (20160506 14:34:32.740) 2016-05-06 14:34:32 140258805017344 [Note] WSREP: Prepared SST request: xtrabackup-v2|10.1.5.42:4444/xtrabackup_sst//1 2016-05-06 14:34:32 140258805017344 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2016-05-06 14:34:32 140258805017344 [Note] WSREP: REPL Protocols: 7 (3, 2) 2016-05-06 14:34:32 140258905679616 [Note] WSREP: Service thread queue flushed. 2016-05-06 14:34:32 140258805017344 [Note] WSREP: Assign initial position for certification: 8, protocol version: 3 2016-05-06 14:34:32 140258905679616 [Note] WSREP: Service thread queue flushed. 2016-05-06 14:34:32 140258805017344 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (b5013090-12e9-11e6-ba99-bf6bb7612fa3): 1 (Operation not permitted) at galera/src/replicator_str.cpp:prepare_for_IST():482. IST will be unavailable. 2016-05-06 14:34:32 140258832279296 [Note] WSREP: Member 0.0 () requested state transfer from '*any*'. Selected 1.0 ()(SYNCED) as donor. 2016-05-06 14:34:32 140258832279296 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 8) 2016-05-06 14:34:32 140258805017344 [Note] WSREP: Requesting state transfer: success, donor: 1 WSREP_SST: [INFO] Proceeding with SST (20160506 14:34:33.153) WSREP_SST: [INFO] Evaluating socat -u TCP-LISTEN:4444,reuseaddr stdio | xbstream -x; RC=( ${PIPESTATUS[@]} ) (20160506 14:34:33.152) WSREP_SST: [INFO] Cleaning the existing datadir and innodb-data/log directories (20160506 14:34:33.163) removed `/var/lib/mysql/ib_logfile0' removed `/var/lib/mysql/aria_log_control' removed `/var/lib/mysql/aria_log.00000001' removed `/var/lib/mysql/mysql-bin.index' removed `/var/lib/mysql/ib_logfile1' removed `/var/lib/mysql/ibdata1' removed `/var/lib/mysql/mysql.sock' removed `/var/lib/mysql/mysql-bin.state' removed `/var/lib/mysql/mysql-bin.000001' WSREP_SST: [INFO] Waiting for SST streaming to complete! (20160506 14:34:33.252) 2016-05-06 14:34:34 140258842769152 [Note] WSREP: (a4dbb726, 'tcp://0.0.0.0:4567') turning message relay requesting off 2016-05-06 14:34:44 140258832279296 [Note] WSREP: 1.0 (): State transfer to 0.0 () complete. 2016-05-06 14:34:44 140258832279296 [Note] WSREP: Member 1.0 () synced with group. ls: cannot access mysql-bin.[0-9]*: No such file or directory WSREP_SST: [INFO] Preparing the backup at /var/lib/mysql//.sst (20160506 14:34:45.012) WSREP_SST: [INFO] Evaluating innobackupex --no-version-check --apply-log $rebuildcmd ${DATA} &>${DATA}/innobackup.prepare.log (20160506 14:34:45.021) rm: cannot remove `/var/lib/mysql//innobackup.prepare.log': No such file or directory rm: cannot remove `/var/lib/mysql//innobackup.move.log': No such file or directory WSREP_SST: [INFO] Moving the backup to /var/lib/mysql/ (20160506 14:34:50.126) WSREP_SST: [INFO] Evaluating innobackupex --no-version-check --move-back --force-non-empty-directories ${DATA} &>${DATA}/innobackup.move.log (20160506 14:34:50.135) WSREP_SST: [ERROR] Cleanup after exit with status:2 (20160506 14:34:50.167) 2016-05-06 14:34:50 140258796631808 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '10.1.5.42' --datadir '/var/lib/mysql/' --parent '28455' --binlog 'mysql-bin' : 2 (No such file or directory) 2016-05-06 14:34:50 140258796631808 [ERROR] WSREP: Failed to read uuid:seqno and wsrep_gtid_domain_id from joiner script. 2016-05-06 14:34:50 140260071184416 [ERROR] WSREP: SST failed: 2 (No such file or directory) 2016-05-06 14:34:50 140260071184416 [ERROR] Aborting
Error in my_thread_global_end(): 1 threads didn't exit
The Joiner innobackup.move.log has this error
innobackupex version 2.4.2 based on MySQL server 5.7.11 Linux (x86_64) (revision id: 8e86a84) Error: Move file ib_logfile0 to ./ib_logfile0 failed: Destination file exists 14:34:50 UTC - xtrabackup got signal 11 ; This could be because you hit a bug or data is corrupted. This error can also be caused by malfunctioning hardware. Attempting to collect some information that could help diagnose the problem. As this is a crash and something is definitely wrong, the information collection process might fail.
Thread pointer: 0x0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0 thread_stack 0x10000 innobackupex(my_print_stacktrace+0x35)[0xedd1a5] innobackupex(handle_fatal_signal+0x273)[0xa3fce3] /lib64/libpthread.so.0(+0xf7e0)[0x7f5527d2b7e0] /lib64/libc.so.6(cfree+0x1c)[0x7f552618593c] innobackupex(_Z9copy_backv+0x377)[0x74f437] innobackupex(main+0xc0f)[0x73e00f] /lib64/libc.so.6(__libc_start_main+0xfd)[0x7f5526128d5d] innobackupex[0x72db39]
_______________________________________________ Mailing list: https://launchpad.net/~maria-discuss Post to : maria-discuss@lists.launchpad.net Unsubscribe : https://launchpad.net/~maria-discuss More help : https://help.launchpad.net/ListHelp
participants (2)
-
Andrew Garner -
Erik Mathis