Creating Replica With Rsync (But It Fails)
Hi there, I am attempting to create a replica from a MariaDB 10.3.31 master. The replicate is running MariaDB 10.3.39 on the same OS (Ubuntu 20.04) and like hardware with identical MariaDB configuration. The strategy I'm using to create the replica is one that I have done before using earlier version of MariaDB and MySQL. But this time it is failing. It roughly looks like this: 1. rsync /var/lib/mysql from master to replica 2. On master: FLUSH TABLES WITH READ LOCK; SHOW MASTER STATUS; 3. On master: Shutdown 4. rerun rsync from master to replica; then restart master 5. Start mariadb on replica 6. CHANGE MASTER on replica 7. Celebrate and have a nap When starting the replica which now has all the master data rsynced over, I am getting the following error repeated over and over again: 2023-12-26 22:14:54 0 [ERROR] InnoDB: Page [page id: space=43077, page number=3] log sequence number 19989139444854 is in the future! Current system log sequence number 19246539202516. 2023-12-26 22:14:54 0 [ERROR] InnoDB: Your database may be corrupt or you may have copied the InnoDB tablespace but not the InnoDB log files. Please refer to https://mariadb.com/kb/en/library/innodb-recovery-modes/ for information about forcing recovery. Then I get the following errors for many (but not all) tables: 2023-12-26 22:16:32 0 [ERROR] InnoDB: Expected tablespace id 3986547 but found 3986540 in the file ./ddx_practice_91804/alerts.ibd 2023-12-26 22:16:32 0 [Note] InnoDB: trying to read page [page id: space=3986547, page number=0] in the background in a non-existing or being-dropped tablespace The replica is totally useless, even though it is an exact copy, running on like hardware, same OS and similar major version of MariaDB. I've in the past used innodb_fast_shutdown = 0 on the master before final rsync, but I was told by MariaDB engineers that should not matter. I'm unclear why I'm seeing this corruption. Any ideas? Thanks, Mike Confidentiality Notice: The information contained in this electronic message and any attachments to this message are intended only for the individual(s) addressed in the message and may contain proprietary and confidential information. If you are not the intended recipient, you should not disseminate, distribute, or copy this e-mail. Please notify the sender and destroy this message. WARNING: Computer viruses can be transmitted via email. The recipient should scan this email before opening it. The company accepts no liability for any damage caused by any virus transmitted by this email.
Michael Caplan via discuss <discuss@lists.mariadb.org> writes:
I am attempting to create a replica from a MariaDB 10.3.31 master. The replicate is running MariaDB 10.3.39 on the same OS (Ubuntu 20.04) and
1. rsync /var/lib/mysql from master to replica 2. On master: FLUSH TABLES WITH READ LOCK; SHOW MASTER STATUS; 3. On master: Shutdown 4. rerun rsync from master to replica; then restart master 5. Start mariadb on replica
When starting the replica which now has all the master data rsynced over, I am getting the following error repeated over and over again:
2023-12-26 22:14:54 0 [ERROR] InnoDB: Page [page id: space=43077, page number=3] log sequence number 19989139444854 is in the future! Current system log sequence number 19246539202516. 2023-12-26 22:14:54 0 [ERROR] InnoDB: Your database may be corrupt or you may have copied the InnoDB tablespace but not the InnoDB log files. Please refer to https://mariadb.com/kb/en/library/innodb-recovery-modes/ for information about forcing recovery.
To debug this, I would suggest to compare the rsync copy with the original in step 4, eg. something like: diff --recursive <original> <copy> (after the rerun rsync, but before restarting the master). Most likely, something goes wrong with the rsync not copying everything it should, and this can hopefully help understand what goes wrong. Since the master restarts ok on the original, the most likely explanation is that the copy is not identical. For example, if rsync's "quick check" or --checksum heuristics is used and fails to detect some file modifications. - Kristian.
Hi Kristian, Thank you very much for your thoughts on this. You were indeed 100% correct. The copy was not completely identical. The short of it is that the original rsync was done off of a replica. Then all future rsyncs where done off master. Two things learned: * I was not aware of the different strategies rsync can use to determine difference. I assumed a default of a more rigorous checksum and would have been safe to resolve difference between the replica rsync then moved to master. * That an existing replica does not share the same log sequence number as master (duh!) Very much appreciate you catching my mistake and taking the time to explain to me what the issue was. Happy New Year! Mike ________________________________ From: Kristian Nielsen <knielsen@knielsen-hq.org> Sent: 27 December 2023 3:52 AM To: Michael Caplan via discuss <discuss@lists.mariadb.org> Cc: Michael Caplan <Michael.Caplan@henryscheinone.com> Subject: EXTERNAL: Re: [MariaDB discuss] Creating Replica With Rsync (But It Fails) CAUTION: This email originated from outside the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. When in doubt, contact the IT and Information Security Departments. [You don't often get email from knielsen@knielsen-hq.org. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ] Michael Caplan via discuss <discuss@lists.mariadb.org> writes:
I am attempting to create a replica from a MariaDB 10.3.31 master. The replicate is running MariaDB 10.3.39 on the same OS (Ubuntu 20.04) and
1. rsync /var/lib/mysql from master to replica 2. On master: FLUSH TABLES WITH READ LOCK; SHOW MASTER STATUS; 3. On master: Shutdown 4. rerun rsync from master to replica; then restart master 5. Start mariadb on replica
When starting the replica which now has all the master data rsynced over, I am getting the following error repeated over and over again:
2023-12-26 22:14:54 0 [ERROR] InnoDB: Page [page id: space=43077, page number=3] log sequence number 19989139444854 is in the future! Current system log sequence number 19246539202516. 2023-12-26 22:14:54 0 [ERROR] InnoDB: Your database may be corrupt or you may have copied the InnoDB tablespace but not the InnoDB log files. Please refer to https://protect.checkpoint.com/v2/___https://mariadb.com/kb/en/library/innod... for information about forcing recovery.
To debug this, I would suggest to compare the rsync copy with the original in step 4, eg. something like: diff --recursive <original> <copy> (after the rerun rsync, but before restarting the master). Most likely, something goes wrong with the rsync not copying everything it should, and this can hopefully help understand what goes wrong. Since the master restarts ok on the original, the most likely explanation is that the copy is not identical. For example, if rsync's "quick check" or --checksum heuristics is used and fails to detect some file modifications. - Kristian. Confidentiality Notice: The information contained in this electronic message and any attachments to this message are intended only for the individual(s) addressed in the message and may contain proprietary and confidential information. If you are not the intended recipient, you should not disseminate, distribute, or copy this e-mail. Please notify the sender and destroy this message. WARNING: Computer viruses can be transmitted via email. The recipient should scan this email before opening it. The company accepts no liability for any damage caused by any virus transmitted by this email.
participants (2)
-
Kristian Nielsen -
Michael Caplan