Hi Kristian,

Thank you very much for your thoughts on this.  

You were indeed 100% correct.  The copy was not completely identical.   The short of it is that the original rsync was done off of a replica.  Then all future rsyncs where done off master.   

Two things learned:

Very much appreciate you catching my mistake and taking the time to explain to me what the issue was.


Happy New Year!

Mike
 
From: Kristian Nielsen <knielsen@knielsen-hq.org>
Sent: 27 December 2023 3:52 AM
To: Michael Caplan via discuss <discuss@lists.mariadb.org>
Cc: Michael Caplan <Michael.Caplan@henryscheinone.com>
Subject: EXTERNAL: Re: [MariaDB discuss] Creating Replica With Rsync (But It Fails)
 
CAUTION: This email originated from outside the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. When in doubt, contact the IT and Information Security Departments.


[You don't often get email from knielsen@knielsen-hq.org. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]

Michael Caplan via discuss <discuss@lists.mariadb.org> writes:

> I am attempting to create a replica from a MariaDB 10.3.31 master. The
> replicate is running MariaDB 10.3.39 on the same OS (Ubuntu 20.04) and

>   1.  rsync /var/lib/mysql from master to replica
>   2.  On master: FLUSH TABLES WITH READ LOCK; SHOW MASTER STATUS;
>   3.  On master: Shutdown
>   4.  rerun rsync from master to replica; then restart master
>   5.  Start mariadb on replica

> When starting the replica which now has all the master data rsynced
> over, I am getting the following error repeated over and over again:
>
> 2023-12-26 22:14:54 0 [ERROR] InnoDB: Page [page id: space=43077, page
> number=3] log sequence number 19989139444854 is in the future! Current
> system log sequence number 19246539202516.
> 2023-12-26 22:14:54 0 [ERROR] InnoDB: Your database may be corrupt or
> you may have copied the InnoDB tablespace but not the InnoDB log
> files. Please refer to
> https://protect.checkpoint.com/v2/___https://mariadb.com/kb/en/library/innodb-recovery-modes/___.YzJ1OmhzcHMxNjU5MTIzMTI0MzE2OmM6bzo1NDI2YWU5OTgyZjk4NWM3NmVkNjQ2NTNiMWEwOWNjYTo2Ojk1NGM6M2NjNWY2MjA5NDZmOTliN2JjOWU1Nzg3NDU5ZTVlYzE1ZjViOWUxNjEwZWFlMjBmZjFkYjY3OWI2ZGViYzQ4YTpwOlQ for
> information about forcing recovery.

To debug this, I would suggest to compare the rsync copy with the original
in step 4, eg. something like:

  diff --recursive <original> <copy>

(after the rerun rsync, but before restarting the master). Most likely,
something goes wrong with the rsync not copying everything it should, and
this can hopefully help understand what goes wrong.

Since the master restarts ok on the original, the most likely explanation is
that the copy is not identical. For example, if rsync's "quick check" or
--checksum heuristics is used and fails to detect some file modifications.

 - Kristian.

Confidentiality Notice: The information contained in this electronic message and any attachments to this message are intended only for the individual(s) addressed in the message and may contain proprietary and confidential information. If you are not the intended recipient, you should not disseminate, distribute, or copy this e-mail. Please notify the sender and destroy this message. WARNING: Computer viruses can be transmitted via email. The recipient should scan this email before opening it. The company accepts no liability for any damage caused by any virus transmitted by this email.