Hi,
We are seeing recurrent corruption on one of our DB nodes, running
RHEL9, under vmware, MariaDB 10.11.10.
> 2025-05-13 6:32:08 0 [ERROR] InnoDB: tried to purge non-delete-marked record in index `cardChipNumber` of table `mydb`.`idm_usc_kaart_status`: tuple: TUPLE (info_bits=0, 2 fields): {[17]36147687118764292(0x3336313437363837313138373634323932),[4] " (0x9722A2FB)}, record: COMPACT RECORD(info_bits=0, 2 fields): {[17]36147687118764292(0x3336313437363837313138373634323932),[4] " (0x9722A2FB)}
> 2025-05-13 6:32:08 0 [ERROR] InnoDB: Flagged corruption of `cardChipNumber` in table `mydb`.`idm_usc_kaart_status` in purge
> 2025-05-13 10:39:20 442752 [ERROR] Got error 180 when reading table './mydb/idm_usc_kaart_status'
> 2025-05-13 12:44:51 0 [ERROR] InnoDB: tried to purge non-delete-marked record in index `inschrijving_id` of table `mydb`.`tinschrijving`: tuple: TUPLE (info_bits=0, 2 fields): {[4] ](0x800D0C5D),[4] \Y[(0x805C595B)}, record: COMPACT RECORD(info_bits=0, 2 fields): {[4] ](0x800D0C5D),[4] \Y[(0x805C595B)}
> 2025-05-13 12:44:51 0 [ERROR] InnoDB: Flagged corruption of `inschrijving_id` in table `mydb`.`tinschrijving` in purge
> 2025-05-13 12:45:59 453516 [ERROR] Got error 180 when reading table './mydb/tinschrijving'
> 2025-05-13 12:46:03 453524 [ERROR] Got error 180 when reading table './mydb/tinschrijving'
> 2025-05-13 12:46:21 453540 [ERROR] Got error 180 when reading table './mydb/tinschrijving'
> 2025-05-13 12:46:26 453541 [ERROR] Got error 180 when reading table './mydb/tinschrijving'
and last thursday suddenly again:
> 2025-05-29 6:32:22 290727 [ERROR] Got error 180 when reading table './mydb/idm_usc'
It starts without an obvious (to us) reason.
We have verified that we are running (with what seems to be) safe
settings, like: innodb_doublewrite = ON and innodb_flush_method = O_DIRECT |
The system has enough ram, swap configured but none in use, no full
disks/partitions.
At a certain moment (after may 13) it became so bad, that I took a full
mysqldump of all DB's:
> mysqldump --all-databases --single-transaction --routines --triggers --events > /mnt/sdc/full_admin_backup.sql
emptied the datadir, and imported it all fresh again:
> mysql < /mnt/sdc/full_admin_backup.sql
And to my surprise, last week thursday, it happened again.
Filesystem is xfs, mounted like:
> /dev/sdb on /data type xfs (rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota)
No fs corruption detected checked with
> xfs_scrub /data
and
> xfs_spaceman -c 'health -c ' /data/ | grep -v ok
We are unsure what causes this. It should not be possible (I guess?) for
userspace / php scripts (this is LAMP setup) to cause issues like this.
We are unsure how to further troubleshoot.
Anyone with valuable suggestions or insights? Or more information
required..?
Thanks in advance!
MJ
