InnoDB buffer pool usage drops regularly since 10.11.10 / 11.4.4
Hi, We see an odd pattern where InnoDB buffer pool usage drops regularly. Attached are two screenshots of memory usage on an 11.4.4 node: one at 20:15 with an InnoDB buffer pool of 47,5 GiB, and 5 minutes later with a 3,69 GiB one. The times at which this occurs are -seemingly- random. Other internal memory structures are stable. On at least one node with this issue, the InnoDB buffer pool is smaller (48 GiB) than the sum of data + indexes (87,57 GiB). As far as I can see (graphing history is limited), this has started occurring after upgrading from MariaDB 10.11.9 to 10.11.10 and from 11.4.3 to 11.4.4 respectively. At first sight, in the pre-release notes for 10.11.11 and 11.4.5, I don't see any related issues. To make absolutely sure this is related to the upgrade, we will be downgrading a troubled node tonight, to see if it fixes anything. I will report back. In userland, these drops cause a domino effect: no cache -> more I/O -> slower queries -> nuisance in application. Has anyone else seen this pattern, possibly after upgrading to the aforementioned versions? Although 1) neither of these versions are bleeding edge and 2) I haven't seen any such reports, asking is worth a shot as this started occurring on several machines after upgrading with no configuration nor (obvious) usage changes. With kind regards, William David Edwards
Hi, As promised yesterday, I would follow up to verify that InnoDB buffer pool usage would no longer drop when downgrading from 11.4.4 to 11.4.3. Confirmed. See attachment. The time at which usage stabilised is the time at which MariaDB was downgraded. Anyone recognise this behaviour? Thank you, William William David Edwards via discuss schreef op 2024-12-09 16:37:
Hi,
We see an odd pattern where InnoDB buffer pool usage drops regularly. Attached are two screenshots of memory usage on an 11.4.4 node: one at 20:15 with an InnoDB buffer pool of 47,5 GiB, and 5 minutes later with a 3,69 GiB one. The times at which this occurs are -seemingly- random. Other internal memory structures are stable.
On at least one node with this issue, the InnoDB buffer pool is smaller (48 GiB) than the sum of data + indexes (87,57 GiB).
As far as I can see (graphing history is limited), this has started occurring after upgrading from MariaDB 10.11.9 to 10.11.10 and from 11.4.3 to 11.4.4 respectively. At first sight, in the pre-release notes for 10.11.11 and 11.4.5, I don't see any related issues. To make absolutely sure this is related to the upgrade, we will be downgrading a troubled node tonight, to see if it fixes anything. I will report back.
In userland, these drops cause a domino effect: no cache -> more I/O -> slower queries -> nuisance in application.
Has anyone else seen this pattern, possibly after upgrading to the aforementioned versions? Although 1) neither of these versions are bleeding edge and 2) I haven't seen any such reports, asking is worth a shot as this started occurring on several machines after upgrading with no configuration nor (obvious) usage changes.
With kind regards,
William David Edwards _______________________________________________ discuss mailing list -- discuss@lists.mariadb.org To unsubscribe send an email to discuss-leave@lists.mariadb.org
Met vriendelijke groeten, William David Edwards
Hi William, Your finding could be explained by the fact that https://jira.mariadb.org/browse/MDEV-34753 was fixed in MariaDB Server 11.4.4. In the 11.4.4 server error log, do you see any messages like the following: [Note] InnoDB: Memory pressure event freed ... pages Best regards, Marko
Hi Marko, Marko Mäkelä schreef op 2024-12-10 15:02:
Hi William,
Your finding could be explained by the fact that https://jira.mariadb.org/browse/MDEV-34753 was fixed in MariaDB Server 11.4.4. In the 11.4.4 server error log, do you see any messages like the following: [Note] InnoDB: Memory pressure event freed ... pages
That's most definitely it. Some systems are subject to memory pressure, and I've never seen InnoDB adjust its buffer pool based on that, but I now gather that is actually intended behaviour. Thanks for pointing me in the right direction!
Best regards,
Marko
Met vriendelijke groeten, William David Edwards
Dear William, memory loss always happens when a message comes up with [Note] InnoDB: Memory pressure event freed 2210366 pages After this memory loss can be seen with top (no zombies) MiB Mem: 62785 total, 957,2 free, 9839.1 used, 40878.3 buff/cache About 11GB missing. We have mariadb 10.11.11 . We didn't saw this before because we had problems with https://jira.mariadb.org/browse/MDEV-35508 (see also https://jira.mariadb.org/browse/MDEV-35829) Best regards Hansjuergen
participants (3)
-
hansjuergen.riess＠forum-media.com -
Marko Mäkelä -
William David Edwards