[Maria-discuss] Mounting tmpdir via NFS
Hello, I have a question that I've been struggling to get a clear answer to, or even any indication someone has done this before, and would appreciate any experience or thoughts. Are there any known issues with defining tmpdir on multiple servers to point to a single NFS-mounted network share? Would each instance co-habit peacefully and not try to change any files from another instance? Why do I want to do this? I have around 80 Centos/Rocky vms which run Mariadb 10.6.x. Due to the nature of the work they do, some of them require a significant amount of tmpdir (up to 75Gb) relatively briefly (once or twice a day for a few minutes) This growth is not always predictable, so generally the first I know about it is when Mariadb reports lowspace and the query stops. We then grow the /tmp partition that is defined as tmpdir and if we're quick enough, the query completes, but often the job dies. Then that extra space stays locked away forever, even if that server never tries to use that much tmpdir again. This does not feel efficient and I am pondering better ways of doing this. One such thought would be to have a single HA NFS share on our network, mounted onto each VM as something like /mariadb-tempdir, and set $tmpdir to it. In my mind, this would make a better use of one large allocation of space used by many things often, rather than lots of individual small to large sized bits of space used by one thing, rarely. The chance of two or more servers filling that space feels small, but if it happened, one would wait until space was cleared - which might actually happen in this shared scenario when it never does in a single-server situation. In my particular world, we have fast storage and a fast network. Speed tests on large and tiny files over NFS give surprisingly similar performance to writing to local vm disks, so I'm not worried about the overhead of NFS and even if it was, slow might still be preferred to full. Thank you, Simon
Op 28 mrt. 2023 om 17:02 heeft Simon Avery <Simon.Avery@atass-sports.co.uk> het volgende geschreven:
Hello,
I have a question that I’ve been struggling to get a clear answer to, or even any indication someone has done this before, and would appreciate any experience or thoughts.
Are there any known issues with defining tmpdir on multiple servers to point to a single NFS-mounted network share? Would each instance co-habit peacefully and not try to change any files from another instance?
Why do I want to do this?
I have around 80 Centos/Rocky vms which run Mariadb 10.6.x. Due to the nature of the work they do, some of them require a significant amount of tmpdir (up to 75Gb) relatively briefly (once or twice a day for a few minutes)
This growth is not always predictable, so generally the first I know about it is when Mariadb reports lowspace and the query stops. We then grow the /tmp partition that is defined as tmpdir and if we’re quick enough, the query completes, but often the job dies. Then that extra space stays locked away forever, even if that server never tries to use that much tmpdir again. This does not feel efficient and I am pondering better ways of doing this.
One such thought would be to have a single HA NFS share on our network, mounted onto each VM as something like /mariadb-tempdir, and set $tmpdir to it.
In my mind, this would make a better use of one large allocation of space used by many things often, rather than lots of individual small to large sized bits of space used by one thing, rarely. The chance of two or more servers filling that space feels small, but if it happened, one would wait until space was cleared – which might actually happen in this shared scenario when it never does in a single-server situation.
In my particular world, we have fast storage and a fast network. Speed tests on large and tiny files over NFS give surprisingly similar performance to writing to local vm disks, so I’m not worried about the overhead of NFS and even if it was, slow might still be preferred to full.
Any reason you’re thinking of using a single share for all MariaDB machines? How about setting up a single NFS VM with lots of space, and mounting a specific directory per MariaDB machine?
Thank you,
Simon _______________________________________________ Mailing list: https://launchpad.net/~maria-discuss Post to : maria-discuss@lists.launchpad.net Unsubscribe : https://launchpad.net/~maria-discuss More help : https://help.launchpad.net/ListHelp
On Tuesday 28 March 2023 at 17:40:33, William Edwards wrote:
Op 28 mrt. 2023 om 17:02 heeft Simon Avery <Simon.Avery@atass-sports.co.uk> het volgende geschreven:
Hello,
Are there any known issues with defining tmpdir on multiple servers to point to a single NFS-mounted network share? Would each instance co-habit peacefully and not try to change any files from another instance?
Any reason you’re thinking of using a single share for all MariaDB machines?
How about setting up a single NFS VM with lots of space, and mounting a specific directory per MariaDB machine?
Alternatively, if disk space is a concern, how about creating an NFS share of, say, 150Gbytes, containing one subdirectory per server, and then mount each subdirectory as /tmp on the respective machine? That avoids allocating 75Gbytes per server, but also keeps each machine's view separate from the others. Antony. -- The words "e pluribus unum" on the Great Seal of the United States are from a poem by Virgil entitled "Moretum", which is about cheese and garlic salad dressing. Please reply to the list; please *don't* CC me.
Hi William Op 28 mrt. 2023 om 17:02 heeft Simon Avery <Simon.Avery@atass-sports.co.uk<mailto:Simon.Avery@atass-sports.co.uk>> het volgende geschreven: Are there any known issues with defining tmpdir on multiple servers to point to a single NFS-mounted network share? Would each instance co-habit peacefully and not try to change any files from another instance? Why do I want to do this? I have around 80 Centos/Rocky vms which run Mariadb 10.6.x. Due to the nature of the work they do, some of them require a significant amount of tmpdir (up to 75Gb) relatively briefly (once or twice a day for a few minutes) This growth is not always predictable, so generally the first I know about it is when Mariadb reports lowspace and the query stops. We then grow the /tmp partition that is defined as tmpdir and if we’re quick enough, the query completes, but often the job dies. Then that extra space stays locked away forever, even if that server never tries to use that much tmpdir again. This does not feel efficient and I am pondering better ways of doing this. One such thought would be to have a single HA NFS share on our network, mounted onto each VM as something like /mariadb-tempdir, and set $tmpdir to it. In my mind, this would make a better use of one large allocation of space used by many things often, rather than lots of individual small to large sized bits of space used by one thing, rarely. The chance of two or more servers filling that space feels small, but if it happened, one would wait until space was cleared – which might actually happen in this shared scenario when it never does in a single-server situation. In my particular world, we have fast storage and a fast network. Speed tests on large and tiny files over NFS give surprisingly similar performance to writing to local vm disks, so I’m not worried about the overhead of NFS and even if it was, slow might still be preferred to full. Any reason you’re thinking of using a single share for all MariaDB machines? How about setting up a single NFS VM with lots of space, and mounting a specific directory per MariaDB machine? Thank you, Simon _______________________________________________ Mailing list: https://launchpad.net/~maria-discuss Post to : maria-discuss@lists.launchpad.net<mailto:maria-discuss@lists.launchpad.net> Unsubscribe : https://launchpad.net/~maria-discuss More help : https://help.launchpad.net/ListHelp
(Apologies for half-formed email sent prematurely) Hi William Thanks for your reply Op 28 mrt. 2023 om 17:02 heeft Simon Avery <Simon.Avery@atass-sports.co.uk<mailto:Simon.Avery@atass-sports.co.uk>> het volgende geschreven: Are there any known issues with defining tmpdir on multiple servers to point to a single NFS-mounted network share? Would each instance co-habit peacefully and not try to change any files from another instance? * Any reason you’re thinking of using a single share for all MariaDB machines? * How about setting up a single NFS VM with lots of space, and mounting a specific directory per MariaDB machine? It’s certainly an option, although I’d probably go for Antony’s suggestion of separate subdirs if there was any chance of interference. That might still be preferable for good housekeeping reasons. The downside is that both require setup on the NFS host when adding a new machine, and whilst that’s not out of the question, I’m wondering if I *need* to do that? Thanks, Simon
participants (3)
-
Antony Stone -
Simon Avery -
William Edwards