[Maria-discuss] Most reliable SST (or IST ?) method for MariaDB Cluster
Hi all, I'm using MariaDB Cluster with 3 servers and I often have split brain because of network. When a network split brain happen, only one node is disconnected. What is for you the best choice for SST or maybe IST method for this configuration ? (N.B.: I already tried rsync method but i'm not really satisfied with it). I have another question, I tried xtrabackup-v2 but it seems this method can't be use for bootstrapping while database is on. Does xtrabackup method can do this ? Regards, -- CHUNKZ.NET - script kiddie and computer technician Bertrand Caplet, Flers (FR) Feel free to send encrypted/signed messages Key ID: 37F70C30 GPG FP: 134A 4027 518B 5F4D D409 558D BA9B 7BF0 37F7 0C30
Hi Bertrand, You don't want your partitioned node to do SST, because IST is obviously faster. IST should always happen, unless the node has lost state (crash, inconsistent data, etc) or the latest writeset is not found in the galera cache. So, rule of thumb is to have a galera cache big enough, so it can hold the latest writesets while your node is partitioned. Galera cache (the name is a bit misleading, it's more like a binary log) is just a file on disk, so the limitation is about disk space only. Set it like this: wsrep_provider_options="gcache.size=5G" About xtrabackup methods. the legacy xtrabackup method should not be used anymore, as it is deprecated because of many bugs/issue. Use xtrabackup-v2 in all cases. You can improve the method by setting some [xtrabackup] options in your my.cnf file, such as: # Enables compression, requires qpress package compress # Enables parallel transfers (n=number of threads) parallel=n xtrabackup-v2 method does not lock the donor database while it is processing, so it can be used for "hot" SST (on the contrary, rsync needs a global lock) Regards, On Sat, Dec 5, 2015 at 2:46 PM Bertrand Caplet <bertrand.caplet@chunkz.net> wrote:
Hi all, I'm using MariaDB Cluster with 3 servers and I often have split brain because of network. When a network split brain happen, only one node is disconnected. What is for you the best choice for SST or maybe IST method for this configuration ? (N.B.: I already tried rsync method but i'm not really satisfied with it).
I have another question, I tried xtrabackup-v2 but it seems this method can't be use for bootstrapping while database is on. Does xtrabackup method can do this ?
Regards,
-- CHUNKZ.NET - script kiddie and computer technician Bertrand Caplet, Flers (FR) Feel free to send encrypted/signed messages Key ID: 37F70C30 GPG FP: 134A 4027 518B 5F4D D409 558D BA9B 7BF0 37F7 0C30
_______________________________________________ Mailing list: https://launchpad.net/~maria-discuss Post to : maria-discuss@lists.launchpad.net Unsubscribe : https://launchpad.net/~maria-discuss More help : https://help.launchpad.net/ListHelp
-- Guillaume Lefranc Remote DBA Services Manager MariaDB Corporation
Hi Guillaume, What variable(s) should we monitor to estimate the rate at which gcache is written? or to otherwise estimate how far back a given gcache.size goes in history? Tom From: Maria-discuss <maria-discuss-bounces+fsb=thefsb.org@lists.launchpad.net> on behalf of Guillaume Lefranc <guillaume.lefranc@mariadb.com> Date: Saturday, December 5, 2015 at 10:23 AM To: Bertrand Caplet <bertrand.caplet@chunkz.net>, maria-discuss email list <maria-discuss@lists.launchpad.net> Subject: Re: [Maria-discuss] Most reliable SST (or IST ?) method for MariaDB Cluster Hi Bertrand, You don't want your partitioned node to do SST, because IST is obviously faster. IST should always happen, unless the node has lost state (crash, inconsistent data, etc) or the latest writeset is not found in the galera cache. So, rule of thumb is to have a galera cache big enough, so it can hold the latest writesets while your node is partitioned. Galera cache (the name is a bit misleading, it's more like a binary log) is just a file on disk, so the limitation is about disk space only. Set it like this: wsrep_provider_options="gcache.size=5G" About xtrabackup methods. the legacy xtrabackup method should not be used anymore, as it is deprecated because of many bugs/issue. Use xtrabackup-v2 in all cases. You can improve the method by setting some [xtrabackup] options in your my.cnf file, such as: # Enables compression, requires qpress package compress # Enables parallel transfers (n=number of threads) parallel=n xtrabackup-v2 method does not lock the donor database while it is processing, so it can be used for "hot" SST (on the contrary, rsync needs a global lock) Regards, On Sat, Dec 5, 2015 at 2:46 PM Bertrand Caplet <bertrand.caplet@chunkz.net> wrote:
Hi all, I'm using MariaDB Cluster with 3 servers and I often have split brain because of network. When a network split brain happen, only one node is disconnected. What is for you the best choice for SST or maybe IST method for this configuration ? (N.B.: I already tried rsync method but i'm not really satisfied with it).
I have another question, I tried xtrabackup-v2 but it seems this method can't be use for bootstrapping while database is on. Does xtrabackup method can do this ?
Regards,
-- CHUNKZ.NET <http://CHUNKZ.NET> - script kiddie and computer technician Bertrand Caplet, Flers (FR) Feel free to send encrypted/signed messages Key ID: 37F70C30 GPG FP: 134A 4027 518B 5F4D D409 558D BA9B 7BF0 37F7 0C30
_______________________________________________ Mailing list: https://launchpad.net/~maria-discuss Post to : maria-discuss@lists.launchpad.net Unsubscribe : https://launchpad.net/~maria-discuss More help : https://help.launchpad.net/ListHelp -- Guillaume Lefranc Remote DBA Services Manager MariaDB Corporation _______________________________________________ Mailing list: https://launchpad.net/~maria-discuss Post to : maria-discuss@lists.launchpad.net Unsubscribe : https://launchpad.net/~maria-discuss More help : https://help.launchpad.net/ListHelp
Hi Tom, Use the wsrep_replicated_bytes and wsrep_received_bytes status variables. The sum of those two variables will give you the total bytes written to gcache. By using a derivative function you should be able to figure out the write rate. Hope this helps, On Sat, Dec 5, 2015 at 6:46 PM Tom Worster <fsb@thefsb.org> wrote:
Hi Guillaume,
What variable(s) should we monitor to estimate the rate at which gcache is written? or to otherwise estimate how far back a given gcache.size goes in history?
Tom
From: Maria-discuss < maria-discuss-bounces+fsb=thefsb.org@lists.launchpad.net> on behalf of Guillaume Lefranc <guillaume.lefranc@mariadb.com> Date: Saturday, December 5, 2015 at 10:23 AM To: Bertrand Caplet <bertrand.caplet@chunkz.net>, maria-discuss email list <maria-discuss@lists.launchpad.net> Subject: Re: [Maria-discuss] Most reliable SST (or IST ?) method for MariaDB Cluster
Hi Bertrand,
You don't want your partitioned node to do SST, because IST is obviously faster. IST should always happen, unless the node has lost state (crash, inconsistent data, etc) or the latest writeset is not found in the galera cache. So, rule of thumb is to have a galera cache big enough, so it can hold the latest writesets while your node is partitioned. Galera cache (the name is a bit misleading, it's more like a binary log) is just a file on disk, so the limitation is about disk space only. Set it like this: wsrep_provider_options="gcache.size=5G"
About xtrabackup methods. the legacy xtrabackup method should not be used anymore, as it is deprecated because of many bugs/issue. Use xtrabackup-v2 in all cases. You can improve the method by setting some [xtrabackup] options in your my.cnf file, such as:
# Enables compression, requires qpress package compress # Enables parallel transfers (n=number of threads) parallel=n
xtrabackup-v2 method does not lock the donor database while it is processing, so it can be used for "hot" SST (on the contrary, rsync needs a global lock)
Regards,
On Sat, Dec 5, 2015 at 2:46 PM Bertrand Caplet <bertrand.caplet@chunkz.net> wrote:
Hi all, I'm using MariaDB Cluster with 3 servers and I often have split brain because of network. When a network split brain happen, only one node is disconnected. What is for you the best choice for SST or maybe IST method for this configuration ? (N.B.: I already tried rsync method but i'm not really satisfied with it).
I have another question, I tried xtrabackup-v2 but it seems this method can't be use for bootstrapping while database is on. Does xtrabackup method can do this ?
Regards,
-- CHUNKZ.NET - script kiddie and computer technician Bertrand Caplet, Flers (FR) Feel free to send encrypted/signed messages Key ID: 37F70C30 GPG FP: 134A 4027 518B 5F4D D409 558D BA9B 7BF0 37F7 0C30
_______________________________________________ Mailing list: https://launchpad.net/~maria-discuss Post to : maria-discuss@lists.launchpad.net Unsubscribe : https://launchpad.net/~maria-discuss More help : https://help.launchpad.net/ListHelp
-- Guillaume Lefranc Remote DBA Services Manager MariaDB Corporation _______________________________________________ Mailing list: https://launchpad.net/~maria-discuss Post to : maria-discuss@lists.launchpad.net Unsubscribe : https://launchpad.net/~maria-discuss More help : https://help.launchpad.net/ListHelp
-- Guillaume Lefranc Remote DBA Services Manager MariaDB Corporation
Hi Guillaume, Yes, it helps. Thanks! I can log these every hour (say) and plot the curve over a week to get an idea of rate and its variability. Tom From: Guillaume Lefranc <guillaume.lefranc@mariadb.com> Date: Saturday, December 5, 2015 at 12:57 PM To: Tom Worster <fsb@thefsb.org>, Bertrand Caplet <bertrand.caplet@chunkz.net>, maria-discuss email list <maria-discuss@lists.launchpad.net> Subject: Re: [Maria-discuss] Most reliable SST (or IST ?) method for MariaDB Cluster Hi Tom, Use the wsrep_replicated_bytes and wsrep_received_bytes status variables. The sum of those two variables will give you the total bytes written to gcache. By using a derivative function you should be able to figure out the write rate. Hope this helps, On Sat, Dec 5, 2015 at 6:46 PM Tom Worster <fsb@thefsb.org> wrote:
Hi Guillaume,
What variable(s) should we monitor to estimate the rate at which gcache is written? or to otherwise estimate how far back a given gcache.size goes in history?
Tom
From: Maria-discuss <maria-discuss-bounces+fsb=thefsb.org@lists.launchpad.net> on behalf of Guillaume Lefranc <guillaume.lefranc@mariadb.com> Date: Saturday, December 5, 2015 at 10:23 AM To: Bertrand Caplet <bertrand.caplet@chunkz.net>, maria-discuss email list <maria-discuss@lists.launchpad.net> Subject: Re: [Maria-discuss] Most reliable SST (or IST ?) method for MariaDB Cluster
Hi Bertrand,
You don't want your partitioned node to do SST, because IST is obviously faster. IST should always happen, unless the node has lost state (crash, inconsistent data, etc) or the latest writeset is not found in the galera cache. So, rule of thumb is to have a galera cache big enough, so it can hold the latest writesets while your node is partitioned. Galera cache (the name is a bit misleading, it's more like a binary log) is just a file on disk, so the limitation is about disk space only. Set it like this: wsrep_provider_options="gcache.size=5G"
About xtrabackup methods. the legacy xtrabackup method should not be used anymore, as it is deprecated because of many bugs/issue. Use xtrabackup-v2 in all cases. You can improve the method by setting some [xtrabackup] options in your my.cnf file, such as:
# Enables compression, requires qpress package compress # Enables parallel transfers (n=number of threads) parallel=n
xtrabackup-v2 method does not lock the donor database while it is processing, so it can be used for "hot" SST (on the contrary, rsync needs a global lock)
Regards,
On Sat, Dec 5, 2015 at 2:46 PM Bertrand Caplet <bertrand.caplet@chunkz.net> wrote:
Hi all, I'm using MariaDB Cluster with 3 servers and I often have split brain because of network. When a network split brain happen, only one node is disconnected. What is for you the best choice for SST or maybe IST method for this configuration ? (N.B.: I already tried rsync method but i'm not really satisfied with it).
I have another question, I tried xtrabackup-v2 but it seems this method can't be use for bootstrapping while database is on. Does xtrabackup method can do this ?
Regards,
-- CHUNKZ.NET <http://CHUNKZ.NET> - script kiddie and computer technician Bertrand Caplet, Flers (FR) Feel free to send encrypted/signed messages Key ID: 37F70C30 GPG FP: 134A 4027 518B 5F4D D409 558D BA9B 7BF0 37F7 0C30
_______________________________________________ Mailing list: https://launchpad.net/~maria-discuss Post to : maria-discuss@lists.launchpad.net Unsubscribe : https://launchpad.net/~maria-discuss More help : https://help.launchpad.net/ListHelp -- Guillaume Lefranc Remote DBA Services Manager MariaDB Corporation _______________________________________________ Mailing list: https://launchpad.net/~maria-discuss Post to : maria-discuss@lists.launchpad.net Unsubscribe : https://launchpad.net/~maria-discuss More help : https://help.launchpad.net/ListHelp -- Guillaume Lefranc Remote DBA Services Manager MariaDB Corporation
participants (3)
-
Bertrand Caplet -
Guillaume Lefranc -
Tom Worster