- developers - lists.mariadb.org

[Maria-developers] MDEV-5277 - Ensure that all MySQL 5.6 options are supported by the MariaDB 10.0 server
by Sergey Vojtovich 20 Nov '13

20 Nov '13

Hi! I'm gathering 5.6 options that are not present in 10.0 and the way we want to handle them. Could you check options listed below and if you're aware of their destiny indicate if options: * not used or needed in MariaDB * to be implemented in later 10.x versions * tuned via different 10.x specific options The following options exist in 5.6 but not in 10.0: * default-tmp-storage-engine * log-raw * default_authentication_plugin * ignore-db-dir * binlog-max-flush-queue-time * binlog-row-image * explicit-defaults-for-timestamp * master-info-repository * relay-log-info-repository * binlog-rows-query-log-events * binlog-order-commits * log-throttle-queries-not-using-indexes * end-markers-in-json * optimizer-trace * optimizer-trace-features * optimizer-trace-offset * optimizer-trace-limit * optimizer-trace-max-mem-size * eq-range-index-dive-limit * server-uuid * server-id-bits * slave-rows-search-algorithms * table-open-cache-instances * slave-allow-batching * slave-checkpoint-period * slave-checkpoint-group * slave-parallel-workers * slave-pending-jobs-size-max * enforce-gtid-consistency * gtid-mode * disconnect-on-expired-password * sha256-password-private-key-path (sha256_password plugin) * sha256-password-public-key-path (sha256_password plugin) The following options exist in 5.5 and 5.6 but not in 10.0: * abort-slave-event-count => debug-abort-slave-event-count * disconnect-slave-event-count => debug-disconnect-slave-event-count * exit-info => debug-exit-info * max-binlog-dump-events => debug-max-binlog-dump-events * sporadic-binlog-dump-fail => debug-sporadic-binlog-dump-fail * new * skip-new * skip-stack-trace The following options were added after 5.6.10: * innodb-buffer-pool-evict // UNIV_DEBUG * innodb-ft-total-cache-size * innodb-ft-result-cache-limit * innodb-log-compressed-pages * innodb-log-checkpoint-now // UNIV_DEBUG * rpl-stop-slave-timeout * validate-user-plugins The following NO_CMD_LINE options exist in 5.6 but not in 10.0 (won't add them): * gtid-executed * gtid-next * gtid-next-list * gtid-owned * gtid-purged * log-bin-basename * relay-log-basename * transaction-allow-batching Thanks, Sergey

4 12

Re: [Maria-developers] Rev 3974: MDEV-5182 - build of 10.0.4/r3863 fails @ 'cmake'
by Sergei Golubchik 20 Nov '13

20 Nov '13

Hi, Sergey! On Nov 18, Sergey Vojtovich wrote: > ------------------------------------------------------------ > revno: 3974 > revision-id: svoj(a)mariadb.org-20131118114801-1kny2cvx8kpjsw7t > parent: jplindst(a)mariadb.org-20131115132442-5d4hqg5idjta2dni > committer: Sergey Vojtovich <svoj(a)mariadb.org> > branch nick: 5.5-mdev5182 > timestamp: Mon 2013-11-18 15:48:01 +0400 > message: > MDEV-5182 - build of 10.0.4/r3863 fails @ 'cmake' with > -DINSTALL_SYSCONFDIR/-DDEFAULT_SYSCONFDIR specified > > The reason for a bug is in support-files/CMakeLists.txt that tries to > install files in INSTALL_SYSCONF2DIR, without checking whether this > variable is actually defined. > > Normally INSTALL_SYSCONF2DIR is always defined if INSTALL_SYSCONFDIR is > defined, and support-files/CMakeLists.txt assumes that. But when one > specifies INSTALL_SYSCONFDIR manually on the command line, this > assumption might be broken. Looks ok to me. An alternative solution would be to define INSTALL_SYSCONF2DIR if INSTALL_SYSCONFDIR is defined. Regards, Sergei

2 1

[Maria-developers] Open Questions in the Knowledge Base
by Ian Gilfillan 19 Nov '13

19 Nov '13

There are quite a few unanswered questions in the Knowledge Base https://mariadb.com/kb/en/+questions/ If anyone knowledgeable in those areas can help answer some of these, even just a request for more info, that would be great :) thanks, ian

2 1

Re: [Maria-developers] Accessing rpl_global_gtid_binlog_state is not consistently protected
by Kristian Nielsen 19 Nov '13

19 Nov '13

Pavel Ivanov <pivanof(a)google.com> writes: > sql/log.cc has highly inconsistent pattern of accessing > rpl_global_gtid_binlog_state. In some places it's protected by mutex > rpl_global_gtid_binlog_state.LOCK_binlog_state, in some places it's > protected by global mutex LOCK_rpl_gtid_state, and in some places it's > not protected by any mutex at all. This is a bug that should be fixed, > right? Yes, agree, thanks for finding this. > I can file this in JIRA if you want, just wanted to let you know faster. I've filed it as MDEV-5306. > I've noticed this problem after one of our AddressSanitizer testing > bots reported that rpl_binlog_state::bump_seq_no_if_needed (called > from MYSQL_BIN_LOG::bump_seq_no_counter_if_needed) was using memory > from hash after it was already freed by rpl_binlog_state::reset > (called by MYSQL_BIN_LOG::reset_logs). Can you test again with the latest patch I've just pushed to 10.0-base? I looked briefly into making an mtr test case to catch this, but I did not see any easy way to do that. I went through the code to try and check all uses of rpl_binlog_state for proper locking, but it would be nice to get a confirmation that I did not miss anything. - Kristian.

3 2

Re: [Maria-developers] MariaDB Galera replication
by Pavel Ivanov 18 Nov '13

18 Nov '13

I'm starting a new thread as this is already doesn't have anything to do with the original topic. On Fri, Nov 15, 2013 at 10:46 AM, Alex Yurchenko <alexey.yurchenko(a)codership.com> wrote: >>> Please pardon this arrogant interruption of your discussion and shameless >>> self-promotion, but I just could not help noticing that Galera >>> replication >>> was designed specifically with these goals in mind. And it does seem to >>> achieve them better than semi-sync plugin. Have you considered Galera? >>> What >>> makes you prefer semi-sync over Galera, if I may ask? >> >> >> To be honest I never looked at how Galera works before. I've looked at >> it now and I don't see how it can fit with us. The major disadvantages >> I immediately see: >> 1. Synchronous replication. That means client must wait while >> transaction is applied on all nodes which is unacceptably big latency >> of each transaction. And what if there's a network blip and some node >> becomes inaccessible? All writes will just freeze? I see the statement >> that "failed nodes automatically excluded from the cluster", but to do >> that cluster must wait for some timeout in case it's indeed a network >> blip and node will "quickly" reconnect. And every client must wait for >> cluster to decide what happened with that one node. >> 2. Let's say node fell out of the cluster for 5 minutes and then >> reconnected. I guess it will be treated as "new node", it will >> generate state transfer and the node will start downloading the whole >> database? And while it's trying to download say 500GB of data files >> all other nodes (or maybe just donor?) won't be able to change those >> files locally and thus will blow up its memory consumption. That means >> they could quickly run out of memory and "new node" won't be able to >> finish its "initialization"... >> 3. It looks like there's strong asymmetry in starting cluster nodes -- >> the first one should be started with empty wsrep_cluster_address and >> all others should be started with the address of the first node. So I >> can't start all nodes uniformly and then issue some commands to >> connect them to each other. That's bad. >> 4. What's the transition path? How do I upgrade MySQL/MariaDB >> replicating using usual replication to Galera? It looks like there's >> no such path and the solution is stop the world using regular >> replication and restart it using Galera. Sorry I can't do that with >> our production systems. >> >> I believe these problems are severe enough for us, so that we can't >> work with Galera. > > > Pavel, you seem to be terribly mistaken on almost all accounts: > > 1. *Replication* (i.e. data buffer copying) is indeed synchronous. But > nobody said that commit is. What Galera does is very similar to semi-sync, > except that it does it technically better. I would not dare to suggest > Galera replication if I didn't believe it to be superior to semi-sync in > every respect. Well, apparently we have a different understanding of what the term "synchronous replication" means. This term is all over the Galera doc, but I didn't find the detailed description of how actually Galera replication work. So I assumed that my understanding of the term (which actually seem to be in line with wiki's definitions http://en.wikipedia.org/wiki/Replication_(computing) ) is what was implied there. So I hope you'll be able to describe in detail how Galera replication works. > As an example here's an independent comparison of Galera vs. > semi-sync performance: > http://linsenraum.de/erkules/2011/06/momentum-galera.html. This is a nice blog post written in German and posted in 2011. And while Google Translate gave me an idea what post was about it would be nice to see something more recent and with better description of what was the actual testing set up. > In fact, majority > of Galera users migrated from the regular *asynchronous* MySQL replication, > which I think is a testimony to Galera performance. I don't mean to troll, but this can also mean that everyone who migrated didn't care much about performance and Galera's performance was within sane boundaries... BTW, just found here https://mariadb.com/kb/en/mariadb-galera-cluster-known-limitations/ : "by design performance of the cluster cannot be higher than performance of the slowest node; however, even if you have only one node, its performance can be considerably lower comparing to running the same server in a standalone mode". That contradicts your words. > 2. Node reconnecting to cluster will normally receive only events that it > missed while being disconnected. This seem to contradict to the docs. Again from https://mariadb.com/kb/en/mariadb-galera-cluster-known-limitations/ : "After a temporary split, if the 'good' part of the cluster was still reachable and its state was modified, resynchronization occurs". > 3. You are partially right about it, but isn't it much different from > regular MySQL replication where you first need to set up master and then > connect slaves (even if you have physically launched the servers at the same > time). Operation of setting up master and then connecting slaves consists of mostly only executing CHANGE MASTER TO and then START SLAVE on all slaves after all MySQL instances (including master) were started with the same set of command line flags. This is fundamentally different from starting instances with different arguments, especially when these arguments should be different depending on whether the replica is starting first or there's already some other replica running. > Yet, Galera nodes can be started simultaneously and then joined > together by setting wsrep_cluster_address from mysql client connection. This > is not advertised method, because in that case state snapshot transfer can > be done only by mysqldump. If you set the address in advance, rsync or > xtrabackup can be used to provision the fresh node. This is of course better because I can start all instances with the same command line arguments. But transferring snapshot of a very big database using mysqldump, and causing the node that creates mysqldump to blow up memory consumption during the process, that is still a big problem. > 4. Every Galera node can perfectly work as either master or slave to native > MySQL replication. So migration path is quite clear. Nope, not clear yet. So I'll be able to upgrade all my MySQL instances to a Galera-supporting binary while they are replicating using standard MySQL replication. That's good. Now, how the Galera replication is turned on after that? What will happen if I just set wsrep_cluster_address address on all replicas? What will replicas do, and what will happen with the standard MySQL replication? > It is very sad that you happen to have such gross misconceptions about > Galera. If those were true, how would MariaDB Galera Cluster get paying > customers? Care to share some numbers? Like what's the rough amount of those paying customers? What size is the biggest installation -- number of clusters, replicas, highest QPS load? I'm not asking to share any confidential information, but the rough ballpark of the numbers would be helpful. > May be my reply will convince you to have a second look at it. > (In addition to the above Galera is fully multi-master, does parallel > applying and works great in WAN) I hope your explanation of how Galera replication work will help me understand how great it works over WAN and how you could make full multi-master work without fully synchronous replication in my understanding of that term. Pavel

3 9

[Maria-developers] What is bundled libevent library for?
by Pavel Ivanov 18 Nov '13

18 Nov '13

Sergei, Do you know why MariaDB sources have libevent library bundled with them? I don't see it being used anywhere. I see it was added in this commit by Monty http://bazaar.launchpad.net/~maria-captains/maria/10.0/revision/3492.3.1, but even in that commit I can't find any usages of the library. So should it be removed from MariaDB? Thank you, Pavel

4 4

[Maria-developers] Why local modifications to PCRE library?
by Pavel Ivanov 16 Nov '13

16 Nov '13

Alexander, I see you've included the full source of PCRE library into MariaDB tree and even made some local modifications which are not included into PCRE upstream. Could you please explain why did you do that? Why not allow to build with system standard PCRE library? I understand that you could have needed some modifications and thus could want to include sources from PCRE trunk to allow users to build with modifications when their system-wide library is too old. But as I don't see your modifications on PCRE trunk that means you probably had something different in mind, and decided to incur the pain of constantly updating the included sources and merging security fixes. So why did you decide to do that? Thank you, Pavel

3 7

Re: [Maria-developers] Semisync plugin incompatibility
by Kristian Nielsen 15 Nov '13

15 Nov '13

Pavel Ivanov <pivanof(a)google.com> writes: > We've noticed recently that semisync_master plugin in MariaDB (which > apparently was fully inherited from MySQL) is seriously incompatible > with our understanding of the purpose of semi-sync replication. This > incompatibility was apparently introduced as a fix for > http://bugs.mysql.com/bug.php?id=45672. The "major no-no" that bug So as I understand it, this bug is about what should happen when semisync is enabled, but no slaves are connected. Apparently before the fix of Bug#45672, an error was thrown late during COMMIT. So the transaction was committed (locally on the master), but the client still got an error back. And if I understand correctly, after the fix of Bug#45672, no error is thrown in the case where no slave is connected. > talks about is in our opinion the whole purpose of semi-sync > replication -- if transaction is not replicated to at least one slave > client shouldn't get OK even if transaction is committed locally on > the master. Also master shouldn't just turn off semi-sync replication > whenever it wants. So with "just turn off semi-sync replication whenever it wants" - what are you refering to here? I seem to remember that semisync has a timeout, and it gets disabled if that timeout triggers? My guess is that this is what you have in mind, but I wanted to ask to make sure ... > We will fix this problem for us, but first I wanted to understand > what's your view of the purpose of semi-sync replication and how you > think it should work? I need to know your opinion to understand how I > should fix this issue... Well, personally, I never was much interested in semi-sync. But it is my understanding that there is some interest, so I will answer with what small opinion I have. I suppose the general idea is that when client sees its COMMIT complete, it can know that its transaction exists in at least two places (master binlog + at least one slave relay log). So there is no longer any single point of failure that can cause loss of the transaction. Another point of view I is that semi-sync provides some sort of throttle on how fast the master can generate events compared to how fast the slaves can receive them: http://www.mysqlperformanceblog.com/2012/01/19/how-does-semisynchronous-mys… There was also a suggestion (and a patch is floating around somewhere) for "enhanced semisync replication": https://mariadb.atlassian.net/browse/MDEV-162 This delays not only client acknowledge but also InnoDB commit until the ack from at least one slave, which means that transactions are not visible to other clients until they exist on at least one slave in addition to on the master. Since this is _semi_-sync, not real two-phase commit synchronous replication, the main problem is that there is way to ensure consistency in the general error case. The transaction is already fully committed on the master, it cannot be rolled back. So we are left with the choice of one of two evils: 1. Report an error to the client. Most clients would then probably wrongly assume that the transaction was _not_ committed. There also does not seem to be much the client can do about the error except perhaps log an incident to the monitoring system. On the other hand, then at least the problem is not silently ignored. 2. Report success to the client but complain loudly in the error log (I assume this is what happens in current code). This leaves the client unaware that there is a problem (but presumably the monitoring system will catch the message in the error log). >From this summary, I think I can see the logic of the current behaviour: - It preserves protection against single-point-of-failure. If all slaves are gone, then we already have one failure, and unless we experience a double failure (master also failing before slave recovers), the transaction will eventually be sent to a slave and no overall failure happens. - If the client can anyway not do anything about the problem except notify the monitoring system, the server may as well do the notification itself. But the opposite point of view also has merit. The client asked for semi-sync behavior, but did not get it, and it does not even have a way to know about the problem. That is not good. Does the client currently at least get a warning for the COMMIT? I think it should (eg. the fix for Bug#45672 should at least have been to turn the error into a warning, not remove the error completely). What I think could make sense is if the client got an error during the prepare phase if no slaves are connected. In this case we _can_ roll back the transaction and give an error to the client without any issue of consistency. But it still leaves a small window where the last slave can disappear between the prepare and the commit phase and leave us with the original problem. I hope this helps you ... Maybe you can describe your use-case, and how you need things to work for that case? Personally I have nothing against changing this behaviour to something more logical, I am just not sure what the most logical behaviour is ... - Kristian.

4 8

[Maria-developers] parallel replication monitoring
by Giuseppe Maxia 15 Nov '13

15 Nov '13

What can I use to see the status of parallel replication? I am looking for: * how many workers are used; * in which schema they are working; * what query they are running. So far, I only see mysql.gtid_slave_pos and information_schema.processlist. Anything else available? Thanks in advance Giuseppe

4 6

[Maria-developers] Slave can take a very long time to start replication
by Pavel Ivanov 15 Nov '13

15 Nov '13

Kristian, As I understand currently when slave connects to master and wants to start replicating it passes GTID to start from, master finds binlog file where the earliest GTID is located and then scans through that file to find the exact binlog position to start sending binlog events from. If this binlog file is pretty big then scanning can take a very long time. I guess especially long when several slaves try to start replicating roughly at the same time. We observed 60-90 seconds between initial connection by the slave and the first real binlog events starting to flow. In this period of time slave doesn't receive anything from master and thus it's very easy to confuse such situation with connection loss, hit slave_net_timeout, reopen connection to the master again and force it to start searching through binlog file from the very beginning... Putting aside the argument of what value is good enough for slave_net_timeout, I'd say in any case slave taking 60 seconds to just start receiving binlog events from master is unacceptable. Did you think about this problem before? Maybe you've even planned already to implement some solution for this? Thank you, Pavel

2 1