I discussed with Monty, and we came up with some more suggested changes for the options used to configure the optimistic parallel replication feature (MDEV-6676, https://mariadb.atlassian.net/browse/MDEV-6676). The biggest change is to split up the --slave-parallel-mode into multiple options. I think that is reasonable, probably that option was doing too many things at once. Instead, we could have the following three options: --slave-parallel-mode=all_transactions | follow_master_commits | only_commits | none "all_transactions" is what was called "transactional" before. The slave will try to apply all transactional DML in parallel; in case of conflicts it will roll back the later transaction and retry it. "follow_master_commits" is the 10.0 functionality, apply in parallel transactions that group-committed together on the master (the default). "only_commits" was suggested to me by a user testing parallel replication. It does not attempt to apply transactions in parallel, but still runs the commit steps in parallel, making slave group commit possible and thus saving on fsyncs if durability settings are on. "none" means the parallel replication code is not used (same as --slave-parallel-threads=0, but now configurable per multimaster connection). (This corresponds to empty value in old --slave-parallel-mode). --slave-parallel-domains=on|off (default on) "This replaces the "domain" option of old --slave-parallel-mode. When enabled, parallel replication will apply in parallel transactions whose GTID has different domain ids (GTID mode only). --slave-parallel-wait-if-conflict-on-master=on|off (default on) When enabled, if a transaction had to do a row lock wait on the master, it will not be applied in parallel with any earlier transaction on the slave (idea is that such transaction is likely to get a conflict on the slave, causing a needless retry). (This was the "waiting" option to old --slave-parallel-mode). These options will also be usable per multi-source master connection, like --master1.slave-parallel-mode=all_transactions. The options will be possible to change dynamically also (with SET GLOBAL), though the associated slave threads must be stopped while changing. Also, Monty suggested to rename @@replicate_allow_parallel to @@SESSION.replicate_expect_conflicts=0|1 (default 0) When this option is enabled on the master when a transaction is committed, that transaction will not be applied in parallel with earlier transactions (when --slave-parallel-mode=all_transactions). This can be used to reduce retries on the slave, if an application is about to do a transaction that is likely to cause a conflict and retry on a slave if applied in parallel with earlier transactions. Let me know if there are any comments to these or suggestions for changes. It is best to get these as right as possible before release (seems the intention is to include optimistic parallel replication in 10.1), since it is the user-visible part of the feature. With these option names, the normal way to use optimistic parallel replication would be these two options in my.cnf: slave_parallel_mode=all_transactions slave_parallel_threads=20 (or whatever) This seems reasonably, I think. None of the other options would need be considered except in more special cases. - Kristian.

I still prefer "auto" as default, if you want the fine grained control, I think an optimizer_switch approach is better than adding X new config options, i.e --parallel_mode=option1=true,option3=4 don't you think that there will be new options/variants ? i do don't you think you will want to remove options/variants ? i do I don't like having the word "transaction" in the names of some modes, ALL variants will maintain transaction semantics. Having "transaction" is name of only a few sort of implies that others are not transactional... i think "optimistic" is better than "transactional" or "all_transactions". in the same spirit "conservative" could be "follow_master_commits" i don't understand what "only_commits" is, how can it prepare transaction "queue up" for group commit if only 1 is prepared in parallel ? -- though, i think all suggestions might work if they have good defaults and a solid implementation /Jonas On Thu, Dec 4, 2014 at 9:06 PM, Pavel Ivanov wrote:

On Thu, Dec 4, 2014 at 5:49 AM, Kristian Nielsen wrote:

...

I discussed with Monty, and we came up with some more suggested changes for the options used to configure the optimistic parallel replication feature (MDEV-6676, https://mariadb.atlassian.net/browse/MDEV-6676).

The biggest change is to split up the --slave-parallel-mode into multiple options. I think that is reasonable, probably that option was doing too many things at once.

Instead, we could have the following three options:

--slave-parallel-mode=all_transactions | " | only_commits | none

"all_transactions" is what was called "transactional" before. The slave will try to apply all transactional DML in parallel; in case of conflicts it will roll back the later transaction and retry it.

"follow_master_commits" is the 10.0 functionality, apply in parallel transactions that group-committed together on the master (the default).

"only_commits" was suggested to me by a user testing parallel replication. It does not attempt to apply transactions in parallel, but still runs the commit steps in parallel, making slave group commit possible and thus saving on fsyncs if durability settings are on.

Does this mean that group commit will be possible if slave is able to execute several transactions consecutively while previous transaction commits/fsyncs? I'd suggest to name this option differently because looking just at the list of available values it's not quite clear what could be the difference between follow_master_commits and only_commits. I don't know yet what is the best name for this. Maybe overlap_commits?

i also don't understand what only_commits means, how can it prepare transaction "queue up" for group commit if only 1 is prepared in parallel ?

...

"none" means the parallel replication code is not used (same as --slave-parallel-threads=0, but now configurable per multimaster connection). (This corresponds to empty value in old --slave-parallel-mode).

--slave-parallel-domains=on|off (default on)

"This replaces the "domain" option of old --slave-parallel-mode. When enabled, parallel replication will apply in parallel transactions

whose

...

GTID has different domain ids (GTID mode only).

I don't understand what would be the meaning of combining this flag with --slave-parallel-mode. Does it mean that when this flag is on transactions from different domains are executed on "all_transactions" level of parallelism no matter what value --slave-parallel-mode has? What will happen if this flag off but --slave-parallel-mode=all_transactions?

I feel like you are up to something here, but implementing it using this flag is not quite right.

...

--slave-parallel-wait-if-conflict-on-master=on|off (default on)

When enabled, if a transaction had to do a row lock wait on the master, it will not be applied in parallel with any earlier transaction on the slave (idea is that such transaction is likely to get a conflict on the slave, causing a needless retry). (This was the "waiting" option to old --slave-parallel-mode).

Hm... The fact that a transaction did a lock wait on master doesn't mean that the conflicting transaction was committed on master, or that both of these transactions were committed close enough to even make it possible to be executed in parallel on slaves, right? Are you sure that this flag will be useful?

...

These options will also be usable per multi-source master connection, like --master1.slave-parallel-mode=all_transactions. The options will be possible to change dynamically also (with SET GLOBAL), though the associated slave threads must be stopped while changing.

Also, Monty suggested to rename @@replicate_allow_parallel to

@@SESSION.replicate_expect_conflicts=0|1 (default 0)

When this option is enabled on the master when a transaction is committed, that transaction will not be applied in parallel with earlier transactions (when --slave-parallel-mode=all_transactions). This can be used to reduce retries on the slave, if an application is about to do a transaction that is likely to cause a conflict and retry on a slave if applied in parallel with earlier transactions.

I think this variable will be completely useless and is not worth implementing. How user will understand that the transaction he is about to execute is likely to conflict with another transactions committed at about the same time? I think it will be completely impossible to do that judgement, at the same time it will give too much impact on the slave's behavior into users' hands. Am I missing something? What kind of scenario you are envisioning this variable to be used in?

...

Let me know if there are any comments to these or suggestions for changes. It is best to get these as right as possible before release (seems the intention is to include optimistic parallel replication in 10.1), since it is the user-visible part of the feature.

With these option names, the normal way to use optimistic parallel replication would be these two options in my.cnf:

slave_parallel_mode=all_transactions slave_parallel_threads=20 (or whatever)

This seems reasonably, I think. None of the other options would need be considered except in more special cases.

Hope that helps, Pavel

Pavel Ivanov writes:

Does this mean that group commit will be possible if slave is able to execute several transactions consecutively while previous transaction commits/fsyncs?

Yes. Because the following transaction is in all cases allowed to start as soon as the prior transaction reaches its COMMIT event, see previous mail for details.

I'd suggest to name this option differently because looking just at the list of available values it's not quite clear what could be the difference between follow_master_commits and only_commits. I don't know yet what is the best name for this. Maybe overlap_commits?

I think overlap_commits sounds good, thanks for the suggest.

...

--slave-parallel-domains=on|off (default on)

"This replaces the "domain" option of old --slave-parallel-mode. When enabled, parallel replication will apply in parallel transactions whose GTID has different domain ids (GTID mode only).

I don't understand what would be the meaning of combining this flag with --slave-parallel-mode. Does it mean that when this flag is on transactions from different domains are executed on "all_transactions" level of parallelism no matter what value --slave-parallel-mode has? What will happen if this flag off but --slave-parallel-mode=all_transactions?

These apply on two different levels. With --slave-parallel-domains=on, each replication domain is replicated as completely independent streams, similar to different multi-source replication slaves. The position in each stream is tracked with each one GTID in gtid_slave_pos, and one stream can be arbitrarily ahead of another. The --slave-parallel-mode applies within each stream. Within one stream, commits are strictly ordered, and --slave-parallel-mode specifies how much parallelism is attempted. The --slave-parallel-mode can be set to any value and the server is responsible to ensure that replication works correctly. In contrast, using --slave-parallel-domains, it is the users/DBAs responsibility to ensure that replication domains are set up correctly so that no conflict can occur between them.

I feel like you are up to something here, but implementing it using this flag is not quite right.

Can you elaborate? --slave-parallel-domains controls whether we have one stream or many. --slave-parallel-mode controls what happens inside each stream. Any suggestion how to clarify?

Hm... The fact that a transaction did a lock wait on master doesn't mean that the conflicting transaction was committed on master, or that both of these transactions were committed close enough to even make it possible to be executed in parallel on slaves, right? Are you sure that this flag will be useful?

Right, and no, I'm not sure. Testing will be needed to have a better idea. If two short transactions T1 and T2 conflict on a row, T2 is quite likely to commit just after T1, and thus likely to conflict on the slave. So there is some rationale behind this.

...

@@SESSION.replicate_expect_conflicts=0|1 (default 0)

I think this variable will be completely useless and is not worth implementing. How user will understand that the transaction he is about to execute is likely to conflict with another transactions committed at about the same time? I think it will be completely impossible to do that judgement, at the same time it will give too much impact on the slave's behavior into users' hands. Am I missing something? What kind of scenario you are envisioning this variable to be used in?

My main worry with optimistic parallel replication is if too many conflicts and retries on the slave will outweight the performance gained from parallelism. If this does not happen, I feel it will be awesome. So I was very focused on what to do if we _do_ get a lot of conflicts. So I wanted to give advanced users the possibility to work around hotspot rows basically, if necessary. Like single row that is updated very frequently. I did not think that this was allowing users much impact on the slave's behaviour. This option is only a heuristics, it controls how aggressive the slave will try to parallelise, but it cannot affect correctness. And the user alredy has a lot of ways to affect parallelism in optimistic parallel replication. For example, imagine lots of transactions like this executed serially on the master: UPDATE t1 SET a=a+1 WHERE id=0; UPDATE t1 SET a=a+1 WHERE id=0; UPDATE t1 SET a=a+1 WHERE id=0; ... All of these would conflict on a slave. It seems likely to cause O(N**2) transaction retries on the slave for --slave-parallel-mode=all_transactions --slave-parallel-threads=N. So the idea was that user can already cause trouble for parallelism on the slave; @@replicate_expect_conflicts is intended for the poweruser to be able to hint the slave at how to get less trouble. But I'm open to change it, if you think it's important. Your perspective is rather different from my usual point of view, which is useful input. Jonas Oreland writes:

I still prefer "auto" as default,

Right... I really want "normal" users to be able to just enable "auto" and have things work reasonably. And I really need fine-grained control to enable testing various combinations in real life, to better understand how to implement "auto". I need to find a good way to combine these...

if you want the fine grained control, I think an optimizer_switch approach is better than adding X new config options, i.e --parallel_mode=option1=true,option3=4 don't you think that there will be new options/variants ? i do don't you think you will want to remove options/variants ? i do

Good point. I think --slave-parallel-domains is reasonable. This is a question of semantics, the user must explicitly request it, as it can break replication if not used correctly by applications. But the other options are less clear. They all should behave equally correct. Maybe something like this: 1. --slave-parallel-domains is off by default, can be enabled if application is written to guarantee no conflicts. 2. --parallel-slave=on|off. This defines if parallelism will be done within each replication stream. 3. --slave-parallel-threads=N. This could default to 3*#CPUs or whatever if --parallel-slave is enabled on at least one multi-master slave. 4. --slave-parallel-mode=auto | <lots of fine-grained options>. Then "normal" user will only need to say --parallel-slave=on. --slave-parallel-mode defaults to "auto". I then need to consider how this affects backwards compatibility with 10.0...

I don't like having the word "transaction" in the names of some modes, ALL variants will maintain transaction semantics. Having "transaction" is name of only a few sort of implies that others are not transactional...

i think "optimistic" is better than "transactional" or "all_transactions". in the same spirit "conservative" could be "follow_master_commits"

Nice, I really like "conservative" and "optimistic". And maybe "only_commits" isn't really useful. There seems little reason not to at least use the "conservative" mode, it shouldn't cause any problems over "only_commits". So slave-parallel-mode= none | conservative | optimistic, perhaps...

i don't understand what "only_commits" is, how can it prepare transaction "queue up" for group commit if only 1 is prepared in parallel ?

The following transaction T2 is started as soon as T1 sees its COMMIT event. So if T2 can reach its own commit while T1 is still waiting for LOCK_log (or --binlog-commit-wait-usec), this is possible.

though, i think all suggestions might work if they have good defaults and a solid implementation

Right, I'd like to get the interface right from the start, but too much bikeshedding is also counterproductive. Thanks, Pavel and Jonas, for your comments! - Kristian.

Pavel Ivanov writes:

This is not entirely true, right? Let's say master binlog has transactions T1.1, T1.2, T1.3, T1.4, T2.1, T1.5, T2.2 (where T1.* have domain_id = 1 and T2.* have domain_id = 2) and slave has 3 parallel threads. Then as I understand threads will be assigned to execute T1.1, T1.2 and T1.3. T2.1 won't be scheduled to execute until these 3 transactions (or at least 2 of them T1.1 and T1.2) have been committed. So streams from different domains are not completely independent, right?

One can use --slave-domain-parallel-threads to limit the number of threads that one domain in one multi-source connection can reserve. By default, things work as in your example. With eg. --slave-parallel-threads=3 --slave-domain-parallel-threads=2, two threads will be assigned to run T1.1, T1.2, T1.3, and T1.4, and one free thread will remain to run T2.1 in parallel with them.

As I pointed above the streams from multiple domains are completely independent only when they are coming from multiple masters. When they come from a single master they are not completely independent and that creates a confusion (at least for me) of how these options work together in that case.

It's the same for multiple masters as for single masters. There is a global pool of --slave-parallel-threads=N threads. One domain in one master connection can allocate up to --slave-domain-parallel-threads=M of those to apply transactions in-order. If one master connection is using all available threads, other connections will stall :-/ So the thread management is pretty basic in the current version of parallel replication. And domain-based parallel replication is not as easy to use as I would like. Hopefully this can be improved in a later version.

I guess a big question I want to ask: why would someone want to use multiple domains together with slave-parallel-domains = off? If it's a kind of kill-switch to turn off multi-domain feature completely if it causes troubles for some reason, then I don't think it is baked deep enough to actually work like that. But I don't understand what else could it be used for.

The original motivation for replication domains is multi-source replication. Suppose we have M1->S1, M2->S1, S1->S2, S1->S3: M1 --\ /---S2 +-- S1 ---+ M2 --/ \---S3 Configuring different domains for M1 and M2 is necessary to be able to reconfigure the replication hierarchy, for example to M1->S2, M2->S2; or to S2->S3: M1 --\ /---S2 +--+ M2 --/ \---S1 ---S3 M1 --\ /---S2 ---S3 +-- S1 ---+ M2 --/ This requires a way to track the position in the binlog streams of M1 and M2 independently, hence the need for domain_id. The domains can also be used for parallel replication; this is needed to allow S2 and S3 to have the same parallelism as S1. However, this kind of parallel replication requires support from the application to avoid conflicts. Now concurrent changes on M1 and M2 have to be conflict-free not just on S1, but on _all_ slaves in the hierarchy. I think that such a feature, which can break replication unless the user carefully designs the application to avoid it, requires a switch to turn it on or off.

Both seem to be a very narrow use case to make it worth adding a flag that can significantly hurt the majority of other use cases. I think

I see your point. Another thing that makes the use case even narrower is that it will be kind of random if we actually get the lock wait in T2 on the master. So even if delaying T2 would significantly improve performance on the slave, it is not a reliable mechanism.

this feature will be useful only if master will somehow leave information about which transaction T2 was in conflict with, and then slave would make sure that T2 is not started until T1 has finished. Though this sounds over-complicated already.

Yeah, it does. What I really need is to get some results from testing optimistic parallel replication, to understand how many retries will be needed in various scenarios, and if those retries are a bottleneck for performance.

I understand everything that you say, but I think the difference between our views is that you consider DBAs and database users to be mostly the same people or two small groups sitting in the same room and easily communicating with each other. For me that's not true. For

(I meant getting your input has made me think more about use cases that are different from something like eg. Facebook, with a single carefully controlled application and a team of highly skilled database developers. Which is where I come from originally, though we were a _lot_ smaller than Facebook, obviously :-) If I understand correctly, your use case is one of a team of highly skilled DBAs (for lack of a better name) managing a database service used by a lot of users, each with their own applications, and each without high database skills. Another use case is users that run a single application on their own hosting of MariaDB, but use the database as a commodity without wanting to invest many resources in acquiring detailed MariaDB skills. These use cases are probably a lot more common than "facebook-like" applications.

So when you say "the user already has a lot of ways to affect parallel replication" it translates to me as "there are certain workloads when parallel replication will behave slower than sequential". Yes, I agree with that. If I meet such workload I will have to turn off the parallel replication, or I (with your help) will have to find some generic improvement to make parallel replication work better with such

Right, point taken.

So overall I don't think this variable will be useful for large installations.

I agree it will not be useful for you, nor for the majority of other users. The question is, if --slave-parallel-wait-if-conflict-on-master and @@replicate_expect_conflicts are sufficiently useful for a minority of users to be worth including in _some_ capacity? From the input I have gotten so far, I think I will remove them, unless someone chimes in with a different point of view (we can always add them back later in some form, if they actually turn out to be needed in real life testing). That would leave the following options (in one form or another): --master_connection.slave-parallel-mode = conservative | optimistic --slave-parallel-threads = N --master_connection.parallel-slave = on|off --master_connection.slave-parallel-domains = on|off I am not yet confident enough in the code to not provide a way to disable the new optimistic mode. And the fixed-size thread pool, while limited, is what we have for now. And being able to enable/disable in-order and out-of-order parallel replication independently on a per-master-connection basis seems useful. Or can we do better? Once again, thanks for taking the time to help me improve this important aspect of the parallel replication. Thanks, - Kristian.

Jonas Oreland writes:

i didn't read the fully reply, sorry for that, but still wanted to say that even if @@replicate_expect_conflicts might only be useful to a very small minority of users, i think it might be worth to implement...if it's not too hard/time-consuming...

It is trivial to implement (and I already did).

I though (of course !?!) suggest a different name: @@block-parallel-slave-execution=1, which i think should be advisory, even if name suggest otherwise.

A similar name but different wording is @@skip_parallel_replication. This is consistent with @@skip_replication, which is also advisory (a slave option controls whether it takes effect or not).

and that the implementation of this is the events is marked with such a bit...and that the slave applier will obey unless slave-parallel-mode=aggressive (and 'aggressive' could be yet another mode to slave-parallel-mode)

I like 'aggressive' to control this slave-side. Thanks, - Kristian.

Pavel Ivanov writes:

So the slave coordinator (or I don't remember how you call it) reads relay log ahead of the last executing transaction? I.e. it will read and assign to threads T1.1, T1.2, then it will read T1.3, detect that there are no threads available for execution, but according to what you said it will still put this in the queue for thread 1, right? How long this queuing can be? Does it keep all queued events in memory? Does it depend on the size of the transactions (i.e. how much memory can it consume by this queuing)?

Right. The queueing is limited by the configuration variable --slave-parallel-max-queued, which defaults to 128KB per worker thread. It does not depend on the size of the transactions (it is possible to replicate a large transaction without keeping all of it in memory at once). It does need to keep at least one event in-memory per worker thread, of course, even if an individual event exceeds --slave-parallel-max-queued. So memory consumption for queued events is generally limited by @@slave_parallel_threads * @@slave_parallel_max_queued.

...

M1 --\ /---S2 ---S3 +-- S1 ---+ M2 --/

...

I think that such a feature, which can break replication unless the user carefully designs the application to avoid it, requires a switch to turn it on or off.

Could there really be cases when multi-domain parallel application of transaction is safe on S1, but not safe on S2 or S3?

Such cases can definitely be constructed. For example, suppose S2 is stopped. User does a long ALTER TABLE t1 on M1, carefully waits for that ALTER to complete on M1 and on S1, then starts doing DML to table t1 on M2. Then, when S2 is restarted, it seems likely that it will start executing the DML in domain 2 before the ALTER in domain 2 has completed, which can break the replication. I do agree that in practice, something that breaks domain-based parallel replication on S2 and S3 is likely to also be able to cause problems on S1. On the other hand, there does not seem much harm to provide a switch to turn on or off domain-based parallel replication. (Such mechanism has to be implemented anyway, as it can only be used in GTID mode, in non-GTID mode domain-based parallel replication is always off).

...

What I really need is to get some results from testing optimistic parallel replication, to understand how many retries will be needed in various scenarios, and if those retries are a bottleneck for performance.

Then I'd suggest to not add any special processing of such use case, but add something that will allow to easily monitor what happens. E.g. some status variables which could be plotted over time and show (or at least hint on) whether this is significant bottleneck for performance or not. This could be something like total time (in both wall time and accumulated CPU time) spent executing transactions in parallel, time spent rolling back transactions due to this lock conflict, time spent rolling back transactions because of other reasons (e.g. due to STOP SLAVE or reconnect after master crash), maybe also time spent waiting in one parallel thread while transaction is executing in another thread, etc.

Yes, I agree, we need more of this. I think the monitoring part of the feature is currently rather weak, it probably suffers from it being now a long time since I was doing operations. Hopefully this can be significantly improved in the near future. I wonder if such accumulated-time measurements can be added liberally without significantly affecting performance? - Kristian.

Hi again Kristian, A slightly off topic question that struck me last night: won't all parallel transactions conflict when updating the slave_gtid_pos table ? or is there something i missed... /Jonas On Thu, Dec 4, 2014 at 2:49 PM, Kristian Nielsen wrote:

I discussed with Monty, and we came up with some more suggested changes for the options used to configure the optimistic parallel replication feature (MDEV-6676, https://mariadb.atlassian.net/browse/MDEV-6676).

The biggest change is to split up the --slave-parallel-mode into multiple options. I think that is reasonable, probably that option was doing too many things at once.

Instead, we could have the following three options:

--slave-parallel-mode=all_transactions | follow_master_commits | only_commits | none

"all_transactions" is what was called "transactional" before. The slave will try to apply all transactional DML in parallel; in case of conflicts it will roll back the later transaction and retry it.

"follow_master_commits" is the 10.0 functionality, apply in parallel transactions that group-committed together on the master (the default).

"only_commits" was suggested to me by a user testing parallel replication. It does not attempt to apply transactions in parallel, but still runs the commit steps in parallel, making slave group commit possible and thus saving on fsyncs if durability settings are on.

"none" means the parallel replication code is not used (same as --slave-parallel-threads=0, but now configurable per multimaster connection). (This corresponds to empty value in old --slave-parallel-mode).

--slave-parallel-domains=on|off (default on)

"This replaces the "domain" option of old --slave-parallel-mode. When enabled, parallel replication will apply in parallel transactions whose GTID has different domain ids (GTID mode only).

--slave-parallel-wait-if-conflict-on-master=on|off (default on)

When enabled, if a transaction had to do a row lock wait on the master, it will not be applied in parallel with any earlier transaction on the slave (idea is that such transaction is likely to get a conflict on the slave, causing a needless retry). (This was the "waiting" option to old --slave-parallel-mode).

These options will also be usable per multi-source master connection, like --master1.slave-parallel-mode=all_transactions. The options will be possible to change dynamically also (with SET GLOBAL), though the associated slave threads must be stopped while changing.

Also, Monty suggested to rename @@replicate_allow_parallel to

@@SESSION.replicate_expect_conflicts=0|1 (default 0)

When this option is enabled on the master when a transaction is committed, that transaction will not be applied in parallel with earlier transactions (when --slave-parallel-mode=all_transactions). This can be used to reduce retries on the slave, if an application is about to do a transaction that is likely to cause a conflict and retry on a slave if applied in parallel with earlier transactions.

Let me know if there are any comments to these or suggestions for changes. It is best to get these as right as possible before release (seems the intention is to include optimistic parallel replication in 10.1), since it is the user-visible part of the feature.

With these option names, the normal way to use optimistic parallel replication would be these two options in my.cnf:

slave_parallel_mode=all_transactions slave_parallel_threads=20 (or whatever)

This seems reasonably, I think. None of the other options would need be considered except in more special cases.

- Kristian.