August 2016 - developers - lists.mariadb.org

Re: [Maria-developers] Patch for parallel replication DBUG_ASSERT check on DROP TABLE et al
by Kristian Nielsen 20 Aug '16

20 Aug '16

Monty writes: > Can you please check if this looks correct? > diff --git a/sql/mdl.cc b/sql/mdl.cc > index 28d2006..3b1f9f2 100644 > --- a/sql/mdl.cc > +++ b/sql/mdl.cc > @@ -17,6 +17,7 @@ > #include "sql_class.h" > #include "debug_sync.h" > #include "sql_array.h" > +#include "rpl_rli.h" > #include <hash.h> > #include <mysqld_error.h> > #include <mysql/plugin.h> > @@ -442,6 +443,7 @@ class MDL_lock > virtual void notify_conflicting_locks(MDL_context *ctx) = 0; > > virtual bitmap_t hog_lock_types_bitmap() const = 0; > + bool check_if_conflicting_replication_locks(MDL_context *ctx); This should probably be #ifdef NDEBUG, since you're only using it in debug build? > > /** List of granted tickets for this lock. */ > Ticket_list m_granted; > @@ -2290,6 +2292,44 @@ void MDL_scoped_lock::notify_conflicting_locks(MDL_context *ctx) > } > } > > +/** > + Check if there is any conflicting lock that could cause this thread > + to wait for another thread which is not ready to commit. > + This is always an error, as the upper level of parallel replication > + should not allow a scheduling of a conflicting DDL until all earlier > + transactions has commited. > + > + This function is only called for a slave using parallel replication > + and trying to get an exclusive lock for the table. > +*/ > + > +bool MDL_lock::check_if_conflicting_replication_locks(MDL_context *ctx) > +{ > + Ticket_iterator it(m_granted); > + MDL_ticket *conflicting_ticket; > + > + while ((conflicting_ticket= it++)) > + { > + if (conflicting_ticket->get_ctx() != ctx) > + { > + MDL_context *conflicting_ctx= conflicting_ticket->get_ctx(); > + > + /* > + If the conflicting thread is another parallel replication > + thread for the same master and it's not in commit stage, then > + the current transaction has started too early and something is > + seriously wrong. > + */ > + if (conflicting_ctx->get_thd()->rgi_slave && > + conflicting_ctx->get_thd()->rgi_slave->rli == > + ctx->get_thd()->rgi_slave->rli && > + !conflicting_ctx->get_thd()->rgi_slave->did_mark_start_commit) > + return 1; // Fatal error No, this is not correct. For example, the threads might be in different domains, or using different multi-source connections. You need a check like in thd_report_wait_for(). _If_ T1 and T2 are in the same parallel replication domain, _and_ T1 goes before T2, _and_ there is a DDL lock conflict but T1 has not started to commit, then it indicates a potential problem. Also, the comments are too vague, "Something is seriously wrong" is not a good description. What is wrong? Please describe the issue being checked more precisely. What is the role of each of the two threads involved? What is the situation being checked? Why is it wrong? Also, use some temporaries for conflicting_ctx->get_thd()->rgi_slave etc, so the condition becomes more readable. > + } > + } > + return 0; > +} > + > > /** > Acquire one lock with waiting for conflicting locks to go away if needed. > @@ -2355,6 +2395,11 @@ MDL_context::acquire_lock(MDL_request *mdl_request, ulong lock_wait_timeout) > if (lock->needs_notification(ticket) && lock_wait_timeout) > lock->notify_conflicting_locks(this); > > + DBUG_ASSERT(mdl_request->type != MDL_INTENTION_EXCLUSIVE || > + !(get_thd()->rgi_slave && > + get_thd()->rgi_slave->is_parallel_exec && > + lock->check_if_conflicting_replication_locks(this))); > + > mysql_prlock_unlock(&lock->m_rwlock); > > will_wait_for(ticket); > diff --git a/sql/mdl.h b/sql/mdl.h > index c4d792a..13de602 100644 > --- a/sql/mdl.h > +++ b/sql/mdl.h > @@ -910,7 +910,6 @@ class MDL_context > */ > MDL_wait_for_subgraph *m_waiting_for; > private: > - THD *get_thd() const { return m_owner->get_thd(); } > MDL_ticket *find_ticket(MDL_request *mdl_req, > enum_mdl_duration *duration); > void release_locks_stored_before(enum_mdl_duration duration, MDL_ticket *sentinel); > @@ -919,6 +918,7 @@ class MDL_context > MDL_ticket **out_ticket); > > public: > + THD *get_thd() const { return m_owner->get_thd(); } > void find_deadlock(); > > ulong get_thread_id() const { return thd_get_thread_id(get_thd()); } - Kristian.

2 4

Re: [Maria-developers] [Maria-discuss] Known limitation with TokuDB in Read Free Replication & parallel replication ?
by Rich Prohaska 18 Aug '16

18 Aug '16

Hello Kristian, I am working on a second variant of the kill query design that will only kill the pending lock request if any for the thd being killed. The previous design had a problem when killing the query that triggered the wait for call On Aug 15, 2016 7:26 PM, "Rich Prohaska" <prohaska7(a)gmail.com> wrote: Hello Kristian, I have a prototype of the TokuFT code that will cause ALL lock waiters to call their killed callback here: https://github.com/ prohaska7/tokuft/tree/kill_lockers On Mon, Aug 15, 2016 at 11:51 AM, Rich Prohaska <prohaska7(a)gmail.com> wrote: > Hello Kristian, > See attached snapshot of slave threads and tokudb locks. Thread 16 is > waiting for a tokudb lock held by thread 16, which is waiting for a tokudb > lock held by thread 14. Thread 14 is waiting for a prior transaction to > complete, presumably either thread 15 or 16. So, we have a deadlock that > tokudb can not detect because the ordering constraint is not available to > tokudb. I assume that the optimistic scheduler killed thread 16, but since > tokudb does not implement the kill_query function, the deadlock is only > resolved when the tokudb lock timer pops. > > On Mon, Aug 15, 2016 at 8:16 AM, Rich Prohaska <prohaska7(a)gmail.com> > wrote: > >> Hello Kristian, >> The simplest kill_query implementation for tokudb would just signal all >> of the pending lock request's condition variables. This would cause the >> killed callback to be called. A performance refinement, if necessary, >> would allow thread A (executing the kill_query function) to identify and >> signal a condition variable for a blocked thread B. >> >> On Mon, Aug 15, 2016 at 5:42 AM, Kristian Nielsen < >> knielsen(a)knielsen-hq.org> wrote: >> >>> Rich Prohaska <prohaska7(a)gmail.com> writes: >>> >>> > tokudb lock timeouts are resolving the replication stall. >>> unfortunately, >>> > the tokudb lock timeout is 4 seconds, so the throughput is almost zero. >>> >>> Yes. Sorry for not making it clear that my proof-of-concept patch was >>> incomplete... >>> >>> >> > I suspect that the poor slave replication performance for optimistic >>> >> > replication occurs because TokuDB does not implement the kill_query >>> >> > handlerton function. kill_handlerton gets called to resolve lock >>> wait >>> >>> >> Possibly, but I'm not sure it's that important. The kill will be >>> effective >>> >> as soon as the wait is over. >>> >>> No, you're absolutely right, after testing (and thinking) some more, I >>> realise that indeed the kill_query functionality is important. >>> >>> A possible scenario is, given transactions T1, T2, and T3 in that order: >>> >>> T3 acquires a lock on row R3, T2 similarly acquires R2. >>> Now T3 tries to acquire R2, but has to wait for T2 to release it. >>> Later T1 tries to acquire R3, also has to wait. >>> >>> At this point, we kill T3, since it is holding a lock (R3) needed by an >>> earlier transaction T1. However, T3 will not notice the kill until its >>> own >>> wait (on R2 held by T2) times out. T2 cannot release the lock because it >>> is >>> waiting for T1 to commit first. So we have a deadlock :-/ >>> >>> With InnoDB, the kill causes T3 to wake up immediately and roll back, so >>> that T1 can proceed without much delay. >>> >>> Ok, so something more is needed here. I see there is a killed_callback() >>> which seems to check for the kill, so I'm hoping that can be used with a >>> suitable wakeup of the offending lock_request (or all requests, >>> perhaps). But as I'm completely new to TokuDB, I still need some more >>> time >>> to read the code and try to understand how everything fits together... >>> >>> > TokuFT implements pessimistic locking and 2 phase locking algorithms. >>> This >>> > wiki describes locking and concurrency in a little more detail: >>> > https://github.com/percona/tokudb-engine/wiki/Transactions-a >>> nd-Concurrency. >>> >>> Thanks, this was quite helpful. >>> >>> > Yes, I think they are false positives since the thd_report_wait_for >>> API is >>> > called but it does NOT call the THD::awake function. >>> >>> Ah. Then it's probably normal, caused by the group-commit optimisation. >>> In >>> conservative mode, if two transactions T1 and T2 did not group commit on >>> the >>> master, then cannot be started in parallel on the slave. But T2 can >>> start as >>> soon as T1 has reached COMMIT. Thus, if T2 happens to conflict with T1, >>> there is a small window where T2 can need to wait on T1 until T1 has >>> completed its commit. >>> >>> Thanks, >>> >>> - Kristian. >>> >> >> >

2 2

Re: [Maria-developers] Freeing options & wrappers.
by Kristian Nielsen 16 Aug '16

16 Aug '16

Oleksandr Byelkin <sanja(a)montyprogram.com> writes: >>> Making something more I think do not worth it because: >>> 1) in 10.2 C/C will replace the library and it has no the problem >> But Connector-C should have exactly the same problem. If it frees the >> options on failed connect, it will break async API. If it does not, code >> that skips mysql_close() will leak. How does C/C solve this problem? >> If we are making any changes in 10.0, we should preferably make things work >> the same as they will work in 10.2. > Don't it has rewritten async operation? (at least I got such impression). My understanding is that Georg implemented all the async API also in connector-c. It should be using much the same code (mysql_async.c and such). But this is not about the implementation. This is about the semantics of how libmysqlclient works. My understanding is that connector-c is supposed to be binary compatible with libmysqlclient.so? So either connector-c removes any mysql_options() set when mysql_real_connect() fails without CLIENT_REMEMBER_OPTIONS. Or it does not. In the former case, there is the problem with MYSQL_OPT_NONBLOCK. In the latter case, the problem with leak when mysql_close() is not called. I really think you need to understand the problem fully, including how this is going to work in 10.2, before you make any changes to how the client library works... - Kristian.

2 3

Re: [Maria-developers] c56c966: MDEV-10563 Crash during shutdown in Master_info_index::any_slave_sql_running
by Kristian Nielsen 16 Aug '16

16 Aug '16

Vicențiu Ciorbaru <vicentiu(a)mariadb.org> writes: > This is undefined behavior: > // In GCC v6 this function _always_ returns 1. > ClassA::member_function() { > if (!this) > return 0; > return 1; > } Yeah, we shouldn't do this, agree. > make use of this construct. I have not changed the logic of the code, > however I am not sure whether "give_error_if_slave_running()" is supposed > to return TRUE if master_info_index is deleted (aka, we are performing a > shutdown). Probably there is no good reason to return TRUE. Unfortunately, there is a lot of code like this in replication, where it is not clear what the purpose is, but also very hard to be sure it's not needed. My guess would be - without knowing for sure - that the return value doesn't matter. This should only happen during shutdown anyway, and it seems to be used only for changing the values of some system-variables. If server is shutting down, then values of replication-related system-variables do not really matter. As to the patch, you're basically moving the check for NULL into all the callers. I'm not sure if the check is even needed for all callers, but then, I also do not think it's worth it to try and determine where it is needed and where not. But wouldn't it be nicer to instead change the give_error_if_slave_running() to be a normal function that takes the Master_info_index pointer as an argument (or a static method, if you want)? This way, you can keep the null-pointer check in the function. Also helps avoid a new bug if another caller is added. - Kristian.

2 1

[Maria-developers] Freeing options & wrappers.
by Oleksandr Byelkin 16 Aug '16

16 Aug '16

Hi! (sorry I forgot mailing list so resending the e-mail) So it is about https://jira.mariadb.org/browse/MDEV-10559 Original problem was that database drivers in Python & Perl (probably other) do not call mysql_close for MYSQL object if there was no connection happing (memory leak). MySQL has solution for this - freeing option on failed connection if user did not asked something else (we even has the same flag, but it did not work). So I took MySQL solution. But it appeared not so compatible with our async wrappers. Frankly speaking the code is difficult to read and debug because everything made over macros nor I have found decent description/overview of how it is done (thank you if somebody point to it) so I can not say that understand what it is doing, and all my suggestion based on how I understood it. So what we have: we allocate memory for some options set between mysql_init and mysql_real_connect, instead of "reall connect" called some wrapper which store some pointers to the options then call mysql_real_connect and if "real_connect" free them it fail on exit. I think that best solution is that wrapper 1) store current state of CLIENT_REMEMBER_OPTIONS 2) set the flag 3) call mysql_real_connect 4) if connect fail and there was not "remember options" then wrapper free options by itself Again, I am not 100% sure that it is possible nor really know the place were this should be added (if idea above ok, I probably find ho to make it).

2 4

Re: [Maria-developers] [Maria-discuss] Known limitation with TokuDB in Read Free Replication & parallel replication ?
by Kristian Nielsen 16 Aug '16

16 Aug '16

[Moving the discussion to maria-developers@, hope that is ok/makes sense...] Ok, so here is a proof-of-concept patch for this, which seems to make TokuDB work with optimistic parallel replication. The core of the patch is this line in lock_request.cc lock_wait_callback(callback_data, m_txnid, conflicts.get(i)); which ends up doing this: thd_report_wait_for (requesting_thd, blocking_thd); All the rest of the patch is just getting the right information around between the different parts of the code. I put this on top of Jocelyn Fournier's tokudb_rpl.rpl_parallel_optimistic patches, and pushed it on my github: https://github.com/knielsen/server/tree/toku_opr2 With this patch, the test case passes! So that's promising. Some things still left to do for this to be a good patch: - I think the callback needs to trigger also for an already waiting transaction, in case another transaction arrives later to contend for the same lock, but happens to get the lock earlier. I can look into this. - This patch needs linear time (in number of active transactions) per callback to find the THD from the TXNID, maybe that could be optimised. - Probably the new callback etc. needs some cleanup to better match TokuDB code organisation and style. - And testing, of course. I'll definitely need some help there, as I'm not familiar with how to run TokuDB efficiently. Any thoughts or comments? - Kristian.

2 8

Re: [Maria-developers] Status of InnoDB 5.7 merge to 10.2
by Kristian Nielsen 13 Aug '16

13 Aug '16

Jan Lindström <jan.lindstrom(a)mariadb.com> writes: > I have reached to situation where I need help fixing remaining test > failures and mentoring is current status good enough for 10.2 Beta. I need > help on test failures on parts where I do not follow why test is failing or > server crashes I do not know how to fix. > > First of all current merge result is on bb-10.2-jan with > commit ca4f194460da469e29624d930e6d0027349d047c > > Higher level of status is on https://jira.mariadb.org/browse/MDEV-10024 > > Note that mtr thinks currently that innodb_plugin compiled using static > linking is xtradb (but not have_xtradb as xtradb is not compiled). > Furthermore galera is not compiled (waiting for codership). > > Here are current issues where I need help and possible persons who could > help: > > (1) Parallel replication lock wait/deadlock handling on InnoDB 5.7 with > refactored lock0lock.cc does not work > https://jira.mariadb.org/browse/MDEV-10550 (Monty, Kristian) >From a quick look at the patch, this seems to be because of code removed from innobase_kill_query(). Specifically: > @@ -4799,65 +5511,11 @@ innobase_kill_query( > DBUG_ENTER("innobase_kill_query"); > DBUG_ASSERT(hton == innodb_hton_ptr); > - if (!wsrep_thd_is_BF(trx->mysql_thd, FALSE) && > - trx->abort_type == TRX_SERVER_ABORT) { > - ut_ad(!lock_mutex_own()); > - lock_mutex_enter(); When parallel replication discovers a case where a transaction T1 goes to wait on a row lock held by a later (in the replication stream) transaction T2, T2 will be killed to avoid a deadlock. In this situation, we are already holding lock mutex. So this code is there in 10.1 to prevent re-taking the mutex, code that is removed here in your patch. > + if (trx != NULL) { > + /* Cancel a pending lock request if there are any */ > + lock_trx_handle_wait(trx); > +lock_trx_handle_wait( > +/*=================*/ > + trx_t* trx) /*!< in/out: trx lock state */ > { > dberr_t err; > - ulint heap_no; > > - ut_ad(!dict_index_is_clust(index)); > - ut_ad(!dict_index_is_online_ddl(index)); > - ut_ad(block->frame == page_align(rec)); > - ut_ad(page_rec_is_user_rec(rec) || page_rec_is_supremum(rec)); > - ut_ad(rec_offs_validate(rec, index, offsets)); > - ut_ad(mode == LOCK_X || mode == LOCK_S); > + lock_mutex_enter(); But in the new code here, the protection on lock_mutex_enter is no longer there, lock_mutex_enter() is done unconditionally. The code which sets the flag TRX_SERVER_ABORT seems to be still there, though: > +void > +lock_report_waiters_to_mysql( > +/*=======================*/ > + struct thd_wait_reports* waitee_buf_ptr, /*!< in: set of trxs */ > + THD* mysql_thd, /*!< in: THD */ > + trx_id_t victim_trx_id) /*!< in: Trx selected > + as deadlock victim, if > + any */ > +{ > + struct thd_wait_reports* p; > + struct thd_wait_reports* q; > + ulint i; > + > + p = waitee_buf_ptr; > + while (p) { > + i = 0; > + while (i < p->used) { > + trx_t *w_trx = p->waitees[i]; > + /* There is no need to report waits to a trx already > + selected as a victim. */ > + if (w_trx->id != victim_trx_id) { > + /* If thd_report_wait_for() decides to kill the > + transaction, then we will get a call back into > + innobase_kill_query. We mark this by setting > + current_lock_mutex_owner, so we can avoid trying > + to recursively take lock_sys->mutex. */ > + w_trx->abort_type = TRX_REPLICATION_ABORT; > + thd_report_wait_for(mysql_thd, w_trx->mysql_thd); > + w_trx->abort_type = TRX_SERVER_ABORT; abort_type is set to TRX_REPLICATION_ABORT around thd_report_wait_for(), during which innobase_kill_query() may be invoked with lock mutex already held. So it would appear that a first attempt should be to restore the protection around lock_mutex_enter()... Note that, if w_trx->abort_type is not functional in your current patch, the parallel replication code was originally working without this, you can see in the snippet above the comment about current_lock_mutex_owner, which was (mistakenly) not removed when this code was changed to use Galera stuff. So you can check git history and go back to the original code (using current_lock_mutex_owner) if necessary, perhaps. Or maybe w_trx->abort_type is fine, I'm not familiar with that change. > (2) mysqld: /home/jan/mysql/10.2/sql/handler.cc:2692: int > handler::ha_index_first(uchar*): Assertion `table_share->tmp_table != > NO_TMP_TABLE || m_lock_type != 2' failed. > https://jira.mariadb.org/browse/MDEV-10549 (Serg) > > (3) Some of the debug sync waits do not work InnoDB 5.7 > https://jira.mariadb.org/browse/MDEV-10548 (Monty, Serg) > > (4) Test multi_update_innodb fails with InnoDB 5.7 > https://jira.mariadb.org/browse/MDEV-10550 (Petrunia) > > (5) Test innodb.defrag_mdl-9155 hangs on InnoDB 5.7 > https://jira.mariadb.org/browse/MDEV-10551 (Jan, Serg) > > (6) IS tables are not found on 10.2 InnoDB 5.7 > https://jira.mariadb.org/browse/MDEV-10200 (Jan) Hope this helps, - Kristian.

2 1

[Maria-developers] MariaDB Developers Meetup
by Ian Gilfillan 10 Aug '16

10 Aug '16

The MariaDB Developers Meetup will be taking place from the 6th to the 8th of October in Amsterdam. https://mariadb.org/2016-mariadb-developers-meetup/ Details of where to sign up, and where to suggest topics for the meetup are on that page. Please feel free to suggest topic you're interested in presenting, as well as topics you'd like to see covered. The meetup is scheduled to be just after Percona Live, which also takes place in Amsterdam from the 3rd to the 5th. The Foundation has arranged a discount so that anyone signing up for the MariaDB Developers Meetup and that also wants to attend Percona Live will receive a discount for that event (the Developers Meetup has no charge). Percona Live are also looking for talks or sessions, so if you have a topic that would be of interest to the broader community, feel free to submit it at https://www.percona.com/live/plam16/program

1 1

[Maria-developers] Please review MDEV-10468 Assertion `nr >= 0.0' failed in Item_sum_std::val_real()
by Alexander Barkov 08 Aug '16

08 Aug '16

Hello Alexey, Can you please review a patch for MDEV-10468? Thanks!

2 3

[Maria-developers] Please review MDEV-10500 CASE/IF Statement returns multiple values and shifts further result values to the next column
by Alexander Barkov 08 Aug '16

08 Aug '16

Hi Serg, Please review a patch for MDEV-10500. Thanks.

2 2