- developers - lists.mariadb.org

Re: [Maria-developers] Status of InnoDB 5.7 merge to 10.2
by Kristian Nielsen 13 Aug '16

13 Aug '16

Jan Lindström <jan.lindstrom(a)mariadb.com> writes: > I have reached to situation where I need help fixing remaining test > failures and mentoring is current status good enough for 10.2 Beta. I need > help on test failures on parts where I do not follow why test is failing or > server crashes I do not know how to fix. > > First of all current merge result is on bb-10.2-jan with > commit ca4f194460da469e29624d930e6d0027349d047c > > Higher level of status is on https://jira.mariadb.org/browse/MDEV-10024 > > Note that mtr thinks currently that innodb_plugin compiled using static > linking is xtradb (but not have_xtradb as xtradb is not compiled). > Furthermore galera is not compiled (waiting for codership). > > Here are current issues where I need help and possible persons who could > help: > > (1) Parallel replication lock wait/deadlock handling on InnoDB 5.7 with > refactored lock0lock.cc does not work > https://jira.mariadb.org/browse/MDEV-10550 (Monty, Kristian) >From a quick look at the patch, this seems to be because of code removed from innobase_kill_query(). Specifically: > @@ -4799,65 +5511,11 @@ innobase_kill_query( > DBUG_ENTER("innobase_kill_query"); > DBUG_ASSERT(hton == innodb_hton_ptr); > - if (!wsrep_thd_is_BF(trx->mysql_thd, FALSE) && > - trx->abort_type == TRX_SERVER_ABORT) { > - ut_ad(!lock_mutex_own()); > - lock_mutex_enter(); When parallel replication discovers a case where a transaction T1 goes to wait on a row lock held by a later (in the replication stream) transaction T2, T2 will be killed to avoid a deadlock. In this situation, we are already holding lock mutex. So this code is there in 10.1 to prevent re-taking the mutex, code that is removed here in your patch. > + if (trx != NULL) { > + /* Cancel a pending lock request if there are any */ > + lock_trx_handle_wait(trx); > +lock_trx_handle_wait( > +/*=================*/ > + trx_t* trx) /*!< in/out: trx lock state */ > { > dberr_t err; > - ulint heap_no; > > - ut_ad(!dict_index_is_clust(index)); > - ut_ad(!dict_index_is_online_ddl(index)); > - ut_ad(block->frame == page_align(rec)); > - ut_ad(page_rec_is_user_rec(rec) || page_rec_is_supremum(rec)); > - ut_ad(rec_offs_validate(rec, index, offsets)); > - ut_ad(mode == LOCK_X || mode == LOCK_S); > + lock_mutex_enter(); But in the new code here, the protection on lock_mutex_enter is no longer there, lock_mutex_enter() is done unconditionally. The code which sets the flag TRX_SERVER_ABORT seems to be still there, though: > +void > +lock_report_waiters_to_mysql( > +/*=======================*/ > + struct thd_wait_reports* waitee_buf_ptr, /*!< in: set of trxs */ > + THD* mysql_thd, /*!< in: THD */ > + trx_id_t victim_trx_id) /*!< in: Trx selected > + as deadlock victim, if > + any */ > +{ > + struct thd_wait_reports* p; > + struct thd_wait_reports* q; > + ulint i; > + > + p = waitee_buf_ptr; > + while (p) { > + i = 0; > + while (i < p->used) { > + trx_t *w_trx = p->waitees[i]; > + /* There is no need to report waits to a trx already > + selected as a victim. */ > + if (w_trx->id != victim_trx_id) { > + /* If thd_report_wait_for() decides to kill the > + transaction, then we will get a call back into > + innobase_kill_query. We mark this by setting > + current_lock_mutex_owner, so we can avoid trying > + to recursively take lock_sys->mutex. */ > + w_trx->abort_type = TRX_REPLICATION_ABORT; > + thd_report_wait_for(mysql_thd, w_trx->mysql_thd); > + w_trx->abort_type = TRX_SERVER_ABORT; abort_type is set to TRX_REPLICATION_ABORT around thd_report_wait_for(), during which innobase_kill_query() may be invoked with lock mutex already held. So it would appear that a first attempt should be to restore the protection around lock_mutex_enter()... Note that, if w_trx->abort_type is not functional in your current patch, the parallel replication code was originally working without this, you can see in the snippet above the comment about current_lock_mutex_owner, which was (mistakenly) not removed when this code was changed to use Galera stuff. So you can check git history and go back to the original code (using current_lock_mutex_owner) if necessary, perhaps. Or maybe w_trx->abort_type is fine, I'm not familiar with that change. > (2) mysqld: /home/jan/mysql/10.2/sql/handler.cc:2692: int > handler::ha_index_first(uchar*): Assertion `table_share->tmp_table != > NO_TMP_TABLE || m_lock_type != 2' failed. > https://jira.mariadb.org/browse/MDEV-10549 (Serg) > > (3) Some of the debug sync waits do not work InnoDB 5.7 > https://jira.mariadb.org/browse/MDEV-10548 (Monty, Serg) > > (4) Test multi_update_innodb fails with InnoDB 5.7 > https://jira.mariadb.org/browse/MDEV-10550 (Petrunia) > > (5) Test innodb.defrag_mdl-9155 hangs on InnoDB 5.7 > https://jira.mariadb.org/browse/MDEV-10551 (Jan, Serg) > > (6) IS tables are not found on 10.2 InnoDB 5.7 > https://jira.mariadb.org/browse/MDEV-10200 (Jan) Hope this helps, - Kristian.

2 1

[Maria-developers] MariaDB Developers Meetup
by Ian Gilfillan 10 Aug '16

10 Aug '16

The MariaDB Developers Meetup will be taking place from the 6th to the 8th of October in Amsterdam. https://mariadb.org/2016-mariadb-developers-meetup/ Details of where to sign up, and where to suggest topics for the meetup are on that page. Please feel free to suggest topic you're interested in presenting, as well as topics you'd like to see covered. The meetup is scheduled to be just after Percona Live, which also takes place in Amsterdam from the 3rd to the 5th. The Foundation has arranged a discount so that anyone signing up for the MariaDB Developers Meetup and that also wants to attend Percona Live will receive a discount for that event (the Developers Meetup has no charge). Percona Live are also looking for talks or sessions, so if you have a topic that would be of interest to the broader community, feel free to submit it at https://www.percona.com/live/plam16/program

1 1

[Maria-developers] Please review MDEV-10468 Assertion `nr >= 0.0' failed in Item_sum_std::val_real()
by Alexander Barkov 08 Aug '16

08 Aug '16

Hello Alexey, Can you please review a patch for MDEV-10468? Thanks!

2 3

[Maria-developers] Please review MDEV-10500 CASE/IF Statement returns multiple values and shifts further result values to the next column
by Alexander Barkov 08 Aug '16

08 Aug '16

Hi Serg, Please review a patch for MDEV-10500. Thanks.

2 2

Re: [Maria-developers] 634e130: MDEV-10424 - Assertion `ticket == __null' failed in MDL_request::set_type
by Sergei Golubchik 08 Aug '16

08 Aug '16

Hi, Sergey! On Aug 04, Sergey Vojtovich wrote: > revision-id: 634e130b00bc900d1b7b8a8df6c13b1701f84b30 (mariadb-5.5.50-24-g634e130) > parent(s): 2f55f0b010619a09870ba704b55dd1426a158f1d > committer: Sergey Vojtovich > timestamp: 2016-08-04 15:51:17 +0400 > message: > > MDEV-10424 - Assertion `ticket == __null' failed in MDL_request::set_type > > Reexecution of prepared "ANALYZE TABLE merge_table, table" may miss to > reinitialize "table" for subsequent execution and trigger assertion failure. > > This happens because MERGE engine may adjust table->next_global chain, which > gets cleared by close_thread_tables()/ha_myisammrg::detach_children() later. > Since reinitilization iterates next_global chain, it won't see tables following > merge table. > > Fixed by appending saved next_global chain after merge children. looks ok to me, but let elena do some extra testing of this, please Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] GSoC weekly reports (Unique indexes for blobs)
by Shubham Barai 04 Aug '16

04 Aug '16

GSoC (week 5) Hello everyone, Here is the list of things I have done in the 5th week of GSoC. 1.implemented unique key violation with a hash collision. (actual row comparison is remaining ). 2.modified hash function for data types like varchar and binary data types. 3.fixed a bug which was causing a server to crash for complex unique keys. 4.added support to allow any number of nulls which will not cause any unique key violation. 5.added test cases for above features. On 22 June 2016 at 16:36, Shubham Barai <shubhambaraiss(a)gmail.com> wrote: > Hi, > > can we discuss on IRC first? > > > Regards, > Shubham > > On 22 June 2016 at 13:21, Jan Lindström <jan.lindstrom(a)mariadb.com> wrote: > >> Hi, >> >> Please commit and push these changes to your git branch I have not yet >> seen them, in my opinion as this is only a working branch you can push >> often. I still fail to see any test cases on InnoDB branch, do you have >> more than one branch and if you have why ? Depending on extent of these >> changes my estimate is that you are behind schedule to complete project in >> time. Based on your progress report you are still missing update and delete >> and redo-logging. For alter table you should start from forcing copy-method >> and then if time permits develop on-line method. This naturally only after >> everything else >> has been completed and tested. >> >> R: Jan >> >> On Tue, Jun 21, 2016 at 11:06 PM, Shubham Barai <shubhambaraiss(a)gmail.com >> > wrote: >> >>> GSoC (week 4) >>> >>> Hello everyone, >>> >>> After working on create table operation,next thing I had to work on was >>> insert operations.So I explored some of the functions like >>> row_ins_scan_index_for_duplicates, btr_pcur_get_rec to get clear >>> understanding about how to implement duplicate search on hash index. >>> There was a problem in hash function that I wrote .It would calculate >>> same hash value for two different keys if the prefix length of blob key >>> part was zero. Now it seems to be working after I checked it in debugger.I >>> still have to modify it for data types like varchar etc. >>> I have added test cases for insert operations in myisam. >>> In MyIsam, I found one problem in update operation. When updating a >>> row,if the key is conflicting then server crashes because some pointer goes >>> invalid in compare_record. I haven't fixed this issue yet. >>> >>> I also modified some functions in dict0load.cc to adjust some members >>> of dict_index_t for a new index type.The main problem is that index entry >>> for hash based index cointains only two fields(hash value and row id) while >>> dict_index_t contains hash field and other user defined fields which are >>> used to calculate hash value.Some of the operations like alter table( e.g. >>> rename column) needs to get access to all fields while other functions like >>> rec_get_offsets and row_build_index_entry_low needs to get access to only >>> hash field and row id. I am still working on this to find efficient >>> solution to this problem. >>> >>> On 16 June 2016 at 23:29, Sergei Golubchik <vuvova(a)gmail.com> wrote: >>> >>>> Hi, Shubham! >>>> >>>> What I wanted to say on IRC was: >>>> >>>> here's what the comment of cmp_dtuple_rec_with_match_low() says: >>>> >>>> ............... If rec has an externally stored field we do not >>>> compare it but return with value 0 if such a comparison should be >>>> made. >>>> >>>> Note that blobs are externally stored fields in InnoDB, so, I think, >>>> this means that you cannot use cmp_dtuple_rec() to compare blobs. >>>> >>>> Regards, >>>> Sergei >>>> Chief Architect MariaDB >>>> and security(a)mariadb.org >>>> >>> >>> >>> _______________________________________________ >>> Mailing list: https://launchpad.net/~maria-developers >>> Post to : maria-developers(a)lists.launchpad.net >>> Unsubscribe : https://launchpad.net/~maria-developers >>> More help : https://help.launchpad.net/ListHelp >>> >>> >> >

1 5

Re: [Maria-developers] bdef42c: MDEV-10365 - Race condition in error handling of INSERT DELAYED
by Sergei Golubchik 03 Aug '16

03 Aug '16

Hi, Sergey! On Aug 03, Sergey Vojtovich wrote: > revision-id: bdef42c54f8938d28d9a421271b11e09e88735a4 (mariadb-5.5.50-19-gbdef42c) > parent(s): 45ec5d673a1f5c736baec8ea9e2065bca4350324 > committer: Sergey Vojtovich > timestamp: 2016-08-03 11:49:35 +0400 > message: > > MDEV-10365 - Race condition in error handling of INSERT DELAYED > > Shared variables of Delayed_insert may be updated without mutex protection > when delayed insert thread gets an error. > > Re-acquire mutex earlier, so that shared variables are protected. ok to push Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] fc42991: MDEV-7901: re-implement analyze table for low impact
by Sergei Golubchik 03 Aug '16

03 Aug '16

Hi, Oleksandr! Looks mostly ok, see few comments below On Aug 01, Oleksandr Byelkin wrote: > revision-id: fc42991720838e165ad448b6707602cded92faa4 (mariadb-10.2.1-9-gfc42991) > parent(s): 08683a726773f8cdf16a4a3dfb3920e5f7842481 > committer: Oleksandr Byelkin > timestamp: 2016-08-01 19:25:45 +0200 > message: > > MDEV-7901: re-implement analyze table for low impact > > Table before collecting engine independent statistics now is reopened in read mode, > InnoDB allow write operations in this case. > > --- > mysql-test/r/stat_tables_innodb_debug.result | 27 +++ > mysql-test/t/stat_tables_innodb_debug.test | 37 ++++ > sql/sql_admin.cc | 295 +++++++++++++++------------ > storage/innobase/handler/ha_innodb.cc | 1 + > storage/xtradb/handler/ha_innodb.cc | 2 +- > 5 files changed, 227 insertions(+), 135 deletions(-) > > diff --git a/mysql-test/r/stat_tables_innodb_debug.result b/mysql-test/r/stat_tables_innodb_debug.result > new file mode 100644 > index 0000000..b02848d > --- /dev/null > +++ b/mysql-test/r/stat_tables_innodb_debug.result > @@ -0,0 +1,27 @@ > +# > +# MDEV-7901: re-implement analyze table for low impact > +# > +create table t1(a int primary key) engine=innodb; > +insert into t1 values (0),(1),(2),(3),(4),(5),(6),(7),(8),(9); you use t1 only to populate t2. You don't need to create a table for thatr, use seq_0_to_9 - https://mariadb.com/kb/en/mariadb/sequence/ > +create table t2 (a int, b int, c int, key IDX1(a), key IDX2(a,b)) engine=innodb; > +insert into t2 select a/10, a/2, a from test.t1; > +SET DEBUG_SYNC='statistics_collection_start1 SIGNAL analyzing WAIT_FOR written'; > +analyze table t2 persistent for all; > +connect con1, localhost, root,,; > +connection con1; > +SET DEBUG_SYNC= 'now WAIT_FOR analyzing'; > +select count(*) from t2; > +count(*) > +10 > +insert into t2 values (333,333,333); > +update t2 set a=1; > +SET DEBUG_SYNC= 'now SIGNAL written'; > +connection default; > +Table Op Msg_type Msg_text > +test.t2 analyze status Engine-independent statistics collected > +test.t2 analyze status OK > +select count(*) from t2; > +count(*) > +11 > +set debug_sync='RESET'; > +drop table t1,t2; good direct test > diff --git a/sql/sql_admin.cc b/sql/sql_admin.cc > index 578b78d..933f3d0 100644 > --- a/sql/sql_admin.cc > +++ b/sql/sql_admin.cc > @@ -812,7 +773,73 @@ static bool mysql_admin_table(THD* thd, TABLE_LIST* tables, > > if (compl_result_code == HA_ADMIN_OK && collect_eis) > { ok, I've quickly checked the code and I think (not sure) that there is no need to reopen for innodb. But as you do, please, show the benefits of it - add a test case for MyISAM too (remember, TL_READ allows concurrent inserts). besides, this reopening code needs a comment explaining why you're doing it. > + enum_sql_command save_sql_command= lex->sql_command; > + trans_commit_stmt(thd); > + trans_commit(thd); > + thd->open_options|= extra_open_options; > + close_thread_tables(thd); > + table->table= NULL; > + thd->mdl_context.release_transactional_locks(); > + lex->sql_command= save_sql_command; where is lex->sql_command changed? > + table->mdl_request.init(MDL_key::TABLE, table->db, table->table_name, > + MDL_SHARED_NO_READ_WRITE, MDL_TRANSACTION); > + table->mdl_request.set_type(MDL_SHARED_READ); > + > + table->lock_type= TL_READ; > + open_error= open_only_one_table(thd, table, > + repair_table_use_frm, > + (view_operator_func != NULL)); can one analyze a view? how does that work? > + thd->open_options&= ~extra_open_options; ... Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 1

[Maria-developers] MariaDB memory usage when rebuilding a large table
by Mark Wadham 03 Aug '16

03 Aug '16

Hi, Apologies if this is the wrong place to ask this question but it seemed like the most apt. We are working on operational processes for online schema changes with MariaDB 10.1.16 and have encountered memory issues which are proving difficult to reliably solve. If anyone on this list could explain to me in detail how to calculate the memory footprint of modifying a large table with DDL statements I would be very grateful. Our test case is a 3-node galera cluster on which we want to alter a ~160GB table that has . Each node has 64GB ram and (originally) no swap space allocated. Even with the innodb buffer pool lowered to 16GB we consistently reached an OOM condition trying to add a column to the table. If we lower the innodb buffer pool to 16GB *and* provision 16GB of swap, the DDL completes. With the innodb buffer pool left at 50GB and allocated swap of 16GB, same OOM condition. Since this is an operational procedure we are testing the process under heavy load, but the node that is applying the DDL is first placed into RSU mode so the writesets simply build up in the gcache while it's executing (and obviously traffic from the production environment would be directed away from the node while this is happening). We are also increasing the gcache to 800GB on the node as part of this process to ensure that we have plenty of time for it to complete. I'm fairly sure that the gcache is simply a continuous write-to-disk process rather than anything involving significant memory, so *hope* that's not a contributing factor here. I am however quite surprised that executing DDL on a table can be so memory-intensive. Our next test run will be with the innodb buffer pool set to 0 on the node executing the DDL with no swap allocated to see if it can complete purely in-memory. Logically it should be able to if this is a memory constraint rather than a bug. But fundamentally we need to be able to determine with confidence how much memory we need to successfully execute DDL on a table of {n} size. Thanks, Mark

1 1

Re: [Maria-developers] e252dae: MDEV-9872 - Add common optimized CRC32 function interface
by Sergei Golubchik 03 Aug '16

03 Aug '16

Hi, Sergey! So, what did you do here? Did you change InnoDB's version of crc32 or just moved it to extra/ ? You removed mysys/my_crc32.c - where will crc32 come from on x86? Why did you name new function crc32_vpmsum instead of just naming it crc32 and avoding #ifdef's elsewhere? On Jul 05, Sergey Vojtovich wrote: > revision-id: e252dae86392a2b50d647a380db36fb8b01bf405 (mariadb-10.2.1-5-ge252dae) > parent(s): 72c94872835f27b536f617978dbc664ddb57da06 > committer: Sergey Vojtovich > timestamp: 2016-07-05 18:09:23 +0400 > message: > > MDEV-9872 - Add common optimized CRC32 function interface > > Move crc32-vpmsum to extra. > Compile static crc32-vpmsum instead of adding sources directly. > Make use of crc32-vpmsum via my_checksum(). > > Based on contribution by Daniel Black. Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 1