- developers - lists.mariadb.org

Re: [Maria-developers] 7910c99: MDEV-10043 - main.events_restart fails sporadically in buildbot (crashes upon
by Sergei Golubchik 20 Jun '16

20 Jun '16

Hi, Sergey! On Jun 17, Sergey Vojtovich wrote: > revision-id: 7910c99bd1fa30911225e512997f466cb1b7996d (mariadb-5.5.49-51-g7910c99) > parent(s): 7ff86b49912df5de7b5a8b9c9134e72134b7ba5c > committer: Sergey Vojtovich > timestamp: 2016-06-17 18:54:11 +0400 > message: > > MDEV-10043 - main.events_restart fails sporadically in buildbot (crashes upon > shutdown) > > There was race condition between shutdown thread and event worker threads. > > Shutdown thread waits for thread_count to become 0 in close_connections(). It > may happen so that event worker thread was started but didn't increment > thread_count by this time. In this case shutdown thread may miss wait for this > working thread and continue deinitialization. Worker thread in turn may continue > execution and crash on deinitialized data. > > Fixed by incrementing thread_count before thread is actually created like it is > done for connection threads. > > Also let event scheduler not to inc/dec running threads counter for symmetry > with other "service" threads. ok to push. I'll have a questionable pleasure of merging it into 10.2 :) (it conflicts with 562c1df). Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] MDEV-10020 InnoDB NOT IN Query Crash When One Item Is NULL
by Alexander Barkov 20 Jun '16

20 Jun '16

Hi Sergei, Please review a patch for mdev-10020. The problem was that the loop in get_func_mm_tree() accessed an improperly initialized instance of String, which is a bzero'ed part of the in_vector::base array. Strings in in_vector::base are initialized in Item_func_in::fix_length_and_dec(): in in_vector::in_vector() using sql_calloc, rather than a String constructor, so its str_charset member of this String is NULL. Strings in in_vector::base are later initialized in Item_func_in::fix_length_and_dec(), using array->set(), in this code: uint j=0; for (uint i=1 ; i < arg_count ; i++) { array->set(j,args[i]); if (!args[i]->null_value) // Skip NULL values j++; else have_null= 1; } if ((array->used_count= j)) array->sort(); NULLs are not taken into account, so array->used_count can be smaller than array->count. This patch fixes the loop in opt_range.cc, in get_func_mm_tree(), to access only properly initialized elements in in_vector::base, preventing access to its bzero'ed non-initialized tail. Thanks.

2 1

Re: [Maria-developers] fa129a7: MDEV-9728 - Hard crash in metadata_lock_info
by Sergei Golubchik 20 Jun '16

20 Jun '16

Hi, Sergey! Looks ok to push, thanks! On Jun 17, Sergey Vojtovich wrote: > revision-id: fa129a7d22656eff9a7d1fb137d22976e0f0cec1 (mariadb-10.0.25-7-gfa129a7) > parent(s): 56ec3bdebd4a17ff59a487a988069f77bfc69aab > committer: Sergey Vojtovich > timestamp: 2016-06-17 12:52:35 +0400 > message: > > MDEV-9728 - Hard crash in metadata_lock_info > > metadata_lock_info plugin called MDL_context::find_ticket() to obtain lock > duration, which in turn iterates foreign thread private lists. These lists > can be updated by owner thread without protection. > > Fixed by iterating threads (instead of MDL locks and tickets) and obtaining > data through APC. > > Also fixed mdl_iterate_lock() to initialize iterator under prlock protection. > Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] [Commits] b21e7af: MDEV-9969 mysql_install_db error processing ignore_db_dirs.
by Sergei Golubchik 16 Jun '16

16 Jun '16

Hi, Alexey! On Jun 16, Alexey Botchkov wrote: > revision-id: b21e7af20ed4e694376b6a2a246af9dd481d7bb0 (mariadb-10.0.25-7-gb21e7af) > parent(s): 0e50b924820d365bcc1659eec2cad606f6887597 > committer: Alexey Botchkov > timestamp: 2016-06-16 13:41:45 +0400 > message: > > MDEV-9969 mysql_install_db error processing ignore_db_dirs. > > Changes to the mysql_install_db scripts so they don't repeat > arguments twice. ok to push Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] Parallel replication slave stops with duplicate key errors
by Sergey Petrunia 16 Jun '16

16 Jun '16

Hi Kristian, I was trying out parallel replication and got an error that I am not sure what to do with. == Setup == I started a master with binlog_format=row, I ran sysbench there to load the dataset ("prepare" mode, about 10M rows IIRC) then I restarted the master and ran sysbench in "run" mode a few times. Then I restarted the master again an ran a "CREATE TABLE" statement manually. Just so that I have third binlog. The master state is: mysql> show binary logs; +-------------+-----------+ | Log_name | File_size | +-------------+-----------+ | pslp.000001 | 194918772 | | pslp.000002 | 28895350 | | pslp.000003 | 589 | +-------------+-----------+ == Non-parallel test run == Then I connected with a slave, with all default settings. The slave had empty datadir, so I made it a slave with this command: CHANGE MASTER TO MASTER_HOST='127.0.0.1', MASTER_PORT=3310, MASTER_USER='root'; All server settings were pretty much default. It took a slave one hour to catch up, but it did so, and encountered no errors. Ok, this was serial replication == Parallel test run == Then I killed the slave, deleted its data directory, put an empty directory instead I added this to slave.cnf file: slave_parallel_threads=4 slave_parallel_mode=aggressive and started the slave. It ran in parallel for a few minutes, I saw this: mysql> show processlist; +----+-------------+-----------+------+---------+------+-----------------------------------------------+------------------+----------+ | Id | User | Host | db | Command | Time | State | Info | Progress | +----+-------------+-----------+------+---------+------+-----------------------------------------------+------------------+----------+ | 3 | root | localhost | test | Query | 0 | init | show processlist | 0.000 | | 4 | system user | | NULL | Connect | 15 | Waiting for master to send event | NULL | 0.000 | | 5 | system user | | NULL | Connect | 0 | Write_rows_log_event::write_row(-1) | NULL | 0.000 | | 6 | system user | | NULL | Connect | 0 | Waiting for work from SQL thread | NULL | 0.000 | | 7 | system user | | NULL | Connect | 0 | Write_rows_log_event::write_row(-1) | NULL | 0.000 | | 8 | system user | | NULL | Connect | 0 | Waiting for work from SQL thread | NULL | 0.000 | | 9 | system user | | NULL | Connect | 14 | Waiting for room in worker thread event queue | NULL | 0.000 | +----+-------------+-----------+------+---------+------+-----------------------------------------------+------------------+----------+ 7 rows in set (0.00 sec) but then it stopped. The error log has this: 2016-06-15 15:09:58 140702586379008 [Note] Slave SQL thread initialized, starting replication in log 'FIRST' at position 0, relay log './psergey-desktop-relay-bin.000001' position: 4 2016-06-15 15:09:59 140702586985216 [Note] InnoDB: Online DDL : Start 2016-06-15 15:09:59 140702586985216 [Note] InnoDB: Online DDL : Start reading clustered index of the table and create temporary files 2016-06-15 15:09:59 140702586985216 [Note] InnoDB: Online DDL : End of reading clustered index of the table and create temporary files 2016-06-15 15:09:59 140702586985216 [Note] InnoDB: Online DDL : Start applying row log 2016-06-15 15:09:59 140702586985216 [Note] InnoDB: Online DDL : End of applying row log 2016-06-15 15:09:59 140702586985216 [Note] InnoDB: Online DDL : Completed 2016-06-15 15:11:18 140702586985216 [ERROR] Slave SQL: Error during XID COMMIT: failed to update GTID state in mysql.gtid_slave_pos: 1062: Duplicate entry '0-1021' for key 'PRIMARY', Gtid 0-1-1021, Internal MariaDB error code: 1942 2016-06-15 15:11:18 140702586985216 [ERROR] Slave (additional info): Duplicate entry '0-1021' for key 'PRIMARY' Error_code: 1062 2016-06-15 15:11:18 140702586985216 [Warning] Slave: Duplicate entry '0-1021' for key 'PRIMARY' Error_code: 1062 2016-06-15 15:11:18 140702586985216 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'pslp.000002' position 8266 2016-06-15 15:11:18 140702586682112 [ERROR] Slave (additional info): Commit failed due to failure of an earlier commit on which this one depends Error_code: 1964 2016-06-15 15:11:18 140702586682112 [Warning] Slave: Duplicate entry '821827' for key 'PRIMARY' Error_code: 1062 2016-06-15 15:11:18 140702586682112 [Warning] Slave: Deadlock found when trying to get lock; try restarting transaction Error_code: 1213 2016-06-15 15:11:18 140702586682112 [Warning] Slave: Commit failed due to failure of an earlier commit on which this one depends Error_code: 1964 2016-06-15 15:11:18 140702586682112 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'pslp.000002' position 8266 2016-06-15 15:11:18 140702602623744 [ERROR] Slave (additional info): Commit failed due to failure of an earlier commit on which this one depends Error_code: 1964 2016-06-15 15:11:18 140702602623744 [Warning] Slave: Commit failed due to failure of an earlier commit on which this one depends Error_code: 1964 2016-06-15 15:11:18 140702602623744 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'pslp.000002' position 8266 2016-06-15 15:11:18 140702602926848 [ERROR] Slave (additional info): Commit failed due to failure of an earlier commit on which this one depends Error_code: 1964 2016-06-15 15:11:18 140702602926848 [Warning] Slave: Commit failed due to failure of an earlier commit on which this one depends Error_code: 1964 2016-06-15 15:11:18 140702602926848 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'pslp.000002' position 8266 2016-06-15 15:11:18 140702586985216 [ERROR] Slave (additional info): Commit failed due to failure of an earlier commit on which this one depends Error_code: 1964 2016-06-15 15:11:18 140702586985216 [Warning] Slave: Duplicate entry '0-1021' for key 'PRIMARY' Error_code: 1062 2016-06-15 15:11:18 140702586985216 [Warning] Slave: Commit failed due to failure of an earlier commit on which this one depends Error_code: 1964 2016-06-15 15:11:18 140702586985216 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'pslp.000002' position 8266 2016-06-15 15:11:18 140702586682112 [ERROR] Slave (additional info): Commit failed due to failure of an earlier commit on which this one depends Error_code: 1964 2016-06-15 15:11:18 140702586682112 [Warning] Slave: Duplicate entry '821827' for key 'PRIMARY' Error_code: 1062 ... and there were lots of these errors. Eventually it stopped with: 2016-06-15 15:11:22 140702586682112 [Warning] Slave: Commit failed due to failure of an earlier commit on which this one depends Error_code: 1964 2016-06-15 15:11:22 140702586682112 [Warning] Slave: Commit failed due to failure of an earlier commit on which this one depends Error_code: 1964 2016-06-15 15:11:22 140702586682112 [Warning] Slave: Commit failed due to failure of an earlier commit on which this one depends Error_code: 1964 2016-06-15 15:11:22 140702586682112 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'pslp.000002' position 8266 2016-06-15 15:11:22 140702586379008 [Note] Slave SQL thread exiting, replication stopped in log 'pslp.000002' at position 8266 The status is: mysql> show slave status\G *************************** 1. row *************************** Slave_IO_State: Waiting for master to send event Master_Host: 127.0.0.1 Master_User: root Master_Port: 3310 Connect_Retry: 60 Master_Log_File: pslp.000003 Read_Master_Log_Pos: 589 Relay_Log_File: psergey-desktop-relay-bin.000003 Relay_Log_Pos: 8549 Relay_Master_Log_File: pslp.000002 Slave_IO_Running: Yes Slave_SQL_Running: No Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 1942 Last_Error: Error during XID COMMIT: failed to update GTID state in mysql.gtid_slave_pos: 1062: Duplicate entry '0-1021' for key 'PRIMARY' Skip_Counter: 0 Exec_Master_Log_Pos: 8266 Relay_Log_Space: 223815659 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: NULL Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 1942 Last_SQL_Error: Error during XID COMMIT: failed to update GTID state in mysql.gtid_slave_pos: 1062: Duplicate entry '0-1021' for key 'PRIMARY' Replicate_Ignore_Server_Ids: Master_Server_Id: 1 Master_SSL_Crl: Master_SSL_Crlpath: Using_Gtid: No Gtid_IO_Pos: Replicate_Do_Domain_Ids: Replicate_Ignore_Domain_Ids: Parallel_Mode: aggressive Can you advise, what does this error mean? (is there any benefit in cc'ing this to maria-developers@ ? if yes, feel free to..) BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

2 1

Re: [Maria-developers] [Commits] 607e22e: MDEV-8633: information_schema.index_statistics doesn't delete
by Sergei Golubchik 15 Jun '16

15 Jun '16

Hi, Jan! On Apr 26, Jan Lindström wrote: > revision-id: 607e22e1426ef3d7e11e3a2e9d7f884d36cf3573 (mariadb-10.0.24-43-g607e22e) > parent(s): ce38adddfa91949b30956abd0e4cab201effcdef > committer: Jan Lindström > timestamp: 2016-04-26 13:47:30 +0300 > message: > > MDEV-8633: information_schema.index_statistics doesn't delete > item when drop table indexes or drop table; > > Problem was that table and index statistics is removed from > persistent tables but not from memory cache. Added functions > to remove table and index statistics from memory cache. That's of course, ok. Just minor comments, see below: > diff --git a/sql/sql_show.cc b/sql/sql_show.cc > index b8926b9..ac4503d 100644 > --- a/sql/sql_show.cc > +++ b/sql/sql_show.cc > @@ -3442,6 +3442,100 @@ int fill_schema_table_stats(THD *thd, TABLE_LIST *tables, COND *cond) > DBUG_RETURN(0); > } > > +/* Remove all indexes from global index statistics */ "all indexes for a given table" > +static > +int del_global_index_stats(THD *thd, uchar* cache_key, uint cache_key_length) > +{ > + int res = 0; > + DBUG_ENTER("del_global_index_stats"); > + > + mysql_mutex_lock(&LOCK_global_index_stats); > + > + for (uint i= 0; i < global_index_stats.records && !res;) > + { > + INDEX_STATS *index_stats = > + (INDEX_STATS*) my_hash_element(&global_index_stats, i); > + > + /* We search correct db\0table_name\0 string */ > + if (index_stats && > + index_stats->index_name_length >= cache_key_length && > + !memcmp(index_stats->index, cache_key, cache_key_length)) > + { > + res= my_hash_delete(&global_index_stats, (uchar*)index_stats); > + /* > + In our HASH implementation on deletion one elements > + is moved into a place where a deleted element was, > + and the last element is moved into the empty space. > + Thus we need to re-examine the current element, but > + we don't have to restart the search from the beginning. > + */ > + } > + else > + i++; > + } > + > + mysql_mutex_unlock(&LOCK_global_index_stats); > + DBUG_RETURN(res); > +} > + > +/* Remove a table from global table statistics */ > + > +int del_global_table_stat(THD *thd, LEX_STRING *db, LEX_STRING *table) > +{ > + TABLE_STATS *table_stats; > + int res = 0; > + uchar *cache_key; > + uint cache_key_length; > + DBUG_ENTER("del_global_TABLE_stat"); correct letter case: "del_global_table_stat". > + > + cache_key_length= db->length + 1 + table->length + 1; > + > + if(!(cache_key= (uchar *)my_malloc(cache_key_length, > + MYF(MY_WME | MY_ZEROFILL)))) > + { > + /* Out of memory error already given */ > + res = 1; > + goto end; > + } > + > + memcpy(cache_key, db->str, db->length); > + memcpy(cache_key + db->length + 1, table->str, table->length); > + > + res= del_global_index_stats(thd, cache_key, cache_key_length); > + > + mysql_mutex_lock(&LOCK_global_table_stats); > + > + if(!res && (table_stats= (TABLE_STATS*) my_hash_search(&global_table_stats, here and below you, probably, don't want if (!res) - if the table is dropped, you should delete as many its entries as possible, even if something fails for an unknown reason. > + cache_key, > + cache_key_length))) > + res= my_hash_delete(&global_table_stats, (uchar*)table_stats); > + > + my_free(cache_key); > + mysql_mutex_unlock(&LOCK_global_table_stats); > + > +end: > + DBUG_RETURN(res); > +} > + > +/* Remove a index from global index statistics */ > + > +int del_global_index_stat(THD *thd, TABLE* table, KEY* key_info) The difference is del_global_index_stat and del_global_index_stats is rather subtle. Could you rename them to make it clearer? For example, you can rename the static one to del_global_index_stats_for_table. > +{ > + INDEX_STATS *index_stats; > + uint key_length= table->s->table_cache_key.length + key_info->name_length + 1; > + int res = 0; > + DBUG_ENTER("del_global_index_stat"); > + mysql_mutex_lock(&LOCK_global_index_stats); > + > + if((index_stats= (INDEX_STATS*) my_hash_search(&global_index_stats, > + key_info->cache_name, > + key_length))) > + res= my_hash_delete(&global_index_stats, (uchar*)index_stats); > + > + mysql_mutex_unlock(&LOCK_global_index_stats); > + DBUG_RETURN(res); > +} > > /* Fill information schema table with index statistics */ > > diff --git a/sql/sql_statistics.cc b/sql/sql_statistics.cc > index e86c840..644ecf2 100644 > --- a/sql/sql_statistics.cc > +++ b/sql/sql_statistics.cc > @@ -29,6 +29,7 @@ > #include "sql_statistics.h" > #include "opt_range.h" > #include "my_atomic.h" > +#include "sql_show.h" > > /* > The system variable 'use_stat_tables' can take one of the > @@ -3193,6 +3194,9 @@ int delete_statistics_for_table(THD *thd, LEX_STRING *db, LEX_STRING *tab) > rc= 1; > } > > + if (!rc) again here and below, better to do it without if (!rc) > + rc= del_global_table_stat(thd, db, tab); > + > thd->restore_stmt_binlog_format(save_binlog_format); > > close_system_tables(thd, &open_tables_backup); > @@ -3339,6 +3343,9 @@ int delete_statistics_for_index(THD *thd, TABLE *tab, KEY *key_info, > } > } > > + if (!rc) > + rc= del_global_index_stat(thd, tab, key_info); > + > thd->restore_stmt_binlog_format(save_binlog_format); > Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] Sachin weekly report
by Sergei Golubchik 14 Jun '16

14 Jun '16

Hi, Sachin! Sounds good. Please continue with tests. It's difficult to see what is working and what isn't without them... What do you plan to do this week? Row comparison? Hidden columns? On Jun 14, Sachin Setia wrote: > Hello Sergei, > Weekly Report for third week of gsoc > > 1. Improved code as suggested by you. > 2.Update worked. > 3.Tried with with tests but ./mtr failed , so tried some bug fixes now it > is working. > 4.Tried unique index for text , and trying for varchar length > file-> > max_key_part_length() > > > Regards > sachin > Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] GSoC weekly reports (Unique indexes for blobs)
by Shubham Barai 14 Jun '16

14 Jun '16

GSoC (week 2) Hello everyone, Finally, I solved the issue with a mismatch of a key number between sql layer and MyISAM file.I also found a simpler approach for this with the help of my mentor. One of the other issues that I had to deal with was compatibility issue of key definitions between .frm file and information stored in MyISAM storage engine. The issue was due to check_defination which compares .frm file and MyISAM definitions of a table.The previous prototype of check_definition didn't contain mi_uniquedef. I have modified this function to compare mi_uniquedef.The main problem was due to the difference in a number of keys which we pass as an argument in ha_myisam::open. I have also started working on InnoDB.I discussed some of the implementation approaches with my mentor. I made changes to some of the functions in ha_innobase.cc. Regards, Shubham On 31 May 2016 at 03:32, Sergei Golubchik <serg(a)mariadb.org> wrote: > Hi, Shubham! > > Good, thanks! > > On May 31, Shubham Barai wrote: > > GSoC (week 1) > > > > Hello everyone, > > > > After developing a prototype for MyISAM ,there were some issues to be > > resolved. One of the issues is related to error regarding a duplicate > > entry. So if there is a duplicate entry for a particular key in a row, it > > generates an error for another key. > > > > After debugging, I found out that it was due to mismatch of a key number > > between sql layer and MyISAM file. Keys are stored in the order of > user's > > input in sql layer. In table2myisam function, this order is changed as > each > > key is either stored as mi_keydef or mi_uniquedef. In mi_create, for each > > unique definition, an extra key is created in the form of mi_keydef and > > stored in MYI file which creates the issue of mismatch. > > > > I tried to solve the issue by adding extra member sql_key_no to mi_keydef > > and mi_uniquedef.I also modified mi_keydef_write and mi_keydef_read to > > support a new member but after compiling, I got some error related to a > key > > file. Also with the help of my mentor, I found out that it could result > in > > compatibility issues related to old MyISAM file. > > > > So currently, I am trying to solve this using another approach. > > Regards, > Sergei > Chief Architect MariaDB > and security(a)mariadb.org >

2 2

[Maria-developers] Please review MDEV-10181 Illegal mix of collation for a field and an ASCII string as a view field
by Alexander Barkov 10 Jun '16

10 Jun '16

Hi Sergei, Please review a patch for MDEV-10181. Thanks!

2 1

[Maria-developers] Myisam-repair-threads question
by Alexandre hadjinlian guerra 10 Jun '16

10 Jun '16

Hi Im currently doing a reload process to an empty myisam table with 4 indexes on Maria db 10 1 13 on win64 and noted that on the last step that the status reports that its using repair to rebuild the table. Myisam-repair-threads remain the default but noted that many threads were running .im not sure if all of them were for repair. Is there any recommendation which i should take using the referred setting with a greater value?would setting it to zero make maridb decide which is there best value? Would the indexes get too much fragmented ? Thanks!

2 1