December 2017 - developers - lists.mariadb.org

Re: [Maria-developers] a01f9ff: MDEV-11071 - Assertion `thd->transaction.stmt.is_empty()' failed in
by Sergei Golubchik 27 Dec '17

27 Dec '17

Hi, Sergey! On Dec 26, Sergey Vojtovich wrote: > revision-id: a01f9ff80b892bf32dd2754da17438e8884c003a (mariadb-10.2.11-51-ga01f9ff) > parent(s): c3bd0b0301936d0dcd0918bb8c966c272610d61b > committer: Sergey Vojtovich > timestamp: 2017-12-26 14:38:45 +0400 > message: > > MDEV-11071 - Assertion `thd->transaction.stmt.is_empty()' failed in > Locked_tables_list::unlock_locked_tables > > Similarly to regular DROP TABLE, don't leave locked tables mode if CREATE OR > REPLACE dropped temporary table but failed to cerate new one. > > The problem is that there's no track of which temporary table was "locked" by > LOCK TABLES. Looks reasonable. Could you also add a test where t2 is locked and t2 is dropped? Just to show that pos_in_locked_tables works for temporary tables. Ok to push after that. Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] Please document HOUR() vs EXTRACT(HOUR)
by Alexander Barkov 22 Dec '17

22 Dec '17

Hello Ian, Can you please document the difference between functions HOUR() and EXTRACT(HOUR)? Please see details here: https://jira.mariadb.org/browse/MDEV-14731 Thanks!

1 0

[Maria-developers] mysql and mariadb got different result
by 落寞回头不如华丽转身 20 Dec '17

20 Dec '17

mysql but mariadb

2 1

[Maria-developers] Fwd: Need Help
by Deepak Gupta 20 Dec '17

20 Dec '17

Hi There, I am working on a project where I have some wired requirement for my project. Issue : I am looking for the generic script that I can run with the help of Admin rights on the data base and I want the return output in such a way that it will give me the complete data base structure along with the table relation ship. For an example , there is one HR schema , with in that schema there might be 100 tables example , Employee / Salary / Address / Department etc and every table has primary and foreign key concept. So I need the script should be such a way that it will fetch all these information and as soon as I run the same script in my local system it will replicate the same data base structure for me in my local system. Kindly suggest how can I achieve this requirement ? Any help or suggestions are more than welcome. Looking foreword to here you soon. Thanks D

2 1

Re: [Maria-developers] [Commits] 203a658: MDEV-11084 Select statement with partition selection against MyISAM table opens all partitions.
by Sergei Golubchik 19 Dec '17

19 Dec '17

Hi, Alexey! On Dec 18, Alexey Botchkov wrote: > revision-id: 203a6585d3deedb36e30d5ebf388de44b6810403 (mariadb-10.1.23-51-g203a658) > parent(s): e4d10e09cf318aad237143254c45458d16009f70 > committer: Alexey Botchkov > timestamp: 2017-12-18 19:31:26 +0400 > message: > > MDEV-11084 Select statement with partition selection against MyISAM table opens all partitions. > > Now SELECT FROM t PARTITION(x) only opens the 'x' file. > The table->partition_names parameter is sent to ha_open > so it only opens the required partitions. > If the table is reopened, the > change_partition_to_open(partition_names) is called that > closes and opens partitions specified for the query. Looks ok to me, thanks! Please add some tests, can status variables show the difference in the number of opened partitions? And make sure that the whole tree compiles without partitioning. Then ok to push into 10.3 Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] [Commits] 7a7ad82: MDEV-13478 Full SST sync fails because of the error in the cleaning part
by Sergei Golubchik 19 Dec '17

19 Dec '17

Hi, Sachin! Unfortunately, your comment is rather difficult to understand. What about this one: Problem: The command was: find $paths -mindepth 1 -regex $cpat -prune -o -exec rm -rf {} \+ Which was supposed to work as * skipping $paths directories themselves (-mindepth 1) * see if the dir/file name matches $cpat (-regex) * if yes - don't dive into the directory, skip it (-prune) * otherwise (-o) * remove it and everything inside (-exec) Now -exec ... \+ works like this: every new found path is appended to the end of the command line. when accumulated command line length reaches `getconf ARG_MAX` (~2Gb) it's executed, and find continues, appending to a new command line. What happens here, find appends some directory to the command line, then dives into it, and starts appending files from that directory. At some point command line overflows, rm -rf gets executed and removes the whole directory. Now find tries to continue scanning the directory that was already removed. Fix: don't dive into directories that will be recursively removed anyway, use -prune for them. Basically, we should be pruning both paths that have matched $cpat and paths that have not matched it. This is achived by pruning unconditionally, before the regex is tested: find $paths -mindepth 1 -prune -regex $cpat -o -exec rm -rf {} \+ On Dec 19, sachin wrote: > revision-id: 7a7ad82029a6c78d31d6736a562c12d02c4d968c (mariadb-galera-5.5.58-3-g7a7ad82) > parent(s): e6e026ae51a77969749de201d491a176483bbc69 > author: Sachin Setiya > committer: Sachin Setiya > timestamp: 2017-12-19 22:30:43 +0530 > message: > > MDEV-13478 Full SST sync fails because of the error in the cleaning part > > Problem:- The problem is in wsrep_sst_xtrabackup-v2.sh we use > find $ib_home_dir $ib_log_dir $ib_undo_dir $DATA -mindepth 1 -regex $cpat -prune -o -exec rm -rfv {} 1>&2 \+ > the problem is that since we have '\+' in end that means all output > will be expanded after rm -rfv . If we have really large database( > quite a no of tables with big names) then this create a problem. > This will result in calling 'rm -rvf' two time(or may be more). So non > deterministicly this might that upto directory name went to rm and > remaining was truncated. For example consider a folder xyz with lots > of files. We executed > find xyz -exec rm -rfv {} \+ > Since we have like millions of file in rm, So it will be greater then > ARG_MAX , so there will be multiple rm invocation. Say in nth invocation > this might happen rm ....... xyz/ {So it get truncated at xyz/} > Above will remove the whole xyz directory and remaining invocation > of rm will return error. How ever this type of error is non deterministic > so is bug. > Solution:- > In above example if instead of removing each file in xyz if we remove > xyz then we have our solution :). > Actually if we shift the -prune term in find we can get to solution. > Why currently find is working like this > find ( -regex && -prune) || -exec > that means if -regex is true (-prune will always return true) then exec > (and hence rm )wont work. But if regex fails then exec is applied with > out -prune which makes rm delete each single file instead of folder. > So we change the position of regex and prune then prune will always be > applied whether regex is true or not. > find (-prune && -regex) || -exec > > Patch Credit:- Serg > > --- > scripts/wsrep_sst_xtrabackup-v2.sh | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/scripts/wsrep_sst_xtrabackup-v2.sh b/scripts/wsrep_sst_xtrabackup-v2.sh > index 327c92f..00d8fe2 100644 > --- a/scripts/wsrep_sst_xtrabackup-v2.sh > +++ b/scripts/wsrep_sst_xtrabackup-v2.sh > @@ -863,7 +863,7 @@ then > > > wsrep_log_info "Cleaning the existing datadir and innodb-data/log directories" > - find $ib_home_dir $ib_log_dir $ib_undo_dir $DATA -mindepth 1 -regex $cpat -prune -o -exec rm -rfv {} 1>&2 \+ > + find $ib_home_dir $ib_log_dir $ib_undo_dir $DATA -mindepth 1 -prune -regex $cpat -o -exec rm -rfv {} 1>&2 \+ > > tempdir=$(parse_cnf mysqld log-bin "") > if [[ -n ${tempdir:-} ]];then Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 1

Re: [Maria-developers] 665402a: MDEV-14265 - RPMLint warning: shared-lib-calls-exit
by Sergei Golubchik 18 Dec '17

18 Dec '17

Hi, Sergey! On Dec 18, Sergey Vojtovich wrote: > revision-id: 665402aa9b83565c1cef3caa06cd998d3808f0d7 (mariadb-10.1.29-31-g665402a) > parent(s): ef9e78c9d41a5ac644f08068e3dabad948b0e30a > committer: Sergey Vojtovich > timestamp: 2017-12-18 20:23:36 +0400 > message: > > MDEV-14265 - RPMLint warning: shared-lib-calls-exit > > find_type_with_warning() client helper did exit(1) on error, exit(1) moved to > clients. It was find_type_or_exit(), and you could've had less changes if you'd replaced it with an inline function. Basically, copied the function into the header and made it static inline. > mysql_read_default_options() did exit(1) on error, error is passed through and > handled now. ok > my_str_malloc_default() did exit(1) on error, replaced my_str_ allocator > functions with normal my_malloc()/my_realloc()/my_free(). I think you need to replace my_str_malloc_default() with my_malloc(..., MYF(MY_FAE)), not my_malloc(..., MYF(0)) > sql_connect.cc did many exit(1) on hash initialisation failure. Removed error > check since my_hash_init() never fails. > > my_malloc() did exit(1) on error. Replaced with abort(). > > my_load_defaults() did exit(1) on error, replaced with abort(). I'd say, return 2 instead of exit/abort. my_load_defaults already does it, because my_search_option_files returns 2 for OOM. > my_load_defaults() still does exit(0) when invoked with --print-defaults. dunno, return -1, for example, and exit in the caller? Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 1

[Maria-developers] MDEV-14031 Password policy causes replication failure
by Alexander Barkov 18 Dec '17

18 Dec '17

Hi Sergei, Please review a patch for MDEV-14031. Thanks!

1 0

Re: [Maria-developers] [Commits] 2603dc0: MDEV-11084 Select statement with partition selection against MyISAM table opens all partitions.
by Sergei Golubchik 15 Dec '17

15 Dec '17

Hi, Alexey! That looked reasonable enough. I didn't find anything big. Small comments - see below. On Jun 01, Alexey Botchkov wrote: > revision-id: 2603dc0d56ec90f308678a3c002f1b75af69f554 (mariadb-10.1.23-51-g2603dc0) > parent(s): e4d10e09cf318aad237143254c45458d16009f70 > committer: Alexey Botchkov > timestamp: 2017-06-01 17:33:34 +0400 > message: > > MDEV-11084 Select statement with partition selection against MyISAM table opens all partitions. > > Now SELECT FROM t PARTITION(x) only opens the 'x' file. > The table->partition_names parameter is sent to ha_open > so it only opens the required partitions. > If the table is reopened, the > change_partition_to_open(partition_names) is called that > closes and opens partitions specified for the query. > > diff --git a/sql/ha_partition.cc b/sql/ha_partition.cc > index 29054c7..53a128c 100644 > --- a/sql/ha_partition.cc > +++ b/sql/ha_partition.cc > @@ -3481,12 +3498,16 @@ int ha_partition::open(const char *name, int mode, uint test_if_locked) > } > name_buffer_ptr+= strlen(name_buffer_ptr) + 1; > } > + file_sample= m_file[0]; are you sure that m_file[0] is opened? > } > else > { > file= m_file; > do > { > + if (!bitmap_is_set(&m_opened_partitions, file - m_file)) > + continue; > + > create_partition_name(name_buff, name, name_buffer_ptr, NORMAL_PART_NAME, > FALSE); > table->s->connect_string = m_connect_string[(uint)(file-m_file)]; > @@ -3494,20 +3515,21 @@ int ha_partition::open(const char *name, int mode, uint test_if_locked) > test_if_locked | HA_OPEN_NO_PSI_CALL))) > goto err_handler; > bzero(&table->s->connect_string, sizeof(LEX_STRING)); > - if (m_file == file) > - m_num_locks= (*file)->lock_count(); > + if (!file_sample) > + m_num_locks= (file_sample= (*file))->lock_count(); > DBUG_ASSERT(m_num_locks == (*file)->lock_count()); > - name_buffer_ptr+= strlen(name_buffer_ptr) + 1; > - } while (*(++file)); > + } while (name_buffer_ptr+= strlen(name_buffer_ptr) + 1, *(++file)); why? > } > > file= m_file; > - ref_length= (*file)->ref_length; > - check_table_flags= (((*file)->ha_table_flags() & > + ref_length= file_sample->ref_length; > + check_table_flags= ((file_sample->ha_table_flags() & > ~(PARTITION_DISABLED_TABLE_FLAGS)) | > (PARTITION_ENABLED_TABLE_FLAGS)); > while (*(++file)) > { > + if (!bitmap_is_set(&m_opened_partitions, file - m_file)) > + continue; > /* MyISAM can have smaller ref_length for partitions with MAX_ROWS set */ > set_if_bigger(ref_length, ((*file)->ref_length)); > /* > @@ -6782,6 +6809,63 @@ void ha_partition::get_dynamic_partition_info(PARTITION_STATS *stat_info, > } > > > +void ha_partition::set_partitions_to_open(List<String> *partition_names) > +{ > + m_partitions_to_open= partition_names; > +} > + > + > +int ha_partition::change_partitions_to_open(List<String> *partition_names) > +{ > + char name_buff[FN_REFLEN]; > + handler **file; > + char *name_buffer_ptr; > + int error= 0; > + > + if (m_is_clone_of) > + return 0; > + > + m_partitions_to_open= partition_names; > + if ((error= m_part_info->set_partition_bitmaps(partition_names))) > + goto err_handler; > + > + if (bitmap_cmp(&m_opened_partitions, &m_part_info->read_partitions) != 0) > + return 0; > + > + if ((error= read_par_file(table->s->normalized_path.str))) > + goto err_handler; > + > + name_buffer_ptr= m_name_buffer_ptr; > + file= m_file; > + do > + { > + int n_file= file-m_file; > + int is_open= bitmap_is_set(&m_opened_partitions, n_file); > + int should_be_open= bitmap_is_set(&m_part_info->read_partitions, n_file); > + > + if (is_open && !should_be_open) > + (*file)->ha_close(); > + else if (!is_open && should_be_open) > + { > + create_partition_name(name_buff, table->s->normalized_path.str, > + name_buffer_ptr, NORMAL_PART_NAME, FALSE); > + table->s->connect_string = m_connect_string[(uint)(file-m_file)]; > + if ((error= (*file)->ha_open(table, name_buff, m_mode, > + m_open_test_lock | HA_OPEN_NO_PSI_CALL))) > + goto err_handler; > + bzero(&table->s->connect_string, sizeof(LEX_STRING)); > + } > + } while (name_buffer_ptr+= strlen(name_buffer_ptr) + 1, *(++file)); Could you reuse that, instead of copying from ::open? Perhaps ::open() can reset the bitmap to be empty and then call ::change_partitions_to_open() ? Or the common code could be moved into a third function that these both will use... > + > + bitmap_copy(&m_opened_partitions, &m_part_info->read_partitions); > + > + clear_handler_file(); > + > +err_handler: > + return error; > +} > + > + > /** > General function to prepare handler for certain behavior. > > diff --git a/sql/partition_info.cc b/sql/partition_info.cc > index bc0db9e..7fb8f70 100644 > --- a/sql/partition_info.cc > +++ b/sql/partition_info.cc > @@ -283,6 +281,27 @@ bool partition_info::set_partition_bitmaps(TABLE_LIST *table_list) > > > /** > + Set read/lock_partitions bitmap over non pruned partitions > + > + @param table_list Possible TABLE_LIST which can contain > + list of partition names to query > + > + @return Operation status > + @retval FALSE OK > + @retval TRUE Failed to allocate memory for bitmap or list of partitions > + did not match > + > + @note OK to call multiple times without the need for free_bitmaps. > +*/ > +bool partition_info::set_partition_bitmaps_from_table(TABLE_LIST *table_list) > +{ > + List<String> *partition_names= table_list ? > + NULL : table_list->partition_names; > + return set_partition_bitmaps(partition_names); > +} This looks unused. > + > + > +/** > Checks if possible to do prune partitions on insert. > > @param thd Thread context > diff --git a/sql/sql_base.cc b/sql/sql_base.cc > index 9c56b7d..2f9ec3e 100644 > --- a/sql/sql_base.cc > +++ b/sql/sql_base.cc > @@ -2533,6 +2533,12 @@ bool open_table(THD *thd, TABLE_LIST *table_list, Open_table_context *ot_ctx) > { > DBUG_ASSERT(table->file != NULL); > MYSQL_REBIND_TABLE(table->file); > + if (table->part_info) > + { > + /* Set all [named] partitions as used. */ > + if (table->file->change_partitions_to_open(table_list->partition_names)) > + DBUG_RETURN(true); > + } > } > else > { > @@ -2549,7 +2555,8 @@ bool open_table(THD *thd, TABLE_LIST *table_list, Open_table_context *ot_ctx) > HA_TRY_READ_ONLY), > (READ_KEYINFO | COMPUTE_TYPES | > EXTRA_RECORD), > - thd->open_options, table, FALSE); > + thd->open_options, table, FALSE, > + table_list->partition_names); Please, make sure you code compiles without partitioning too. (in this case I wouldn't make the last argument of open_table_from_share conditional, but I'd rather make table_list->partition_names unconditional). > > if (error) > { > @@ -2598,9 +2605,11 @@ bool open_table(THD *thd, TABLE_LIST *table_list, Open_table_context *ot_ctx) > #ifdef WITH_PARTITION_STORAGE_ENGINE > if (table->part_info) > { > +#ifdef CODE_TO_REMOVE > /* Set all [named] partitions as used. */ > if (table->part_info->set_partition_bitmaps(table_list)) > DBUG_RETURN(true); > +#endif /*CODE_TO_REMOVE*/ why not to remove at once? > } > else if (table_list->partition_names) > { Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] 14acb68: MDEV-14593 human-readable XA RECOVER.
by Sergei Golubchik 14 Dec '17

14 Dec '17

Hi, Alexey! On Dec 12, Alexey Botchkov wrote: > revision-id: 14acb687199f0d30444f8c0849c0d1c1284f3e2b (mariadb-10.2.11-26-g14acb68) > parent(s): 4d016e6ed2d32298977a66e125cbfcae39e23847 > committer: Alexey Botchkov > timestamp: 2017-12-12 14:31:41 +0400 > message: > > MDEV-14593 human-readable XA RECOVER. > > The 'data' field in the XA RECOVER resultset changed > to be charset_bin. It seems to me right and also > --binary-as-hex starts working. > The XA RECOVER FORMAT='SQL' option implemented. > It returns the XID string that fits to be an argument for the > XA ... statements. Looks ok, just a couple of comments: > +xa recover format='SQL'; Please add a test for format=SQL (without quotes). We normally document EXPLAIN FORMAT=JSON without quotes (although it works with quotes too), so XA RECOVER should work the same way. > +ER_UNKNOWN_XID_FORMAT > + eng "Unknown XID format name: '%s'" Don't add a new error message, reuse ER_UNKNOWN_EXPLAIN_FORMAT: - eng "Unknown EXPLAIN format name: '%s'" - rus "Неизвестное имя формата команды EXPLAIN: '%s'" + eng "Unknown %s format name: '%s'" + rus "Неизвестное имя формата команды %s: '%s'" note, you'll need either to remove/rephrase the russian error message, or use "XA RECOVER" as the first argument, not "XID". Ok to push (into bb-10.2-ext) with these changes. Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0