- developers - lists.mariadb.org

Re: [Maria-developers] 9bf4b92cbc5: MDEV-17554 Auto-create new partition for system versioned tables with history partitioned by INTERVAL/LIMIT
by Sergei Golubchik 31 Mar '21

31 Mar '21

Hi, Aleksey! On Jun 29, Aleksey Midenkov wrote: > revision-id: 9bf4b92cbc5 (mariadb-10.5.2-168-g9bf4b92cbc5) > parent(s): 478301d9b9a > author: Aleksey Midenkov <midenok(a)gmail.com> > committer: Aleksey Midenkov <midenok(a)gmail.com> > timestamp: 2020-04-17 17:04:24 +0300 > message: > > MDEV-17554 Auto-create new partition for system versioned tables with history partitioned by INTERVAL/LIMIT > > == Syntax change == > > Keyword AUTO_INCREMENT (or AUTO) enables partition auto-creation. I still think that AUTO_INCREMENT here looks rather out-of-place, I don't know how to explain to users why AUTO_INCREMENT is allowed here. But if we'll have it, we'll have to support it here for a long time for compatibility reasons. "Because somebody might be already using it" I'd suggest to support only AUTO here. Or, if the parser can do it easily, a much more readable syntax would be create table t1 (x int) with system versioning auto partition by system_time interval 1 hour; this would be the best, easy to read, matches the documentation (where it could be called "auto partitioning"). If it won't cause any difficult parser problems, I'd prefer us to use that. > create or replace table t1 (x int) with system versioning > partition by system_time interval 1 hour auto_increment; > > create or replace table t1 (x int) with system versioning > partition by system_time limit 1000 auto; > > Or with explicit partitions: > > create or replace table t1 (x int) with system versioning > partition by system_time interval 1 hour auto > (partition p0 history, partition pn current); > > == Description == > > Before executing history-generating DML command add N history > partitions, so that N would be sufficient for potentially generated > history. N > 1 may be required when history is rotated by INTERVAL and > timestamp was jumped to future further than interval value. > > It is assumed that one DML command will not generate history rows more > than specified LIMIT. Otherwise history would overflow generated > partition, a warning would be printed, and user action would be > required to rebuild partitions. > > Auto-creation is implemented by synchronous > fast_alter_partition_table() call from the thread of the executed DML > command before the command itself (by the fallback and retry mechanism > similar to Discovery feature, see Open_table_context). > > Creating history partitions was made by the principle of minimal > disruption of the main business process. I.e. when procedure of > creation fails it will not (if possible) fail the original DML > command. Warning info will display what happened and the last history > partition may overflow. In such case partition rebuild is required to > correctly display history info. User application may detect warning > code ER_VERS_HIST_PART_FAILED and stop execution if that is preferred. > > The name for newly added partitions are generated like default > partition names with extension of MDEV-22155 (which avoids name > clashes by extending assignment counter to next free-enough gap). > > These DML commands trigger auto-creation: > > * DELETE (including multi-delete, excluding DELETE HISTORY) > * UPDATE (including multi-update) > * REPLACE (including REPLACE .. SELECT) INSERT ... ON DUPLICATE KEY UPDATE ? > diff --git a/mysql-test/suite/versioning/common.inc b/mysql-test/suite/versioning/common.inc > index 355b571e5a0..b35a5138015 100644 > --- a/mysql-test/suite/versioning/common.inc > +++ b/mysql-test/suite/versioning/common.inc > @@ -6,6 +6,7 @@ if (!$TEST_VERSIONING_SO) > source include/have_innodb.inc; > > set @@session.time_zone='+00:00'; > +set @@global.time_zone='+00:00'; Why is that? I'd expect your auto-adding of partitions to happen in the context of a session, using session time zone. Is it for replication? > select ifnull(max(transaction_id), 0) into @start_trx_id from mysql.transaction_registry; > set @test_start=now(6); > > diff --git a/mysql-test/suite/versioning/r/delete_history.result b/mysql-test/suite/versioning/r/delete_history.result > index cb865a835b3..90c9e4777bb 100644 > --- a/mysql-test/suite/versioning/r/delete_history.result > +++ b/mysql-test/suite/versioning/r/delete_history.result > @@ -63,8 +63,6 @@ insert into t values (1); > update t set a= 2; > update t set a= 3; > delete history from t; > -Warnings: > -Warning 4114 Versioned table `test`.`t`: last HISTORY partition (`p1`) is out of LIMIT, need more HISTORY partitions good riddance, it didn't make much sense anyway > # The above warning is one command late (MDEV-20345) ^^^ > select * from t for system_time all; > a > diff --git a/sql/ha_partition.h b/sql/ha_partition.h > index 85cb736b5bd..5fb893b4f0e 100644 > --- a/sql/ha_partition.h > +++ b/sql/ha_partition.h > @@ -1623,7 +1623,7 @@ class ha_partition :public handler could you also, please, fix (in a separate commit) ha_partition::part_records() to take a proper partition_element* argument, not void* ? > for (; part_id < part_id_end; ++part_id) > { > handler *file= m_file[part_id]; > - DBUG_ASSERT(bitmap_is_set(&(m_part_info->read_partitions), part_id)); > + bitmap_set_bit(&(m_part_info->read_partitions), part_id); > file->info(HA_STATUS_VARIABLE | HA_STATUS_NO_LOCK | HA_STATUS_OPEN); > part_recs+= file->stats.records; > } > diff --git a/sql/partition_info.cc b/sql/partition_info.cc > index e9dbf2b49c3..72312ee7ac4 100644 > --- a/sql/partition_info.cc > +++ b/sql/partition_info.cc > @@ -814,10 +814,16 @@ bool partition_info::has_unique_name(partition_element *element) > vers_info->interval Limit by fixed time interval > vers_info->hist_part (out) Working history partition > */ > -void partition_info::vers_set_hist_part(THD *thd) > +uint partition_info::vers_set_hist_part(THD *thd, bool auto_inc) let's not all it auto_inc. it'll add noise to every grep for auto_inc issues. What about auto_add ? > { > + DBUG_ASSERT(!thd->lex->last_table()->vers_conditions.delete_history); > + > + uint create_count= 0; > + auto_inc= auto_inc && vers_info->auto_inc; > + > if (vers_info->limit) > { > + DBUG_ASSERT(!vers_info->interval.is_set()); > ha_partition *hp= (ha_partition*)(table->file); > partition_element *next= NULL; > List_iterator<partition_element> it(partitions); > @@ -836,22 +842,26 @@ void partition_info::vers_set_hist_part(THD *thd) > { > if (next == vers_info->now_part) > { > - my_error(WARN_VERS_PART_FULL, MYF(ME_WARNING|ME_ERROR_LOG), > - table->s->db.str, table->s->table_name.str, > - vers_info->hist_part->partition_name, "LIMIT"); > + if (auto_inc) > + create_count= 1; > + else > + my_error(WARN_VERS_PART_FULL, MYF(ME_WARNING|ME_ERROR_LOG), > + table->s->db.str, table->s->table_name.str, > + vers_info->hist_part->partition_name, "LIMIT"); > } > else > vers_info->hist_part= next; > } > - return; > + // reserve at least one history partition > + if (auto_inc && create_count == 0 && > + vers_info->hist_part->id + 1 == vers_info->now_part->id) > + create_count= 1; Questionable. What does it solve? > } > - > - if (vers_info->interval.is_set()) > + else if (vers_info->interval.is_set() && > + vers_info->hist_part->range_value <= thd->query_start()) > { > - if (vers_info->hist_part->range_value > thd->query_start()) > - return; > - > partition_element *next= NULL; > + bool error= true; > List_iterator<partition_element> it(partitions); > while (next != vers_info->hist_part) > next= it++; > @@ -860,12 +870,200 @@ void partition_info::vers_set_hist_part(THD *thd) > { > vers_info->hist_part= next; > if (next->range_value > thd->query_start()) > - return; > + { > + error= false; > + break; but here you don't "reserve at least one history partition" Why INTERVAL is different? > + } > + } > + if (error) > + { > + if (auto_inc) > + { > + DBUG_ASSERT(thd->query_start() >= vers_info->hist_part->range_value); > + my_time_t diff= thd->query_start() - vers_info->hist_part->range_value; > + if (diff > 0) > + { > + size_t delta= vers_info->interval.seconds(); > + create_count= diff / delta + 1; > + if (diff % delta) > + create_count++; > + } > + else > + create_count= 1; > + } > + else > + { > + my_error(WARN_VERS_PART_FULL, MYF(ME_WARNING|ME_ERROR_LOG), > + table->s->db.str, table->s->table_name.str, > + vers_info->hist_part->partition_name, "INTERVAL"); > + } > } > - my_error(WARN_VERS_PART_FULL, MYF(ME_WARNING|ME_ERROR_LOG), > - table->s->db.str, table->s->table_name.str, > - vers_info->hist_part->partition_name, "INTERVAL"); > } > + > + return create_count; > +} > + > + > +/** > + @brief Run fast_alter_partition_table() to add new history partitions > + for tables requiring them. > +*/ > +void vers_add_auto_parts(THD *thd, TABLE_LIST* tl, uint num_parts) > +{ > + HA_CREATE_INFO create_info; > + Alter_info alter_info; > + String query; > + partition_info *save_part_info= thd->work_part_info; > + Query_tables_list save_query_tables; > + Reprepare_observer *save_reprepare_observer= thd->m_reprepare_observer; > + Diagnostics_area new_stmt_da(thd->query_id, false, true); > + Diagnostics_area *save_stmt_da= thd->get_stmt_da(); > + bool save_no_write_to_binlog= thd->lex->no_write_to_binlog; > + const CSET_STRING save_query= thd->query_string; > + thd->m_reprepare_observer= NULL; > + thd->lex->reset_n_backup_query_tables_list(&save_query_tables); > + thd->in_sub_stmt|= SUB_STMT_AUTO_HIST; > + thd->lex->no_write_to_binlog= !thd->is_current_stmt_binlog_format_row(); > + TABLE *table= tl->table; > + > + DBUG_ASSERT(!thd->is_error()); > + /* NB: we have to preserve m_affected_rows, m_row_count_func, m_last_insert_id, etc */ > + thd->set_stmt_da(&new_stmt_da); > + new_stmt_da.set_overwrite_status(true); Is it needed? You've just started using a new Diagnostics_area, the status should be DA_EMPTY here, nothing to overwrite. > + > + { > + DBUG_ASSERT(table->s->get_table_ref_type() == TABLE_REF_BASE_TABLE); > + DBUG_ASSERT(table->versioned()); > + DBUG_ASSERT(table->part_info); > + DBUG_ASSERT(table->part_info->vers_info); > + alter_info.reset(); > + alter_info.partition_flags= ALTER_PARTITION_ADD|ALTER_PARTITION_AUTO_HIST; > + create_info.init(); > + create_info.alter_info= &alter_info; > + Alter_table_ctx alter_ctx(thd, tl, 1, &table->s->db, &table->s->table_name); > + > + MDL_REQUEST_INIT(&tl->mdl_request, MDL_key::TABLE, tl->db.str, > + tl->table_name.str, MDL_SHARED_NO_WRITE, MDL_TRANSACTION); > + if (thd->mdl_context.acquire_lock(&tl->mdl_request, > + thd->variables.lock_wait_timeout)) > + goto exit; > + table->mdl_ticket= tl->mdl_request.ticket; > + > + create_info.db_type= table->s->db_type(); > + create_info.options|= HA_VERSIONED_TABLE; > + DBUG_ASSERT(create_info.db_type); > + > + create_info.vers_info.set_start(table->s->vers_start_field()->field_name); > + create_info.vers_info.set_end(table->s->vers_end_field()->field_name); > + > + partition_info *part_info= new partition_info(); > + if (unlikely(!part_info)) > + { > + my_error(ER_OUT_OF_RESOURCES, MYF(0)); > + goto exit; > + } > + part_info->use_default_num_partitions= false; > + part_info->use_default_num_subpartitions= false; > + part_info->num_parts= num_parts; > + part_info->num_subparts= table->part_info->num_subparts; > + part_info->subpart_type= table->part_info->subpart_type; > + if (unlikely(part_info->vers_init_info(thd))) > + { > + my_error(ER_OUT_OF_RESOURCES, MYF(0)); > + goto exit; > + } > + > + // NB: set_ok_status() requires DA_EMPTY > + thd->get_stmt_da()->reset_diagnostics_area(); Is it needed? You've just started using a new Diagnostics_area, the status should be DA_EMPTY here. > + > + thd->work_part_info= part_info; > + if (part_info->set_up_defaults_for_partitioning(thd, table->file, NULL, > + table->part_info->next_part_no(num_parts))) > + { > + push_warning(thd, Sql_condition::WARN_LEVEL_WARN, > + ER_VERS_HIST_PART_FAILED, > + "Auto-increment history partition: " > + "setting up defaults failed"); > + my_error(ER_VERS_HIST_PART_FAILED, MYF(ME_WARNING), > + tl->db.str, tl->table_name.str); > + goto exit; > + } > + bool partition_changed= false; > + bool fast_alter_partition= false; > + if (prep_alter_part_table(thd, table, &alter_info, &create_info, > + &partition_changed, &fast_alter_partition)) > + { > + push_warning(thd, Sql_condition::WARN_LEVEL_WARN, ER_VERS_HIST_PART_FAILED, > + "Auto-increment history partition: " > + "alter partitition prepare failed"); > + my_error(ER_VERS_HIST_PART_FAILED, MYF(ME_WARNING), > + tl->db.str, tl->table_name.str); > + goto exit; > + } > + if (!fast_alter_partition) > + { > + push_warning(thd, Sql_condition::WARN_LEVEL_WARN, ER_VERS_HIST_PART_FAILED, > + "Auto-increment history partition: " > + "fast alter partitition is not possible"); > + my_error(ER_VERS_HIST_PART_FAILED, MYF(ME_WARNING), > + tl->db.str, tl->table_name.str); > + goto exit; > + } > + DBUG_ASSERT(partition_changed); > + if (mysql_prepare_alter_table(thd, table, &create_info, &alter_info, > + &alter_ctx)) > + { > + push_warning(thd, Sql_condition::WARN_LEVEL_WARN, ER_VERS_HIST_PART_FAILED, > + "Auto-increment history partition: " > + "alter prepare failed"); > + my_error(ER_VERS_HIST_PART_FAILED, MYF(ME_WARNING), > + tl->db.str, tl->table_name.str); > + goto exit; > + } > + > + // Forge query string for rpl logging > + if (!thd->lex->no_write_to_binlog) > + { > + query.set(STRING_WITH_LEN("ALTER TABLE `"), &my_charset_latin1); better use StringBuffer<...> query; because you're doing lots of reallocs now. > + > + if (query.append(table->s->db) || > + query.append(STRING_WITH_LEN("`.`")) || > + query.append(table->s->table_name) || this is wrong, the table and db names can have backticks inside. use append_identifier() instead. > + query.append("` ADD PARTITION PARTITIONS ") || > + query.append_ulonglong(part_info->num_parts) || > + query.append(" AUTO")) > + { > + my_error(ER_OUT_OF_RESOURCES, MYF(ME_ERROR_LOG)); > + goto exit; > + } > + CSET_STRING qs(query.c_ptr(), query.length(), &my_charset_latin1); > + thd->set_query(qs); > + } Should it be binlogged at all? May be just leave it to the slave to auto-add the partition as needed? Looks a bit suspicious now, you log an ALTER TABLE possibly in the middle of a transaction. It will be replayed differently, not as auto-adding, in particular, it will commit. So, possibly different results on the slave, perhaps different gtid. Do clear it out with Andrei please. Or don't binlog auto-adding at all. > + > + if (fast_alter_partition_table(thd, table, &alter_info, &create_info, > + tl, &table->s->db, &table->s->table_name)) > + { > + push_warning(thd, Sql_condition::WARN_LEVEL_WARN, ER_VERS_HIST_PART_FAILED, > + "Auto-increment history partition: " > + "alter partition table failed"); > + my_error(ER_VERS_HIST_PART_FAILED, MYF(ME_WARNING), > + tl->db.str, tl->table_name.str); > + } > + } > + > + if (!thd->transaction.stmt.is_empty()) > + trans_commit_stmt(thd); 1. why would this operation register a transaction? 2. you amended a couple of checks like - if (thd->in_sub_stmt) + if (thd->in_sub_stmt & ~SUB_STMT_AUTO_HIST) but what will happen if it's, for example, an insert inside a trigger? So a real sub_stmt and SUB_STMT_AUTO_HIST? > + > +exit: > + thd->work_part_info= save_part_info; > + thd->m_reprepare_observer= save_reprepare_observer; > + thd->lex->restore_backup_query_tables_list(&save_query_tables); > + thd->in_sub_stmt&= ~SUB_STMT_AUTO_HIST; > + if (!new_stmt_da.is_warning_info_empty()) > + save_stmt_da->copy_sql_conditions_from_wi(thd, new_stmt_da.get_warning_info()); > + thd->set_stmt_da(save_stmt_da); > + thd->lex->no_write_to_binlog= save_no_write_to_binlog; > + thd->set_query(save_query); > } > > > diff --git a/sql/partition_info.h b/sql/partition_info.h > index 7ae2d168068..ca68e61cbe2 100644 > --- a/sql/partition_info.h > +++ b/sql/partition_info.h > @@ -72,9 +73,34 @@ struct Vers_part_info : public Sql_alloc > my_time_t start; > INTERVAL step; > enum interval_type type; > - bool is_set() { return type < INTERVAL_LAST; } > + bool is_set() const { return type < INTERVAL_LAST; } > + size_t seconds() const > + { > + if (step.second) > + return step.second; > + if (step.minute) > + return step.minute * 60; > + if (step.hour) > + return step.hour * 3600; > + if (step.day) > + return step.day * 3600 * 24; > + // comparison is used in rough estimates, it doesn't need to be calendar-correct Are you sure the approximate value is enough here? You use it to estimate how many partitions to create. It's not a big deal if you create more than necessary (although it's not nice and we'll definitely get bug reports if that will happen). But you surely don't want to create less. > + if (step.month) > + return step.month * 3600 * 24 * 30; > + DBUG_ASSERT(step.year); > + return step.year * 86400 * 30 * 365; > + } > + bool lt(size_t secs) const > + { > + return seconds() < secs; > + } > + bool ge(size_t seconds) const > + { > + return !(this->lt(seconds)); > + } lt() and ge() don't seem to be used anywhere. > } interval; > ulonglong limit; > + bool auto_inc; > partition_element *now_part; > partition_element *hist_part; > }; > diff --git a/sql/sql_base.cc b/sql/sql_base.cc > index b89be77f282..5583d70f8eb 100644 > --- a/sql/sql_base.cc > +++ b/sql/sql_base.cc > @@ -1862,6 +1862,25 @@ bool open_table(THD *thd, TABLE_LIST *table_list, Open_table_context *ot_ctx) > DBUG_PRINT("info",("Using locked table")); > #ifdef WITH_PARTITION_STORAGE_ENGINE > part_names_error= set_partitions_as_used(table_list, table); > + if (table->part_info && > + table->part_info->part_type == VERSIONING_PARTITION && > + !table_list->vers_conditions.delete_history && > + table_list->lock_type >= TL_WRITE_ALLOW_WRITE && > + table_list->mdl_request.type == MDL_SHARED_WRITE) > + { > + switch (thd->lex->sql_command) > + { > + case SQLCOM_DELETE: > + case SQLCOM_UPDATE: > + case SQLCOM_REPLACE: > + case SQLCOM_REPLACE_SELECT: > + case SQLCOM_DELETE_MULTI: > + case SQLCOM_UPDATE_MULTI: > + /* Rotation is still needed under LOCK TABLES */ > + table->part_info->vers_set_hist_part(thd, false); ALTER TABLE works under LOCK TABLES, so this should be possible too, but let's have it as a separate task. > + default:; > + } > + } > #endif > goto reset; > } > @@ -2104,6 +2123,37 @@ bool open_table(THD *thd, TABLE_LIST *table_list, Open_table_context *ot_ctx) > tc_add_table(thd, table); > } > > +#ifdef WITH_PARTITION_STORAGE_ENGINE > + if (table->part_info && > + table->part_info->part_type == VERSIONING_PARTITION && > + !table_list->vers_conditions.delete_history && > + table_list->lock_type >= TL_WRITE_ALLOW_WRITE && > + table_list->mdl_request.type >= MDL_SHARED_WRITE && > + table_list->mdl_request.type < MDL_EXCLUSIVE) > + { > + switch (thd->lex->sql_command) > + { > + case SQLCOM_LOCK_TABLES: > + case SQLCOM_DELETE: > + case SQLCOM_UPDATE: > + case SQLCOM_REPLACE: > + case SQLCOM_REPLACE_SELECT: > + case SQLCOM_DELETE_MULTI: > + case SQLCOM_UPDATE_MULTI: this is quite complex condition, better not to duplicate it. May be something like bool need_set_hist_part(TABLE_LIST *table_list, enum_sql_command sql_command) { if (table_list->table->part_info && table_list->table->part_info->part_type == VERSIONING_PARTITION && !table_list->vers_conditions.delete_history && table_list->lock_type >= TL_WRITE_ALLOW_WRITE && table_list->mdl_request.type >= MDL_SHARED_WRITE && table_list->mdl_request.type < MDL_EXCLUSIVE) { switch(sql_command) { case SQLCOM_LOCK_TABLES: case SQLCOM_DELETE: case SQLCOM_UPDATE: case SQLCOM_REPLACE: case SQLCOM_REPLACE_SELECT: case SQLCOM_DELETE_MULTI: case SQLCOM_UPDATE_MULTI: return true; } default:; } return false; } > + ot_ctx->vers_create_count= table->part_info->vers_set_hist_part(thd, true); > + if (ot_ctx->vers_create_count) > + { > + ot_ctx->request_backoff_action(Open_table_context::OT_ADD_HISTORY_PARTITION, > + table_list); > + MYSQL_UNBIND_TABLE(table->file); > + tc_release_table(table); There's no MYSQL_UNBIND_TABLE/tc_release_table after other request_backoff_action invocations. Why this one is special? > + DBUG_RETURN(TRUE); > + } > + default:; > + } > + } > +#endif > + > if (!(flags & MYSQL_OPEN_HAS_MDL_LOCK) && > table->s->table_category < TABLE_CATEGORY_INFORMATION) > { > diff --git a/sql/sql_class.h b/sql/sql_class.h > index 14516811262..145ac5c5f64 100644 > --- a/sql/sql_class.h > +++ b/sql/sql_class.h > @@ -3346,6 +3347,10 @@ class THD: public THD_count, /* this must be first */ > > #ifdef WITH_PARTITION_STORAGE_ENGINE > partition_info *work_part_info; > + /** > + List of tables requiring new history partition. > + */ > + List<TABLE_SHARE> vers_auto_part_tables; this doesn't seem to be used anywhere > #endif > > #ifndef EMBEDDED_LIBRARY > diff --git a/sql/sql_yacc.yy b/sql/sql_yacc.yy > index 7cc1faea79b..3f6c1793432 100644 > --- a/sql/sql_yacc.yy > +++ b/sql/sql_yacc.yy > @@ -7521,14 +7529,17 @@ add_partition_rule: > > add_part_extra: > /* empty */ > - | '(' part_def_list ')' > + | '(' part_def_list ')' opt_vers_auto_inc > { > - LEX *lex= Lex; > - lex->part_info->num_parts= lex->part_info->partitions.elements; > + Lex->part_info->num_parts= Lex->part_info->partitions.elements; > + if ($4) > + Lex->alter_info.partition_flags|= ALTER_PARTITION_AUTO_HIST; > } > - | PARTITIONS_SYM real_ulong_num > + | PARTITIONS_SYM real_ulong_num opt_vers_auto_inc > { > Lex->part_info->num_parts= $2; > + if ($3) > + Lex->alter_info.partition_flags|= ALTER_PARTITION_AUTO_HIST; I'm confused. I thought that ALTER_PARTITION_AUTO_HIST is what you set in vers_add_auto_parts() to mark auto-adding of a new partition. But here you set it in the parser, when a user specifies AUTO in partition specifications. So it seems you're using this flag for two very different purposes. How do you distinguish between them? And why did you decide to do it this way? > } > ; > > diff --git a/sql/sql_partition.cc b/sql/sql_partition.cc > index fbed614489d..25017ee5425 100644 > --- a/sql/sql_partition.cc > +++ b/sql/sql_partition.cc > @@ -4825,6 +4829,13 @@ uint prep_alter_part_table(THD *thd, TABLE *table, Alter_info *alter_info, > DBUG_RETURN(TRUE); > } > > + if (alter_info->partition_flags & ALTER_PARTITION_AUTO_HIST && > + (!table->part_info || !table->part_info->vers_info)) > + { > + my_error(ER_SYNTAX_ERROR, MYF(0)); can it even happen? Or should it be DBUG_ASSERT(0) here? > + DBUG_RETURN(TRUE); > + } > + > partition_info *alt_part_info= thd->lex->part_info; > /* > This variable is TRUE in very special case when we add only DEFAULT > @@ -5312,7 +5323,9 @@ that are reorganised. > now_part= el; > } > } > - if (*fast_alter_table && tab_part_info->vers_info->interval.is_set()) > + if (*fast_alter_table && > + !(alter_info->partition_flags & ALTER_PARTITION_AUTO_HIST) && > + tab_part_info->vers_info->interval.is_set()) this !ALTER_PARTITION_AUTO_HIST - do you mean not vers_add_auto_parts() or not sql statement that uses AUTO ? > { > partition_element *hist_part= tab_part_info->vers_info->hist_part; > if (hist_part->range_value <= thd->query_start()) > @@ -5347,7 +5360,8 @@ that are reorganised. > */ > if (!(alter_info->partition_flags & ALTER_PARTITION_TABLE_REORG)) > { > - if (!alt_part_info->use_default_partitions) > + if (!alt_part_info->use_default_partitions && > + !(alter_info->partition_flags & ALTER_PARTITION_AUTO_HIST)) Sorry, I don't understand that if() at all. What does it do? > { > DBUG_PRINT("info", ("part_info: %p", tab_part_info)); > tab_part_info->use_default_partitions= FALSE; Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 4

Re: [Maria-developers] f8901333a82: Add support for minimum field width for strings to my_vsnprintf()
by Sergei Golubchik 31 Mar '21

31 Mar '21

Hi, Michael! On Mar 29, Michael Widenius wrote: > revision-id: f8901333a82 (mariadb-10.5.2-534-gf8901333a82) > parent(s): a194c5cfd41 > author: Michael Widenius <michael.widenius(a)gmail.com> > committer: Michael Widenius <michael.widenius(a)gmail.com> > timestamp: 2021-03-24 15:53:35 +0200 > message: > > Add support for minimum field width for strings to my_vsnprintf() > > diff --git a/mysql-test/main/almost_full.result b/mysql-test/main/almost_full.result > index b2d7092aa51..4b5c8417412 100644 > --- a/mysql-test/main/almost_full.result > +++ b/mysql-test/main/almost_full.result > @@ -7,24 +7,24 @@ INSERT INTO t1 SET b=repeat('a',600); > ERROR HY000: The table 't1' is full > CHECK TABLE t1 EXTENDED; > Table Op Msg_type Msg_text > -test.t1 check warning Datafile is almost full, 65448 of 65534 used > +test.t1 check warning Datafile is almost full, 65448 of 65534 used This happens because mi_check.c has mi_check_print_warning(param, "Datafile is almost full, %10s of %10s used", ... could you please remove explicit width specification from these messages? this looks rather ugly in warnings. This applies both to MyISAM and Aria. > test.t1 check status OK > UPDATE t1 SET b=repeat('a', 800) where a=10; > ERROR HY000: The table 't1' is full > diff --git a/mysql-test/main/grant4.result b/mysql-test/main/grant4.result > index 981bbdeddf5..ba64010f1bb 100644 > --- a/mysql-test/main/grant4.result > +++ b/mysql-test/main/grant4.result > @@ -187,8 +187,8 @@ connection con1; > check table mysqltest_db1.t1; > Table Op Msg_type Msg_text > mysqltest_db1.t1 check warning 1 client is using or hasn't closed the table properly > -mysqltest_db1.t1 check error Size of indexfile is: 1024 Should be: 2048 > -mysqltest_db1.t1 check warning Size of datafile is: 14 Should be: 7 > +mysqltest_db1.t1 check error Size of indexfile is: 1024 Should be: 2048 > +mysqltest_db1.t1 check warning Size of datafile is: 14 Should be: 7 Same here. > mysqltest_db1.t1 check error Corrupt > # The below statement should fail before repairing t1. > # Otherwise info about such repair will be missing from its result-set. > diff --git a/mysql-test/main/myisam_recover.result b/mysql-test/main/myisam_recover.result > index da96682186c..d3c385bc49b 100644 > --- a/mysql-test/main/myisam_recover.result > +++ b/mysql-test/main/myisam_recover.result > @@ -69,9 +69,9 @@ flush table t1; > # check table is needed to mark the table as crashed. > check table t1; > Table Op Msg_type Msg_text > -test.t1 check warning Size of datafile is: 42 Should be: 21 > -test.t1 check error Record-count is not ok; is 6 Should be: 3 > -test.t1 check warning Found 6 key parts. Should be: 3 > +test.t1 check warning Size of datafile is: 42 Should be: 21 > +test.t1 check error Record-count is not ok; is 6 Should be: 3 > +test.t1 check warning Found 6 key parts. Should be: 3 Same. Must be in many messages in mi_check.c and ma_check.c > test.t1 check error Corrupt > # > diff --git a/strings/my_vsnprintf.c b/strings/my_vsnprintf.c > index a2e3f9b738d..1621db2e412 100644 > --- a/strings/my_vsnprintf.c > +++ b/strings/my_vsnprintf.c > @@ -224,12 +224,27 @@ static char *backtick_string(CHARSET_INFO *cs, char *to, const char *end, > */ > > static char *process_str_arg(CHARSET_INFO *cs, char *to, const char *end, > - size_t width, char *par, uint print_type, > - my_bool nice_cut) > + longlong length_arg, size_t width, char *par, > + uint print_type, my_bool nice_cut) > { > int well_formed_error; > uint dots= 0; > size_t plen, left_len= (size_t) (end - to) + 1, slen=0; > + my_bool left_fill= 1; > + size_t length; > + > + /* > + The sign of the length argument specific the string should be right > + or left adjusted > + */ > + if (length_arg < 0) You're trying to support left alignment, but you have yourself added a test that shows that it doesn't work. So this code is dead, never used. See below. > + { > + length= -length_arg; > + left_fill= 0; > + } > + else > + length= (size_t) length_arg; > + > if (!par) > par = (char*) "(null)"; > > @@ -693,7 +720,8 @@ size_t my_vsnprintf_ex(CHARSET_INFO *cs, char *to, size_t n, > if (*fmt == 's' || *fmt == 'T') /* String parameter */ > { > reg2 char *par= va_arg(ap, char *); > - to= process_str_arg(cs, to, end, width, par, print_type, (*fmt == 'T')); > + to= process_str_arg(cs, to, end, (longlong) length, width, par, see? length is unsigned type size_t, the sign before length is ignored by the parser. Whether you cast length to a signed type or not and whether you checks for length < 0 or not, left alignment still won't work. That's why you have added this test: > + test1("Negative width is ignored for strings < x> < y>", > + "Negative width is ignored for strings <%-4s> <%-5s>", "x", "y"); Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 2

Re: [Maria-developers] 1bbc852ef71: Changed field_index to use field_index_t instead of uint16
by Sergei Golubchik 31 Mar '21

31 Mar '21

Hi, Michael! On Mar 29, Michael Widenius wrote: > revision-id: 1bbc852ef71 (mariadb-10.5.2-525-g1bbc852ef71) > parent(s): 039f4a054bb > author: Michael Widenius <michael.widenius(a)gmail.com> > committer: Michael Widenius <michael.widenius(a)gmail.com> > timestamp: 2021-03-24 14:31:54 +0200 > message: > > Changed field_index to use field_index_t instead of uint16 Just curious. Why? > diff --git a/sql/unireg.cc b/sql/unireg.cc > index 51dd1fb6e8c..f1a0c5c5269 100644 > --- a/sql/unireg.cc > +++ b/sql/unireg.cc > @@ -146,8 +146,8 @@ get_fieldno_by_name(HA_CREATE_INFO *create_info, List<Create_field> &create_fiel > { > if (field_name.streq(sql_field->field_name)) > { > - DBUG_ASSERT(field_no <= uint16(~0U)); > + DBUG_ASSERT(field_no < NO_CACHED_FIELD_INDEX); this look suspicios. NO_CACHED_FIELD_INDEX is ((uint)(-1)) which is always greater than anything you can put into field_index_t. You want to ensure that field_no is valid when casted into field_index_t. I'd suggest to redefine NO_CACHED_FIELD_INDEX as ((field_index_t)~0U) > - return uint16(field_no); > + return (field_index_t) field_no; why not to declare field_no to be field_index_t? then you won't need a cast. > } > } > > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] 1d577194654: Revert MDEV-16592 "Change Item::with_sum_func to a virtual method"
by Sergei Golubchik 31 Mar '21

31 Mar '21

Hi, Michael! On Mar 29, Michael Widenius wrote: > revision-id: 1d577194654 (mariadb-10.5.2-526-g1d577194654) > parent(s): 1bbc852ef71 > author: Michael Widenius <michael.widenius(a)gmail.com> > committer: Michael Widenius <michael.widenius(a)gmail.com> > timestamp: 2021-03-24 15:03:04 +0200 > message: > > Revert MDEV-16592 "Change Item::with_sum_func to a virtual method" > > diff --git a/sql/item.cc b/sql/item.cc > index e5be8699d80..e0ba4ff2c89 100644 > --- a/sql/item.cc > +++ b/sql/item.cc > @@ -975,9 +975,7 @@ class Item :public Value_source, > void share_name_with(const Item *item) > { > name= item->name; > - common_flags= static_cast<uint8> > - ((common_flags & ~IS_AUTO_GENERATED_NAME) | > - (item->common_flags & IS_AUTO_GENERATED_NAME)); > + is_autogenerated_name= item->is_autogenerated_name; this, apparently, should be in the commit 039f4a054bb "Removed Item::common_flags and replaced it with bit fields" > } Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] 25fe7dbad5d: Reduce usage of strlen()
by Sergei Golubchik 31 Mar '21

31 Mar '21

Hi, Michael! On Mar 31, Michael Widenius wrote: > commit 25fe7dbad5d > Author: Michael Widenius <michael.widenius(a)gmail.com> > Date: Wed Aug 12 20:29:55 2020 +0300 > > Reduce usage of strlen() I'll only mention actual mistakes, no style comments or suggestions or question, just want to get it over with. > diff --git a/sql/item.h b/sql/item.h > index 9a6d0b4abfe..d6ff10bf3ba 100644 > --- a/sql/item.h > +++ b/sql/item.h > @@ -1093,7 +1093,7 @@ class Item :public Value_source, > void share_name_with(const Item *item) > { > name= item->name; > - copy_flags(item, item_base_t::IS_AUTOGENERATED_NAME); > + copy_flags(item, item_base_t::IS_EXPLICIT_NAME); belongs to a previous commit > } > virtual void cleanup(); > virtual void make_send_field(THD *thd, Send_field *field); > diff --git a/sql/item_windowfunc.cc b/sql/item_windowfunc.cc > index 262481d019b..a9e008b0b92 100644 > --- a/sql/item_windowfunc.cc > +++ b/sql/item_windowfunc.cc > @@ -99,13 +99,15 @@ Item_window_func::fix_fields(THD *thd, Item **ref) > > if (window_spec->window_frame && is_frame_prohibited()) > { > - my_error(ER_NOT_ALLOWED_WINDOW_FRAME, MYF(0), window_func()->func_name()); > + my_error(ER_NOT_ALLOWED_WINDOW_FRAME, MYF(0), > + window_func()->func_name()); This should be func_name().str, otherwise it'll crash or print garbage on Windows. Same below, in a couple of places. > return true; > } Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] dbb62d20f36: Fixes that enables my_new.cc (new wrapper using my_malloc)
by Sergei Golubchik 29 Mar '21

29 Mar '21

Hi, Michael! On Mar 28, Michael Widenius wrote: > revision-id: dbb62d20f36 (mariadb-10.5.2-518-gdbb62d20f36) > parent(s): 0c37211c2a2 > author: Michael Widenius <michael.widenius(a)gmail.com> > committer: Michael Widenius <michael.widenius(a)gmail.com> > timestamp: 2021-03-24 14:31:53 +0200 > message: > > Fixes that enables my_new.cc (new wrapper using my_malloc) > > This is not enabled by default as there are leaks in the > server that needs to be fixed first. One can compile > with -DREALLY_USE_MYSYS_NEW -DSF_REMEMBER_FRAMES=14 to find the > memory leaks from 'new' The change in my_new.cc is ok. The change in my_global.h and CMakeLists.txt is confusing and redundant. You unconditionaly define USE_MYSYS_NEW then conditionally undefine it again? Everything already worked just fine. By default cxx new is used, if one wants to use mysys version, it's done with -DHAVE_CXX_NEW=0 this works for me in 10.5 and fails to compile my_new.cc, as it should. Please, only keed your my_new.cc changes and say in the commit message "One can compile with -DHAVE_CXX_NEW=0 to find memory leaks from 'new'" As for -DSF_REMEMBER_FRAMES=14, perhaps it's better to do it automatically? Like #ifdef USE_MYSYS_NEW #define SF_REMEMBER_FRAMES 14 #else #define SF_REMEMBER_FRAMES 8 #endif or may be even just always make it 14. It should be easy to enable this feature, right? > diff --git a/CMakeLists.txt b/CMakeLists.txt > index a0b540f12d2..c51fb98abdc 100644 > --- a/CMakeLists.txt > +++ b/CMakeLists.txt > @@ -366,10 +366,8 @@ ENDIF() > # Run platform tests > INCLUDE(configure.cmake) > > -# force -DUSE_MYSYS_NEW unless already done by HAVE_CXX_NEW > -IF(NOT HAVE_CXX_NEW) > - ADD_DEFINITIONS(-DUSE_MYSYS_NEW) > -ENDIF() > +# Always use USE_MYSYS_NEW > +ADD_DEFINITIONS(-DUSE_MYSYS_NEW) > > # Find header files from the bundled libraries > # (wolfssl, readline, pcre2, etc) > diff --git a/include/my_global.h b/include/my_global.h > index e999e555bf7..6e3b48a00fd 100644 > --- a/include/my_global.h > +++ b/include/my_global.h > @@ -487,6 +487,9 @@ typedef unsigned short ushort; > duplicate declaration of __cxa_pure_virtual, solved by declaring it a > weak symbol. > */ > +#ifndef REALLY_USE_MYSYS_NEW > +#undef USE_MYSYS_NEW > +#endif > #if defined(USE_MYSYS_NEW) && ! defined(DONT_DECLARE_CXA_PURE_VIRTUAL) > C_MODE_START > int __cxa_pure_virtual () __attribute__ ((weak)); > diff --git a/mysys/my_static.c b/mysys/my_static.c > index 3c4b9efc1f8..96a23cdb283 100644 > --- a/mysys/my_static.c > +++ b/mysys/my_static.c > @@ -48,6 +48,7 @@ PSI_memory_key key_memory_my_err_head; > PSI_memory_key key_memory_my_file_info; > PSI_memory_key key_memory_pack_frm; > PSI_memory_key key_memory_charsets; > +PSI_memory_key key_memory_new; where do you initialize it? I'd suggest statically, like PSI_memory_key key_memory_new= PSI_INSTRUMENT_MEM; > > #ifdef _WIN32 > PSI_memory_key key_memory_win_SECURITY_ATTRIBUTES; Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 4

Re: [Maria-developers] 0c37211c2a2: Fixed my_addr_resolve
by Sergei Golubchik 28 Mar '21

28 Mar '21

Hi, Michael! On Mar 28, Michael Widenius wrote: > revision-id: 0c37211c2a2 (mariadb-10.5.2-517-g0c37211c2a2) > parent(s): 555c4470130 > author: Michael Widenius <michael.widenius(a)gmail.com> > committer: Michael Widenius <michael.widenius(a)gmail.com> > timestamp: 2021-03-24 14:31:53 +0200 > message: > > Fixed my_addr_resolve > > At least on openSUSE Leap 15.1 one should not use the dli_fbase offset > provided by dladdr() for addresses in the main program. > Without this patch the stack traces from sf_report_leaked_memory() > are not usable. This is very strange. First, the whole purpose of this code is to use dli_fbase offset for the main program. That's why it was added in the first place. PIE binaries can be loaded at any address in memory and the address you see with addr2line or nm or readelf is not the same address you'll see in gdb or in a strack trace. We have to subtract the base address first. But may be openSUSE is different? So, I built on openSUSE Leap and the stack trace was perfectly usable without your patch. I tried both `kill -ABRT` and an artificially introduced memory leak. So I suspect there was some misunderstanding and the problem was somewhere else (I cannot know what, of course, but just as an example - the build was bad and recompilation fixed it). Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 2

Re: [Maria-developers] ee538938345: MDEV-21117: refine the server binlog-based recovery for semisync
by Sergei Golubchik 24 Mar '21

24 Mar '21

Hi, Andrei! On Mar 24, Andrei Elkin wrote: > >> Here it goes. > >> > >> As after master crash and failover to slave, the demoted-to-slave > >> ex-master must be ready to face and accept its own (generated by) > >> events, without generally necessary --replicate-same-server-id. > >> > >> So the acceptance conditions are refined/relaxed for the semisync > >> slave connected to master in the GTID mode which ensures (under > >> gtid_strict_mode ON) that there can't be any duplicate events for > >> execution on such an ex-master slave. Non-GTID "binlog-truncated" > >> ex-master semisync slave has to be configured with > >> --replicate-same-server-id. > > > > Why would you generally care about same server id in the gtid mode? > > Maybe you did not care in above to the 'connection' word in 'the gtid > connection mode' which is about > > Change-Master to ... master_use_gtid= slave_pos > > So the "legacy" pre-gtid one corresponds to master_use_gtid= no. > [Thinking into future, it may stay for some more time actually to > serve for manual recovery of the slave.] As discussed on slack, let's now only support fully automatic recovery for the case of master_use_gtid=slave_pos. If one uses master_use_gtid=none the failover is non-trivial anyway. We can always add flipping of replicate-same-server-id later, if needed. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] ee538938345: MDEV-21117: refine the server binlog-based recovery for semisync
by Sergei Golubchik 23 Mar '21

23 Mar '21

Hi, Andrei! On Mar 22, Andrei Elkin wrote: > > Here it goes. > > As after master crash and failover to slave, the demoted-to-slave > ex-master must be ready to face and accept its own (generated by) > events, without generally necessary --replicate-same-server-id. > > So the acceptance conditions are refined/relaxed for the semisync slave > connected to master in the GTID mode which ensures (under gtid_strict_mode ON) > that there can't be any duplicate events for execution > on such an ex-master slave. > Non-GTID "binlog-truncated" ex-master semisync slave has to be > configured with --replicate-same-server-id. Why would you generally care about same server id in the gtid mode? If gtid is not present locally, then it should be applied, right? Why to look at server id at all? It's a thing of the past, from pre-gtid times. > >> >> +--let $query1 = INSERT INTO t VALUES (20) > >> >> +--let $query2 = DELETE FROM t2 WHERE f = 666 /* no such record */ > >> >> +--source binlog_truncate_active_log.inc > >> >> + > >> >> +--echo # Case B. > >> >> +# The inverted sequence ends up to truncate only $query2 > >> >> +--let $this_search_pattern = Successfully truncated.*to remove transactions starting from GTID 0-1-10 > >> >> +--let $query1 = DELETE FROM t2 WHERE f = 0 > >> > > >> > why not `= 666 /* no such record */` ? > >> > is `= 0` important here? > >> > >> Yes, there's a fine detail here. > >> > >> 'The inverted sequence ends up' to leave in binlog > >> > >> $query1 = DELETE FROM t2 WHERE f = 0 > >> > >> which is as ineffective as 666. The patch recognizes the ineffectiveness > >> through a special value of engines involved counter in Gtid event. > >> In principle the truncation could > >> start from it rather than the following $query2. > >> But that would mean more complicated coding. > >> So the semantics of the zero is to be good, as we're so used to :-)! > > > > Sorry, I didn't understand anything. What does the value in the WHERE > > clause has to do with engines and where the binlog is truncated? ok. after discussing the issue on slack, the answer appears to be "the value in WHERE has nothing to do with any of that, it doesn't matter whether it's 0 or 666 as long as it doesn't match any rows" then, I'd say, better change 0 to 667 in the test. We wouldn't be having this discussion at all if you'd have some number that appeared arbitrary (and "667" is fine). My confusion was exactly caused by the fact that it looked like =0 matters, that it was important to hav 0 there. > >> >> +--let $query2 = INSERT INTO t VALUES (20) > >> >> +--source binlog_truncate_active_log.inc > > ... > >> >> diff --git a/sql/sql_class.h b/sql/sql_class.h > >> >> index 140394fefc1..87fa88d4c89 100644 > >> >> --- a/sql/sql_class.h > >> >> +++ b/sql/sql_class.h > >> >> @@ -4632,7 +4632,7 @@ class THD :public Statement, > >> >> LF_PINS *tdc_hash_pins; > >> >> LF_PINS *xid_hash_pins; > >> >> bool fix_xid_hash_pins(); > >> >> - > >> >> + bool is_1pc_ro_trans; > >> > > >> > I don't like how it looks, it's in THD (for every single connection > >> > and more) and it's only used in some exotic cases. It's set in two > >> > places around ha_commit_one_phase(), when it would be better to set > >> > it inside ha_commit_one_phase(), in one place only. > >> > > >> > But I cannot suggest how to get rid of it yet as I don't understand > >> > what it's for, I've asked a question about it below, in > >> > Gtid_log_event::Gtid_log_event > >> > >> The flag is to help sorting out ineffective (no engine data change) > >> STATEMENT binlog-format logged transaction. I thought it's practical > >> enough to face and important to handle. > >> Through an example, > >> let the table `t' be initially empty, and > >> let binlog contain > >> > >> trx1: INSERT INTO t SET a = 1; /* slave acked, trx got committed in engine */ > >> trx2: INSERT INTO t SET a = 2; /* not acked, trx remains prepared in engine */ > >> trx3: DELETE FROM t WHERE a = 0; /* ineffective, does not have Xid event */ > >> > >> prior a crash with their states as commented. > >> The truncate trx should be trx2 as the trx3's status > >> suggests it's harmless for involving into > >> truncation. > >> However we have yet to distinguish it from unsafe (to truncate) > >> event group ("transaction") that also ends with COMMIT query not Xid event. > > > > Hmm. I see. But thd isn't a universal dumpster for passing > > arguments to the function, don't use it like that. > > Well, I had to resort to that. To explain, > > > Please, pass this "read-only transaction with no xid" flag to the > > Gtid_log_event constructor as an argument. Or recalculate it inside > > the constructor, it's not such an expensive thing to do. > > To pass a flag ensues adding an extra arg to > struct handlerton::commit > which I don't feel easy about considering the reason. yes, one more reason to remove binlog_hton. But for now, like with binlog_commit_by_xid, I'd say, call binlog_commit directly and make binlog_hton->commit a no-op. Like: --- a/sql/handler.cc +++ b/sql/handler.cc @@ -1935,6 +1935,7 @@ commit_one_phase_2(THD *thd, bool all, THD_TRANS *trans, if (ha_info) { + binlog_commit(ht, thd, all); for (; ha_info; ha_info= ha_info_next) { int err; > >> >> + if (!truncate_validated) > >> >> + { > >> >> + DBUG_ASSERT(round <= 2); > >> >> + /* > >> >> + Either truncate was not set or was reset, else > >> >> + it gets incremented, otherwise it may only set to an earlier > >> >> + offset only at the turn out of the 1st round. > >> > > >> > sorry, I cannot parse that :( > >> > >> The comments are for the assert in a branch that acts when the truncate position > >> (gtid) is not yet finalized. > >> Let me annotate under each assert's line. > >> > >> > > >> >> + */ > >> >> + DBUG_ASSERT(truncate_gtid.seq_no == 0 || > >> > >> 'truncate was not set or was reset' ^, that is truncate_gtid. > >> The 'was reset' is actually not present in the assert - that'd be > >> reflected with > >> truncate_reset_done == true > >> > >> > >> >> + last_gtid_coord >= binlog_truncate_coord || > >> > >> 'it gets incremented' ^ > > > > why does that mean "it gets incremented" ? > > I think my English failed me. > > It is getting incremented I understand that "it gets incremented" means "it = it + 1", no problem. How this assert verifies that "it gets incremented" ? > meant to say (in my last reply as well). > The use case is that at this point there's a truncate candidate /* 1 */ > and it has not been settled yet /* 2 */, > > !truncate_validated == true /* 2 */ && truncate_gtid.seq_no > 0 /* 1 */ > > and the current gtid is still "better" to take over as the candidate in > this block. The candidate's offset must not decrease (unless... > > >> >> + (round == 2 && truncate_set_in_1st)); > >> > >> And if it's neither of the above there must've been truncate candidate > >> set in the 1st round. > > > > ... > ... the above ). > > >> >> switch (typ) > >> >> { > >> >> + case FORMAT_DESCRIPTION_EVENT: > >> >> + if (round > 1) > >> >> + { > >> >> + if (fdle != fdle_arg) > >> >> + delete fdle; > >> >> + fdle= (Format_description_log_event*) ev; > >> > > >> > why is it needed? all binlog files should have the same > >> > Format_description_log_event anyway. > >> > >> Actually contrary to what was stated (the patch removed that claim) in > >> the parent revision of this commit, the FD:s of the > >> recovery binlog files may have different checksums. I.e > >> > >> set @@global.binlog_checksum = value > >> > >> can be issued to rotate binlog to a new hot file having different checksum value > >> while the binlog *CheckPoint* file is behind. So this is possible to > >> have at recovery: > >> > >> B_1(CP, checksum = NONE), B_2(hot, checksum = CRC32) > >> > >> which also manifests a normal use case bug I have not reported any bug (though wanted to). > > > > okay, checksum can differ. But how does it affect > > Format_description_log_event ? > > FD holds knowledge of whether events in the file are with the checksum > info. In my example above B_1 events would be (are) considered as checksummed > by the base. Okay. Please, 1. add a comment about it 2. add a test case for it ideally, also, move it into a separate commit Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] f1db957c5da: MDEV-21469: Implement crash-safe binary logging of the user XA
by Sergei Golubchik 22 Mar '21

22 Mar '21

Hi, Andrei! On Mar 16, Andrei Elkin wrote: > > On Mar 14, Sujatha wrote: > >> revision-id: f1db957c5da (mariadb-10.5.2-247-gf1db957c5da) > >> parent(s): dc92235f21e > >> author: Sujatha <sujatha.sivakumar(a)mariadb.com> > >> committer: Andrei Elkin <andrei.elkin(a)mariadb.com> > >> timestamp: 2020-12-21 16:10:46 +0200 > >> message: > >> > >> MDEV-21469: Implement crash-safe binary logging of the user XA ... > >> diff --git a/sql/log.cc b/sql/log.cc > >> index 731bb3e98f0..c69b8518cf4 100644 > >> --- a/sql/log.cc > >> +++ b/sql/log.cc > >> @@ -2019,28 +2027,49 @@ static int binlog_xa_recover_dummy(handlerton *hton __attribute__((unused)), > >> return 0; > >> } > >> > >> - > >> +/* > >> + The function invokes binlog_commit() and returns its result > >> + when it has not yet called it already. > >> + binlog_cache_mngr::completed_by_xid remembers the fact of > >> + the 1st of maximum two subsequent calls. > >> +*/ > >> static int binlog_commit_by_xid(handlerton *hton, XID *xid) > >> { > >> + int rc= 0; > >> THD *thd= current_thd; > >> - > >> - (void) thd->binlog_setup_trx_data(); > >> + binlog_cache_mngr *cache_mngr= thd->binlog_setup_trx_data(); > >> > >> DBUG_ASSERT(thd->lex->sql_command == SQLCOM_XA_COMMIT); > >> > >> - return binlog_commit(hton, thd, TRUE); > >> -} > >> + if (!cache_mngr->completed_by_xid) > > > > On what code path can you have completed_by_xid == false here? > > Let me answer broadly, the purpose of `completed_by_xid' is to > make sure binlog_hton->commit_by_xid() does not invoke lower level > binlog_commit() for the 2nd time at the following function. I understand that. And I mean that if commit_by_xid is always true, then binlog_hton->commit_by_xid can be a noop, because you always invoke binlog_commit_by_xid (or binlog_rollback_by_xid) explicitly. > I leave only two essential statements prepared with explanatory comments: > > ha_commit_or_rollback_by_xid() > { > // binlog_commit() has to be called as first hton ... > > binlog_hton->commit_by_xid(binlog_hton, xid); > > // ... So the above raises the flag (sets up a fence) to block binlog_hton > // run for the 2nd time in the loop here: > > plugin_foreach(NULL, commit ? xacommit_handlerton : xarollback_handlerton, > MYSQL_STORAGE_ENGINE_PLUGIN, &xaop); > > Now I am thinking a (much) better option exists to block that 2nd time invocation > of binlog_commit() with > merely testing the type of hton in > > static my_bool xacommit_handlerton(THD *unused1, plugin_ref plugin, void *arg) > { > handlerton *hton= plugin_hton(plugin); > - if (hton->recover) > + if (hton->recover && hton != binlog_hton) > { > hton->commit_by_xid(); > > and the same to xarollback_handlerton. > I am now committed to turn to it. No, see above. Ideally, binlog won't have a hton and it won't be in the list of engines at all. But if that's a too big step to make now, then just make binlog_hton->commit_by_xid noop (or NULL, if that works) and invoke binlog_commit_by_xid() explicitly by name. > >> + { > >> + rc= binlog_commit(hton, thd, TRUE); > >> + cache_mngr->completed_by_xid= true; > >> + } > >> > >> + return rc; > >> +} > >> > >> @@ -10333,14 +10386,108 @@ start_binlog_background_thread() ... > >> + > >> +/** > >> + Performs recovery based on transaction coordinator log for 2pc. At the > >> + time of crash, if the binary log was in active state, then recovery for > >> + "implicit" 'xid's and explicit 'XA' transactions is initiated, > >> + otherwise merely the gtid binlog state is updated. > >> + For 'xid' and 'XA' based recovery the following steps are performed. > >> + > >> + Identify the active binlog checkpoint file. > >> + Scan the binary log from the beginning. > >> + From GTID_LIST and GTID_EVENTs reconstruct the gtid binlog state. > >> + Prepare a list of 'xid's for recovery. > >> + Prepare a list of explicit 'XA' transactions for recovery. > >> + Recover the 'xid' transactions. > >> + The explicit 'XA' transaction recovery is initiated once all the server > >> + components are initialized. Please check 'execute_xa_for_recovery()'. > > > > why explicit XA recovery is delayed? > > It has to wait for at least until after > init_update_queries(); > in init_server_components() without which Better move init_update_queries() to happen before recovery. As far as I can see, init_update_queries() only sets flags in the array of commands, it does not depend on anything and can be done anytime. Ideally all the recovery could happen in one place, not split over two separate functions that are invoked at separate points in time during the startup. > >> @@ -10561,6 +10734,219 @@ int TC_LOG_BINLOG::recover(LOG_INFO *linfo, const char *last_log_name, > >> + { > >> + enable_apply_event= true; > >> + // partially engine-prepared XA is first cleaned out prior replay > >> + thd->lex->sql_command= SQLCOM_XA_ROLLBACK; > >> + ha_commit_or_rollback_by_xid(&gev->xid, 0); > >> + } > >> + else > >> + --recover_xa_count; > >> + } > >> + } > >> + else if (gev->flags2 & Gtid_log_event::FL_COMPLETED_XA) > >> + { > >> + if (member->state == XA_COMPLETE && > >> + member->in_engine_prepare > 0) > >> + enable_apply_event= true; > > > > why? you cannot replay a fully prepared and partially committed > > transaction > > We might make use of slave_exec_mode = IDEMPOTENT but only ROW format > events are subject to the mode. Yes, so why you're trying to replay it instead of just completing the commit? Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 269328884e5: MDEV-21365: Check $MARIADB_HOME/my.cnf in addition to $MYSQL_HOME/my.cnf
by Sergei Golubchik 22 Mar '21

22 Mar '21

Hi, Rucha! On Mar 22, Rucha Deodhar wrote: > revision-id: 269328884e5 (mariadb-10.5.2-269-g269328884e5) > parent(s): 46fab5b32a8 > author: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > committer: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > timestamp: 2021-03-22 16:18:18 +0530 > message: > > MDEV-21365: Check $MARIADB_HOME/my.cnf in addition to $MYSQL_HOME/my.cnf > > Currently for the server, we only check if $MYSQL_HOME is set. Added a check > if $MARIADB_HOME is set and try to read the configuration file from this > directory. > To keep behavior consistent with MariaDB Connector/C, checking $MARIADB_HOME > after $MYSQL_HOME. The patch is fine. But the commit comment says exactly the opposite of what you've implemented. You check MARIADB_HOME before MYSQL_HOME (which is correct), not after. Please fix the comment and ok to push. > diff --git a/mysys/my_default.c b/mysys/my_default.c > index c76ab8c6eb1..ec94aee290c 100644 > --- a/mysys/my_default.c > +++ b/mysys/my_default.c > @@ -1051,8 +1051,17 @@ static const char **init_default_directories(MEM_ROOT *alloc) > > #endif > > - if ((env= getenv("MYSQL_HOME"))) > + /* > + If value of $MARIADB_HOME environment variable name is NULL, check > + for $MYSQL_HOME > + */ > + if ((env= getenv("MARIADB_HOME"))) > errors += add_directory(alloc, env, dirs); > + else > + { > + if ((env= getenv("MYSQL_HOME"))) > + errors += add_directory(alloc, env, dirs); > + } > > /* Placeholder for --defaults-extra-file=<path> */ > errors += add_directory(alloc, "", dirs); > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] e5fc78f84e3: MDEV-20220: Merge 5.7 P_S replication table 'replication_applier_status_by_worker
by Sergei Golubchik 22 Mar '21

22 Mar '21

Hi, Sujatha! Could you split this patch, please? 1. Just add replication_applier_status_by_worker table. With CHANNEL_NAME column and (perhaps, not sure if it applies) with WORKER_ID column. Without extra columns and backup. 2. Add extra columns. 3. Add backup. With these three commits you'll have exactly the same diff as now, just split (and with CHANNEL_NAME column). But really, I wonder whether this backup functionality is needed at all? In MySQL there is persistent information about these workers, it doesn't go away when they're stopped, it's stored persistently in a table, indexed by worker_id. If we don't have anything persistent like that and all workers completely disappear into oblivion, then, may be, replication_applier_status_by_worker should not show anything when they aren't running? If you agree, then you'll only need two commits. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org On Mar 21, Sujatha wrote: > revision-id: e5fc78f84e3 (mariadb-10.5.2-303-ge5fc78f84e3) > parent(s): 8b8969929d7 > author: Sujatha <sujatha.sivakumar(a)mariadb.com> > committer: Sujatha <sujatha.sivakumar(a)mariadb.com> > timestamp: 2020-11-27 12:59:42 +0530 > message: > > MDEV-20220: Merge 5.7 P_S replication table 'replication_applier_status_by_worker > > Fix: > === > Iterate through rpl_parallel_thread_pool and display slave worker thread > specific information as part of 'replication_applier_status_by_worker' table. > > --------------------------------------------------------------------------------- > |Column Name: | Description: | > |-------------------------------------------------------------------------------| > | | | > |THREAD_ID | Thread_Id as displayed in 'performance_schema.threads'| > | | table for thread with name | > | | 'thread/sql/rpl_parallel_thread' | > | | | > | | THREAD_ID will be NULL when worker threads are stopped| > | | due to error/force stop | > | | | > |SERVICE_STATE | Thread is running or not | > | | | > |LAST_SEEN_TRANSACTION | Last GTID executed by worker | > | | | > |LAST_ERROR_NUMBER | Last Error that occurred on a particular worker | > | | | > |LAST_ERROR_MESSAGE | Last error specific message | > | | | > |LAST_ERROR_TIMESTAMP | Time stamp of last error | > | | | > |WORKER_IDLE_TIME | Total idle time in seconds that the worker thread has | > | | spent waiting for work from SQL thread | > | | | > |LAST_TRANS_RETRY_COUNT | Total number of retries attempted by last transaction | > --------------------------------------------------------------------------------- > > > In case STOP SLAVE is executed worker threads will be gone, hence worker > threads will be unavailable. Querying the table at this stage will give empty > rows. To address this case when worker threads are about to stop, due to an > error or forced stop, create a backup pool and preserve the data which is > relevant to populate performance schema table. Clear the backup pool upon > slave start.

2 4

Re: [Maria-developers] e5fc78f84e3: MDEV-20220: Merge 5.7 P_S replication table 'replication_applier_status_by_worker
by sujatha 22 Mar '21

22 Mar '21

Hello Andrei, Good Morning. On 22/03/21 3:50 pm, Andrei Elkin wrote: > Sujatha, howdy. > > Just one note from me at this point, > >> ==> https://dev.mysql.com/doc/refman/5.7/en/replication-channels.html >> >> A multi-source replica can also be set up as a multi-threaded >> replica, by setting the slave_parallel_workers system variable to a >> value greater than 0. When you do this on a multi-source replica, each >> channel on the replica has the specified number of applier >> threads, plus a coordinator thread to manage them. You cannot >> configure the number of applier threads for individual channels. >> >> In case of MySQL, worker threads are dedicated to particular channel. >> >> In MariaDB "The pool of replication worker threads is shared among all >> multi-source master connections, and among all replication domains >> that can replicate in parallel using out-of-order". >> >> Because of this I didn't include CHANNEL_NAME. In MariaDB Slave_worker >> thread's don't have WORKER_ID they only have 'thread_id'. > Would it make sense to report > > LAST_CHANNEL (or LAST_SOURCE) similar to other 'LAST_' like > > |LAST_SEEN_TRANSACTION | Last GTID executed by worker | > > ? Yes, as 'LAST_SOURCE', we can display 'mi->connection_name'. In MariaDB documentation 'CONNECTION_NAME' is used to identify a source, so I used 'CONNECTION_NAME' instead of 'CHANNEL_NAME'(MySQL'). Thank you. S.Sujatha > >>>> Fix: >>>> === >>>> Iterate through rpl_parallel_thread_pool and display slave worker thread >>>> specific information as part of 'replication_applier_status_by_worker' table. >>>> >>>> --------------------------------------------------------------------------------- >>>> |Column Name: | Description: | >>>> |-------------------------------------------------------------------------------| >>>> | | | >>>> |THREAD_ID | Thread_Id as displayed in 'performance_schema.threads'| >>>> | | table for thread with name | >>>> | | 'thread/sql/rpl_parallel_thread' | >>>> | | | >>>> | | THREAD_ID will be NULL when worker threads are stopped| >>>> | | due to error/force stop | >>>> | | | >>>> |SERVICE_STATE | Thread is running or not | >>>> | | | >>>> |LAST_SEEN_TRANSACTION | Last GTID executed by worker | >>>> | | | >>>> |LAST_ERROR_NUMBER | Last Error that occurred on a particular worker | >>>> | | | >>>> |LAST_ERROR_MESSAGE | Last error specific message | >>>> | | | >>>> |LAST_ERROR_TIMESTAMP | Time stamp of last error | >>>> | | | >>>> |WORKER_IDLE_TIME | Total idle time in seconds that the worker thread has | >>>> | | spent waiting for work from SQL thread | >>>> | | | >>>> |LAST_TRANS_RETRY_COUNT | Total number of retries attempted by last transaction | >>>> --------------------------------------------------------------------------------- > Cheers, > > Andrei

1 0

Re: [Maria-developers] d86fe72c027: MDEV-22010: use executables MariaDB named in scripts
by Sergei Golubchik 21 Mar '21

21 Mar '21

Hi, Rucha! On Mar 21, Rucha Deodhar wrote: > revision-id: d86fe72c027 (mariadb-10.5.2-386-gd86fe72c027) > parent(s): de407e7cb4d > author: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > committer: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > timestamp: 2021-03-18 00:58:43 +0530 > message: > > MDEV-22010: use executables MariaDB named in scripts > > As a part of this MDEV following changes were made: > 1) Mariadb named executables used instead of mysql named executables in scripts > 2) renamed mysql-test-run and mysql-stress-test to mariadb-test-run and > mariadb-stress-test and created a symlink. > > diff --git a/.gitignore b/.gitignore > index d58dd2aa2b8..1a840d1a51d 100644 > --- a/.gitignore > +++ b/.gitignore > @@ -600,3 +600,7 @@ tests/mariadb-client-test > versioninfo_dll.rc > versioninfo_exe.rc > win/packaging/ca/symlinks.cc > +mysql-test/mysql-stress-test.pl > +mysql-test/mysql-test-run > +mysql-test/mysql-test-run.pl > +mysql-test/mtr Please, add these lines together with other mysql-test/ lines in the file. Then you'll see that mysql-test-run and mtr are already ignored. > diff --git a/scripts/mysql_install_db.sh b/scripts/mysql_install_db.sh > index 5f183afe8fc..147bd7351bc 100644 > --- a/scripts/mysql_install_db.sh > +++ b/scripts/mysql_install_db.sh > @@ -338,11 +338,11 @@ then > cannot_find_file resolveip @resolveip_locations@ > exit 1 > fi > - mysqld=`find_in_dirs mysqld @mysqld_locations@` > + mysqld=`find_in_dirs mariadbd @mysqld_locations@` > if test -z "$mysqld" > then > - cannot_find_file mysqld @mysqld_locations@ > - exit 1 > + cannot_find_file mysqld @mysqld_locations@ you wanted to say "cannot_find_file mariadbd @mysqld_locations@" > + exit 1 > fi > langdir=`find_in_dirs --dir errmsg.sys @errmsg_locations@` > if test -z "$langdir" > @@ -579,7 +579,7 @@ else > echo > echo " shell> $mysqld --skip-grant-tables --general-log &" > echo > - echo "and use the command line tool $bindir/mysql" > + echo "and use the command line tool $bindir/mariadb or $bindir/mysql" I don't think you need any "or mysql" here. > echo "to connect to the mysql database and look at the grant tables:" > echo > echo " shell> $bindir/mysql -u root mysql" > @@ -613,7 +613,8 @@ then > echo "PLEASE REMEMBER TO SET A PASSWORD FOR THE MariaDB root USER !" > echo "To do so, start the server, then issue the following commands:" > echo > - echo "'$bindir/mysqladmin' -u root password 'new-password'" > + echo "'$bindir/mariadb-admin' -u root password 'new-password' or" > + echo "'$bindir/mariadb-admin' -u root -h $hostname password 'new-password' or" and not here. > echo "'$bindir/mysqladmin' -u root -h $hostname password 'new-password'" > echo > echo "Alternatively you can run:" > diff --git a/scripts/mysql_secure_installation.sh b/scripts/mysql_secure_installation.sh > index b2a9edf4953..6098e0148b5 100644 > --- a/scripts/mysql_secure_installation.sh > +++ b/scripts/mysql_secure_installation.sh > @@ -159,15 +159,15 @@ then > cannot_find_file my_print_defaults $basedir/bin $basedir/extra > exit 1 > fi > - mysql_command=`find_in_basedir mysql bin` > + mysql_command=`find_in_basedir mariadb bin` > if test -z "$mysql_command" > then > - cannot_find_file mysql $basedir/bin > - exit 1 > + cannot_find_file mysql $basedir/bin should be cannot_find_file mariadb $basedir/bin" > + exit 1 > fi > else > print_defaults="@bindir@/my_print_defaults" > - mysql_command="@bindir@/mysql" > + mysql_command="@bindir@/mariadb" > fi > > if test ! -x "$print_defaults" > diff --git a/scripts/mysqld_multi.sh b/scripts/mysqld_multi.sh > index 0b33f61710e..d83663e03fc 100644 > --- a/scripts/mysqld_multi.sh > +++ b/scripts/mysqld_multi.sh > @@ -728,7 +728,7 @@ sub example > > [mysqld_multi] > mysqld = @bindir@/mysqld_safe mariadbd-safe ? (here and everywhere in this file) Also, you've missed mysqld_safe in FIND_PROC set in scripts/CMakeLists.txt > -mysqladmin = @bindir@/mysqladmin > +mysqladmin = @bindir@/mariadb-admin > user = multi_admin > password = my_password > > diff --git a/scripts/mysqld_safe.sh b/scripts/mysqld_safe.sh > index 126eb924814..7763fb9db91 100644 > --- a/scripts/mysqld_safe.sh > +++ b/scripts/mysqld_safe.sh > @@ -531,7 +531,11 @@ else > ledir='@libexecdir@' > fi > > -helper=`find_in_bin mysqld_safe_helper` > +helper=`find_in_bin mariadbd-safe-helper` > +if test -x helper > +then > + helper=`find_in_bin mysqld_safe_helper` > +fi no, remove the if, no need to fallback (and you've got the condition wrong anyway) > print_defaults=`find_in_bin my_print_defaults` > # Check if helper exists > command -v $helper --help >/dev/null 2>&1 > diff --git a/scripts/wsrep_sst_common.sh b/scripts/wsrep_sst_common.sh > index 5e134570881..b8621a76d6f 100644 > --- a/scripts/wsrep_sst_common.sh > +++ b/scripts/wsrep_sst_common.sh > @@ -273,16 +273,16 @@ SCRIPTS_DIR="$(cd $(dirname "$0"); pwd -P)" > EXTRA_DIR="$SCRIPTS_DIR/../extra" > CLIENT_DIR="$SCRIPTS_DIR/../client" > > -if [ -x "$CLIENT_DIR/mysql" ]; then > - MYSQL_CLIENT="$CLIENT_DIR/mysql" > +if [-x "$CLIENT_DIR/mariadb"]; then "[" is a name of an executable, always leave a space after it. there's no [-x executable. > + MYSQL_CLIENT="$CLIENT_DIR/mariadb" > else > - MYSQL_CLIENT=mysql > + MYSQL_CLIENT=mariadb > fi Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 2858f9fb7e0: MDEV-22189: Change error messages inside code to have mariadb instead of
by Sergei Golubchik 21 Mar '21

21 Mar '21

Hi, Rucha! First, please check the buildbot and make sure all tests pass. You've changed MySQL->MariaDB in result files, but without corresponding changes in the code your result files now differ from the actual test output. One more comment below On Mar 21, Rucha Deodhar wrote: > revision-id: 2858f9fb7e0 (mariadb-10.5.2-274-g2858f9fb7e0) > parent(s): 41485551c58 > author: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > committer: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > timestamp: 2021-03-18 13:58:34 +0530 > message: > > MDEV-22189: Change error messages inside code to have mariadb instead of > mysql > > Fix: Changed error messages, rerecorded results and changed other relevant > files. > > diff --git a/debian/additions/innotop/innotop b/debian/additions/innotop/innotop > index 19229a57a82..a045a479be9 100644 > --- a/debian/additions/innotop/innotop > +++ b/debian/additions/innotop/innotop > @@ -987,12 +987,12 @@ sub parse_tx_text { > # Parsing the line that begins 'MySQL thread id' is complicated. The only > # thing always in the line is the thread and query id. See function > # innobase_mysql_print_thd in InnoDB source file sql/ha_innodb.cc. > - my ( $thread_line ) = $txn =~ m/^(MySQL thread id .*)$/m; > + my ( $thread_line ) = $txn =~ m/^(MariaDB thread id .*)$/m; > my ( $mysql_thread_id, $query_id, $hostname, $ip, $user, $query_status ); > > if ( $thread_line ) { > # These parts can always be gotten. > - ( $mysql_thread_id, $query_id ) = $thread_line =~ m/^MySQL thread id $d, .*?query id $d/m; > + ( $mysql_thread_id, $query_id ) = $thread_line =~ m/^MariaDB thread id $d, .*?query id $d/m; > > # If it's a master/slave thread, "Has (read|sent) all" may be the thread's > # proc_info. In these cases, there won't be any host/ip/user info > @@ -4144,15 +4144,15 @@ my $innodb_parser = InnoDBParser->new; > > my $nonfatal_errs = join('|', > 'Access denied for user', > - 'Unknown MySQL server host', > + 'Unknown MariaDB server host', > 'Unknown database', > - 'Can\'t connect to local MySQL server through socket', > - 'Can\'t connect to MySQL server on', > + 'Can\'t connect to local MariaDB server through socket', > + 'Can\'t connect to MariaDB server on', > 'MySQL server has gone away', > 'Cannot call SHOW INNODB STATUS', > 'Access denied', > 'AutoCommit', > - 'Lost connection to MySQL server', > + 'Lost connection to MariaDB server', > 'Too many connections', > ); innotop is generally a separate tool that technically could work both with MariaDB and MySQL. So, better use (?:MariaDB|MySQL) in all regexp patterns above. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] MDEV-13467 review.
by Alexey Botchkov 20 Mar '21

20 Mar '21

One thing i'd still like you to do differently. + if (g->get_class_info()->m_type_id == Geometry::wkb_multipoint) + { + const char point_size= 4 + WKB_HEADER_SIZE + POINT_DATA_SIZE+1; //1 for the type + char point_temp[point_size]; + memset(point_temp+4, Geometry::wkb_point, 1); + memcpy(point_temp+5, static_cast<const Gis_multi_point *>(g)->get_data_ptr()+5, 4); + memcpy(point_temp+4+WKB_HEADER_SIZE, g->get_data_ptr()+4+WKB_ HEADER_SIZE, + POINT_DATA_SIZE); + point_temp[point_size-1]= '\0'; + Geometry_buffer gbuff; + Geometry *gg= Geometry::construct(&gbuff, point_temp, point_size-1); > Gis_multi_point::geometry_n(uint32 num, String *result) stores the point in String *, > but I need Geometry* to use further, so still would need some recalculation. You could call the ::construct with the string->prr() instead of point_temp. Like str->q_append(0); //SRID g->geometry_n(str, 1); gg= Geometry::construct(&gbuff, str->ptr(), str->length()); But I propose an even simpler method. { Gis_pont p; p.set_data_ptr(g->get_data_ptr()+4+WKB_HEADER_SIZE, POINT_DATA_SIZE); calculate_haversine(&p, radius); } > Regarding radius, Daniel has suggestion: when r=NULL, result= NULL, but r=0, > how should that be treated? > Here is MySQL example (no error, but warning) There i see that MySQL returns the error when the radius is 0 or below. Right? I think it's ok for us to do same. > In a single function we are calculating radian for x,y, I think it is faster than to call function 2 times for x and y. I doubt so. Really would like to compare the speed here. Still i'm ok with that single function if you like it better. Best regards. HF On Tue, Mar 2, 2021 at 1:16 PM Anel Husakovic <anel(a)mariadb.org> wrote: > Hi Alexey, > thanks for your review, hope everything is ok now. > Here is the new patch 1f618e36577a2a322807 > <https://github.com/MariaDB/server/commit/1f618e36577a2a32280705c640d6f9a748…> according > to your review and comments below. > > On Sat, Jan 9, 2021 at 2:09 PM Alexey Botchkov <holyfoot(a)mariadb.com> > wrote: > >> Hi, Anel! >> >> Sorry for that long delay with this review. Had real difficult weeks of >> my life :( >> Now it seems to be over, so getting back to normal. >> Below is my opinions of the code i see in the branch >> bb-10.1-anel-MDEV-13467-gis-feature-st_sphere-v2 >> Hope it's the recent version. >> >> + Adapter for functions that takes two or three arguments. >> +*/ >> + >> +class Create_func_arg2_or_3 : public Create_func >> +{ >> Do you think we need the separate Create_fund_arg2_or_3 class? >> If you don't see any other class to inherit from it, >> i'd recommend to leave the Create_func_distance_sphere alone. >> No need to have two constructors in the Item_func_sphere_distance >> too. Just use the list argument as it's done for the >> Item_func_json_contains (in 10.2). >> > > Ack ^ droped the Create_func_arg2_or_3 and inherited from > Create_native_function which can use a variable number of parameters. > Thanks > > >> Anyway i belive we should get rid of duplicating code >> about the 'param1' and 'param2'. They can be popped/checked >> with no if (arg_count == 2) condition. >> >> + double distance= 0.0; >> + null_value= (args[0]->null_value || args[1]->null_value); >> + if (null_value) >> + { >> + // Returned item must be set to Null >> + DBUG_ASSERT(maybe_null); >> don't think it's needed. >> + return 0; >> Should be return 0.0 i belive. Or goto handle_errors. >> Thet goes to all but one 'return' statements in this function. >> >> + } >> >> > ... >> > if (!arg1 || !arg2) >> >> This last condition should never happen since we have not-null_value. >> So i'd remove that 'if' section completely. >> And anyway it should do the 'return' in that branch. >> >> + null_value= args[2]->null_value; >> + if (null_value) >> + { >> + return 0; >> + } >> >> seems nicer to me this way >> if (args[2]->null_value) >> { >> null_value= true; >> goto handle_errors; >> } >> >> > Ack. > Regarding radius, Daniel has suggestion > <https://github.com/MariaDB/server/commit/e42b69136a31cb596e940d1ceafffa5c11…>: > when r=NULL, result= NULL, but r=0, how should that be treated? > Here is MySQL > <https://dbfiddle.uk/?rdbms=mysql_8.0&fiddle=d467903dce23464df0ebf396220da886>example > (no error, but warning) > > > >> + if (!(g1= Geometry::construct(&buffer1, arg1->ptr(), arg1->length())) >> || >> + !(g2= Geometry::construct(&buffer2, arg2->ptr(), arg2->length()))) >> + { >> + goto handle_errors; >> >> we should launch the ER_GIS_INVALID_DATA here. >> >> > ER_GIS_INVALID_DATA is introduced in 10.2, I will change this (and test > case) when pushing to 10.2, but I would also push to 10.1 since there may > be users using it in 10.1. > Also when inspecting this error in 10.2 noted that lot tests have > commented this error, so will remove that comment in the same commit. > > mysql-test/suite/innodb_gis/t/gis.test > > mysql-test/suite/innodb_gis/t/1.test > > mysql-test/suite/innodb_gis/t/bug16236208.test > > mysql-test/suite/innodb_gis/t/precise.test > > mysql-test/suite/innodb_gis/t/rtree.test > > > + // Generate error message in case different geometry is used? >> Yes, i think the explainig message here is good to have. >> >> +double Item_func_sphere_distance::spherical_distance(Geometry *g1, >> + Geometry *g2, >> + const double >> sphere_radius) >> +double Item_func_sphere_distance::spherical_distance_points(Geometry *g1, >> + Geometry *g2, >> + const double >> r) >> Why these two are member static functions, not simply static? >> If you plan any 'external' use for them, please add comments about >> what exacly they do. >> >> > Changed function to be member function only. > > > >> +double Item_func_sphere_distance::spherical_distance(Geometry *g1, >> + Geometry *g2, >> + const double >> sphere_radius) >> +{ >> + double res= 0.0; >> + switch (g1->get_class_info()->m_type_id) >> + { >> + case Geometry::wkb_point: >> + res= spherical_distance_points(g1, g2, sphere_radius); >> + break; >> + >> + case Geometry::wkb_multipoint: >> + res= spherical_distance_points(g1, g2, sphere_radius); >> + break; >> >> Why do we need this function? >> We confirmed already that the type is either wkb_point or >> wkb_multipolygon, >> so just can call spherical_distance_points. No? >> >> > Indeed, left from earlier implementation (v1). > > >> >>>>> >> >> >> +class Item_func_sphere_distance: public Item_real_func >> +{ >> + double sphere_radius= 6370986.0; // Default radius equals Earth radius >> >> Some compilers don't like this syntax, and we didn't use it yet, so >> i'd recomment to move this to the constructor or even better to the >> ::val_real(). >> >> +double Gis_point::calculate_haversine(const Geometry *g, >> + const double sphere_radius) >> +{ >> + DBUG_ASSERT(sphere_radius > 0); >> + double x1r= 0.0; >> + double x2r= 0.0; >> + double y1r= 0.0; >> + double y2r= 0.0; >> That lot of unnecessary initializations seems excessive. >> >> Ack > > >> + if (g->get_class_info()->m_type_id == Geometry::wkb_multipoint) >> + { >> + const char point_size= 4 + WKB_HEADER_SIZE + POINT_DATA_SIZE+1; //1 >> for the type >> + char point_temp[point_size]; >> + memset(point_temp+4, Geometry::wkb_point, 1); >> + memcpy(point_temp+5, static_cast<const Gis_multi_point >> *>(g)->get_data_ptr()+5, 4); >> + memcpy(point_temp+4+WKB_HEADER_SIZE, >> g->get_data_ptr()+4+WKB_HEADER_SIZE, >> + POINT_DATA_SIZE); >> + point_temp[point_size-1]= '\0'; >> + Geometry_buffer gbuff; >> + Geometry *gg= Geometry::construct(&gbuff, point_temp, point_size-1); >> >> There is the Gis_multi_point::geometry_n() function, should be used here >> to get the point. >> >> Gis_multi_point::geometry_n(uint32 num, String *result) stores the point > in String *, but I need Geometry* to use further, so still would need some > recalculation. > > I didn’t find a multipoint API to convert String * to Geometry* so I did > my own calculation after debugging from row data. > > > I don't like that lot of static_cast-s. >> Isn't it nicer to have just the function of >> double calculate_harvesine_points(Geometry *p1, Geometry *p2, double >> radius) >> double calculate_harvesine_multipoint_point(Geometry *mp, Geometry *p, >> double radius) >> double calculate_harvesine_multipoint_multipoint(Geometry *mp1, Geometry >> *mp2, double radius) >> which would call each other? >> >> > I would prefer to have 2 static member functions and use static cast where > needed, if you don't mind. > The same could be done with 3 functions but with the code repeated. > > >> The ::get_xy_radian() method looks unneeded to me. >> It's enough to have the to_radian(double d) function. >> >> > In a single function we are calculating radian for x,y, I think it is > faster than to call function 2 times for x and y. > > >> BTW it's possible to do the multipoint/multipoint calculations faster >> than n^2. >> Not necessary to do it right now, just not to remember. >> >> > Regarding this there are some implementations that don't use Multipoints > <https://postgis.net/docs/manual-1.4/ST_Distance_Sphere.html> as both > arguments, MySQL > <https://dev.mysql.com/doc/refman/8.0/en/spatial-convenience-functions.html>also > use only Multipoints,Points combination, not MultiPoints,MultiPoints, so we > need to decide what to support. > However, there are algorithms to calculate max > <http://citeseerx.ist.psu.edu/viewdoc/download;jsessionid=CA2AE9B932194B2970…> > ,min <https://core.ac.uk/download/pdf/82507217.pdf> distance and > O(nlong) using convex hull <https://en.wikipedia.org/wiki/Convex_hull> > (which we have implemented <https://mariadb.com/kb/en/st_convexhull/>) > and using rotating calipers > <https://en.wikipedia.org/wiki/Rotating_calipers> from N points from > convex hull it could be O(n). > > Also there is suggested usage of iteration > <https://github.com/MariaDB/server/commit/e42b69136a31cb596e940d1ceafffa5c11…>from > Daniel. > > > <https://github.com/MariaDB/server/commit/e42b69136a31cb596e940d1ceafffa5c11…> > > > >> Best regards. >> HF >> >> > > -- > *Anel Husaković, * > *Software developer* > *MariaDB Foundation* >

1 0

Re: [Maria-developers] 0fc221a012d: MDEV-8334: Rename utf8 to utf8mb3
by Sergei Golubchik 19 Mar '21

19 Mar '21

Hi, Rucha! Thanks for combining all commits in one and a cleanup. See comments below. On Mar 18, Rucha Deodhar wrote: > revision-id: 0fc221a012d (mariadb-10.5.2-402-g0fc221a012d) > parent(s): a1542f8a573 > author: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > committer: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > timestamp: 2021-03-16 23:31:12 +0530 > message: > > MDEV-8334: Rename utf8 to utf8mb3 > > This patch changes the main name of 3 byte character set from utf8 to > utf8mb3. New old_mode UTF8_IS_UTF8MB3 is added and set TRUE by default, > so that utf8 would mean utf8mb3. If not set, utf8 would mean utf8mb4. > diff --git a/client/mysqlcheck.c b/client/mysqlcheck.c > index fb3103a318d..a8990d8cb6b 100644 > --- a/client/mysqlcheck.c > +++ b/client/mysqlcheck.c > @@ -437,7 +437,7 @@ static int get_options(int *argc, char ***argv) > if (!default_charset) > { > if (opt_fix_db_names || opt_fix_table_names) > - default_charset= (char*) "utf8"; > + default_charset= (char*) "utf8mb3"; why not to keep it utf8? > else > default_charset= (char*) MYSQL_AUTODETECT_CHARSET_NAME; > } > diff --git a/client/mysqldump.c b/client/mysqldump.c > index 7c363973da2..900456b31b2 100644 > --- a/client/mysqldump.c > +++ b/client/mysqldump.c > @@ -3235,7 +3235,7 @@ static uint get_table_structure(const char *table, const char *db, char *table_t > { > fprintf(sql_file, > "/*!40101 SET @saved_cs_client = @@character_set_client */;\n" > - "/*!40101 SET character_set_client = utf8 */;\n" > + "/*!40101 SET character_set_client = utf8mb3 */;\n" why not to keep it utf8? > "%s%s;\n" > "/*!40101 SET character_set_client = @saved_cs_client */;\n", > is_log_table ? "CREATE TABLE IF NOT EXISTS " : "", > diff --git a/extra/mariabackup/backup_mysql.cc b/extra/mariabackup/backup_mysql.cc > index 3083326a7e0..c62252257b9 100644 > --- a/extra/mariabackup/backup_mysql.cc > +++ b/extra/mariabackup/backup_mysql.cc > @@ -117,7 +117,7 @@ xb_mysql_connect() > mysql_options(connection, MYSQL_PLUGIN_DIR, xb_plugin_dir); > } > mysql_options(connection, MYSQL_OPT_PROTOCOL, &opt_protocol); > - mysql_options(connection,MYSQL_SET_CHARSET_NAME, "utf8"); > + mysql_options(connection,MYSQL_SET_CHARSET_NAME, "utf8mb3"); why not to keep it utf8? > > msg("Connecting to MySQL server host: %s, user: %s, password: %s, " > "port: %s, socket: %s", opt_host ? opt_host : "localhost", > @@ -1506,7 +1506,7 @@ write_xtrabackup_info(MYSQL *connection, const char * filename, bool history, > "incremental ENUM('Y', 'N') DEFAULT NULL," > "format ENUM('file', 'tar', 'xbstream') DEFAULT NULL," > "compressed ENUM('Y', 'N') DEFAULT NULL" > - ") CHARACTER SET utf8 ENGINE=innodb", false); > + ") CHARACTER SET utf8mb3 ENGINE=innodb", false); why not to keep it utf8? > > > #define ESCAPE_BOOL(expr) ((expr)?"'Y'":"'N'") > diff --git a/extra/mariabackup/xtrabackup.cc b/extra/mariabackup/xtrabackup.cc > index 62fdace654d..0f500c35ff6 100644 > --- a/extra/mariabackup/xtrabackup.cc > +++ b/extra/mariabackup/xtrabackup.cc > @@ -1716,7 +1716,7 @@ static int create_bootstrap_file() > if(!f) > return -1; > > - fputs("SET NAMES UTF8;\n",f); > + fputs("SET NAMES UTF8MB3;\n",f); why not to keep it utf8? > enumerate_ibd_files(append_export_table); > for (std::set<std::string>::iterator it = tables_for_export.begin(); > it != tables_for_export.end(); it++) > diff --git a/mysql-test/main/create-uca.test b/mysql-test/main/create-uca.test > index 0acb51f7286..f73f6114962 100644 > --- a/mysql-test/main/create-uca.test > +++ b/mysql-test/main/create-uca.test > @@ -1,5 +1,5 @@ > # Prerequisites > -let collation=utf8_unicode_ci; > +let collation=utf8mb3_unicode_ci; that's fine, as we generally don't want tests to depend on the old_mode. > --source include/have_collation.inc > > # Initial cleanup > diff --git a/mysql-test/main/ctype_ldml.result b/mysql-test/main/ctype_ldml.result > index 22b7a316111..7c284520733 100644 > --- a/mysql-test/main/ctype_ldml.result > +++ b/mysql-test/main/ctype_ldml.result > @@ -8,7 +8,6 @@ Variable_name Value > character_sets_dir MYSQL_TEST_DIR/std_data/ldml/ > show collation like 'utf8_phone_ci'; > Collation Charset Id Default Compiled Sortlen > -utf8_phone_ci utf8 352 8 I suppose the test should show there is a _phone_ci collation. As you've renamed it, you need to adjust the test to do show collation like 'utf8mb3_phone_ci'; > CREATE TABLE t1 ( > name VARCHAR(64), > phone VARCHAR(64) CHARACTER SET utf8 COLLATE utf8_phone_ci > @@ -37,7 +36,6 @@ Bar +7-912-800-80-01 > DROP TABLE t1; > show collation like 'utf8_test_ci'; > Collation Charset Id Default Compiled Sortlen > -utf8_test_ci utf8 353 8 same > create table t1 (c1 char(1) character set utf8 collate utf8_test_ci); > insert into t1 values ('a'); > select * from t1 where c1='b'; > @@ -526,7 +524,6 @@ DROP TABLE t1; > SET NAMES utf8 COLLATE utf8_phone_ci; > SHOW COLLATION LIKE 'utf8_phone_ci'; > Collation Charset Id Default Compiled Sortlen > -utf8_phone_ci utf8 352 8 and here > SET NAMES utf8; > SELECT hex(weight_string(_utf8mb4'a' collate utf8mb4_test_400_ci)); > hex(weight_string(_utf8mb4'a' collate utf8mb4_test_400_ci)) > diff --git a/mysql-test/main/show_check.result b/mysql-test/main/show_check.result > index d031c792922..1c85ef596c9 100644 > --- a/mysql-test/main/show_check.result > +++ b/mysql-test/main/show_check.result > @@ -874,12 +874,11 @@ set names utf8; > ---------------------------------------------------------------- > SHOW CHARACTER SET LIKE 'utf8'; > Catalog Database Table Table_alias Column Column_alias Type Length Max length Is_null Flags Decimals Charsetnr > -def information_schema CHARACTER_SETS CHARACTER_SETS CHARACTER_SET_NAME Charset 253 96 4 N 1 0 33 > -def information_schema CHARACTER_SETS CHARACTER_SETS DESCRIPTION Description 253 180 13 N 1 0 33 > -def information_schema CHARACTER_SETS CHARACTER_SETS DEFAULT_COLLATE_NAME Default collation 253 96 15 N 1 0 33 > -def information_schema CHARACTER_SETS CHARACTER_SETS MAXLEN Maxlen 8 3 1 N 32769 0 63 > +def information_schema CHARACTER_SETS CHARACTER_SETS CHARACTER_SET_NAME Charset 253 96 0 N 1 0 33 > +def information_schema CHARACTER_SETS CHARACTER_SETS DESCRIPTION Description 253 180 0 N 1 0 33 > +def information_schema CHARACTER_SETS CHARACTER_SETS DEFAULT_COLLATE_NAME Default collation 253 96 0 N 1 0 33 > +def information_schema CHARACTER_SETS CHARACTER_SETS MAXLEN Maxlen 8 3 0 N 32769 0 63 > Charset Description Default collation Maxlen > -utf8 UTF-8 Unicode utf8_general_ci 3 again, I suspect this test should now do `SHOW CHARACTER SET LIKE 'utf8mb3'; > ---------------------------------------------------------------- > SHOW COLLATION LIKE 'latin1_bin'; > Catalog Database Table Table_alias Column Column_alias Type Length Max length Is_null Flags Decimals Charsetnr > diff --git a/mysql-test/suite/funcs_1/r/charset_collation.result b/mysql-test/suite/funcs_1/r/charset_collation.result > index 31bd30c5acf..6b52e80d6ba 100644 > --- a/mysql-test/suite/funcs_1/r/charset_collation.result > +++ b/mysql-test/suite/funcs_1/r/charset_collation.result > @@ -9,7 +9,6 @@ ORDER BY character_set_name; > CHARACTER_SET_NAME DEFAULT_COLLATE_NAME DESCRIPTION MAXLEN > binary binary Binary pseudo charset 1 > latin1 latin1_swedish_ci cp1252 West European 1 > -utf8 utf8_general_ci UTF-8 Unicode 3 and here too, changing the test is in order > > SELECT * > FROM information_schema.collations > diff --git a/mysql-test/suite/funcs_1/r/is_column_privileges.result b/mysql-test/suite/funcs_1/r/is_column_privileges.result > index b6be9118048..46b2d515041 100644 > --- a/mysql-test/suite/funcs_1/r/is_column_privileges.result > +++ b/mysql-test/suite/funcs_1/r/is_column_privileges.result > @@ -45,7 +45,7 @@ COLUMN_PRIVILEGES CREATE TEMPORARY TABLE `COLUMN_PRIVILEGES` ( > `COLUMN_NAME` varchar(64) NOT NULL DEFAULT '', > `PRIVILEGE_TYPE` varchar(64) NOT NULL DEFAULT '', > `IS_GRANTABLE` varchar(3) NOT NULL DEFAULT '' > -) ENGINE=MEMORY DEFAULT CHARSET=utf8 > +) ENGINE=MEMORY DEFAULT CHARSET=utf8mb3 Just a thought. Did you also fix all --embedded, all --ps, and all --big --big tests to pass? > SHOW COLUMNS FROM information_schema.COLUMN_PRIVILEGES; > Field Type Null Key Default Extra > GRANTEE varchar(190) NO > diff --git a/mysql-test/suite/innodb/r/innodb_ctype_ldml.result b/mysql-test/suite/innodb/r/innodb_ctype_ldml.result > index 502f57156c3..8a96c023134 100644 > --- a/mysql-test/suite/innodb/r/innodb_ctype_ldml.result > +++ b/mysql-test/suite/innodb/r/innodb_ctype_ldml.result > @@ -8,7 +8,6 @@ Variable_name Value > character_sets_dir MYSQL_TEST_DIR/std_data/ldml/ > show collation like 'utf8_phone_ci'; > Collation Charset Id Default Compiled Sortlen > -utf8_phone_ci utf8 352 8 deja vu. I rememeber I've commented on this very thing already :) please, update this test too. > CREATE TABLE t1 ( > name VARCHAR(64), > phone VARCHAR(64) CHARACTER SET utf8 COLLATE utf8_phone_ci > Binary files a/mysql-test/suite/sys_vars/r/character_set_results_basic.result and b/mysql-test/suite/sys_vars/r/character_set_results_basic.result differ > diff --git a/mysql-test/suite/sys_vars/r/old_mode_basic.result b/mysql-test/suite/sys_vars/r/old_mode_basic.result > index 39c8e554be2..a6b95f1c60c 100644 > --- a/mysql-test/suite/sys_vars/r/old_mode_basic.result > +++ b/mysql-test/suite/sys_vars/r/old_mode_basic.result > @@ -167,8 +167,100 @@ NO_PROGRESS_INFO > SET @@global.old_mode = @global_start_value; > SELECT @@global.old_mode; > @@global.old_mode > - > +UTF8_IS_UTF8MB3 > SET @@session.old_mode = @session_start_value; > SELECT @@session.old_mode; > @@session.old_mode > - > +UTF8_IS_UTF8MB3 > +# > +# Beginning of 10.6 test > +# > +# MDEV-8334: Rename utf8 to utf8mb3 > +# Ah! Great. > +# Save and display old values > +SET @save_old_mode = @@OLD_MODE; > +SET @save_character_set_server = @@character_set_server; > +SET @save_character_set_client = @@character_set_client; > +SET @save_character_set_results = @@character_set_results; > +SET @save_character_set_connection = @@character_set_connection; > +SET @save_character_set_filesystem = @@character_set_filesystem; > +SET @save_character_set_database = @@character_set_database; > +SET @save_collation_connection = @@collation_connection; > +SET @save_collation_server = @@collation_server; > +SET @save_collation_database = @@collation_database; > +SELECT @@OLD_MODE; > +@@OLD_MODE > +UTF8_IS_UTF8MB3 > +SELECT @@character_set_server,@@character_set_client,@@character_set_results, > +@@character_set_connection, @@character_set_filesystem, @@character_set_database, > +@@collation_connection, @@collation_server, @@collation_database; > +@@character_set_server @@character_set_client @@character_set_results @@character_set_connection @@character_set_filesystem @@character_set_database @@collation_connection @@collation_server @@collation_database > +latin1 latin1 latin1 latin1 binary latin1 latin1_swedish_ci latin1_swedish_ci latin1_swedish_ci > +# > +# UTF8MB3 alias for UTF8 > +# > +SET @@character_set_server = utf8; > +SET @@character_set_client = utf8; > +SET @@character_set_results = utf8; > +SET @@character_set_connection = utf8; > +SET @@character_set_filesystem = utf8; > +SET @@character_set_database = utf8; > +SET @@collation_connection = utf8_general_ci; > +SET @@collation_server = utf8_unicode_ci; > +SET @@collation_database = utf8_bin; > +SELECT @@character_set_server, @@character_set_client, @@character_set_results, > +@@character_set_connection, @@character_set_filesystem, @@character_set_database, > +@@collation_connection, @@collation_server, @@collation_database; > +@@character_set_server @@character_set_client @@character_set_results @@character_set_connection @@character_set_filesystem @@character_set_database @@collation_connection @@collation_server @@collation_database > +utf8mb3 utf8mb3 utf8mb3 utf8mb3 utf8mb3 utf8mb3 utf8mb3_general_ci utf8mb3_unicode_ci utf8mb3_bin > +CREATE DATABASE db1 CHARACTER SET = 'utf8' COLLATE = 'utf8_general_ci'; > +ALTER DATABASE db1 CHARACTER SET = 'utf8' COLLATE = 'utf8_unicode_ci'; > +CREATE TABLE tb1 (id1 INT) CHARACTER SET 'utf8' COLLATE 'utf8_bin'; > +SHOW CREATE TABLE tb1; > +Table Create Table > +tb1 CREATE TABLE `tb1` ( > + `id1` int(11) DEFAULT NULL > +) ENGINE=MyISAM DEFAULT CHARSET=utf8mb3 COLLATE=utf8mb3_bin > +DROP TABLE tb1; > +DROP DATABASE db1; > +# > +# UTF8MB4 is alias for UTF8 > +# > +SET @@OLD_MODE=0; > +SET @@character_set_server = utf8; > +SET @@character_set_client = utf8; > +SET @@character_set_results = utf8; > +SET @@character_set_connection = utf8; > +SET @@character_set_filesystem = utf8; > +SET @@character_set_database = utf8; > +SET @@collation_connection = utf8_general_ci; > +SET @@collation_server = utf8_unicode_ci; > +SET @@collation_database = utf8_bin; > +SELECT @@character_set_server, @@character_set_client, @@character_set_results, > +@@character_set_connection, @@character_set_filesystem, @@character_set_database, > +@@collation_connection, @@collation_server, @@collation_database; > +@@character_set_server @@character_set_client @@character_set_results @@character_set_connection @@character_set_filesystem @@character_set_database @@collation_connection @@collation_server @@collation_database > +utf8mb4 utf8mb4 utf8mb4 utf8mb4 utf8mb4 utf8mb4 utf8mb4_general_ci utf8mb4_unicode_ci utf8mb4_bin > +CREATE DATABASE db1 CHARACTER SET = 'utf8' COLLATE = 'utf8_general_ci'; > +ALTER DATABASE db1 CHARACTER SET = 'utf8' COLLATE = 'utf8_unicode_ci'; > +CREATE TABLE tb1 (id1 INT) CHARACTER SET 'utf8' COLLATE 'utf8_bin'; > +SHOW CREATE TABLE tb1; > +Table Create Table > +tb1 CREATE TABLE `tb1` ( > + `id1` int(11) DEFAULT NULL > +) ENGINE=MyISAM DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin > +DROP TABLE tb1; > +DROP DATABASE db1; > +SET @@OLD_MODE = @save_old_mode; > +SET @@character_set_server = @save_character_set_server; > +SET @@character_set_client = @save_character_set_client; > +SET @@character_set_results = @save_character_set_results; > +SET @@character_set_connection = @save_character_set_connection; > +SET @@character_set_filesystem = @save_character_set_filesystem; > +SET @@character_set_database = @save_character_set_database; > +SET @@collation_connection = @save_collation_connection; > +SET @@collation_server = @save_collation_server; > +SET @@collation_database = @save_collation_database; > +# > +# End of 10.6 test > +# > diff --git a/mysys/charset.c b/mysys/charset.c > index 32cfeb56e2d..dbb2749d217 100644 > --- a/mysys/charset.c > +++ b/mysys/charset.c > @@ -763,22 +764,13 @@ get_charset_number_internal(const char *charset_name, uint cs_flags) > } > > > -static const char* > -get_charset_name_alias(const char *name) > -{ > - if (!my_strcasecmp(&my_charset_latin1, name, "utf8mb3")) > - return "utf8"; > - return NULL; > -} > - > - > uint get_charset_number(const char *charset_name, uint cs_flags) > { > uint id; > my_pthread_once(&charsets_initialized, init_available_charsets); > if ((id= get_charset_number_internal(charset_name, cs_flags))) > return id; > - if ((charset_name= get_charset_name_alias(charset_name))) > + if ((charset_name= !my_strcasecmp(&my_charset_latin1, charset_name, "utf8") ? "utf8mb3" : NULL)) Huh? Why do you not check MY_UTF8_IS_UTF8MB3 here? > return get_charset_number_internal(charset_name, cs_flags); > return 0; > } > @@ -820,7 +812,7 @@ static CHARSET_INFO *find_collation_data_inheritance_source(CHARSET_INFO *cs) > char name[MY_CS_NAME_SIZE + 1]; > memcpy(name, beg, end - beg); > name[end - beg]= '\0'; > - return inheritance_source_by_id(cs, get_collation_number(name)); > + return inheritance_source_by_id(cs, get_collation_number(name,MYF(0))); and not here? > } > return NULL; > } > @@ -961,7 +953,28 @@ my_collation_get_by_name(MY_CHARSET_LOADER *loader, > CHARSET_INFO *get_charset_by_name(const char *cs_name, myf flags) > { > MY_CHARSET_LOADER loader; > + my_bool utf8_is_utf8mb3= flags & MY_UTF8_IS_UTF8MB3 ? 1 : 0; > + char *copy_of_name= (char*)cs_name; > + char start[6], result[64]; > + char *temp_cs_name; > + > my_charset_loader_init_mysys(&loader); > + > + if (!strcasecmp("utf8",copy_of_name)) > + cs_name = (const char*)(utf8_is_utf8mb3 ? "utf8mb3" : "utf8mb4"); > + > + strncpy(start, cs_name, 5); > + temp_cs_name= (char *)(utf8_is_utf8mb3 ? "utf8mb3_":"utf8mb4_"); > + > + if (!strncasecmp("utf8_", start,5)) > + { > + copy_of_name+= 5; > + result[63]='\0'; > + strcpy(result, temp_cs_name); > + strcat(result, copy_of_name); > + result[strlen(copy_of_name)+strlen(temp_cs_name)]='\0'; > + cs_name= (const char *) result; > + } And why do you do all that ^^^ ? Old code didn't try to change utf8mb3 to utf8 here, because, I suppose, my_collation_get_by_name() below did all that. Why did you add alias resolution where none was? > return my_collation_get_by_name(&loader, cs_name, flags); > } > > @@ -1005,12 +1018,16 @@ get_charset_by_csname(const char *cs_name, uint cs_flags, myf flags) > { > MY_CHARSET_LOADER loader; > my_charset_loader_init_mysys(&loader); > + > + if (!strcasecmp("utf8",cs_name)) > + cs_name= (const char*)(flags & MY_UTF8_IS_UTF8MB3 ? "utf8mb3" : "utf8mb4"); same here > + > return my_charset_get_by_name(&loader, cs_name, cs_flags, flags); > } > > > /** > - Resolve character set by the character set name (utf8, latin1, ...). > + Resolve character set by the character set name (utf8mb3, latin1, ...). > > The function tries to resolve character set by the specified name. If > there is character set with the given name, it is assigned to the "cs" > @@ -1453,8 +1472,8 @@ static const MY_CSET_OS_NAME charsets[] = > > {"US-ASCII", "latin1", my_cs_approx}, > > - {"utf8", "utf8", my_cs_exact}, > - {"utf-8", "utf8", my_cs_exact}, > + {"utf8mb3", "utf8mb3", my_cs_exact}, eh, no. Try to understand what this array is for. > + {"utf-8", "utf8mb3", my_cs_exact}, > #endif > {NULL, NULL, 0} > }; > diff --git a/plugin/handler_socket/client/hslongrun.cpp b/plugin/handler_socket/client/hslongrun.cpp > index b7c02951340..7f88d48fff2 100644 > --- a/plugin/handler_socket/client/hslongrun.cpp > +++ b/plugin/handler_socket/client/hslongrun.cpp > @@ -897,7 +897,7 @@ hs_longrun_init_table(const config& conf, int num_prepare, > "v1 varchar(32) not null," > "v2 varchar(32) not null," > "v3 varchar(32) not null" > - ") character set utf8 collate utf8_bin engine = innodb"); > + ") character set utf8mb3 collate utf8_bin engine = innodb"); just keep it utf8 > for (int i = 0; i < num_prepare; ++i) { > const std::string i_str = to_stdstring(i); > const std::string v1 = "pv1_" + i_str; > diff --git a/plugin/handler_socket/client/hstest.pl b/plugin/handler_socket/client/hstest.pl > index 1363e153c44..5924d8a0ce5 100755 > --- a/plugin/handler_socket/client/hstest.pl > +++ b/plugin/handler_socket/client/hstest.pl > @@ -52,7 +52,7 @@ for my $action (@actions) { > "k $keytype primary key" . > ",v varchar(32) not null" . > $moreflds . > - ") character set utf8 collate utf8_bin " . > + ") character set utf8mb3 collate utf8_bin " . and here. forget about handlersocket > "engine = $engine"); > } elsif ($action eq "insert") { > print("INSERT $db.$table tablesize=$tablesize\n"); > diff --git a/plugin/win_auth_client/common.cc b/plugin/win_auth_client/common.cc > index 8b7319252ac..ddd34aec7da 100644 > --- a/plugin/win_auth_client/common.cc > +++ b/plugin/win_auth_client/common.cc > @@ -384,7 +384,7 @@ char* wchar_to_utf8(const wchar_t *string, size_t *len) > buf= (char*)malloc(buf_len + 1); > if (!buf) > { > - DBUG_PRINT("error",("Out of memory when converting string '%S' to utf8", > + DBUG_PRINT("error",("Out of memory when converting string '%S' to utf8mb3", Nope, see what this function is doing. > string)); > return NULL; > } > @@ -408,7 +408,7 @@ char* wchar_to_utf8(const wchar_t *string, size_t *len) > > #ifndef DBUG_OFF > Error_message_buf error_buf; > - DBUG_PRINT("error", ("Could not convert string '%S' to utf8" > + DBUG_PRINT("error", ("Could not convert string '%S' to utf8mb3" same > ", WideCharToMultiByte() failed with error %X (%s)", > string, GetLastError(), > get_last_error_message(error_buf))); > @@ -451,7 +451,7 @@ wchar_t* utf8_to_wchar(const char *string, size_t *len) > > if (!buf) > { > - DBUG_PRINT("error",("Out of memory when converting utf8 string '%s'" > + DBUG_PRINT("error",("Out of memory when converting utf8mb3 string '%s'" same > " to wide-char representation", string)); > return NULL; > } > diff --git a/scripts/fill_help_tables.sql b/scripts/fill_help_tables.sql > index d0efb750330..ad7c4fce9a4 100644 > --- a/scripts/fill_help_tables.sql > +++ b/scripts/fill_help_tables.sql > @@ -22,7 +22,7 @@ don't change help tables, please. they'll be regenerated from the documentation > > -- mysql -u root -p mysql < file_name > > -set names 'utf8'; > +set names 'utf8mb3'; > > set sql_log_bin = 0; > > diff --git a/scripts/mysql_system_tables.sql b/scripts/mysql_system_tables.sql > index e390f36a98b..7c8532577a1 100644 > --- a/scripts/mysql_system_tables.sql > +++ b/scripts/mysql_system_tables.sql > @@ -209,7 +209,7 @@ SET @create_transaction_registry="CREATE TABLE IF NOT EXISTS transaction_registr > UNIQUE KEY (commit_id), > INDEX (begin_timestamp), > INDEX (commit_timestamp, transaction_id) > -) ENGINE=INNODB DEFAULT CHARSET=utf8 COLLATE=utf8_bin STATS_PERSISTENT=0"; > +) ENGINE=INNODB DEFAULT CHARSET=utf8mb3 COLLATE=utf8_bin STATS_PERSISTENT=0"; here and in all other places in this file and other .sql files: you need to change the collation too, not just the charset. > > SET @str=IF(@have_innodb <> 0, @create_innodb_table_stats, "SET @dummy = 0"); > PREPARE stmt FROM @str; > diff --git a/sql/mysqld.cc b/sql/mysqld.cc > index 0bf21e02002..cc7568990b4 100644 > --- a/sql/mysqld.cc > +++ b/sql/mysqld.cc > @@ -4039,7 +4039,10 @@ static int init_common_variables() > *next_character_set_name++= '\0'; > if (!(default_charset_info= > get_charset_by_csname(default_character_set_name, > - MY_CS_PRIMARY, MYF(MY_WME)))) > + MY_CS_PRIMARY, > + global_system_variables.old_behavior & > + OLD_MODE_UTF8_IS_UTF8MB3 ? > + MYF(MY_UTF8_IS_UTF8MB3 | MY_WME) : MYF(MY_WME)))) may be you'd better get this multi-line assignment out of if() ? e.g. myf utf8_flag= global_system_variables.old_behavior & OLD_MODE_UTF8_IS_UTF8MB3 : 0; default_charset_info= get_charset_by_csname(default_character_set_name, MY_CS_PRIMARY, MYF(utf8_flag | MY_WME)); if (!default_charset_info) ... or add a helper thd->utf8_alias() as I suggested below > { > if (next_character_set_name) > { > @@ -4056,7 +4059,10 @@ static int init_common_variables() > if (default_collation_name) > { > CHARSET_INFO *default_collation; > - default_collation= get_charset_by_name(default_collation_name, MYF(0)); > + default_collation= get_charset_by_name(default_collation_name, > + global_system_variables.old_behavior & > + OLD_MODE_UTF8_IS_UTF8MB3 ? > + MYF(MY_UTF8_IS_UTF8MB3) : MYF(0)); and here you'll be able to use utf8_flag without the conditional operator > if (!default_collation) > { > #ifdef WITH_PERFSCHEMA_STORAGE_ENGINE > @@ -4097,7 +4103,10 @@ static int init_common_variables() > > if (!(character_set_filesystem= > get_charset_by_csname(character_set_filesystem_name, > - MY_CS_PRIMARY, MYF(MY_WME)))) > + MY_CS_PRIMARY, > + global_system_variables.old_behavior & > + OLD_MODE_UTF8_IS_UTF8MB3 ? > + MYF(MY_UTF8_IS_UTF8MB3 | MY_WME) : MYF(MY_WME)))) and here > return 1; > global_system_variables.character_set_filesystem= character_set_filesystem; > > @@ -7415,7 +7424,9 @@ static void usage(void) > DBUG_ENTER("usage"); > if (!(default_charset_info= get_charset_by_csname(default_character_set_name, > MY_CS_PRIMARY, > - MYF(MY_WME)))) > + global_system_variables.old_behavior & > + OLD_MODE_UTF8_IS_UTF8MB3 ? > + MYF(MY_UTF8_IS_UTF8MB3 | MY_WME) : MYF(MY_WME)))) split this one too? > exit(1); > if (!default_collation_name) > default_collation_name= (char*) default_charset_info->name; > diff --git a/sql/sql_class.h b/sql/sql_class.h > index 50b746fe514..15aa9ca8199 100644 > --- a/sql/sql_class.h > +++ b/sql/sql_class.h > @@ -1007,6 +1008,36 @@ inline void update_global_memory_status(int64 size) > my_atomic_add64_explicit(ptr, size, MY_MEMORY_ORDER_RELAXED); > } > > +inline const char* get_alias_collation_or_charset_name(const char* name, > + bool utf8_is_utf8mb3) > +{ > + char *copy_of_name= (char*)name; > + char start[6], result[64]; > + char *temp_cs_name; > + > + if (!strchr(name,'_')) > + { > + if (!strcasecmp("utf8",name)) > + name = utf8_is_utf8mb3 ? "utf8mb3" : "utf8mb4"; > + return name; > + } > + else > + { > + strncpy(start, name, 5); > + temp_cs_name= (char *)(utf8_is_utf8mb3 ? "utf8mb3_":"utf8mb4_"); > + if (!strncasecmp("utf8_", start,5)) > + { > + copy_of_name+= 5; > + result[63]='\0'; > + strcpy(result, temp_cs_name); > + strcat(result, copy_of_name); > + result[strlen(copy_of_name)+strlen(temp_cs_name)]='\0'; > + strcpy((char*)name,result); > + } > + } > + return name; > +} Please, no. First, you cannot just copy `result` into `name`, because `result` is longer, you'll overwrite whatever was in memory after name's value. Second, you don't need to resolve aliases here, charset code already does it, don't duplicate that. Just pass MY_UTF8_IS_UTF8MB3 down to my_collation_get_by_name() below. > + > /** > Get collation by name, send error to client on failure. > @param name Collation name > diff --git a/sql/sql_db.cc b/sql/sql_db.cc > index 9bf16220535..f471d8edc66 100644 > --- a/sql/sql_db.cc > +++ b/sql/sql_db.cc > @@ -583,9 +583,14 @@ bool load_db_opt(THD *thd, const char *path, Schema_specification_st *create) > default-collation commands. > */ > if (!(create->default_table_charset= > - get_charset_by_csname(pos+1, MY_CS_PRIMARY, MYF(0))) && > + get_charset_by_csname(pos+1, MY_CS_PRIMARY, > + thd->variables.old_behavior & > + OLD_MODE_UTF8_IS_UTF8MB3 ? > + MYF(MY_UTF8_IS_UTF8MB3) : MYF(0))) && > !(create->default_table_charset= > - get_charset_by_name(pos+1, MYF(0)))) > + get_charset_by_name(pos+1, thd->variables.old_behavior & > + OLD_MODE_UTF8_IS_UTF8MB3 ? > + MYF(MY_UTF8_IS_UTF8MB3) : MYF(0)))) again, please move myf flags manipulations out of if() > { > sql_print_error("Error while loading database options: '%s':",path); > sql_print_error(ER_THD(thd, ER_UNKNOWN_CHARACTER_SET),pos+1); > @@ -595,7 +600,9 @@ bool load_db_opt(THD *thd, const char *path, Schema_specification_st *create) > else if (!strncmp(buf,"default-collation", (pos-buf))) > { > if (!(create->default_table_charset= get_charset_by_name(pos+1, > - MYF(0)))) > + thd->variables.old_behavior & > + OLD_MODE_UTF8_IS_UTF8MB3 ? > + MYF(MY_UTF8_IS_UTF8MB3) : MYF(0)))) and here > { > sql_print_error("Error while loading database options: '%s':",path); > sql_print_error(ER_THD(thd, ER_UNKNOWN_COLLATION),pos+1); > diff --git a/sql/sql_lex.cc b/sql/sql_lex.cc > index 6871699dc5b..ad322eda097 100644 > --- a/sql/sql_lex.cc > +++ b/sql/sql_lex.cc > @@ -2789,7 +2789,10 @@ int Lex_input_stream::scan_ident_middle(THD *thd, Lex_ident_cli_st *str, > body_utf8_append(m_cpp_text_start, m_cpp_tok_start + length); > ErrConvString csname(str->str + 1, str->length - 1, &my_charset_bin); > CHARSET_INFO *cs= get_charset_by_csname(csname.ptr(), > - MY_CS_PRIMARY, MYF(0)); > + MY_CS_PRIMARY, > + thd->variables.old_behavior & > + OLD_MODE_UTF8_IS_UTF8MB3 ? > + MYF(MY_UTF8_IS_UTF8MB3) : MYF(0)); may be it could be a helper in thd, like THD::utf8_alias() { return variables.old_behavior & OLD_MODE_UTF8_IS_UTF8MB3 ? MY_UTF8_IS_UTF8MB3 : 0; } > if (cs) > { > *introducer= cs; > diff --git a/storage/connect/mysql-test/connect/my.cnf b/storage/connect/mysql-test/connect/my.cnf > index 6310772d01f..83f0aa8ab30 100644 > --- a/storage/connect/mysql-test/connect/my.cnf > +++ b/storage/connect/mysql-test/connect/my.cnf > @@ -14,4 +14,4 @@ MASTER_MYSOCK= @mysqld.1.socket > SLAVE_MYPORT= @mysqld.2.port > SLAVE_MYSOCK= @mysqld.2.socket > > -PGCLIENTENCODING= UTF8 > +PGCLIENTENCODING= UTF8MB3 eh... really? Are you sure you've tested it and it worked? by the name of it I suspect it's a postgresql client encoding. > diff --git a/storage/connect/mysql-test/connect/t/odbc_postgresql.sql b/storage/connect/mysql-test/connect/t/odbc_postgresql.sql > index 1c302294393..3c78120a7a2 100644 > --- a/storage/connect/mysql-test/connect/t/odbc_postgresql.sql > +++ b/storage/connect/mysql-test/connect/t/odbc_postgresql.sql > @@ -4,13 +4,13 @@ > -- Run this script as a admin user: > -- psql -U postgres < odbc_postgresql.sql > > -SET NAMES 'UTF8'; > +SET NAMES 'UTF8MB3'; same, postgresql > > DROP DATABASE IF EXISTS mtr; > DROP USER IF EXISTS mtr; > > CREATE USER mtr WITH PASSWORD 'mtr'; > -CREATE DATABASE mtr OWNER=mtr ENCODING='UTF8'; > +CREATE DATABASE mtr OWNER=mtr ENCODING='UTF8MB3'; same > GRANT ALL ON DATABASE mtr TO mtr; > \c mtr > SET role mtr; > diff --git a/storage/innobase/fts/fts0opt.cc b/storage/innobase/fts/fts0opt.cc > index e5164fcc4fa..a26a05c2cf9 100644 > --- a/storage/innobase/fts/fts0opt.cc > +++ b/storage/innobase/fts/fts0opt.cc > @@ -330,7 +330,7 @@ fts_word_t* > fts_word_init( > /*==========*/ > fts_word_t* word, /*!< in: word to initialize */ > - byte* utf8, /*!< in: UTF-8 string */ > + byte* utf8mb3, /*!< in: UTF-8 string */ don't rename variables, please > ulint len) /*!< in: length of string in bytes */ > { > mem_heap_t* heap = mem_heap_create(sizeof(fts_node_t)); > diff --git a/storage/mroonga/mysql-test/mroonga/storage/r/collation_utf8_unicode_ci_french.result b/storage/mroonga/mysql-test/mroonga/storage/r/collation_utf8_unicode_ci_french.result > index 3f24de87035..3659aa5aee8 100644 > --- a/storage/mroonga/mysql-test/mroonga/storage/r/collation_utf8_unicode_ci_french.result > +++ b/storage/mroonga/mysql-test/mroonga/storage/r/collation_utf8_unicode_ci_french.result > @@ -7,5 +7,4 @@ FULLTEXT INDEX (content) > INSERT INTO diaries VALUES ("Je suis un garçon."); > SELECT * FROM diaries WHERE MATCH (content) AGAINST ("garcon"); > content > -Je suis un garçon. looks like a bug > DROP TABLE diaries; > diff --git a/storage/mroonga/mysql-test/mroonga/storage/r/collation_utf8_unicode_ci_japanese.result b/storage/mroonga/mysql-test/mroonga/storage/r/collation_utf8_unicode_ci_japanese.result > index 94ef2608b81..79dac1e63a7 100644 > --- a/storage/mroonga/mysql-test/mroonga/storage/r/collation_utf8_unicode_ci_japanese.result > +++ b/storage/mroonga/mysql-test/mroonga/storage/r/collation_utf8_unicode_ci_japanese.result > @@ -7,5 +7,4 @@ FULLTEXT INDEX (content) > INSERT INTO diaries VALUES ("ひらがなとカタカナを覚えました。"); > SELECT * FROM diaries WHERE MATCH (content) AGAINST ("かたかな"); > content > -ひらがなとカタカナを覚えました。 that too > DROP TABLE diaries; > diff --git a/storage/mroonga/vendor/groonga/CMakeLists.txt b/storage/mroonga/vendor/groonga/CMakeLists.txt > index d271d4c4eb9..fc134b81cde 100644 > --- a/storage/mroonga/vendor/groonga/CMakeLists.txt > +++ b/storage/mroonga/vendor/groonga/CMakeLists.txt > @@ -268,7 +268,7 @@ if(UNIX) > ac_check_funcs(pthread_condattr_setpshared) > endif() > > -option(GRN_WITH_NFKC "use NFKC based UTF8 normalization." ON) > +option(GRN_WITH_NFKC "use NFKC based UTF8MB3 normalization." ON) not here, please > > if(WIN32) > ac_check_headers(winsock2.h) > diff --git a/storage/mroonga/vendor/groonga/benchmark/bench-nfkc.c b/storage/mroonga/vendor/groonga/benchmark/bench-nfkc.c > index ebae95b273b..3997b933e87 100644 > --- a/storage/mroonga/vendor/groonga/benchmark/bench-nfkc.c > +++ b/storage/mroonga/vendor/groonga/benchmark/bench-nfkc.c > @@ -83,11 +83,11 @@ static void > bench_char_type(gpointer user_data) > { > uint64_t code_point; > - char utf8[7]; > + char utf8mb3[7]; don't rename variables > > for (code_point = 1; code_point < MAX_UNICODE; code_point++) { > - ucs2utf8(code_point, (unsigned char *)utf8); > - grn_nfkc50_char_type(utf8); > + ucs2utf8(code_point, (unsigned char *)utf8mb3); > + grn_nfkc50_char_type(utf8mb3); > } > } > > diff --git a/storage/mroonga/vendor/groonga/vendor/plugins/groonga-normalizer-mysql/tool/dump_difference_utf8.rb b/storage/mroonga/vendor/groonga/vendor/plugins/groonga-normalizer-mysql/tool/dump_difference_utf8.rb > index 4b6fde8c7b0..ce20a2c5b40 100644 > --- a/storage/mroonga/vendor/groonga/vendor/plugins/groonga-normalizer-mysql/tool/dump_difference_utf8.rb > +++ b/storage/mroonga/vendor/groonga/vendor/plugins/groonga-normalizer-mysql/tool/dump_difference_utf8.rb > @@ -36,8 +36,8 @@ parser.sorted_pages.each do |page, characters| > next if base == sort > n_differences += 1 > utf8s = [base, upper, lower, sort] > - formatted_code_points = utf8s.collect do |utf8| > - "%#07x" % Unicode.from_utf8(utf8) > + formatted_code_points = utf8s.collect do |utf8mb3| > + "%#07x" % Unicode.from_utf8(utf8mb3) just revert all changes under storage/mroonga/vendor/* > end > if sort.bytesize > base.bytesize > n_expanded_sort_characters += 1 > diff --git a/storage/sphinx/mysql-test/sphinx/my.cnf b/storage/sphinx/mysql-test/sphinx/my.cnf > index f60380b7171..22cc06914f4 100644 > --- a/storage/sphinx/mysql-test/sphinx/my.cnf > +++ b/storage/sphinx/mysql-test/sphinx/my.cnf > @@ -7,7 +7,7 @@ xmlpipe_command = cat @ENV.MTR_SUITE_DIR/testdata.xml > [index test1] > source = src1 > docinfo = extern > -charset_type = utf-8 > +charset_type = utf-8mb3 revert > path = @ENV.MYSQLTEST_VARDIR/searchd/test1 > > [indexer] > diff --git a/storage/spider/mysql-test/spider/bg/my.cnf b/storage/spider/mysql-test/spider/bg/my.cnf > index 246099c623e..39f5bd01c67 100644 > --- a/storage/spider/mysql-test/spider/bg/my.cnf > +++ b/storage/spider/mysql-test/spider/bg/my.cnf > @@ -75,15 +75,15 @@ MASTER_1_MYSOCK= @mysqld.1.1.socket > MASTER_1_ENGINE_TYPE= Spider > #MASTER_1_ENGINE_TYPE= MyISAM > MASTER_1_ENGINE= ENGINE=Spider > -MASTER_1_CHARSET= DEFAULT CHARSET=utf8 > +MASTER_1_CHARSET= DEFAULT CHARSET=utf8mb3 Make sure to update both charset and collation here and in all other .cnf files in the spider suite > MASTER_1_ENGINE2= ENGINE=MyISAM > -MASTER_1_CHARSET2= DEFAULT CHARSET=utf8 > -MASTER_1_CHARSET3= DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci > +MASTER_1_CHARSET2= DEFAULT CHARSET=utf8mb3 > +MASTER_1_CHARSET3= DEFAULT CHARSET=utf8mb3 COLLATE=utf8_unicode_ci > SLAVE1_1_MYPORT= @mysqld.4.1.port > SLAVE1_1_MYSOCK= @mysqld.4.1.socket > SLAVE1_1_ENGINE_TYPE= MyISAM > SLAVE1_1_ENGINE= ENGINE=MyISAM > -SLAVE1_1_CHARSET= DEFAULT CHARSET=utf8 > +SLAVE1_1_CHARSET= DEFAULT CHARSET=utf8mb3 > USE_CHILD_GROUP2= 1 > OUTPUT_CHILD_GROUP2= 0 > CHILD2_1_MYPORT= @mysqld.2.1.port > diff --git a/storage/spider/spd_init_query.h b/storage/spider/spd_init_query.h > index 19b04d50b82..f12cef377e2 100644 > --- a/storage/spider/spd_init_query.h > +++ b/storage/spider/spd_init_query.h > @@ -559,7 +559,7 @@ static LEX_STRING spider_init_queries[] = { > " table_name char(64) not null default ''," > " primary key (table_id)," > " unique uk1(db_name, table_name)" > - " ) engine=Aria transactional=1 default charset=utf8 collate=utf8_bin;" > + " ) engine=Aria transactional=1 default charset=utf8mb3 collate=utf8_bin;" always change both charset and collation (everywhere in this file) > " create table if not exists mysql.spider_rewrite_table_tables(" > " table_id bigint unsigned not null," > " partition_id bigint unsigned not null auto_increment," > diff --git a/tests/mysql_client_test.c b/tests/mysql_client_test.c > index 0043786d477..5d9213591fb 100644 > --- a/tests/mysql_client_test.c > +++ b/tests/mysql_client_test.c > @@ -19236,7 +19236,7 @@ static void test_bug12337762() > rc= mysql_query(mysql, "create table charset_tab("\ > "txt1 varchar(32) character set Latin1,"\ > "txt2 varchar(32) character set Latin1 collate latin1_bin,"\ > - "txt3 varchar(32) character set utf8 collate utf8_bin"\ > + "txt3 varchar(32) character set utf8mb3 collate utf8_bin"\ both charset and collation > ")"); > > DIE_UNLESS(rc == 0); Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

[Maria-developers] Old buildbot 10.5 branch failure review
by Otto Kekäläinen 19 Mar '21

19 Mar '21

Hello! I noticed there are quite a lot of upgrade test failures on the 10.5 branch, which is a stable release and should definitely be all green all the time. Here is quick review of those issues with suggested fixes. Maybe Daniel or Elena can do them? They are all easy ones and would make 7 currently red CI steps turn into green. ## Overview: http://buildbot.askmonty.org/buildbot/grid?branch=10.5&category=main&catego… ## kvm-deb-buster-amd6 minor-upgrade-all (http://buildbot.askmonty.org/buildbot/builders/kvm-deb-buster-amd64/builds/…) ++ diff -U1000 /home/buildbot/mariadb.org-tools-master/buildbot/baselines/ldd.10.5.buster.amd64 /home/buildbot/ldd.new --- /home/buildbot/mariadb.org-tools-master/buildbot/baselines/ldd.10.5.buster.amd64 2021-03-16 21:03:04.000000000 -0400 +++ /home/buildbot/ldd.new 2021-03-17 17:23:42.066634818 -0400 - liblz4.so.1 => /usr/lib/x86_64-linux-gnu/liblz4.so.1 Seems some dependency stopped including liblz4. This didn't affect MariaDB. This is a valid change and the fix is to "reset" the ldd.10.5.buster.amd64 file on buildbot.askmonty.org so that there would be no diff no more. ## kvm-deb-buster-amd64 minor-upgrade-all - liblz4.so.1 => /usr/lib/x86_64-linux-gnu/liblz4.so.1 Same as above. ## kvm-deb-sid-x86 - minor-upgrade-all and ## kvm-deb-sid-amd64 - minor-upgrade-all: - libhogweed.so.5 => /lib/x86_64-linux-gnu/libhogweed.so.5 + libhogweed.so.6 => /lib/x86_64-linux-gnu/libhogweed.so.6 - libnettle.so.7 => /lib/x86_64-linux-gnu/libnettle.so.7 + libnettle.so.8 => /lib/x86_64-linux-gnu/libnettle.so.8 Similar issue to buster minor-upgrade-all failure. This is a valid dependency change and the fix is to "reset" the ldd.10.5.sid.amd64 and ldd.10.5.sid.x86 filse on buildbot.askmonty.org so that there would be no diff no more. ## kvm-zyp-opensuse150-amd64 minor-upgrade-all: --- /tmp/ldd.baseline 2021-03-17 19:42:17.468328791 +0000 +++ /home/buildbot/ldd.new 2021-03-17 19:42:16.928328910 +0000 === /usr/sbin/mysqld === /usr/bin/mysql + libedit.so.0 => /usr/lib64/libedit.so.0 === /usr/bin/mysqlaccess === /usr/bin/mysqladmin === /usr/bin/mysqlbinlog === /usr/bin/mysqlcheck === /usr/bin/mysql_client_test === /usr/bin/mysql_client_test_embedded === /usr/bin/mysql_config === /usr/bin/mysql_convert_table_format === /usr/bin/mysqld_multi === /usr/bin/mysqld_safe === /usr/bin/mysqld_safe_helper === /usr/bin/mysqldump === /usr/bin/mysqldumpslow === /usr/bin/mysql_embedded + libedit.so.0 => /usr/lib64/libedit.so.0 Seems legit, library has correct path now. Fix by "resetting" the ldd.baseline. # kvm-zyp-sles150-amd64 minor-upgrade-all: --- /tmp/ldd.baseline 2021-03-17 20:59:56.412361298 +0000 +++ /home/buildbot/ldd.new 2021-03-17 20:59:56.024361282 +0000 === /usr/sbin/mysqld - libpcre2-8.so.0 => /usr/lib64/libpcre2-8.so.0 === /usr/bin/mysql === /usr/bin/mysqlaccess === /usr/bin/mysqladmin === /usr/bin/mysqlbinlog === /usr/bin/mysqlcheck === /usr/bin/mysql_client_test === /usr/bin/mysql_client_test_embedded - libpcre2-8.so.0 => /usr/lib64/libpcre2-8.so.0 === /usr/bin/mysql_config === /usr/bin/mysql_convert_table_format === /usr/bin/mysqld_multi === /usr/bin/mysqld_safe === /usr/bin/mysqld_safe_helper === /usr/bin/mysqldump === /usr/bin/mysqldumpslow === /usr/bin/mysql_embedded - libpcre2-8.so.0 => /usr/lib64/libpcre2-8.so.0 === /usr/bin/mysql_find_rows === /usr/bin/mysql_fix_extensions === /usr/bin/mysqlhotcopy === /usr/bin/mysqlimport === /usr/bin/mysql_install_db === /usr/bin/mysql_ldb === /usr/bin/mysql_plugin === /usr/bin/mysql_secure_installation === /usr/bin/mysql_setpermission === /usr/bin/mysqlshow === /usr/bin/mysqlslap === /usr/bin/mysqltest - libpcre2-8.so.0 => /usr/lib64/libpcre2-8.so.0 - libpcre2-posix.so.2 => /usr/lib64/libpcre2-posix.so.2 === /usr/bin/mysqltest_embedded - libpcre2-8.so.0 => /usr/lib64/libpcre2-8.so.0 - libpcre2-posix.so.2 => /usr/lib64/libpcre2-posix.so.2 Seems legit, library has correct path now. Fix by "resetting" the ldd.baseline. ## kvm-deb-sid-amd64 minor-upgrade-columnstore: The following packages have unmet dependencies: mariadb-plugin-columnstore : Depends: mariadb-server-10.5 (= 1:10.5.8+maria~sid) but 1:10.5.9+maria~sid is to be installed E: Unable to correct problems, you have held broken packages. The mariadb-plugin-columnstore binary packages don't exist for kvm-deb-sid-amd64, as they were skipped when making the 10.5.9 release. Either build them or disable the test until 10.5.10 is out and they are again built. By disabling the test there would be less "red" and less time wasted by developers checking the red ones on why their commits (potentially) fail on CI. Thanks!

4 3

[Maria-developers] 0038c24f2d3: MDEV-16146: MariaDB slave stops with following errors.
by sujatha 18 Mar '21

18 Mar '21

revision-id: 0038c24f2d388b9b0f3bff9219a53c8900153d02 (mariadb-10.5.4-539-g0038c24f2d3) parent(s): 190a8312f598fc4892331225104297f6288f23ac author: Sujatha committer: Sujatha timestamp: 2021-03-18 18:41:30 +0530 message: MDEV-16146: MariaDB slave stops with following errors. Problem: ======== 180511 11:07:58 [ERROR] Slave I/O: Unexpected master's heartbeat data: heartbeat is not compatible with local info;the event's data: log_file_name mysql-bin.000009 log_pos 1054262041, Error_code: 1623 Analysis: ========= In replication setup when master server doesn't have any events to send to slave server it sends an 'Heartbeat_log_event'. This event carries the current binary log filename and offset details. The offset values is stored within 4 bytes of event header. When the size of binary log is higher than UINT32_MAX the log_pos values will not fit in 4 bytes memory. It overflows and hence slave stops with an error. Fix: === Since we cannot extend the fixed header of Log_event class an additional header named 'extra_header' is introduced. 'extra_header' contains HB_FLAGS - 2 bytes HB_LONG_LOG_POS_OFFSET - 8 bytes This 'extra_header' is added only in a case where log_pos > UINT32_MAX and 'HB_LONG_LOG_POS_OFFSET_F' will be enabled. On slave while reading the 'Heartbeat_log_event' if 'HB_LONG_LOG_POS_OFFSET_F' is found to be set then value of 'log_pos' is extracted from 'extra_header'. --- .../suite/rpl/r/rpl_incompatible_heartbeat.result | 20 +++++++++ .../suite/rpl/t/rpl_incompatible_heartbeat.test | 47 ++++++++++++++++++++++ sql/log_event.h | 19 ++++++++- sql/log_event_server.cc | 8 +++- sql/slave.cc | 14 +++++++ sql/sql_repl.cc | 26 ++++++++++-- 6 files changed, 128 insertions(+), 6 deletions(-) diff --git a/mysql-test/suite/rpl/r/rpl_incompatible_heartbeat.result b/mysql-test/suite/rpl/r/rpl_incompatible_heartbeat.result new file mode 100644 index 00000000000..fe2b0436f2d --- /dev/null +++ b/mysql-test/suite/rpl/r/rpl_incompatible_heartbeat.result @@ -0,0 +1,20 @@ +include/master-slave.inc +[connection master] +connection master; +SET @saved_dbug = @@GLOBAL.debug_dbug; +SET @@global.debug_dbug= '+d,simulate_pos_4G'; +connection slave; +include/stop_slave.inc +SET @saved_dbug = @@GLOBAL.debug_dbug; +SET @@global.debug_dbug= '+d,simulate_pos_4G'; +CHANGE MASTER TO MASTER_HEARTBEAT_PERIOD=0.001; +include/start_slave.inc +connection master; +SET @@GLOBAL.debug_dbug = @saved_dbug; +connection slave; +SET @@GLOBAL.debug_dbug = @saved_dbug; +connection master; +CREATE TABLE t (f INT) ENGINE=INNODB; +INSERT INTO t VALUES (10); +DROP TABLE t; +include/rpl_end.inc diff --git a/mysql-test/suite/rpl/t/rpl_incompatible_heartbeat.test b/mysql-test/suite/rpl/t/rpl_incompatible_heartbeat.test new file mode 100644 index 00000000000..b1aa5435ff4 --- /dev/null +++ b/mysql-test/suite/rpl/t/rpl_incompatible_heartbeat.test @@ -0,0 +1,47 @@ +# ==== Purpose ==== +# +# Test verifies that slave IO thread can process heartbeat events with log_pos +# values higher than UINT32_MAX. +# +# ==== Implementation ==== +# +# Steps: +# 0 - Stop slave threads. Configure a small master_heartbeat_period. +# 1 - Using debug points, simulate a huge binlog offset higher than +# UINT32_MAX on master. +# 2 - Start the slave and observe that slave IO thread is able to process +# the offset received through heartbeat event. +# +# ==== References ==== +# +# MDEV-16146: MariaDB slave stops with incompatible heartbeat +# +--source include/have_debug.inc +--source include/have_innodb.inc +--source include/have_binlog_format_mixed.inc +# Test simulates binarylog offsets higher than UINT32_MAX +--source include/have_64bit.inc +--source include/master-slave.inc + +--connection master +SET @saved_dbug = @@GLOBAL.debug_dbug; +SET @@global.debug_dbug= '+d,simulate_pos_4G'; + +--connection slave +--source include/stop_slave.inc +SET @saved_dbug = @@GLOBAL.debug_dbug; +SET @@global.debug_dbug= '+d,simulate_pos_4G'; +CHANGE MASTER TO MASTER_HEARTBEAT_PERIOD=0.001; +--source include/start_slave.inc + +--connection master +sleep 1; +SET @@GLOBAL.debug_dbug = @saved_dbug; +--sync_slave_with_master +SET @@GLOBAL.debug_dbug = @saved_dbug; + +--connection master +CREATE TABLE t (f INT) ENGINE=INNODB; +INSERT INTO t VALUES (10); +DROP TABLE t; +--source include/rpl_end.inc diff --git a/sql/log_event.h b/sql/log_event.h index 4e193232f4b..72d8766cea5 100644 --- a/sql/log_event.h +++ b/sql/log_event.h @@ -576,6 +576,22 @@ class String; #define MARIA_SLAVE_CAPABILITY_MINE MARIA_SLAVE_CAPABILITY_GTID +#define HB_EXTRA_HEADER_LEN 10 +/* + Extra header contains flags specific to Heartbeat_log_event and + respective data. The length needs to be updated when new memebers are + added to this header +*/ +#define HB_FLAGS_OFFSET 0 +/* + When the size of 'log_pos' within Heartbeat_log_event exceeds UINT_MAX32 it + cannot be stored in standard header as 'log_pos' is of 4 bytes size. Hence + extra_header is introduced. First 2 bytes represent flags. In case of long + 'log_pos' value 'HB_LONG_LOG_POS_OFFSET_F' bit within the flag will be set. + The log_pos is stored witin 'long_log_pos' variables. +*/ +#define HB_LONG_LOG_POS_OFFSET_F 0x1 +#define HB_LONG_LOG_POS_OFFSET 2 /** @enum Log_event_type @@ -5718,7 +5734,8 @@ bool copy_cache_to_file_wrapped(IO_CACHE *body, class Heartbeat_log_event: public Log_event { public: - Heartbeat_log_event(const char* buf, uint event_len, + uint16 hb_flags; + Heartbeat_log_event(const char* buf, ulong event_len, const Format_description_log_event* description_event); Log_event_type get_type_code() { return HEARTBEAT_LOG_EVENT; } bool is_valid() const diff --git a/sql/log_event_server.cc b/sql/log_event_server.cc index 6dee9b9adf6..2701f4f2f9a 100644 --- a/sql/log_event_server.cc +++ b/sql/log_event_server.cc @@ -8493,14 +8493,18 @@ void Ignorable_log_event::pack_info(Protocol *protocol) #if defined(HAVE_REPLICATION) -Heartbeat_log_event::Heartbeat_log_event(const char* buf, uint event_len, +Heartbeat_log_event::Heartbeat_log_event(const char* buf, ulong event_len, const Format_description_log_event* description_event) :Log_event(buf, description_event) { + uint16 hb_flags; uint8 header_size= description_event->common_header_len; - ident_len = event_len - header_size; + ident_len = event_len - (header_size + HB_EXTRA_HEADER_LEN); set_if_smaller(ident_len,FN_REFLEN-1); log_ident= buf + header_size; + hb_flags= uint2korr(buf + header_size + ident_len); + if (hb_flags & HB_LONG_LOG_POS_OFFSET_F) + log_pos= uint8korr(buf + header_size + ident_len + HB_LONG_LOG_POS_OFFSET); } #endif diff --git a/sql/slave.cc b/sql/slave.cc index 1da030084ef..f365a1e7b5c 100644 --- a/sql/slave.cc +++ b/sql/slave.cc @@ -6493,6 +6493,12 @@ static int queue_event(Master_info* mi,const char* buf, ulong event_len) TODO: handling `when' for SHOW SLAVE STATUS' snds behind */ + DBUG_EXECUTE_IF("simulate_pos_4G", + { + inc_pos= mi->master_log_pos; // temp_save + mi->master_log_pos= ((ulong)2394967295); + };); + if (memcmp(mi->master_log_name, hb.get_log_ident(), hb.get_ident_len()) || mi->master_log_pos > hb.log_pos) { /* missed events of heartbeat from the past */ @@ -6504,6 +6510,14 @@ static int queue_event(Master_info* mi,const char* buf, ulong event_len) error_msg.append_ulonglong(hb.log_pos); goto err; } + DBUG_EXECUTE_IF("simulate_pos_4G", + { + if (hb.log_pos > UINT32_MAX) + { + mi->master_log_pos= inc_pos; // restore + DBUG_SET("-d, simulate_pos_4G"); + } + };); /* Heartbeat events doesn't count in the binlog size, so we don't have to diff --git a/sql/sql_repl.cc b/sql/sql_repl.cc index 010ea794254..d68ae05b60e 100644 --- a/sql/sql_repl.cc +++ b/sql/sql_repl.cc @@ -824,12 +824,14 @@ get_slave_until_gtid(THD *thd, String *out_str) */ static int send_heartbeat_event(binlog_send_info *info, NET* net, String* packet, - const struct event_coordinates *coord, + struct event_coordinates *coord, enum enum_binlog_checksum_alg checksum_alg_arg) { DBUG_ENTER("send_heartbeat_event"); ulong ev_offset; + char extra_buf[HB_EXTRA_HEADER_LEN]; + uint16 hb_flags= 0; if (reset_transmit_packet(info, info->flags, &ev_offset, &info->errmsg)) DBUG_RETURN(1); @@ -850,19 +852,30 @@ static int send_heartbeat_event(binlog_send_info *info, size_t event_len = ident_len + LOG_EVENT_HEADER_LEN + (do_checksum ? BINLOG_CHECKSUM_LEN : 0); int4store(header + SERVER_ID_OFFSET, global_system_variables.server_id); + if (coord->pos <= UINT32_MAX) + { + int4store(header + LOG_POS_OFFSET, coord->pos); // log_pos + } + else + { + hb_flags|= HB_LONG_LOG_POS_OFFSET_F; + int2store(extra_buf + HB_FLAGS_OFFSET, hb_flags); + int8store(extra_buf + HB_LONG_LOG_POS_OFFSET, coord->pos); + event_len+= HB_EXTRA_HEADER_LEN; + } int4store(header + EVENT_LEN_OFFSET, event_len); int2store(header + FLAGS_OFFSET, 0); - int4store(header + LOG_POS_OFFSET, coord->pos); // log_pos - packet->append(header, sizeof(header)); packet->append(p, ident_len); // log_file_name + packet->append(extra_buf, sizeof(extra_buf)); if (do_checksum) { char b[BINLOG_CHECKSUM_LEN]; ha_checksum crc= my_checksum(0, (uchar*) header, sizeof(header)); crc= my_checksum(crc, (uchar*) p, ident_len); + crc= my_checksum(crc, (uchar*) extra_buf, sizeof(extra_buf)); int4store(b, crc); packet->append(b, sizeof(b)); } @@ -2525,6 +2538,13 @@ static int wait_new_events(binlog_send_info *info, /* in */ } #endif mysql_bin_log.unlock_binlog_end_pos(); + + DBUG_EXECUTE_IF("simulate_pos_4G", + { + coord.pos= ((ulong)5394967295); + DBUG_SET("-d, simulate_pos_4G"); + };); + ret= send_heartbeat_event(info, info->net, info->packet, &coord, info->current_checksum_alg);

1 0

Re: [Maria-developers] 9acabefd276: MDEV-22010: use executables MariaDB named in scripts
by Sergei Golubchik 17 Mar '21

17 Mar '21

Hi, Rucha! On Mar 17, Rucha Deodhar wrote: > > mysql-test-run and mysql-stress-test was also supposed to be renamed as > part of this task. > I need to check again if debs and rpms package symlink. > > > > > > diff --git a/mysql-test/.mtr b/mysql-test/.mtr > > > new file mode 120000 > > > index 00000000000..68986c8cd7e > > > --- /dev/null > > > +++ b/mysql-test/.mtr > > > @@ -0,0 +1 @@ > > > +./mariadb-test-run > > > > What is that? What is .mtr file? > > I think after i built it got committed along with other files. please remove it from the commit > > > diff --git a/mysql-test/mysql-stress-test.pl b/mysql-test/ > > mysql-stress-test.pl > > > new file mode 120000 > > > index 00000000000..e704a5d70bf > > > --- /dev/null > > > +++ b/mysql-test/mysql-stress-test.pl > > > @@ -0,0 +1 @@ > > > +./mariadb-stress-test.pl > > > > looks like you've checked in files you shouldn't have. Like symlinks. > > As mysql-test-run and mysql-stress-test was supposed to be renamed too. So, > I created a symlink > so that ./mysql-test-run and ./mysql-stress-test would still work if > anybody uses it. symlinks - yes. But they are created by cmake during the build, they should not be added to a commit. please, remove. You might want to add them to the .gitignore file though. > > > \ No newline at end of file > > > diff --git a/scripts/msql2mysql.sh b/scripts/msql2mysql.sh > > > index 72a609fa6e7..11923fa15c7 100644 > > > --- a/scripts/msql2mysql.sh > > > +++ b/scripts/msql2mysql.sh > > > @@ -16,4 +16,4 @@ > > > # Free Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston > > > # MA 02110-1335 USA. > > > > > > -@bindir@/replace msqlConnect mysql_connect msqlListDBs mysql_list_dbs > > msqlNumRows mysql_num_rows msqlFetchRow mysql_fetch_row msqlFetchField > > mysql_fetch_field msqlFreeResult mysql_free_result msqlListFields > > mysql_list_fields msqlListTables mysql_list_tables msqlErrMsg > > 'mysql_error(mysql)' msqlStoreResult mysql_store_result msqlQuery > > mysql_query msqlField mysql_field msqlSelect mysql_select msqlSelectDB > > mysql_select_db msqlNumFields mysql_num_fields msqlClose mysql_close > > msqlDataSeek mysql_data_seek m_field MYSQL_FIELD m_result MYSQL_RES m_row > > MYSQL_ROW msql mysql mSQL mySQL MSQL MYSQL msqlCreateDB mysql_create_db > > msqlDropDB mysql_drop_db msqlFieldSeek mysql_field_seek -- $* > > > +@bindir@/replace msqlConnect mysql_connect msqlListDBs mysql_list_dbs > > msqlNumRows mysql_num_rows msqlFetchRow mysql_fetch_row msqlFetchField > > mysql_fetch_field msqlFreeResult mysql_free_result msqlListFields > > mysql_list_fields msqlListTables mysql_list_tables msqlErrMsg > > 'mysql_error(mysql)' msqlStoreResult mysql_store_result msqlQuery > > mysql_query msqlField mysql_field msqlSelect mysql_select msqlSelectDB > > mysql_select_db msqlNumFields mysql_num_fields msqlClose mysql_close > > msqlDataSeek mysql_data_seek m_field MYSQL_FIELD m_result MYSQL_RES m_row > > MYSQL_ROW msql mysql mysql mariadb mSQL mySQL MSQL MYSQL msqlCreateDB > > mysql_create_db msqlDropDB mysql_drop_db msqlFieldSeek mysql_field_seek -- > > $* > > > > Why? > > So that mysql in bin directory would get replaced with mariadb no, that's not it. See other renames, msqlListTables -> mysql_list_tables, msqlFetchRow -> mysql_fetch_row etc this (prehistoric) script was supposed to help migrating programs that work with mSQL (https://en.wikipedia.org/wiki/MSQL) to MySQL by renaming C API calls in the source code. In short, revert this change, it doesn't belong to this script. > > > diff --git a/scripts/mysql_install_db.sh b/scripts/mysql_install_db.sh > > > index 5f183afe8fc..084af42d01c 100644 > > > --- a/scripts/mysql_install_db.sh > > > +++ b/scripts/mysql_install_db.sh > > > @@ -338,11 +338,15 @@ then > > > cannot_find_file resolveip @resolveip_locations@ > > > exit 1 > > > fi > > > - mysqld=`find_in_dirs mysqld @mysqld_locations@` > > > + mysqld=`find_in_dirs mariadbd @mysqld_locations@` > > > if test -z "$mysqld" > > > then > > > - cannot_find_file mysqld @mysqld_locations@ > > > - exit 1 > > > + mysqld=`find_in_dirs mysqld @mysqld_locations@` > > > + if test -z "$mysqld" > > > + then > > > + cannot_find_file mysqld @mysqld_locations@ > > > > Hmm. At least, it should be mariadbd in the cannot_find_file line. > > > > Why do you search for mysqld after mariadbd wasn't found? > > > I added it for additional check, just in case there is mysqld and not > mariadbd or if it could affect compatibility. I don't think it could. These scripts you've edited are installed together with MariaDB, there is no supported configuration where these scripts exist and mariadb* binaries don't. That is, I believe this extra check is not needed. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 9acabefd276: MDEV-22010: use executables MariaDB named in scripts
by Sergei Golubchik 17 Mar '21

17 Mar '21

Hi, Rucha! On Mar 17, Rucha Deodhar wrote: > revision-id: 9acabefd276 (mariadb-10.5.2-386-g9acabefd276) > parent(s): de407e7cb4d > author: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > committer: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > timestamp: 2021-02-11 15:19:29 +0530 > message: > > MDEV-22010: use executables MariaDB named in scripts > > As a part of this MDEV following changes were made: > 1) Mariadb named executables used instead of mysql named executables in scripts > 2) renamed mysql-test-run and mysql-stress-test to mariadb-test-run and > mariadb-stress-test and created a symlink. > > diff --git a/debian/mariadb-test.links b/debian/mariadb-test.links > index 718a5355474..b3577667d01 100644 > --- a/debian/mariadb-test.links > +++ b/debian/mariadb-test.links > @@ -2,5 +2,7 @@ usr/bin/mariadb-client-test usr/bin/mysql_client_test > usr/bin/mariadb-client-test-embedded usr/bin/mysql_client_test_embedded > usr/bin/mariadb-test usr/bin/mysqltest > usr/bin/mariadb-test-embedded usr/bin/mysqltest_embedded > -usr/share/mysql/mysql-test/mysql-test-run.pl usr/share/mysql/mysql-test/mtr > -usr/share/mysql/mysql-test/mysql-test-run.pl usr/share/mysql/mysql-test/mysql-test-run > +usr/share/mysql/mysql-test/mariadb-test-run.pl usr/share/mysql/mysql-test/mysql-test-run.pl > +usr/share/mysql/mysql-test/mariadb-stress-test.pl usr/share/mysql/mysql-test/mysql-stress-test.pl > +usr/share/mysql/mysql-test/mariadb-test-run.pl usr/share/mysql/mysql-test/mysql-test-run > +usr/share/mysql/mysql-test/mariadb-test-run.pl usr/share/mysql/mysql-test/mtr Not sure that was part of the task, but ok. Did you check that both debs and rpms package all new symlinks correctly? > diff --git a/mysql-test/.mtr b/mysql-test/.mtr > new file mode 120000 > index 00000000000..68986c8cd7e > --- /dev/null > +++ b/mysql-test/.mtr > @@ -0,0 +1 @@ > +./mariadb-test-run What is that? What is .mtr file? > \ No newline at end of file > diff --git a/mysql-test/mysql-stress-test.pl b/mysql-test/mysql-stress-test.pl > new file mode 120000 > index 00000000000..e704a5d70bf > --- /dev/null > +++ b/mysql-test/mysql-stress-test.pl > @@ -0,0 +1 @@ > +./mariadb-stress-test.pl looks like you've checked in files you shouldn't have. Like symlinks. > \ No newline at end of file > diff --git a/mysql-test/mysql-test-run.pl b/mysql-test/mysql-test-run.pl > new file mode 120000 > index 00000000000..56150a0ecad > --- /dev/null > +++ b/mysql-test/mysql-test-run.pl > @@ -0,0 +1 @@ > +./mariadb-test-run.pl another one > \ No newline at end of file > diff --git a/scripts/msql2mysql.sh b/scripts/msql2mysql.sh > index 72a609fa6e7..11923fa15c7 100644 > --- a/scripts/msql2mysql.sh > +++ b/scripts/msql2mysql.sh > @@ -16,4 +16,4 @@ > # Free Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston > # MA 02110-1335 USA. > > -@bindir@/replace msqlConnect mysql_connect msqlListDBs mysql_list_dbs msqlNumRows mysql_num_rows msqlFetchRow mysql_fetch_row msqlFetchField mysql_fetch_field msqlFreeResult mysql_free_result msqlListFields mysql_list_fields msqlListTables mysql_list_tables msqlErrMsg 'mysql_error(mysql)' msqlStoreResult mysql_store_result msqlQuery mysql_query msqlField mysql_field msqlSelect mysql_select msqlSelectDB mysql_select_db msqlNumFields mysql_num_fields msqlClose mysql_close msqlDataSeek mysql_data_seek m_field MYSQL_FIELD m_result MYSQL_RES m_row MYSQL_ROW msql mysql mSQL mySQL MSQL MYSQL msqlCreateDB mysql_create_db msqlDropDB mysql_drop_db msqlFieldSeek mysql_field_seek -- $* > +@bindir@/replace msqlConnect mysql_connect msqlListDBs mysql_list_dbs msqlNumRows mysql_num_rows msqlFetchRow mysql_fetch_row msqlFetchField mysql_fetch_field msqlFreeResult mysql_free_result msqlListFields mysql_list_fields msqlListTables mysql_list_tables msqlErrMsg 'mysql_error(mysql)' msqlStoreResult mysql_store_result msqlQuery mysql_query msqlField mysql_field msqlSelect mysql_select msqlSelectDB mysql_select_db msqlNumFields mysql_num_fields msqlClose mysql_close msqlDataSeek mysql_data_seek m_field MYSQL_FIELD m_result MYSQL_RES m_row MYSQL_ROW msql mysql mysql mariadb mSQL mySQL MSQL MYSQL msqlCreateDB mysql_create_db msqlDropDB mysql_drop_db msqlFieldSeek mysql_field_seek -- $* Why? > diff --git a/scripts/mysql_install_db.sh b/scripts/mysql_install_db.sh > index 5f183afe8fc..084af42d01c 100644 > --- a/scripts/mysql_install_db.sh > +++ b/scripts/mysql_install_db.sh > @@ -338,11 +338,15 @@ then > cannot_find_file resolveip @resolveip_locations@ > exit 1 > fi > - mysqld=`find_in_dirs mysqld @mysqld_locations@` > + mysqld=`find_in_dirs mariadbd @mysqld_locations@` > if test -z "$mysqld" > then > - cannot_find_file mysqld @mysqld_locations@ > - exit 1 > + mysqld=`find_in_dirs mysqld @mysqld_locations@` > + if test -z "$mysqld" > + then > + cannot_find_file mysqld @mysqld_locations@ Hmm. At least, it should be mariadbd in the cannot_find_file line. Why do you search for mysqld after mariadbd wasn't found? > + exit 1 > + fi > fi > langdir=`find_in_dirs --dir errmsg.sys @errmsg_locations@` > if test -z "$langdir" > @@ -579,7 +583,7 @@ else > echo > echo " shell> $mysqld --skip-grant-tables --general-log &" > echo > - echo "and use the command line tool $bindir/mysql" > + echo "and use the command line tool $bindir/mariadb or $bindir/mysql" again, why both? > echo "to connect to the mysql database and look at the grant tables:" > echo > echo " shell> $bindir/mysql -u root mysql" > @@ -613,7 +617,9 @@ then > echo "PLEASE REMEMBER TO SET A PASSWORD FOR THE MariaDB root USER !" > echo "To do so, start the server, then issue the following commands:" > echo > + echo "'$bindir/mariadb-admin' -u root password 'new-password' or" > echo "'$bindir/mysqladmin' -u root password 'new-password'" > + echo "'$bindir/mariadb-admin' -u root -h $hostname password 'new-password' or" > echo "'$bindir/mysqladmin' -u root -h $hostname password 'new-password'" and here > echo > echo "Alternatively you can run:" > diff --git a/scripts/mysql_secure_installation.sh b/scripts/mysql_secure_installation.sh > index b2a9edf4953..12364f1be43 100644 > --- a/scripts/mysql_secure_installation.sh > +++ b/scripts/mysql_secure_installation.sh > @@ -159,15 +159,19 @@ then > cannot_find_file my_print_defaults $basedir/bin $basedir/extra > exit 1 > fi > - mysql_command=`find_in_basedir mysql bin` > + mysql_command=`find_in_basedir mariadb bin` > if test -z "$mysql_command" > then > - cannot_find_file mysql $basedir/bin > - exit 1 > + mysql_command=`find_in_basedir mysql bin` > + if test -z "$mysql_command" > + then > + cannot_find_file mysql $basedir/bin > + exit 1 > + fi same as above > fi > else > print_defaults="@bindir@/my_print_defaults" > - mysql_command="@bindir@/mysql" > + mysql_command="@bindir@/mariadb" > fi > > if test ! -x "$print_defaults" > diff --git a/scripts/mysqld_safe.sh b/scripts/mysqld_safe.sh > index 126eb924814..7763fb9db91 100644 > --- a/scripts/mysqld_safe.sh > +++ b/scripts/mysqld_safe.sh > @@ -531,7 +531,11 @@ else > ledir='@libexecdir@' > fi > > -helper=`find_in_bin mysqld_safe_helper` > +helper=`find_in_bin mariadbd-safe-helper` > +if test -x helper > +then > + helper=`find_in_bin mysqld_safe_helper` > +fi and here > print_defaults=`find_in_bin my_print_defaults` > # Check if helper exists > command -v $helper --help >/dev/null 2>&1 > diff --git a/scripts/wsrep_sst_common.sh b/scripts/wsrep_sst_common.sh > index 5e134570881..cf38bda8f70 100644 > --- a/scripts/wsrep_sst_common.sh > +++ b/scripts/wsrep_sst_common.sh > @@ -273,13 +273,21 @@ SCRIPTS_DIR="$(cd $(dirname "$0"); pwd -P)" > EXTRA_DIR="$SCRIPTS_DIR/../extra" > CLIENT_DIR="$SCRIPTS_DIR/../client" > > -if [ -x "$CLIENT_DIR/mysql" ]; then > +if [-x "$CLIENT_DIR/mariadb"]; then > + MYSQL_CLIENT="$CLIENT_DIR/mariadb" > +elif [-x mariadb] then; > + MYSQL_CLIENT=mariadb > +elif [ -x "$CLIENT_DIR/mysql" ]; then > MYSQL_CLIENT="$CLIENT_DIR/mysql" > else > MYSQL_CLIENT=mysql > fi > > -if [ -x "$CLIENT_DIR/mysqldump" ]; then > +if [ -x "$CLIENT_DIR/mariadb-dump" ]; then > + MYSQLDUMP="$CLIENT_DIR/mariadb-dump" > +elif [-x mariadb-dump]; then > + MYSQLDUMP=mariadb-dump > +elif [ -x "$CLIENT_DIR/mysqldump" ]; then > MYSQLDUMP="$CLIENT_DIR/mysqldump" > else > MYSQLDUMP=mysqldump and here Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] ee538938345: MDEV-21117: refine the server binlog-based recovery for semisync
by Sergei Golubchik 16 Mar '21

16 Mar '21

Hi, Andrei! Few comments/questions below. I still have troubles understanding the logic of your solution, so I did not ask any questions about those your replies - as I'd be just repeating the same thing over and over. After I'll start getting it finally, I'll go over your reply again. Meanwhile - minor comments/questions, not related to the main algorithm. On Mar 04, Andrei Elkin wrote: > >> To facilitate the failover on the slave side conditions to accept > >> own events (having been discarded by the above recovery) are > >> relaxed to let so for the semisync slave that connects to master in > >> gtid mode. gtid_strict_mode is further recommended to secure from > >> inadvertent re-applying out of order gtids in general. Non-gtid > >> mode connected semisync slave would require > >> --replicate-same-server-id (mind --log-slave-updates must be OFF > >> then). > > > > Sorry, I failed to understand this paragraph :( > > 'gtid_strict_mode is further recommended to secure > from inadvertent re-applying out of order gtids in general.' > > attempted to promote GTID connection with master, > which of course should not be of this patch's > business, but in a way it is 'cos the non-gtid's mode > (CHANGE MASTER TO ... master_use_gtid = NO) is covered by the patch > with a limitation of '--log-slave-updates must be OFF'. > > The GTID-ON connection is covered, and without any `--replicate-same-server-id'. This doesn't seem to clarify anything. I mean the whole paragraph, starting from "To facilitate" and to "then)." Could you try to rephrase it to make it clearer, please? > >> +--let $query1 = INSERT INTO t VALUES (20) > >> +--let $query2 = DELETE FROM t2 WHERE f = 666 /* no such record */ > >> +--source binlog_truncate_active_log.inc > >> + > >> +--echo # Case B. > >> +# The inverted sequence ends up to truncate only $query2 > >> +--let $this_search_pattern = Successfully truncated.*to remove transactions starting from GTID 0-1-10 > >> +--let $query1 = DELETE FROM t2 WHERE f = 0 > > > > why not `= 666 /* no such record */` ? > > is `= 0` important here? > > Yes, there's a fine detail here. > > 'The inverted sequence ends up' to leave in binlog > > $query1 = DELETE FROM t2 WHERE f = 0 > > which is as ineffective as 666. The patch recognizes the ineffectiveness > through a special value of engines involved counter in Gtid event. > In principle the truncation could > start from it rather than the following $query2. > But that would mean more complicated coding. > So the semantics of the zero is to be good, as we're so used to :-)! Sorry, I didn't understand anything. What does the value in the WHERE clause has to do with engines and where the binlog is truncated? > > > >> +--let $query2 = INSERT INTO t VALUES (20) > >> +--source binlog_truncate_active_log.inc ... > >> +--connection master2 > >> +SET DEBUG_SYNC= "now WAIT_FOR master1_ready"; > >> +if ($pre_q2) > > > > is $pre_q2 a flag or a query string? > > it's used as a flag, it's set as a query string. > > It's actually both with the idea execute it when defined. > Legit, right? legit, but it's not both. Were it both, you'd have if ($pre_q2) { eval $pre_q2; } but you have, if ($pre_q2) { CALL sp_blank_xa; } > >> +{ > >> + CALL sp_blank_xa; > >> +} ... > >> diff --git a/mysql-test/suite/binlog/t/binlog_truncate_multi_engine.test b/mysql-test/suite/binlog/t/binlog_truncate_multi_engine.test > >> new file mode 100644 > >> index 00000000000..fe153e5703c > >> --- /dev/null > >> +++ b/mysql-test/suite/binlog/t/binlog_truncate_multi_engine.test > >> +# > >> +# ==== Implementation ==== > >> +# > >> +# Steps: > >> +# 0 - Create two tables in innodb and rocksdb engines, > >> +# > >> +# In loop for A,B,C cases (below) do 1-5: > > > > there's no loop now (which is good, don't add it please :) [x] ? > >> +# The 1st trx binlogs, rotate binlog and hold on before committing at engine > >> +SET DEBUG_SYNC= "commit_after_release_LOCK_after_binlog_sync SIGNAL master1_ready WAIT_FOR master1_go"; > > > > you don't wait for master1_ready anywhere. intentional? > > No, actually > > SET DEBUG_SYNC= "now WAIT_FOR master1_ready"; > > must be run by below master2 (sim to how master3 waits for master2). meaning [x] ? > >> +--source include/have_binlog_format_row.inc > >> +--let $rpl_topology=1->2 > >> +--source include/rpl_init.inc > > > > isn't this a normal master-slave.inc topology? > > Initially yes. > > I am guessing you mean why not to > --source it. > The test heavily uses 'server_`index`' format to automate > replication direction. as far as I remember, master-slave.inc also defines server_1 and server_2 connections. > >> diff --git a/sql/sql_class.h b/sql/sql_class.h > >> index 140394fefc1..87fa88d4c89 100644 > >> --- a/sql/sql_class.h > >> +++ b/sql/sql_class.h > >> @@ -4632,7 +4632,7 @@ class THD :public Statement, > >> LF_PINS *tdc_hash_pins; > >> LF_PINS *xid_hash_pins; > >> bool fix_xid_hash_pins(); > >> - > >> + bool is_1pc_ro_trans; > > > > I don't like how it looks, it's in THD (for every single connection > > and more) and it's only used in some exotic cases. It's set in two > > places around ha_commit_one_phase(), when it would be better to set > > it inside ha_commit_one_phase(), in one place only. > > > > But I cannot suggest how to get rid of it yet as I don't understand > > what it's for, I've asked a question about it below, in > > Gtid_log_event::Gtid_log_event > > The flag is to help sorting out ineffective (no engine data change) > STATEMENT binlog-format logged transaction. I thought it's practical > enough to face and important to handle. > Through an example, > let the table `t' be initially empty, and > let binlog contain > > trx1: INSERT INTO t SET a = 1; /* slave acked, trx got committed in engine */ > trx2: INSERT INTO t SET a = 2; /* not acked, trx remains prepared in engine */ > trx3: DELETE FROM t WHERE a = 0; /* ineffective, does not have Xid event */ > > prior a crash with their states as commented. > The truncate trx should be trx2 as the trx3's status > suggests it's harmless for involving into > truncation. > However we have yet to distinguish it from unsafe (to truncate) > event group ("transaction") that also ends with COMMIT query not Xid event. Hmm. I see. But thd isn't a universal dumpster for passing arguments to the function, don't use it like that. Please, pass this "read-only transaction with no xid" flag to the Gtid_log_event constructor as an argument. Or recalculate it inside the constructor, it's not such an expensive thing to do. > One rather drastic solution could be to not binlog DELETE altogether. > I did not look into this with much of interest earlier, but how about it? > Perhaps it'd more like a bug fixes than breaking legacy. No, I suspect it's too drastic > >> /* Members related to temporary tables. */ > >> public: > >> /* Opened table states. */ > >> diff --git a/sql/slave.cc b/sql/slave.cc > >> index 28e08e32346..9b3f73e5341 100644 > >> --- a/sql/slave.cc > >> +++ b/sql/slave.cc > >> @@ -6944,7 +6945,9 @@ static int queue_event(Master_info* mi,const char* buf, ulong event_len) > >> } > >> else > >> if ((s_id == global_system_variables.server_id && > >> - !mi->rli.replicate_same_server_id) || > >> + (!mi->rli.replicate_same_server_id && > >> + !(semisync_recovery= (rpl_semi_sync_slave_enabled && > >> + mi->using_gtid != Master_info::USE_GTID_NO)))) || > > > > How can "semisync recovery" code path reach queue_event()? > > This is a failover now, after recovery. > The slave which is ex-master tries to adopt "own" (generated by it) events. 1. don't call it "semisync recovery" then 2. add a comment explaining this case (it's a rare and not obvious one) 3. why does it have to be specially recognized anyway? > > > >> event_that_should_be_ignored(buf) || > >> /* > >> the following conjunction deals with IGNORE_SERVER_IDS, if set > >> diff --git a/sql/log_event.h b/sql/log_event.h > >> index 8a342cb5cd3..1588ab85104 100644 > >> --- a/sql/log_event.h > >> +++ b/sql/log_event.h > >> @@ -482,6 +482,16 @@ class String; > >> */ > >> #define LOG_EVENT_IGNORABLE_F 0x80 > >> > >> +/** > >> + @def LOG_EVENT_ACCEPT_OWN_F > >> + > >> + Flag sets by the gtid-mode connected semisync slave for > >> + the same server_id ("own") events which the slave must not have > >> + in its state. Typically such events were never committed by > >> + their originator (this server) and discared at its crash recovery > > > > sorry, I failed to understand that > > This should've been explained above. > The slave which is ex-master truncated at recovery some of its ("own") transactions. > The flag serves to help recognize that at execution of the replication events. 1. yes, this is clearer, thanks. may be you should rewrite your comment putting this explanation in it? 2. why does it have to be specially recognized anyway? ... > > 3. also it's somewhat an unfounded assumption that only r/w engines > > will have the transaction prepared. But we can make it a fact, if we > > won't prepare r/o engines at all (in ha_prepare). > > [?] > > Sorry, I don't get it. > Do you mean that an r/o engine (branch) might need to be counted by > `ha_count_rw()`? But then it would be contributing with binlogged > events of its branch... I can't imagine that. I mean the following. Say, you have two engines participating in a transaction, like in INSERT innodb_table SELECT * FROM rocksdb_table; One is r/w, the other is r/o. It's a 2pc transaction, the server does prepare for everyone, then commit for everyone. I don't think there is any guarantee that prepare for an r/o engine will be a no-op. It is completely up to the engine what to do in this case, it's possible that the engine will optimize it to a no-op, but it is not a certainty. What I think we should do is not to call prepare() for r/o engines at all. Just skip straight to commit/rollback on the second phase. > >> diff --git a/sql/handler.cc b/sql/handler.cc > >> index 6792e80b8fe..247ab7267bf 100644 > >> --- a/sql/handler.cc > >> +++ b/sql/handler.cc > >> @@ -1245,6 +1245,24 @@ int ha_prepare(THD *thd) > >> DBUG_RETURN(error); > >> } > >> > >> +/* > >> + Returns counted number of > >> + read-write recoverable transaction participants. > >> +*/ > >> +uint ha_count_rw(THD *thd, bool all) > > > > the name doesn't match the implementation, please, rename > > [?] > > Please help here. > I could not find what specifically you mean. > It > returns rw_ha_count > > which I think as 'read-write recoverable transaction participants.' I read the name is "a number of rw engines in the ha_info", it does not mention "recoverable", and I from the name I woulnd't have guessed. May be ha_count_rw_2pc ? > >> +{ > >> + unsigned rw_ha_count= 0; > >> + THD_TRANS *trans=all ? &thd->transaction.all : &thd->transaction.stmt; > >> + > >> + for (Ha_trx_info * ha_info= trans->ha_list; ha_info; > >> + ha_info= ha_info->next()) > >> + { > >> + if (ha_info->is_trx_read_write() && ha_info->ht()->recover) > >> + ++rw_ha_count; > >> + } > >> + return rw_ha_count; > >> +} > >> + > >> /** > >> Check if we can skip the two-phase commit. > >> > >> @@ -1960,8 +1982,122 @@ struct xarecover_st > >> XID *list; > >> HASH *commit_list; > >> bool dry_run; > >> + MEM_ROOT *mem_root; > >> + bool error; > >> }; > >> > >> +/** > >> + Inserts a new hash member. > >> + > >> + returns a successfully created and inserted @c xid_recovery_member > >> + into hash @c hash_arg, > >> + or NULL. > >> +*/ > >> +static xid_recovery_member* > >> +xid_member_insert(HASH *hash_arg, my_xid xid_arg, MEM_ROOT *ptr_mem_root) > >> +{ > >> + xid_recovery_member *member= (xid_recovery_member*) > >> + alloc_root(ptr_mem_root, sizeof(xid_recovery_member)); > >> + if (!member) > >> + return NULL; > >> + > >> + member->xid= xid_arg; > >> + member->in_engine_prepare= 1; > >> + member->decided_to_commit= false; > >> + > >> + return my_hash_insert(hash_arg, (uchar*) member) ? NULL : member; > >> +} > >> + > >> +/* > >> + Inserts a new or updates an existing hash member to increment > >> + the member's prepare counter. > >> + > >> + returns false on success, > >> + true otherwise. > >> +*/ > >> +static bool xid_member_replace(HASH *hash_arg, my_xid xid_arg, > >> + MEM_ROOT *ptr_mem_root) > >> +{ > >> + xid_recovery_member* member; > >> + if ((member= (xid_recovery_member *) > >> + my_hash_search(hash_arg, (uchar *)& xid_arg, sizeof(xid_arg)))) > >> + member->in_engine_prepare++; > >> + else > >> + member= xid_member_insert(hash_arg, xid_arg, ptr_mem_root); > >> + > >> + return member == NULL; > >> +} > >> + > >> +/* > >> + Hash iterate function to complete with commit or rollback as decided > >> + (typically at binlog recovery processing) in member->in_engine_prepare. > >> +*/ > >> +static my_bool xarecover_do_commit_or_rollback(void *member_arg, > >> + void *hton_arg) > >> +{ > >> + xid_recovery_member *member= (xid_recovery_member*) member_arg; > >> + handlerton *hton= (handlerton*) hton_arg; > >> + xid_t x; > >> + my_bool rc; > >> + > >> + x.set(member->xid); > >> + rc= member->decided_to_commit ? hton->commit_by_xid(hton, &x) : > >> + hton->rollback_by_xid(hton, &x); > >> + > >> + DBUG_ASSERT(rc || member->in_engine_prepare > 0); > >> + > >> + if (!rc) > >> + { > > > > I don't think you can trust rc on that. > > if it's non-zero, it's an error all right. > > but it's a bit of a stretch to presume that > > nonexisting xid is always an error. > > > > also commit_by_xid returns int, not my_bool. > > [?] I mean, if an engine returns success when you ask it to commit or rollback an unknown xid - this is not a bug, right? Commit means "make all changes by this xid persistent", if there are no changes, there is nothing to do, so doing nothing and reporting success isn't exactly wrong. Same for rollback. > I'd rather to stick with the patch and require from any engine to > return \cite{xa spec} > > [XAER_NOTA] The specified XID is not known by the resource manager > > What'd you say? Okay. I'd say I was wrong, if the standard requires a resource manager to return an error for an unknown XID, then you surely can rely on that :) Sorry for confusion. > >> + > >> + return false; > >> +} > >> + > >> +static my_bool xarecover_do_count_in_prepare(void *member_arg, > >> + void *ptr_count) > >> +{ > >> + xid_recovery_member *member= (xid_recovery_member*) member_arg; > >> + if (member->in_engine_prepare) > >> + { > >> + *(uint*) ptr_count += member->in_engine_prepare; > > > > This is a rather weird semantics. > > it's kind of a number of transactions times number of engines. > > if one transaction wasn't committed in two engines and another > > transaction wasn't rolled back in one engine, the counter will be 3. > > how are you going to explain this to users? > > So `ptr_count` actually is for transaction branches, not transactions. > But the ultimate warning speaks erronously the latter > > + sql_print_warning("Could not complete %u number of transactions in " > "engines.", count_in_prepare); > > So I'd convert the counter to count the transactions: > > (*(uint*) ptr_count) ++ > > [?] yes, that'd be easier to understand, thanks > >> diff --git a/sql/log.cc b/sql/log.cc > >> index 8073f09ab88..a9808ed8823 100644 > >> --- a/sql/log.cc > >> +++ b/sql/log.cc ... > >> + char buf[21]; > >> + > >> + longlong10_to_str(ptr_gtid->seq_no, buf, 10); > >> + sql_print_information("Successfully truncated binlog file:%s " > >> + "to pos:%llu to remove transactions starting from " > >> + "GTID %u-%u-%s", file_name, pos, > >> + ptr_gtid->domain_id, ptr_gtid->server_id, buf); > >> + } > >> + if (!(error= init_io_cache(&cache, file, IO_SIZE, WRITE_CACHE, > >> + (my_off_t) pos, 0, MYF(MY_WME|MY_NABP)))) > >> + { > >> + /* > >> + Write Stop_log_event to ensure clean end point for the binary log being > >> + truncated. > >> + */ > > > > I'm not sure about it. The less you touch the corrupted binlog the better. > > simply truncating it would be enough, wouldn't it? > > Yes, it would. > But I somewhat preferred a tidy-up style. > > [?] > > If you feel strong to remove it will be done. I feel it'd be safer not to write anything into the corrupted binlog. > >> + Stop_log_event se; > >> + se.checksum_alg= cs_alg; > >> + if ((error= write_event(&se, NULL, &cache))) > >> + { > >> + sql_print_error("Failed to write stop event to " > >> + "binary log. Errno:%d", > >> + (cache.error == -1) ? my_errno : error); > >> + goto end; > >> + } > >> + clear_inuse_flag_when_closing(cache.file); > >> + if ((error= flush_io_cache(&cache)) || > >> + (error= mysql_file_sync(file, MYF(MY_WME|MY_SYNC_FILESIZE)))) > >> + { > >> + sql_print_error("Faild to write stop event to " > >> + "binary log. Errno:%d", > >> + (cache.error == -1) ? my_errno : error); > >> + } > >> + } > >> + else > >> + sql_print_error("Failed to initialize binary log " > >> + "cache for writing stop event. Errno:%d", > >> + (cache.error == -1) ? my_errno : error); > > > > 1. I don't see why you need to sync after every operation. > > Yep, there's one fsync before Stop event writing, redundant if Stop will > stay in the final patch. As far as I understand, if Stop is removed, you shouldn't need any syncs at all, just truncate and close. > >> + int next_binlog_or_round(int& round, > >> + const char *last_log_name, > >> + const char *binlog_checkpoint_name, > >> + LOG_INFO *linfo, MYSQL_BIN_LOG *log); > >> + bool is_safe() > > > > what do you mean "safe" ? > > Safe to truncate condition is explained elsewhere to mean there are no > non-transactional group of events within the truncated tail. > Earlier mentioned ineffective COMMIT-query ended "transactions" are safe. may be is_safe_to_truncate() then? ... > >> + if (!truncate_validated) > >> + { > >> + DBUG_ASSERT(round <= 2); > >> + /* > >> + Either truncate was not set or was reset, else > >> + it gets incremented, otherwise it may only set to an earlier > >> + offset only at the turn out of the 1st round. > > > > sorry, I cannot parse that :( > > The comments are for the assert in a branch that acts when the truncate position > (gtid) is not yet finalized. > Let me annotate under each assert's line. > > > > >> + */ > >> + DBUG_ASSERT(truncate_gtid.seq_no == 0 || > > 'truncate was not set or was reset' ^, that is truncate_gtid. > The 'was reset' is actually not present in the assert - that'd be > reflected with > truncate_reset_done == true > > > >> + last_gtid_coord >= binlog_truncate_coord || > > 'it gets incremented' ^ why does that mean "it gets incremented" ? > > >> + (round == 2 && truncate_set_in_1st)); > > And if it's neither of the above there must've been truncate candidate > set in the 1st round. ... > >> switch (typ) > >> { > >> + case FORMAT_DESCRIPTION_EVENT: > >> + if (round > 1) > >> + { > >> + if (fdle != fdle_arg) > >> + delete fdle; > >> + fdle= (Format_description_log_event*) ev; > > > > why is it needed? all binlog files should have the same > > Format_description_log_event anyway. > > Actually contrary to what was stated (the patch removed that claim) in > the parent revision of this commit, the FD:s of the > recovery binlog files may have different checksums. I.e > > set @@global.binlog_checksum = value > > can be issued to rotate binlog to a new hot file having different checksum value > while the binlog *CheckPoint* file is behind. So this is possible to > have at recovery: > > B_1(CP, checksum = NONE), B_2(hot, checksum = CRC32) > > which also manifests a normal use case bug I have not reported any bug (though wanted to). okay, checksum can differ. But how does it affect Format_description_log_event ? Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] ee538938345: MDEV-21117: refine the server binlog-based recovery for semisync
by Sergei Golubchik 15 Mar '21

15 Mar '21

Hi, Andrei! On Mar 01, Andrei Elkin wrote: > > I've reviewed almost everything, see comments below. But not the > > Recovery_context methods. Please explain how it works and how all > > these truncate_validated, truncate_reset_done, truncate_set_in_1st, > > etc all work together. > > ...specifically to this point. Just in case I hope you did not miss to > read recovery_design.txt from the MDEV, which does not go into coding > details that you're effectively about in above. I've read it now. Still I don't understand why you need an extra round of binlog scanning (two rounds if only one file, three rounds if many). Also, this recovery_design.txt is not part of the code, so whoever will look at it later will be just as puzzled as I was. ... > [ G1, G2, ..., G_k, g_{k+1}, ... g_n ] > > here the uppercase `G' stands for committed trx, the smallcase `g' for > prepared,`_k' - sub-scripts in the recovery sequence. As the capital > letter first rules in the single-engine case the first occurrence of a > pattern `G_k,g_k+1' identifies the truncate index. The patch reflects > such fact with raising `truncate_validated' flag. > > But it's more complicated in the multiple binlog files / engines case. > The first `Gg' letter-case drop is not guaranteed to be the only drop > so `truncate_reset_done' and `truncate_set_in_1st' are introduced to > help with truncate index identification. Why is it not guaranteed? > When `truncate_validated' is set that indicates the truncate index is > determined and may not change in the current (1st of 2nd) nor future > rounds. `truncate_reset_done' says that an "inverse" `g_k,G_k+1' pair > is found so that any earlier truncation candidate gets reset (to > "zero"). If there will be later any candidate found in *this* (1st or > 2nd) round in the sequence its index will be obviously greater. > > `truncate_set_in_1st' function is to remember that the truncate > candidate was found in the 1st round (in the "hot" binlog file), but > if the candidate has not been validated `!truncate_validated' it may > be exacted in the 2nd round and then to an earlier transaction. So the > flag helps to handle exception from truncate candidate monotony rule: > e.g the hot binlog B2 contains `[g5,g6,...g_n]' and a ref to binlog > checkpoint file B1 that contains `[G1,g2,G3,g4]'. The first round > truncate candidate of g5 would be first exacted to `g2' before finally > ascertained to `g4' in the 2nd round. (Notice `g2 -> g4' preserves the > truncate index monotony). > > Notice that due to exacting like `g2 -> g4' in the 2nd round of the > above example `g2' got "to be up-cased" into `G2' for committing > (feasible with two trx:s on two different engine scenario - g2 with > Innodb only got prepared, G4 - on Rocksdb, and got committed prior the > crash). That's what the 3rd round is for. > > I hope this will be helpful. Unfortunately, not very much. It describes how you juggle with variables and scanning rounds. But not *what* you're trying to find. And it's kind of difficult to reverse engineer your "how" back into "what" :( Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

[Maria-developers] MDEV-17399: JSON_TABLE: Incorrect code with table elimination
by Sergey Petrunia 15 Mar '21

15 Mar '21

Hi Alexey, > diff --git a/sql/opt_table_elimination.cc b/sql/opt_table_elimination.cc > index 3958797ec44..f2497d524ec 100644 > --- a/sql/opt_table_elimination.cc > +++ b/sql/opt_table_elimination.cc > @@ -637,6 +637,16 @@ void eliminate_tables(JOIN *join) > List_iterator<Item> it(join->fields_list); > while ((item= it++)) > used_tables |= item->used_tables(); > + > + { > + List_iterator<TABLE_LIST> it(*join->join_list); > + TABLE_LIST *tbl; > + while ((tbl= it++)) > + { > + if (tbl->table_function) > + used_tables|= tbl->table_function->used_tables(); > + } > + } > This only walks the tables that are "at the top level" of the join. if a JSON_TABLE(...) is inside a nested outer join, it will not be found. Please walk select_lex->leaf_tables instead. Please add the testcase (I provide one below). Please also add a comment, clarifying what is being done, something like: Table function JSON_TABLE() can have references to other tables. Do not eliminate the tables that JSON_TABLE() refers to. Note: the JSON_TABLE itself cannot be eliminated as it doesn't have unique keys. == Testcase == create table t20 (a int not null); create table t21 (a int not null primary key, js varchar(100)); insert into t20 select seq from seq_1_to_100; insert into t21 select a, '{"a":100}' from t20; create table t31(a int); create table t32(b int); insert into t31 values (1); insert into t32 values (1); explain select t20.a, jt1.ab from t20 left join t21 on t20.a=t21.a join (t31 left join (t32 join JSON_TABLE(t21.js,'$' COLUMNS (ab INT PATH '$.a')) AS jt1) on t31.a<3); BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://petrunia.net

2 3

Re: [Maria-developers] f1db957c5da: MDEV-21469: Implement crash-safe binary logging of the user XA
by Sergei Golubchik 14 Mar '21

14 Mar '21

Hi, Sujatha! Note, this is a review of a combined diff dc92235f21e + f1db957c5da On Mar 14, Sujatha wrote: > revision-id: f1db957c5da (mariadb-10.5.2-247-gf1db957c5da) > parent(s): dc92235f21e > author: Sujatha <sujatha.sivakumar(a)mariadb.com> > committer: Andrei Elkin <andrei.elkin(a)mariadb.com> > timestamp: 2020-12-21 16:10:46 +0200 > message: > > MDEV-21469: Implement crash-safe binary logging of the user XA > > Make XA PREPARE, XA COMMIT and XA ROLLBACK statements > crash-safe when --log-bin is specified. > > At execution of XA PREPARE, XA COMMIT and XA ROLLBACK their replication > events are made written into the binary log prior to execution > of the commands in storage engines. > In case the server crashes, after writing to binary log but before all > involved engines have processed the command, the following recovery > will execute the command's replication events to equalize the states > of involved engines with that of binlog. > That applies to all XA PREPARE *group* and XA COMMIT or ROLLBACK. > > On the implementation level the recovery time binary log parsing > is augmented to pay attention to > the user XA xids to identify the XA transactions' state:s in binary log, and > eventually match them against their states in engines, see > MYSQL_BIN_LOG::recover_explicit_xa_prepare(). > In discrepancy cases the outdated state in the engines is corrected with > resubmitting the transaction prepare group of events, or completion > ones. The multi-engine partly prepared XA PREPARE case > the XA is rolled back first. > The fact of multiple-engine involved is registered into > Gtid_log_event::flags2 as one bit. The boolean value is > sufficient and precise to deal with two engines in XA transaction. > With more than 2 recoverable engines the flag method is still correct though > may be pessimistic, as it treats all recoverable engines as XA > participants. So when the number of such Engines exceeds the number > of prepared engines of the XA that XA is treated > as partially completed, with all that ensued. > As an optimization no new bit is allocated in flags2, instead a > pre-existing ones (of MDEV-742) are reused, observing that > A. XA "COMPLETE" does not require multi-engine hint for its recovery and that > B. the MDEV-742 XA-completion bit is not anyhow used by > XA-PREPARE and its GTID log event. > > Notice the multi-engine flagging is superceded by MDEV-21117 extension > in Gtid log event so this part should be taken from there. > diff --git a/mysql-test/suite/binlog/t/binlog_xa_multi_binlog_crash_recovery.test b/mysql-test/suite/binlog/t/binlog_xa_multi_binlog_crash_recovery.test > new file mode 100644 > index 00000000000..d0852de2fcc > --- /dev/null > +++ b/mysql-test/suite/binlog/t/binlog_xa_multi_binlog_crash_recovery.test > @@ -0,0 +1,86 @@ > +# ==== Purpose ==== > +# > +# Test verifies that XA crash recovery works fine across multiple binary logs. > +# > +# ==== Implementation ==== > +# > +# Steps: > +# 0 - Generate an explicit XA transaction. Using debug simulation hold the "debug sync" != "debug sim" :) > +# execution of XA PREPARE statement after the XA PREPARE is written to > +# the binary log. With this the prepare will not be done in engine. > +# 1 - By executing FLUSH LOGS generate multiple binary logs. > +# 2 - Now make the server to disappear at this point. > +# 3 - Restart the server. During recovery the XA PREPARE from the binary > +# log will be read. It is cross checked with engine. Since it is not > +# present in engine it will be executed once again. > +# 4 - When server is up execute XA RECOVER to check that the XA is > +# prepared in engine as well. > +# 5 - XA COMMIT the transaction and check the validity of the data. > +# > +# ==== References ==== > +# > +# MDEV-21469: Implement crash-safe logging of the user XA > +# > + > +--source include/have_innodb.inc > +--source include/have_debug.inc > +--source include/have_debug_sync.inc > +--source include/have_log_bin.inc > + > +RESET MASTER; > + > +CREATE TABLE t1 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; > +CALL mtr.add_suppression("Found 1 prepared XA transactions"); > + > +connect(con1,localhost,root,,); > +XA START 'xa1'; > +INSERT INTO t1 SET a=1; > +SET DEBUG_SYNC= "simulate_hang_after_binlog_prepare SIGNAL con1_ready WAIT_FOR con1_go"; > +SET GLOBAL DEBUG_DBUG="d,simulate_crash_after_binlog_prepare"; > +XA END 'xa1'; > +--send XA PREPARE 'xa1'; > + > +connection default; > +SET DEBUG_SYNC= "now WAIT_FOR con1_ready"; > +FLUSH LOGS; > +FLUSH LOGS; > +FLUSH LOGS; > + > +--source include/show_binary_logs.inc > +--let $binlog_file= master-bin.000004 > +--let $binlog_start= 4 > +--source include/show_binlog_events.inc > + > +--write_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect > +wait I don't see why you'd need to wait first, it'd be simpler to restart right away. > +EOF > + > +--error 0,2013 > +SET DEBUG_SYNC= "now SIGNAL con1_go"; > +--source include/wait_until_disconnected.inc > + > +--connection con1 > +--error 2013 > +--reap > +--source include/wait_until_disconnected.inc > + > +# > +# Server restart > +# > +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect > +restart > +EOF > + > +connection default; > +--enable_reconnect > +--source include/wait_until_connected_again.inc > + > +XA RECOVER; > +XA COMMIT 'xa1'; > + > +SELECT * FROM t1; > + > +# Clean up. > +connection default; > +DROP TABLE t1; > + > diff --git a/mysql-test/suite/rpl/t/rpl_xa_commit_crash_safe.test b/mysql-test/suite/rpl/t/rpl_xa_commit_crash_safe.test > new file mode 100644 > index 00000000000..e972e3f09de > --- /dev/null > +++ b/mysql-test/suite/rpl/t/rpl_xa_commit_crash_safe.test > @@ -0,0 +1,98 @@ > +# ==== Purpose ==== > +# > +# Test verifies that XA COMMIT statements are crash safe. > +# > +# ==== Implementation ==== > +# > +# Steps: > +# 0 - Generate 2 explicit XA transactions. 'xa1' and 'xa2'. > +# 'xa1' will be prepared and committed. > +# 1 - For 'xa2' let the XA COMMIT be done in binary log and crash the > +# server so that it is not committed in engine. > +# 2 - Restart the server. The recovery code should successfully recover > +# 'xa2'. The COMMIT should be executed during recovery. > +# 3 - Check the data in table. Both rows should be present in table. > +# 4 - Trying to commit 'xa2' should report unknown 'XA' error as COMMIT is > +# already complete during recovery. > +# > +# ==== References ==== > +# > +# MDEV-21469: Implement crash-safe logging of the user XA > + > + > +--source include/have_innodb.inc > +--source include/master-slave.inc master-slave must be always included last > +--source include/have_debug.inc > + > +connect (master2,localhost,root,,); why do you need master2 connection? > +--connection master > +CALL mtr.add_suppression("Found 1 prepared XA transactions"); > + > +CREATE TABLE t ( f INT ) ENGINE=INNODB; > +XA START 'xa1'; > +INSERT INTO t VALUES (20); > +XA END 'xa1'; > +XA PREPARE 'xa1'; > +XA COMMIT 'xa1'; > +--sync_slave_with_master > +--source include/stop_slave.inc > + > +--connection master1 > +XA START 'xa2'; > +INSERT INTO t VALUES (40); > +XA END 'xa2'; > +XA PREPARE 'xa2'; > + > +--write_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect > +wait > +EOF Same. Why do you wait? > + > +SET GLOBAL DEBUG_DBUG="d,simulate_crash_after_binlog_commit_or_rollback"; > +--error 2013 # CR_SERVER_LOST > +XA COMMIT 'xa2'; > +--source include/wait_until_disconnected.inc > + > +--connection master1 > +--source include/wait_until_disconnected.inc > + > +--connection master > +--source include/wait_until_disconnected.inc > + > +# > +# Server restart > +# > +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect > +restart > +EOF > + > +connection default; > +--enable_reconnect > +--source include/wait_until_connected_again.inc > + > +# rpl_end.inc needs to use the connection server_1 > +connection server_1; > +--enable_reconnect > +--source include/wait_until_connected_again.inc > + > +--connection master > +--enable_reconnect > +--source include/wait_until_connected_again.inc > + > +--connection slave > +--source include/start_slave.inc > +--sync_with_master > + > +--connection master > +SELECT * FROM t; > +XA RECOVER; > +--error 1397 # ER_XAER_NOTA > +XA COMMIT 'xa2'; > +SELECT * FROM t; > +--sync_slave_with_master > + > +SELECT * FROM t; > + > +--connection master > +DROP TABLE t; > +--sync_slave_with_master > +--source include/rpl_end.inc > diff --git a/mysql-test/suite/rpl/t/rpl_xa_event_apply_failure.test b/mysql-test/suite/rpl/t/rpl_xa_event_apply_failure.test > new file mode 100644 > index 00000000000..14cebbd9b13 > --- /dev/null > +++ b/mysql-test/suite/rpl/t/rpl_xa_event_apply_failure.test > @@ -0,0 +1,119 @@ > +# ==== Purpose ==== > +# > +# Test verifies that if for some reason an event cannot be applied during > +# recovery, appropriate error is reported. > +# > +# ==== Implementation ==== > +# > +# Steps: > +# 0 - Generate 2 explicit XA transactions. 'xa1' and 'xa2'. > +# 'xa1' will be prepared and committed. > +# 1 - For 'xa2' let the XA COMMIT be done in binary log and crash the > +# server so that it is not committed in engine. > +# 2 - Restart the server. Using debug simulation point make XA COMMIT 'xa2' > +# execution to fail. The server will resume anyway > +# to leave the error in the errlog (see "Recovery: Error.."). > +# 3 - Work around the simulated failure with Commit once again > +# from a connection that turns OFF binlogging. > +# Slave must catch up with the master. > +# > +# ==== References ==== > +# > +# MDEV-21469: Implement crash-safe logging of the user XA > + > + > +--source include/have_innodb.inc > +--source include/master-slave.inc master-slave must be always included last > +--source include/have_debug.inc > + > +connect (master2,localhost,root,,); why do you need master2 connection? > +--connection master > +CALL mtr.add_suppression("Found 1 prepared XA transactions"); > +CALL mtr.add_suppression("Failed to execute binlog query event"); > +CALL mtr.add_suppression("Recovery: Error .Out of memory.."); > +CALL mtr.add_suppression("Crash recovery failed."); > +CALL mtr.add_suppression("Can.t init tc log"); > +CALL mtr.add_suppression("Aborting"); > + > +CREATE TABLE t ( f INT ) ENGINE=INNODB; > +XA START 'xa1'; > +INSERT INTO t VALUES (20); > +XA END 'xa1'; > +XA PREPARE 'xa1'; > +XA COMMIT 'xa1'; > +--sync_slave_with_master > +--source include/stop_slave.inc > + > +--connection master1 > +XA START 'xa2'; > +INSERT INTO t VALUES (40); > +XA END 'xa2'; > +XA PREPARE 'xa2'; > + > +--write_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect > +wait ditto > +EOF > + > +SET GLOBAL DEBUG_DBUG="d,simulate_crash_after_binlog_commit_or_rollback"; > +--error 2013 # CR_SERVER_LOST > +XA COMMIT 'xa2'; > +--source include/wait_until_disconnected.inc > + > +--connection master1 > +--source include/wait_until_disconnected.inc > + > +--connection master > +--source include/wait_until_disconnected.inc > + > +# > +# Server restart > +# > +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect > +restart: --debug-dbug=d,trans_xa_commit_fail > +EOF > + > +connection default; > +--source include/wait_until_disconnected.inc > + > +connection default; > +--enable_reconnect > +--source include/wait_until_connected_again.inc > + > +--connection master > +--enable_reconnect > +--echo *** must be no 'xa2' commit seen, as it's still prepared: > +SELECT * FROM t; > +XA RECOVER; > + > +# Commit it manually now to work around the extra binlog record > +# by turning binlogging OFF by the connection. > + > +SET GLOBAL DEBUG_DBUG=""; > +SET SQL_LOG_BIN=0; > +--error 0 why error 0? What will happen if you don't disable binlog? > +XA COMMIT 'xa2'; > +SET SQL_LOG_BIN=1; > + > + > +# rpl_end.inc needs to use the connection server_1 > +connection server_1; > +--enable_reconnect > +--source include/wait_until_connected_again.inc > + > +--connection master > +--source include/wait_until_connected_again.inc > + > +--connection slave > +--source include/start_slave.inc > +--sync_with_master > + > +--connection master > +SELECT * FROM t; > + > +--sync_slave_with_master > +SELECT * FROM t; > + > +--connection master > +DROP TABLE t; > +--sync_slave_with_master > +--source include/rpl_end.inc > diff --git a/mysql-test/suite/rpl/t/rpl_xa_prepare_commit_prepare.test b/mysql-test/suite/rpl/t/rpl_xa_prepare_commit_prepare.test > new file mode 100644 > index 00000000000..7b987c7f29b > --- /dev/null > +++ b/mysql-test/suite/rpl/t/rpl_xa_prepare_commit_prepare.test > @@ -0,0 +1,95 @@ > +# ==== Purpose ==== > +# > +# Test verifies that XA PREPARE transactions are crash safe. > +# > +# ==== Implementation ==== > +# > +# Steps: > +# 0 - Generate 2 explicit XA transactions. 'xa1' and 'xa2'. > +# 'xa1' will be prepared and committed. > +# 1 - For 'xa2' let the XA PREPARE be done in binary log and crash the > +# server so that it is not prepared in engine. > +# 2 - Restart the server. The recovery code should successfully recover > +# 'xa2'. > +# 3 - When server is up, execute XA RECOVER and verify that 'xa2' is > +# present. > +# 4 - Commit the XA transaction and verify its correctness. > +# > +# ==== References ==== > +# > +# MDEV-21469: Implement crash-safe logging of the user XA > + > +--source include/have_innodb.inc > +--source include/master-slave.inc ditto > +--source include/have_debug.inc > + > +connect (master2,localhost,root,,); ditto > +--connection master > +CALL mtr.add_suppression("Found 1 prepared XA transactions"); > + > +CREATE TABLE t ( f INT ) ENGINE=INNODB; > +XA START 'xa1'; > +INSERT INTO t VALUES (20); > +XA END 'xa1'; > +XA PREPARE 'xa1'; > +XA COMMIT 'xa1'; > +--sync_slave_with_master > +--source include/stop_slave.inc > + > +--connection master1 > +XA START 'xa2'; > +INSERT INTO t VALUES (40); > +XA END 'xa2'; > + > +--write_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect > +wait ditto > +EOF > + > +SET GLOBAL DEBUG_DBUG="d,simulate_crash_after_binlog_prepare"; > +--error 2013 # CR_SERVER_LOST > +XA PREPARE 'xa2'; > +--source include/wait_until_disconnected.inc > + > +--connection master1 > +--source include/wait_until_disconnected.inc > + > +--connection master > +--source include/wait_until_disconnected.inc > + > +# > +# Server restart > +# > +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect > +restart > +EOF > + > +connection default; > +--enable_reconnect > +--source include/wait_until_connected_again.inc > + > +# rpl_end.inc needs to use the connection server_1 > +connection server_1; > +--enable_reconnect > +--source include/wait_until_connected_again.inc > + > +--connection master > +--enable_reconnect > +--source include/wait_until_connected_again.inc > + > +--connection slave > +--source include/start_slave.inc > +--sync_with_master > + > +--connection master > +SELECT * FROM t; > +XA RECOVER; > +XA COMMIT 'xa2'; > +SELECT * FROM t; > +--sync_slave_with_master > + > +SELECT * FROM t; > + > +--connection master > +DROP TABLE t; > +--sync_slave_with_master > +--source include/rpl_end.inc > diff --git a/mysql-test/suite/rpl/t/rpl_xa_prepare_crash_safe.test b/mysql-test/suite/rpl/t/rpl_xa_prepare_crash_safe.test > new file mode 100644 > index 00000000000..9d2c5cce528 > --- /dev/null > +++ b/mysql-test/suite/rpl/t/rpl_xa_prepare_crash_safe.test > @@ -0,0 +1,117 @@ > +# ==== Purpose ==== > +# > +# Test verifies that XA PREPARE transactions are crash safe. > +# > +# ==== Implementation ==== > +# > +# Steps: > +# 0 - Generate 3 explicit XA transactions. 'xa1', 'xa2' and 'xa3'. > +# Using debug simulation hold the execution of second XA PREPARE > +# statement after the XA PREPARE is written to the binary log. > +# With this the prepare will not be done in engine. > +# 1 - For 'xa3' allow the PREPARE statement to be written to binary log and > +# simulate server crash. > +# 2 - Restart the server. The recovery code should successfully recover > +# 'xa2' and 'xa3'. > +# 3 - When server is up, execute XA RECOVER and verify that 'xa2' and 'xa3' > +# are present along with 'xa1'. > +# 4 - Commit all the XA transactions and verify their correctness. > +# > +# ==== References ==== > +# > +# MDEV-21469: Implement crash-safe logging of the user XA > + > + > +--source include/have_innodb.inc > +--source include/master-slave.inc ditto > +--source include/have_debug.inc > + > +connect (master2,localhost,root,,); ok, in this test you actually use master2 :) > +--connection master > +CALL mtr.add_suppression("Found 1 prepared XA transactions"); > +CALL mtr.add_suppression("Found 2 prepared XA transactions"); > +CALL mtr.add_suppression("Found 3 prepared XA transactions"); > + > +CREATE TABLE t ( f INT ) ENGINE=INNODB; > +XA START 'xa1'; > +INSERT INTO t VALUES (20); > +XA END 'xa1'; > +XA PREPARE 'xa1'; > +--sync_slave_with_master > +--source include/stop_slave.inc > + > +--connection master1 > +use test; > +xa start 'xa2'; > +insert into t values (30); > +xa end 'xa2'; > +SET DEBUG_SYNC="simulate_hang_after_binlog_prepare SIGNAL reached WAIT_FOR go"; > +send xa prepare 'xa2'; > + > +--connection master2 > +let $wait_condition= > + SELECT COUNT(*) = 1 FROM INFORMATION_SCHEMA.PROCESSLIST > + WHERE STATE like "debug sync point: simulate_hang_after_binlog_prepare%"; > +--source include/wait_condition.inc eh? you can just 'WAIT_FOR signal', couldn't you? > + > +XA START 'xa3'; > +INSERT INTO t VALUES (40); > +XA END 'xa3'; > + > +--write_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect > +wait ditto > +EOF > + > +SET GLOBAL DEBUG_DBUG="d,simulate_crash_after_binlog_prepare"; > +--error 2013 # CR_SERVER_LOST > +XA PREPARE 'xa3'; > +--source include/wait_until_disconnected.inc > + > +--connection master1 > +--error 2013 > +--reap > +--source include/wait_until_disconnected.inc > + > +--connection master > +--source include/wait_until_disconnected.inc > + > +# > +# Server restart > +# > +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect > +restart > +EOF > + > +connection default; > +--enable_reconnect > +--source include/wait_until_connected_again.inc > + > +# rpl_end.inc needs to use the connection server_1 > +connection server_1; > +--enable_reconnect > +--source include/wait_until_connected_again.inc > + > +--connection master > +--enable_reconnect > +--source include/wait_until_connected_again.inc > + > + > +--connection slave > +--source include/start_slave.inc > +--sync_with_master > + > +--connection master > +SELECT * FROM t; > +XA RECOVER; > +XA COMMIT 'xa1'; > +XA COMMIT 'xa2'; > +XA COMMIT 'xa3'; > +SELECT * FROM t; > +--sync_slave_with_master > + > +SELECT * FROM t; > + > +--connection master > +DROP TABLE t; > +--sync_slave_with_master > +--source include/rpl_end.inc > diff --git a/mysql-test/suite/rpl/t/rpl_xa_rollback_commit_crash_safe.test b/mysql-test/suite/rpl/t/rpl_xa_rollback_commit_crash_safe.test > new file mode 100644 > index 00000000000..1d19f96116e > --- /dev/null > +++ b/mysql-test/suite/rpl/t/rpl_xa_rollback_commit_crash_safe.test > @@ -0,0 +1,97 @@ > +# ==== Purpose ==== > +# > +# Test verifies that XA COMMIT statements are crash safe. XA ROLLBACK > +# > +# ==== Implementation ==== > +# > +# Steps: > +# 0 - Generate 2 explicit XA transactions. 'xa1' and 'xa2'. > +# 'xa1' will be prepared and committed. > +# 1 - For 'xa2' let the XA ROLLBACK be done in binary log and crash the > +# server so that it is not committed in engine. not rolled back > +# 2 - Restart the server. The recovery code should successfully recover > +# 'xa2'. The ROLLBACK should be executed during recovery. > +# 3 - Check the data in table. Only one row should be present in table. > +# 4 - Trying to rollback 'xa2' should report unknown 'XA' error as rollback > +# is already complete during recovery. > +# > +# ==== References ==== > +# > +# MDEV-21469: Implement crash-safe logging of the user XA > + > +--source include/have_innodb.inc > +--source include/master-slave.inc ditto > +--source include/have_debug.inc > + > +connect (master2,localhost,root,,); unused again > +--connection master > +CALL mtr.add_suppression("Found 1 prepared XA transactions"); > + > +CREATE TABLE t ( f INT ) ENGINE=INNODB; > +XA START 'xa1'; > +INSERT INTO t VALUES (20); > +XA END 'xa1'; > +XA PREPARE 'xa1'; > +XA COMMIT 'xa1'; > +--sync_slave_with_master > +--source include/stop_slave.inc > + > +--connection master1 > +XA START 'xa2'; > +INSERT INTO t VALUES (40); > +XA END 'xa2'; > +XA PREPARE 'xa2'; > + > +--write_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect > +wait ditto > +EOF > + > +SET GLOBAL DEBUG_DBUG="d,simulate_crash_after_binlog_commit_or_rollback"; > +--error 2013 # CR_SERVER_LOST > +XA ROLLBACK 'xa2'; > +--source include/wait_until_disconnected.inc > + > +--connection master1 > +--source include/wait_until_disconnected.inc > + > +--connection master > +--source include/wait_until_disconnected.inc > + > +# > +# Server restart > +# > +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect > +restart > +EOF > + > +connection default; > +--enable_reconnect > +--source include/wait_until_connected_again.inc > + > +# rpl_end.inc needs to use the connection server_1 > +connection server_1; > +--enable_reconnect > +--source include/wait_until_connected_again.inc > + > +--connection master > +--enable_reconnect > +--source include/wait_until_connected_again.inc > + > +--connection slave > +--source include/start_slave.inc > +--sync_with_master > + > +--connection master > +SELECT * FROM t; > +XA RECOVER; > +--error 1397 # ER_XAER_NOTA > +XA ROLLBACK 'xa2'; > +SELECT * FROM t; > +--sync_slave_with_master > + > +SELECT * FROM t; > + > +--connection master > +DROP TABLE t; > +--sync_slave_with_master > +--source include/rpl_end.inc > diff --git a/storage/rocksdb/mysql-test/rocksdb_rpl/t/rpl_rocksdb_xa_recover.test b/storage/rocksdb/mysql-test/rocksdb_rpl/t/rpl_rocksdb_xa_recover.test > new file mode 100644 > index 00000000000..c7cefbda4bf > --- /dev/null > +++ b/storage/rocksdb/mysql-test/rocksdb_rpl/t/rpl_rocksdb_xa_recover.test > @@ -0,0 +1,46 @@ > +# MDEV-742, MDEV-21469 XA replication, and xa crash-safe. > +# Tests prove xa state is recovered to a prepared or completed state > +# upon post-crash recovery, incl a multi-engine case. > + > +--source include/have_rocksdb.inc > +--source include/have_innodb.inc > +--source include/have_debug.inc > +--source include/master-slave.inc ditto > +--source include/have_binlog_format_row.inc > + > +--connection slave > +--source include/stop_slave.inc > + > +--connection master > +CALL mtr.add_suppression("Found . prepared XA transactions"); > +CALL mtr.add_suppression("Failed to execute binlog query event"); > +CREATE TABLE t1 (a INT PRIMARY KEY) ENGINE=rocksdb; > +CREATE TABLE t2 (a INT PRIMARY KEY) ENGINE=innodb; > +INSERT INTO t1 SET a = 1; > +INSERT INTO t2 SET a = 1; > + > +--let $xa=xa1 > +--let $when=after_binlog > +--source rpl_rocksdb_xa_recover.inc > + > +--let $xa=xa2 > +--let $when=after_first_engine > +--let $finally_expected=$xa in the list > +--source rpl_rocksdb_xa_recover.inc > + > +--connection slave > +--source include/start_slave.inc > + > +--connection master > +--sync_slave_with_master > +let $diff_tables= master:t1, slave:t1; > +source include/diff_tables.inc; > +let $diff_tables= master:t2, slave:t2; > +source include/diff_tables.inc; > + > +--connection master > +SET SESSION DEBUG_DBUG=""; > +drop table t1,t2; > +--sync_slave_with_master > + > +--source include/rpl_end.inc > diff --git a/storage/rocksdb/mysql-test/rocksdb_rpl/t/rpl_rocksdb_xa_recover.inc b/storage/rocksdb/mysql-test/rocksdb_rpl/t/rpl_rocksdb_xa_recover.inc > new file mode 100644 > index 00000000000..312b8b6a19e > --- /dev/null > +++ b/storage/rocksdb/mysql-test/rocksdb_rpl/t/rpl_rocksdb_xa_recover.inc > @@ -0,0 +1,67 @@ > +# callee of rpl_rocksdb_xa_recover.test > +# requires t1,t2 as defined in the caller. > + > +--let $at=_prepare > +--let $finally_expected=$xa in the list > +--let $init_val=`SELECT MAX(a) from t1 as t_1` why as t_1 ? > +--eval XA START '$xa' > +--eval INSERT INTO t1 SET a=$init_val + 1 > +--eval INSERT INTO t2 SET a=$init_val + 1 > +--eval XA END '$xa' > +--eval SET SESSION DEBUG_DBUG="d,simulate_crash_$when$at" > + > +--exec echo "restart" > $MYSQLTEST_VARDIR/tmp/mysqld.1.expect > +--error 2013 # CR_SERVER_LOST > +--eval XA PREPARE '$xa' > +--source include/wait_until_disconnected.inc > +--let $rpl_server_number = 1 > +--source include/rpl_reconnect.inc > +--echo "*** $finally_expected" > +XA RECOVER; > +--eval XA ROLLBACK '$xa' > +--eval SELECT MAX(a) - $init_val as zero FROM t1 > +--eval SELECT MAX(a) - $init_val as zero FROM t2 > + > + > +--let $at=_commit_or_rollback > +--let $finally_expected=empty list (rolled back) > +--eval XA START '$xa' > +--eval INSERT INTO t1 SET a=$init_val + 2 > +--eval INSERT INTO t2 SET a=$init_val + 2 > +--eval XA END '$xa' > +--eval XA PREPARE '$xa' > + > +--eval SET SESSION DEBUG_DBUG="d,simulate_crash_$when$at" > +--exec echo "restart" > $MYSQLTEST_VARDIR/tmp/mysqld.1.expect > +--error 2013 # CR_SERVER_LOST > +--eval XA ROLLBACK '$xa' > +--source include/wait_until_disconnected.inc > +--let $rpl_server_number = 1 > +--source include/rpl_reconnect.inc > + > +--echo "*** $finally_expected" > +XA RECOVER; > +--eval SELECT MAX(a) - $init_val as zero FROM t1 > +--eval SELECT MAX(a) - $init_val as zero FROM t2 > + > + > +--let $at=_commit_or_rollback > +--let $finally_expected=empty list (committed away) > +--eval XA START '$xa' > +--eval INSERT INTO t1 SET a=$init_val + 3 > +--eval INSERT INTO t2 SET a=$init_val + 3 > +--eval XA END '$xa' > +--eval XA PREPARE '$xa' > + > +--eval SET SESSION DEBUG_DBUG="d,simulate_crash_$when$at" > +--exec echo "restart" > $MYSQLTEST_VARDIR/tmp/mysqld.1.expect > +--error 2013 # CR_SERVER_LOST > +--eval XA COMMIT '$xa' > +--source include/wait_until_disconnected.inc > +--let $rpl_server_number = 1 > +--source include/rpl_reconnect.inc > + > +--echo "*** $finally_expected" > +XA RECOVER; > +--eval SELECT MAX(a) - $init_val as three FROM t1 > +--eval SELECT MAX(a) - $init_val as three FROM t2 > diff --git a/sql/log_event.h b/sql/log_event.h > index 639cbfbe7aa..e1880339e85 100644 > --- a/sql/log_event.h > +++ b/sql/log_event.h > @@ -3614,6 +3616,12 @@ class Gtid_log_event: public Log_event > static const uchar FL_PREPARED_XA= 64; > /* FL_"COMMITTED or ROLLED-BACK"_XA is set for XA transaction. */ > static const uchar FL_COMPLETED_XA= 128; > + /* > + To mark the fact of multiple transactional engine participants > + in the prepared XA. The FL_COMPLETED_XA bit is reused by XA_PREPARE_LOG_EVENT, What do you mean by that? XA_prepare_log_event does not inherit from Gtid_log_event, it cannot use Gtid_log_event flags. And, please, explain in a comment why you can reuse a bit for two different flags. Like "the ambiguity between FL_COMPLETED_XA and FL_MULTI_ENGINE_XA is resolved by the flag FL_PREPARED_XA. if it's set, then 128 means FL_MULTI_ENGINE_XA, if it's not set, then 128 means FL_COMPLETED_XA" > + oth the XA completion events do not need such marking. other > + */ > + static const uchar FL_MULTI_ENGINE_XA= 128; > > #ifdef MYSQL_SERVER > Gtid_log_event(THD *thd_arg, uint64 seq_no, uint32 domain_id, bool standalone, > diff --git a/sql/handler.h b/sql/handler.h > index 0eff7bd930d..9acdd47bfc2 100644 > --- a/sql/handler.h > +++ b/sql/handler.h > @@ -1880,8 +1891,18 @@ class Ha_trx_info > m_ht= ht_arg; > m_flags= (int) TRX_READ_ONLY; /* Assume read-only at start. */ > > - m_next= trans->ha_list; > - trans->ha_list= this; > + if (likely(!past_head)) > + { > + m_next= trans->ha_list; > + trans->ha_list= this; > + } > + else > + { > + DBUG_ASSERT(trans->ha_list); > + > + m_next= trans->ha_list->m_next; > + trans->ha_list->m_next= this; > + } I don't like this fake generalization. There's no point for an arbitrary engine to be put at a specific place in the list. Only binlog is special. At least, remove this new argument and check for binlog_hton here. But really, think of this - binlog is so special, may be it'd be easier not to pretend it's a storage engine at all? No need for a fake binlog_hton, no need to register it, etc. It kind of made sense in 2005, but it doesn't necessarily make it now. > } > > /** Clear, prepare for reuse. */ > diff --git a/sql/handler.cc b/sql/handler.cc > index ffb89b30d92..b981294f0d6 100644 > --- a/sql/handler.cc > +++ b/sql/handler.cc > @@ -1338,6 +1345,9 @@ int ha_prepare(THD *thd) > handlerton *ht= ha_info->ht(); > if (ht->prepare) > { > + DBUG_EXECUTE_IF("simulate_crash_after_first_engine_prepare", > + if (!ha_info->next()) DBUG_SUICIDE();); looks like not simulate_crash_after_first_engine_prepare, but more like simulate_crash_before_last_engine_prepare. it's the same point in time if you have only two engines, but the code still implements "before last", not "after first" > + > if (unlikely(prepare_or_error(ht, thd, all))) > { > ha_rollback_trans(thd, all); > @@ -1368,22 +1378,22 @@ int ha_prepare(THD *thd) > } > > /* > - Like ha_check_and_coalesce_trx_read_only to return counted number of > - read-write transaction participants limited to two, but works in the 'all' > - context. > - Also returns the last found rw ha_info through the 2nd argument. > + Returns counted number of > + read-write recoverable transaction participants optionally limited to two. > + Also optionally returns the last found rw ha_info through the 2nd argument. > */ > -uint ha_count_rw_all(THD *thd, Ha_trx_info **ptr_ha_info) > +uint ha_count_rw_all(THD *thd, Ha_trx_info **ptr_ha_info, bool count_through) I don't understand why you need a new argument. ptr_ha_info==NULL means don't return a value there, this is a standard convention used everywhere. also, it's not an "count rw all" it's are there more rw-all htons than N where N can be 1 or 2. So: * either take N as an argument and return as soon as rw_ha_count > N * or don't overoptimize here and just return full count every time > { > unsigned rw_ha_count= 0; > > for (auto ha_info= thd->transaction.all.ha_list; ha_info; > ha_info= ha_info->next()) > { > - if (ha_info->is_trx_read_write()) > + if (ha_info->is_trx_read_write() && ha_info->ht()->recover) > { > - *ptr_ha_info= ha_info; > - if (++rw_ha_count > 1) > + if (ptr_ha_info) > + *ptr_ha_info= ha_info; > + if (++rw_ha_count > 1 && !count_through) > break; > } > } > @@ -1876,6 +1886,10 @@ commit_one_phase_2(THD *thd, bool all, THD_TRANS *trans, bool is_real_trans) > { > int err; > handlerton *ht= ha_info->ht(); > + > + DBUG_EXECUTE_IF("simulate_crash_after_first_engine_commit_or_rollback", > + if (!ha_info->next()) DBUG_SUICIDE();); 1. same as above 2. why not to use *different* names for these two crashes? in case, you know, you'll want to crash at a specific point in the code, not just somewhere? And in a case you'll actually want to crash just somewhere you can always use "+d,simulate_crash_after_first_engine_commit,simulate_crash_after_first_engine_rollback" (and it should be "before_last", of course, not "after_first") > + > if ((err= ht->commit(ht, thd, all))) > { > my_error(ER_ERROR_DURING_COMMIT, MYF(0), err); > @@ -1987,6 +2001,10 @@ int ha_rollback_trans(THD *thd, bool all) > { > int err; > handlerton *ht= ha_info->ht(); > + > + DBUG_EXECUTE_IF("simulate_crash_after_first_engine_commit_or_rollback", > + if (!ha_info->next()) DBUG_SUICIDE();); ditto > + > if ((err= ht->rollback(ht, thd, all))) > { // cannot happen > my_error(ER_ERROR_DURING_ROLLBACK, MYF(0), err); > @@ -2098,8 +2116,24 @@ int ha_commit_or_rollback_by_xid(XID *xid, bool commit) > xaop.xid= xid; > xaop.result= 1; > > + if (binlog_hton->recover) > + { > + /* > + When the binlogging service is enabled complete the transaction > + by it first. > + */ > + if (commit) > + binlog_hton->commit_by_xid(binlog_hton, xid); > + else > + binlog_hton->rollback_by_xid(binlog_hton, xid); > + } > plugin_foreach(NULL, commit ? xacommit_handlerton : xarollback_handlerton, > MYSQL_STORAGE_ENGINE_PLUGIN, &xaop); > + if (binlog_hton->recover) > + { > + THD *thd= current_thd; > + thd->reset_binlog_completed_by_xid(); > + } 1. this, precisely, shows that binlog does not need a hton anymoreo 2. how can binlog_hton->recover be NULL here? > > return xaop.result; > } > @@ -2222,6 +2258,7 @@ static my_bool xarecover_handlerton(THD *unused, plugin_ref plugin, > > if (hton->recover) > { > + info->recover_htons++; don't forget to subtract 1 for binlog_hton->recover, otherwise it'll make you to rollback everything with a FL_MULTI_ENGINE_XA flag > while ((got= hton->recover(hton, info->list, info->len)) > 0 ) > { > sql_print_information("Found %d prepared transaction(s) in %s", > @@ -2263,7 +2300,21 @@ static my_bool xarecover_handlerton(THD *unused, plugin_ref plugin, > _db_doprnt_("ignore xid %s", xid_to_str(buf, info->list[i])); > }); > xid_cache_insert(info->list + i); > + XID *foreign_xid= info->list + i; swap those two lines ^^^ then you can write xid_cache_insert(foreign_xid); > info->found_foreign_xids++; > + > + /* > + For each foreign xid prepraed in engine, check if it is present in > + xa_prepared_list of binlog. > + */ > + if (info->xa_prepared_list) for simplicity I'd rather always initialize xa_prepared_list and remove this if() > + { > + struct xa_recovery_member *member= NULL; > + if ((member= (xa_recovery_member *) > + my_hash_search(info->xa_prepared_list, foreign_xid->key(), > + foreign_xid->key_length()))) > + member->in_engine_prepare++; > + } > continue; > } > if (IF_WSREP(!(wsrep_emulate_bin_log && > @@ -2362,7 +2424,7 @@ int ha_recover(HASH *commit_list) > info.found_my_xids, opt_tc_log_file); > DBUG_RETURN(1); > } > - if (info.commit_list) > + if (info.commit_list && !info.found_foreign_xids) Why? after the finished crash recovery one can still have transactions in doubt. > sql_print_information("Crash recovery finished."); > DBUG_RETURN(0); > } > diff --git a/sql/log.cc b/sql/log.cc > index 731bb3e98f0..c69b8518cf4 100644 > --- a/sql/log.cc > +++ b/sql/log.cc > @@ -2019,28 +2027,49 @@ static int binlog_xa_recover_dummy(handlerton *hton __attribute__((unused)), > return 0; > } > > - > +/* > + The function invokes binlog_commit() and returns its result > + when it has not yet called it already. > + binlog_cache_mngr::completed_by_xid remembers the fact of > + the 1st of maximum two subsequent calls. > +*/ > static int binlog_commit_by_xid(handlerton *hton, XID *xid) > { > + int rc= 0; > THD *thd= current_thd; > - > - (void) thd->binlog_setup_trx_data(); > + binlog_cache_mngr *cache_mngr= thd->binlog_setup_trx_data(); > > DBUG_ASSERT(thd->lex->sql_command == SQLCOM_XA_COMMIT); > > - return binlog_commit(hton, thd, TRUE); > -} > + if (!cache_mngr->completed_by_xid) On what code path can you have completed_by_xid == false here? > + { > + rc= binlog_commit(hton, thd, TRUE); > + cache_mngr->completed_by_xid= true; > + } > > + return rc; > +} > > +/* > + The function invokes binlog_rollback() and returns its results similarly > + to a commit branch. > +*/ > static int binlog_rollback_by_xid(handlerton *hton, XID *xid) > { > + int rc= 0; > THD *thd= current_thd; > - > - (void) thd->binlog_setup_trx_data(); > + binlog_cache_mngr *cache_mngr= thd->binlog_setup_trx_data(); > > DBUG_ASSERT(thd->lex->sql_command == SQLCOM_XA_ROLLBACK || > (thd->transaction.xid_state.get_state_code() == XA_ROLLBACK_ONLY)); > - return binlog_rollback(hton, thd, TRUE); > + > + if (!cache_mngr->completed_by_xid) > + { > + rc= binlog_rollback(hton, thd, TRUE); > + cache_mngr->completed_by_xid= true; > + } > + > + return rc; > } > > > @@ -2195,6 +2224,12 @@ static int binlog_commit(handlerton *hton, THD *thd, bool all) > if (!all) > cache_mngr->trx_cache.set_prev_position(MY_OFF_T_UNDEF); > > + DBUG_EXECUTE_IF("simulate_crash_after_binlog_commit_or_rollback", > + DBUG_SUICIDE();); > + DEBUG_SYNC(thd, "simulate_hang_after_binlog_prepare"); > + DBUG_EXECUTE_IF("simulate_crash_after_binlog_prepare", > + DBUG_SUICIDE();); wouldn't you say there's one DBUG_EXECUTE_IF too many? Just keep one, "simulate_crash_after_binlog_commit" > + > THD_STAGE_INFO(thd, org_stage); > DBUG_RETURN(error); > } > @@ -2298,6 +2333,9 @@ static int binlog_rollback(handlerton *hton, THD *thd, bool all) > cache_mngr->trx_cache.set_prev_position(MY_OFF_T_UNDEF); > thd->reset_binlog_for_next_statement(); > > + DBUG_EXECUTE_IF("simulate_crash_after_binlog_commit_or_rollback", same as above, use unique dbug keywords, like "simulate_crash_after_binlog_rollback" (because it's rollback here) > + DBUG_SUICIDE();); > + > DBUG_RETURN(error); > } > > @@ -5749,6 +5794,14 @@ binlog_cache_mngr *THD::binlog_setup_trx_data() > DBUG_RETURN(cache_mngr); > } > > +/* > + XA completion flag resetter for ha_commit_or_rollback_by_xid(). > +*/ > +void THD::reset_binlog_completed_by_xid() > +{ > + binlog_setup_trx_data()->reset_completed_by_xid(); > +} I don't think this makes any sense as a THD method > + > /* > Function to start a statement and optionally a transaction for the > binary log. > @@ -10333,14 +10386,108 @@ start_binlog_background_thread() > return 0; > } > > +#ifdef HAVE_REPLICATION > +/** > + Auxiliary function for TC_LOG::recover(). > + @returns a successfully created and inserted @c xa_recovery_member > + into hash @c hash_arg, > + or NULL. > +*/ > +static xa_recovery_member* > +xa_member_insert(HASH *hash_arg, xid_t *xid_arg, xa_binlog_state state_arg, > + MEM_ROOT *ptr_mem_root) > +{ > + xa_recovery_member *member= (xa_recovery_member*) > + alloc_root(ptr_mem_root, sizeof(xa_recovery_member)); > + if (!member) > + return NULL; > + > + member->xid.set(xid_arg); > + member->state= state_arg; > + member->in_engine_prepare= 0; > + return my_hash_insert(hash_arg, (uchar*) member) ? NULL : member; > +} > + > +/* Inserts or updates an existing hash member with a proper state */ > +static bool xa_member_replace(HASH *hash_arg, xid_t *xid_arg, bool is_prepare, > + MEM_ROOT *ptr_mem_root) > +{ > + if(is_prepare) > + { > + if (!(xa_member_insert(hash_arg, xid_arg, XA_PREPARE, ptr_mem_root))) > + return true; > + } > + else > + { > + /* > + Search if XID is already present in recovery_list. If found > + and the state is 'XA_PREPRAED' mark it as XA_COMPLETE. > + Effectively, there won't be XA-prepare event group replay. > + */ > + xa_recovery_member* member; > + if ((member= (xa_recovery_member *) > + my_hash_search(hash_arg, xid_arg->key(), xid_arg->key_length()))) > + { > + if (member->state == XA_PREPARE) > + member->state= XA_COMPLETE; > + } > + else // We found only XA COMMIT during recovery insert to list > + { > + if (!(member= xa_member_insert(hash_arg, > + xid_arg, XA_COMPLETE, ptr_mem_root))) > + return true; > + } > + } > + return false; > +} > +#endif > > +extern "C" uchar *xid_get_var_key(xid_t *entry, size_t *length, define the first argument as const char*, then you won't need to cast in my_hash_init, and you can be sure you pass the correct function with the correct number of arguments, even if my_hash_get_key changes in the future. and cast entry to xid_t* below explicitly. > + my_bool not_used __attribute__((unused))) > +{ > + *length= entry->key_length(); > + return (uchar*) entry->key(); > +} > + > +/** > + Performs recovery based on transaction coordinator log for 2pc. At the > + time of crash, if the binary log was in active state, then recovery for > + "implicit" 'xid's and explicit 'XA' transactions is initiated, > + otherwise merely the gtid binlog state is updated. > + For 'xid' and 'XA' based recovery the following steps are performed. > + > + Identify the active binlog checkpoint file. > + Scan the binary log from the beginning. > + From GTID_LIST and GTID_EVENTs reconstruct the gtid binlog state. > + Prepare a list of 'xid's for recovery. > + Prepare a list of explicit 'XA' transactions for recovery. > + Recover the 'xid' transactions. > + The explicit 'XA' transaction recovery is initiated once all the server > + components are initialized. Please check 'execute_xa_for_recovery()'. why explicit XA recovery is delayed? > + > + Called from @c MYSQL_BIN_LOG::do_binlog_recovery() > + > + @param linfo Store here the found log file name and position to > + the NEXT log file name in the index file. > + > + @param last_log_name Name of the last active binary log at the time of > + crash. > + > + @param first_log Pointer to IO_CACHE of active binary log > + @param fdle Format_description_log_event of active binary log > + @param do_xa Is 2pc recovery needed for 'xid's and explicit XA > + transactions. > + @return indicates success or failure of recovery. > + @retval 0 success > + @retval 1 failure > + > +*/ > int TC_LOG_BINLOG::recover(LOG_INFO *linfo, const char *last_log_name, > IO_CACHE *first_log, > Format_description_log_event *fdle, bool do_xa) > { > Log_event *ev= NULL; > HASH xids; > - MEM_ROOT mem_root; > char binlog_checkpoint_name[FN_REFLEN]; > bool binlog_checkpoint_found; > bool first_round; > @@ -10533,10 +10693,22 @@ int TC_LOG_BINLOG::recover(LOG_INFO *linfo, const char *last_log_name, > > if (do_xa) > { > - if (ha_recover(&xids)) > + if (ha_recover(&xids, &xa_recover_list, &xa_recover_htons)) > goto err2; > > - free_root(&mem_root, MYF(0)); > + DBUG_ASSERT(!xa_recover_list.records || > + (binlog_checkpoint_found && binlog_checkpoint_name[0] != 0)); why is that? > + > + if (!xa_recover_list.records) > + { > + free_root(&mem_root, MYF(0)); > + my_hash_free(&xa_recover_list); > + } > + else > + { > + xa_binlog_checkpoint_name= strmake_root(&mem_root, binlog_checkpoint_name, > + strlen(binlog_checkpoint_name)); > + } > my_hash_free(&xids); > } > return 0; > @@ -10561,6 +10734,219 @@ int TC_LOG_BINLOG::recover(LOG_INFO *linfo, const char *last_log_name, > return 1; > } > > +void MYSQL_BIN_LOG::execute_xa_for_recovery() > +{ > + if (xa_recover_list.records) > + (void) recover_explicit_xa_prepare(); > + free_root(&mem_root, MYF(0)); > + my_hash_free(&xa_recover_list); > +}; > + > +/** > + Performs recovery of user XA transactions. > + 'xa_recover_list' contains the list of XA transactions to be recovered. > + with possible replaying replication event from the binary log. > + > + @return indicates success or failure of recovery. > + @retval false success > + @retval true failure > + > +*/ > +bool MYSQL_BIN_LOG::recover_explicit_xa_prepare() why is it bool, if you only use it in execute_xa_for_recovery() and ignore the return value? > +{ > +#ifndef HAVE_REPLICATION > + /* Can't be supported without replication applier built in. */ > + return false; > +#else > + bool err= true; > + int error=0; > + Relay_log_info *rli= NULL; > + rpl_group_info *rgi; > + THD *thd= new THD(0); /* Needed by start_slave_threads */ > + thd->thread_stack= (char*) &thd; > + thd->store_globals(); > + thd->security_ctx->skip_grants(); > + IO_CACHE log; > + const char *errmsg; > + File file; > + bool enable_apply_event= false; > + Log_event *ev = 0; > + LOG_INFO linfo; > + int recover_xa_count= xa_recover_list.records; > + xa_recovery_member *member= NULL; > + > + if (!(rli= thd->rli_fake= new Relay_log_info(FALSE, "Recovery"))) > + { > + my_error(ER_OUTOFMEMORY, MYF(ME_FATAL), 1); > + goto err2; > + } > + rli->sql_driver_thd= thd; > + static LEX_CSTRING connection_name= { STRING_WITH_LEN("Recovery") }; > + rli->mi= new Master_info(&connection_name, false); > + if (!(rgi= thd->rgi_fake)) > + rgi= thd->rgi_fake= new rpl_group_info(rli); > + rgi->thd= thd; > + thd->system_thread_info.rpl_sql_info= > + new rpl_sql_thread_info(rli->mi->rpl_filter); > + > + if (rli && !rli->relay_log.description_event_for_exec) > + { > + rli->relay_log.description_event_for_exec= > + new Format_description_log_event(4); > + } > + if (find_log_pos(&linfo, xa_binlog_checkpoint_name, 1)) > + { > + sql_print_error("Binlog file '%s' not found in binlog index, needed " > + "for recovery. Aborting.", xa_binlog_checkpoint_name); > + goto err2; > + } > + > + tmp_disable_binlog(thd); > + thd->variables.pseudo_slave_mode= TRUE; > + for (;;) > + { > + if ((file= open_binlog(&log, linfo.log_file_name, &errmsg)) < 0) > + { > + sql_print_error("%s", errmsg); > + goto err1; > + } > + while (recover_xa_count > 0 && > + (ev= Log_event::read_log_event(&log, > + rli->relay_log.description_event_for_exec, > + opt_master_verify_checksum))) > + { > + if (!ev->is_valid()) > + { > + sql_print_error("Found invalid binlog query event %s" > + " at %s:%llu; error %d %s", ev->get_type_str(), > + linfo.log_file_name, > + (ev->log_pos - ev->data_written)); > + goto err1; > + } > + enum Log_event_type typ= ev->get_type_code(); > + ev->thd= thd; > + > + if (typ == FORMAT_DESCRIPTION_EVENT) > + enable_apply_event= true; > + > + if (typ == GTID_EVENT) > + { > + Gtid_log_event *gev= (Gtid_log_event *)ev; > + if (gev->flags2 & > + (Gtid_log_event::FL_PREPARED_XA | Gtid_log_event::FL_COMPLETED_XA)) > + { > + if ((member= > + (xa_recovery_member*) my_hash_search(&xa_recover_list, > + gev->xid.key(), > + gev->xid.key_length()))) > + { > + /* > + When XA PREPARE group of events (as flagged so) check > + its actual binlog state which may be COMPLETED. If the > + state is also PREPARED then analyze through > + in_engine_prepare whether the transaction needs replay. > + */ > + if (gev->flags2 & Gtid_log_event::FL_PREPARED_XA) > + { > + if (member->state == XA_PREPARE) > + { > + // XA prepared is not present in (some) engine then apply it > + if (member->in_engine_prepare == 0) > + enable_apply_event= true; > + else if (gev->flags2 & Gtid_log_event::FL_MULTI_ENGINE_XA && > + xa_recover_htons > member->in_engine_prepare) This needs a comment, something like "FL_MULTI_ENGINE_XA says the transaction must be prepared in more than one engine. We don't know in how many, so if it's not prepared in *all* engines we'll replay it just in case" I presume that (a bit silly) logic will do away when you will port this patch to use a counter instead of a flag. Please, don't push if before that. > + { > + enable_apply_event= true; > + // partially engine-prepared XA is first cleaned out prior replay > + thd->lex->sql_command= SQLCOM_XA_ROLLBACK; > + ha_commit_or_rollback_by_xid(&gev->xid, 0); > + } > + else > + --recover_xa_count; > + } > + } > + else if (gev->flags2 & Gtid_log_event::FL_COMPLETED_XA) > + { > + if (member->state == XA_COMPLETE && > + member->in_engine_prepare > 0) > + enable_apply_event= true; why? you cannot replay a fully prepared and partially committed transaction > + else > + --recover_xa_count; > + } > + } > + } > + } > + > + if (enable_apply_event) > + { > + if ((err= ev->apply_event(rgi))) > + { > + sql_print_error("Failed to execute binlog query event of type: %s," > + " at %s:%llu; error %d %s", ev->get_type_str(), > + linfo.log_file_name, > + (ev->log_pos - ev->data_written), > + thd->get_stmt_da()->sql_errno(), > + thd->get_stmt_da()->message()); > + delete ev; > + goto err1; > + } > + else if (typ == FORMAT_DESCRIPTION_EVENT) > + enable_apply_event=false; how can that happen? > + else if (thd->lex->sql_command == SQLCOM_XA_PREPARE || > + thd->lex->sql_command == SQLCOM_XA_COMMIT || > + thd->lex->sql_command == SQLCOM_XA_ROLLBACK) > + { > + --recover_xa_count; > + enable_apply_event=false; > + > + sql_print_information("Binlog event %s at %s:%llu" > + " successfully applied", > + typ == XA_PREPARE_LOG_EVENT ? > + static_cast<XA_prepare_log_event *>(ev)->get_query() : > + static_cast<Query_log_event *>(ev)->query, > + linfo.log_file_name, (ev->log_pos - ev->data_written)); > + } > + } > + if (typ != FORMAT_DESCRIPTION_EVENT) > + delete ev; > + } > + end_io_cache(&log); > + mysql_file_close(file, MYF(MY_WME)); > + file= -1; > + if (unlikely((error= find_next_log(&linfo, 1)))) > + { > + if (error != LOG_INFO_EOF) > + sql_print_error("find_log_pos() failed (error: %d)", error); > + else > + break; > + } > + } > +err1: > + reenable_binlog(thd); > + /* > + There should be no more XA transactions to recover upon successful > + completion. > + */ > + if (recover_xa_count > 0) > + goto err2; > + sql_print_information("Crash recovery finished."); > + err= false; > +err2: > + if (file >= 0) > + { > + end_io_cache(&log); > + mysql_file_close(file, MYF(MY_WME)); > + } > + thd->variables.pseudo_slave_mode= FALSE; > + delete rli->mi; > + delete thd->system_thread_info.rpl_sql_info; > + rgi->slave_close_thread_tables(thd); > + thd->reset_globals(); > + delete thd; > + > + return err; > +#endif /* !HAVE_REPLICATION */ > +} > > int > MYSQL_BIN_LOG::do_binlog_recovery(const char *opt_name, bool do_xa_recovery) Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

[Maria-developers] MDEV-17399: JSON_TABLE: final input
by Sergey Petrunia 13 Mar '21

13 Mar '21

Hi Alexey, Please find below final bits of input. After these are addressed the patch will probably be good to push (But I'll need a final pass before giving ok to push). == json_table_mysql* tests == * Please remove one of the json_table_mysql and json_table_mysql2 tests. * Please move the remaining test to be in the same suite as json_table.test. == Fix json_table.test == The test has this somewhere in the middle: set optimizer_switch='firstmatch=off'; Please restore the original setting. == ha_json_table::rnd_next() returns 0 on error == Consider this query: select * from json_table( '[ {"name": "X"}, {"name2": "Y"} ]', '$[*]' columns ( col1 varchar(100) path '$.name2' error on empty )) as T; The query produces an error. It happens like so: * the call to ha_json_table::rnd_next() returns 0. * The SQL layer finds out about the error by checking thd->is_error() which returns true. Please make ha_json_table::rnd_next() return an error. == Rename table_function.* == Looking at the contets of sql/table_function.{h,cc}, one can see that there's very little code there that is applicable to generic table function. Almost all code is specifi to JSON_TABLE. Because of that, please rename the files to e.g. json_table.{h,cc}. BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://petrunia.net

2 5

[Maria-developers] MDEV-17399: JSON_TABLE: Odd code in table.cc:create_view_field ?
by Sergey Petrunia 12 Mar '21

12 Mar '21

Hi Alexey, What does the code quoted below do? I don't recall seeing it on previous review iterations. In any case, * It needs a comment about why such special handling is needed. * It needs test coverage - I have reverted these changes and didn't see any test to fail? > diff --git a/sql/table.cc b/sql/table.cc > index 4f65dbd65f4..9c205fc4be6 100644 > --- a/sql/table.cc > +++ b/sql/table.cc > @@ -6722,6 +6722,8 @@ Item *create_view_field(THD *thd, TABLE_LIST *view, Item **field_ref, > LEX_CSTRING *name) > { > bool save_wrapper= thd->lex->first_select_lex()->no_wrap_view_item; > + bool *wrapper_to_set= thd->lex->current_select ? > + &thd->lex->current_select->no_wrap_view_item : &save_wrapper; > Item *field= *field_ref; > DBUG_ENTER("create_view_field"); > > @@ -6737,17 +6739,17 @@ Item *create_view_field(THD *thd, TABLE_LIST *view, Item **field_ref, > } > > DBUG_ASSERT(field); > - thd->lex->current_select->no_wrap_view_item= TRUE; > + *wrapper_to_set= TRUE; > if (!field->is_fixed()) > { > if (field->fix_fields(thd, field_ref)) > { > - thd->lex->current_select->no_wrap_view_item= save_wrapper; > + *wrapper_to_set= save_wrapper; > DBUG_RETURN(0); > } > field= *field_ref; > } > - thd->lex->current_select->no_wrap_view_item= save_wrapper; > + *wrapper_to_set= save_wrapper; > if (save_wrapper) > { > DBUG_RETURN(field); BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://petrunia.net

2 1

[Maria-developers] GSOC21: MDEV-16375 & MDEV-23143
by Hollow Man 10 Mar '21

10 Mar '21

Hi MariaDB community!    Glad to be here! My github account is @HollowMan6. Though I'm new to MariaDB community, I'm interested in MDEV-16375 & MDEV-23143: Function to normalize a json value & missing a JSON_EQUALS function for this year's GSOC project. Here are my first thoughts on these issues:    I have checked part of the codebase and I think the two issues can be merged into one. First we can create a function named JSON_NORMALIZE to normalize the json, which automatically parses the inputed json document, recursively sorts the keys (for objects) / sorts the numbers (for arrays), removes the spaces, and then return the json document string.    Then we create a function named JSON_EQUALS, which can be used to compare 2 json documents for equality realized by first seperately normalize the two json documents using JSON_NORMALIZE, then the 2 can be compared exactly as binary strings.    I have taken some inspirations from the Item_func_json_keys and json_scan_start for parsing json documents, and I think it's possible to sort the keys using std::map in STL for objects.    That's all for my ideas so far. Please correct me if I made some mistakes, and I'm going to work on my ideas later. Cheers! Hollow Man

1 0

Re: [Maria-developers] MDEV-25075: Ignorable index makes the resulting CREATE TABLE invalid
by Sergey Petrunia 10 Mar '21

10 Mar '21

Hi Varun, > diff --git a/sql/lex.h b/sql/lex.h > index 542356c0e433..5a9ec2ec1b35 100644 > --- a/sql/lex.h > +++ b/sql/lex.h > @@ -289,6 +289,7 @@ static SYMBOL symbols[] = { > { "IDENTIFIED", SYM(IDENTIFIED_SYM)}, > { "IF", SYM(IF_SYM)}, > { "IGNORE", SYM(IGNORE_SYM)}, > + { "IGNORED", SYM(IGNORED_SYM)}, > { "IGNORE_DOMAIN_IDS", SYM(IGNORE_DOMAIN_IDS_SYM)}, > { "IGNORE_SERVER_IDS", SYM(IGNORE_SERVER_IDS_SYM)}, > { "IMMEDIATE", SYM(IMMEDIATE_SYM)}, Above this array one can see: >> ... >> NOTE!! >> If you add or delete symbols from this file, you must also update results for >> the perfschema.start_server_low_digest_sql_length test! >> */ >> >> static SYMBOL symbols[] = { and indeed the buildbot shows failures, e.g. http://buildbot.askmonty.org/buildbot/builders/kvm-rpm-centos74-amd64/build… CURRENT_TEST: perfschema.start_server_low_digest_sql_length --- /usr/share/mysql-test/suite/perfschema/r/start_server_low_digest_sql_length.result 2021-03-10 06:23:20.000000000 +0000 +++ /dev/shm/var/4/log/start_server_low_digest_sql_length.reject 2021-03-10 07:37:16.351213429 +0000 @@ -8,5 +8,5 @@ #################################### SELECT event_name, digest, digest_text, sql_text FROM events_statements_history_long; event_name digest digest_text sql_text -statement/sql/select beb5bd93b7e8c45bc5cb6060804988e8 SELECT ? + ? + SELECT ... -statement/sql/truncate faf6cefb662b443f05e97b5c5ab14a59 TRUNCATE TABLE truncat... +statement/sql/select ade774bdfbc132a71810ede8ef469660 SELECT ? + ? + SELECT ... +statement/sql/truncate 0f84807fb4a75d0f391f8a93e7c3c182 TRUNCATE TABLE truncat... Please fix this. BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://petrunia.net

1 1

Re: [Maria-developers] bf677c554ad: UTF8 and utf8
by Sergei Golubchik 08 Mar '21

08 Mar '21

Hi, Rucha! On Mar 08, Rucha Deodhar wrote: > revision-id: bf677c554ad (mariadb-10.5.2-402-gbf677c554ad) > parent(s): a1542f8a573 > author: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > committer: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > timestamp: 2021-02-22 00:37:18 +0530 > message: > > UTF8 and utf8 Two main comments: * your global search-and-replace utf8 to utf8mb3 everywhere has replaced way too much. Please, revert it. * why did you have so many changes in tests with utf8mb4? You set the old-mode to be utf8_is_utf8mb3 by default. It should've applied to tests too, right? So why tests have switched to utf8mb4? Other comments below. > diff --git a/debian/additions/innotop/innotop b/debian/additions/innotop/innotop > index 19229a57a82..4f1e76ea497 100644 > --- a/debian/additions/innotop/innotop > +++ b/debian/additions/innotop/innotop > @@ -264,9 +264,9 @@ sub get_dbh { > MKDEBUG && _d("$dbh: $sql"); > $dbh->do($sql); > MKDEBUG && _d('Enabling charset for STDOUT'); > - if ( $charset eq 'utf8' ) { > - binmode(STDOUT, ':utf8') > - or die "Can't binmode(STDOUT, ':utf8'): $OS_ERROR"; > + if ( $charset eq 'utf8mb3' ) { > + binmode(STDOUT, ':utf8mb3') > + or die "Can't binmode(STDOUT, ':utf8mb3'): $OS_ERROR"; It seems you performed search-and-replace over the whole source tree. I don't think it was particularly meaningful. Here, for example, it was wrong, it's perl code and perl charser utf8, there's no utf8mb3 in perl. also all internal variables you've renamed, like below: > } > else { > binmode(STDOUT) or die "Can't binmode(STDOUT): $OS_ERROR"; > diff --git a/sql/field.h b/sql/field.h > index 4a4f7cee2a5..f672ae7e315 100644 > --- a/sql/field.h > +++ b/sql/field.h > @@ -582,7 +582,7 @@ class Virtual_column_info: public Sql_alloc, > public: > /* Flag indicating that the field is physically stored in the database */ > bool stored_in_db; > - bool utf8; /* Already in utf8 */ > + bool utf8mb3; /* Already in utf8 */ I mean, here ^^^ > bool automatic_name; > Item *expr; > Lex_ident name; /* Name of constraint */ > diff --git a/client/mysql.cc b/client/mysql.cc > index f2938d4c824..b0ee239c0bc 100644 > --- a/client/mysql.cc > +++ b/client/mysql.cc > @@ -1918,6 +1918,10 @@ static int get_options(int argc, char **argv) > > if ((ho_error=handle_options(&argc, &argv, my_long_options, get_one_option))) > return(ho_error); > + if (!strcasecmp("utf8",default_charset)) > + { > + default_charset=(char*)"utf8mb3"; > + } why is that? (and the same everywhere else) > > *mysql_params->p_max_allowed_packet= opt_max_allowed_packet; > *mysql_params->p_net_buffer_length= opt_net_buffer_length; > diff --git a/client/mysqldump.c b/client/mysqldump.c > index 7c363973da2..ad69618fbd1 100644 > --- a/client/mysqldump.c > +++ b/client/mysqldump.c > @@ -3160,7 +3160,7 @@ static uint get_table_structure(const char *table, const char *db, char *table_t > > fprintf(sql_file, > "SET @saved_cs_client = @@character_set_client;\n" > - "SET character_set_client = utf8;\n" > + "SET character_set_client = utf8mb3;\n" why is that? > "/*!50001 CREATE TABLE %s (\n", > result_table); > > diff --git a/include/m_ctype.h b/include/m_ctype.h > index 5fa8f28ff7a..f1a8166d9df 100644 > --- a/include/m_ctype.h > +++ b/include/m_ctype.h > @@ -223,6 +223,7 @@ extern MY_UNI_CTYPE my_uni_ctype[256]; > #define MY_CS_NON1TO1 0x40000 /* Has a complex mapping from characters > to weights, e.g. contractions, expansions, > ignorable characters */ > +#define MY_CS_UTF8_IS_UTF8MB3 0x80000 this isn't a MY_CS_xxx flag, it doesn't belong here, see all other MY_CS_xxx flags - they are *properties* of a charset. They are set as struct charset_info_st my_charset_tis620_thai_ci= { 18,0,0, /* number */ MY_CS_COMPILED|MY_CS_PRIMARY|MY_CS_STRNXFRM|MY_CS_NON1TO1, /* state */ charset_name_tis620, /* cs name */ "tis620_thai_ci", /* name */ that is, when defining a charset. See below > #define MY_CHARSET_UNDEFINED 0 > > /* Character repertoire flags */ > diff --git a/man/mysql.1 b/man/mysql.1 > index 03f23df3660..5ad953ba5eb 100644 > --- a/man/mysql.1 > +++ b/man/mysql.1 > @@ -401,7 +401,7 @@ Use > as the default character set for the client and connection\&. > .sp > A common issue that can occur when the operating system uses > -utf8 > +utf8mb3 this didn't need changing > or another multi\-byte character set is that output from the > \fBmysql\fR > client is formatted incorrectly, due to the fact that the MariaDB client uses the > diff --git a/mysys/charset.c b/mysys/charset.c > index 32cfeb56e2d..9d035406e8d 100644 > --- a/mysys/charset.c > +++ b/mysys/charset.c > @@ -361,7 +361,7 @@ static int add_collation(struct charset_info_st *cs) > newcs->state|= MY_CS_AVAILABLE | MY_CS_LOADED | MY_CS_NONASCII; > #endif > } > - else if (!strcmp(cs->csname, "utf8") || !strcmp(cs->csname, "utf8mb3")) > + else if (!strcmp(cs->csname, "utf8mb3") || !strcmp(cs->csname, "utf8mb3")) one more result of your mechanical search-and-replace, now you compare with "utf8mb3" twice. > { > #if defined (HAVE_CHARSET_utf8mb3) && defined(HAVE_UCA_COLLATIONS) > copy_uca_collation(newcs, newcs->state & MY_CS_NOPAD ? > @@ -767,7 +767,7 @@ static const char* > get_charset_name_alias(const char *name) > { > if (!my_strcasecmp(&my_charset_latin1, name, "utf8mb3")) > - return "utf8"; > + return "utf8mb3"; and again > return NULL; > } > > @@ -1004,13 +1004,19 @@ CHARSET_INFO * > get_charset_by_csname(const char *cs_name, uint cs_flags, myf flags) > { > MY_CHARSET_LOADER loader; > + > + if (!strcasecmp(cs_name, "utf8")) > + { > + cs_name = (const char*)(cs_flags & MY_CS_UTF8_IS_UTF8MB3 ? "utf8mb3" : "utf8mb4"); your flag should be in `myf flags` not in `uint cs_flags` (and defined in my_sys.h with other myf flags) > + } > + > my_charset_loader_init_mysys(&loader); > return my_charset_get_by_name(&loader, cs_name, cs_flags, flags); > } > > > /** > - Resolve character set by the character set name (utf8, latin1, ...). > + Resolve character set by the character set name (utf8mb3, latin1, ...). > > The function tries to resolve character set by the specified name. If > there is character set with the given name, it is assigned to the "cs" > @@ -1453,8 +1459,8 @@ static const MY_CSET_OS_NAME charsets[] = > > {"US-ASCII", "latin1", my_cs_approx}, > > - {"utf8", "utf8", my_cs_exact}, > - {"utf-8", "utf8", my_cs_exact}, > + {"utf8mb3", "utf8mb3", my_cs_exact}, > + {"utf-8mb3", "utf8mb3", my_cs_exact}, utf-8mb3 really? :) > #endif > {NULL, NULL, 0} > }; > diff --git a/plugin/win_auth_client/common.cc b/plugin/win_auth_client/common.cc > index 8b7319252ac..f46b0ec9696 100644 > --- a/plugin/win_auth_client/common.cc > +++ b/plugin/win_auth_client/common.cc > @@ -338,7 +338,7 @@ UPN::UPN(): m_buf(NULL) > m_buf= wchar_to_utf8(buf1, &m_len); > > if(!m_buf) > - ERROR_LOG(ERROR, ("Failed to convert UPN to utf8")); > + ERROR_LOG(ERROR, ("Failed to convert UPN to utf8mb3")); probably wrong too (everything in this file), see the code - it uses windows utf8 charset, not mariadb's utf8 > > // Note: possible error would be indicated by the fact that m_buf is NULL. > return; > diff --git a/sql/mysqld.cc b/sql/mysqld.cc > index 0bf21e02002..fd131497f8c 100644 > --- a/sql/mysqld.cc > +++ b/sql/mysqld.cc > @@ -4053,6 +4056,24 @@ static int init_common_variables() > break; > } > > + if (default_collation_name) > + { > + char *copy_of_name= (char*)default_collation_name; > + char start[6]; > + strncpy(start, default_collation_name, 5); > + char *fname= (char *)"utf8mb3_"; > + if (! strncasecmp("utf8_", start,5)) > + { > + copy_of_name+= 5; > + char result[64]; > + result[63]='\0'; > + strcpy(result, fname); > + strcat(result, copy_of_name); > + result[strlen(copy_of_name)+strlen(fname)]='\0'; > + default_collation_name= (char *) result; > + } > + } 1. wrong indentation 2. shouldn't it be in the get_charset_by_name ? 3. shouldn't it depend on the UTF8_IS_UTF8MB3 ? > + > if (default_collation_name) > { > CHARSET_INFO *default_collation; > diff --git a/sql/sql_class.h b/sql/sql_class.h > index 50b746fe514..f0616b40361 100644 > --- a/sql/sql_class.h > +++ b/sql/sql_class.h > @@ -5248,6 +5249,65 @@ class THD: public THD_count, /* this must be first */ > Item *sp_prepare_func_item(Item **it_addr, uint cols= 1); > bool sp_eval_expr(Field *result_field, Item **expr_item_ptr); > > + CHARSET_INFO *get_charset_by_csname(const char *cs_name, > + uint cs_flags, > + myf myf_flags) const > + { > + return ::get_charset_by_csname(cs_name, > + this->variables.old_behavior & OLD_MODE_UTF8_IS_UTF8MB3 ? > + cs_flags | MY_CS_UTF8_IS_UTF8MB3 : cs_flags, > + myf_flags); > + } > + > + inline const char* get_collation_or_charset_name(const char* name) shouldn't the name be get_collation_name_alias() ? all other method names you've created match corresponding charset-related function names > + { > + char *copy_of_name= (char*)name; > + char start[6]; > + strncpy(start, name, 5); > + char *fname= (char *)(this->variables.old_behavior & OLD_MODE_UTF8_IS_UTF8MB3?"utf8mb3_":"utf8mb4_"); > + if (!strncasecmp("utf8_", start,5)) > + { > + copy_of_name+= 5; > + char result[64]; > + result[63]='\0'; > + strcpy(result, fname); > + strcat(result, copy_of_name); > + result[strlen(copy_of_name)+strlen(fname)]='\0'; > + name= (const char *) result; > + } > + return name; here you return a pointer to the local variable also: indentation, too long line, end-of-line spaces > + } > + > + inline CHARSET_INFO * > + mysqld_collation_get_by_name(const char *name, > + CHARSET_INFO *name_cs= system_charset_info) > + { > + name=this->get_collation_or_charset_name(name); > + return ::mysqld_collation_get_by_name(name, name_cs); > +} > + > +CHARSET_INFO *get_charset_by_name(const char *cs_name, myf flags) > +{ > + > + cs_name=this->get_collation_or_charset_name(cs_name); > + return ::get_charset_by_name(cs_name,flags); here it'd be better to pass MY_UTF8_IS_UTF8MB3 in flags > +} > +my_bool resolve_charset(const char *cs_name, > + CHARSET_INFO *default_cs, > + CHARSET_INFO **cs) > +{ > + cs_name=this->get_collation_or_charset_name(cs_name); > + return ::resolve_charset(cs_name,default_cs,cs); perhaps it'd make sense to introduce flags argument here too? > +} > + > +my_bool resolve_collation(const char *cl_name, > + CHARSET_INFO *default_cl, > + CHARSET_INFO **cl) > +{ > + cl_name= this->get_collation_or_charset_name(cl_name); > + return ::resolve_collation(cl_name,default_cl,cl); > +} please make sure the indentation is correct, I won't comment on it anymore > + > }; > > > diff --git a/sql/sql_db.cc b/sql/sql_db.cc > index 9bf16220535..85e0e0dbd2f 100644 > --- a/sql/sql_db.cc > +++ b/sql/sql_db.cc > @@ -583,9 +583,13 @@ bool load_db_opt(THD *thd, const char *path, Schema_specification_st *create) > default-collation commands. > */ > if (!(create->default_table_charset= > - get_charset_by_csname(pos+1, MY_CS_PRIMARY, MYF(0))) && > + thd->get_charset_by_csname(pos+1, thd->variables.old_behavior & > + OLD_MODE_UTF8_IS_UTF8MB3 ? > + MY_CS_UTF8_IS_UTF8MB3 | MY_CS_PRIMARY : > + MY_CS_PRIMARY, > + MYF(0))) && that's a bit redundant, don't you think? you use both thd-> method that replaces utf_ with a real charset name and *also* you set MY_CS_UTF8_IS_UTF8MB3 flag. there are other places like that, to keep the review shorter I won't comment on them below. > !(create->default_table_charset= > - get_charset_by_name(pos+1, MYF(0)))) > + thd->get_charset_by_name(pos+1, MYF(0)))) > { > sql_print_error("Error while loading database options: '%s':",path); > sql_print_error(ER_THD(thd, ER_UNKNOWN_CHARACTER_SET),pos+1); > diff --git a/sql/sys_vars.cc b/sql/sys_vars.cc > index a75066271d9..fd480bdced2 100644 > --- a/sql/sys_vars.cc > +++ b/sql/sys_vars.cc > @@ -3705,7 +3709,7 @@ static Sys_var_set Sys_old_behavior( > "old_mode", > "Used to emulate old behavior from earlier MariaDB or MySQL versions", > SESSION_VAR(old_behavior), CMD_LINE(REQUIRED_ARG), > - old_mode_names, DEFAULT(0)); > + old_mode_names, DEFAULT(8)); DEFAULT(OLD_MODE_UTF8_IS_UTF8MB3) please > > #if defined(HAVE_OPENSSL) && !defined(EMBEDDED_LIBRARY) > #define SSL_OPT(X) CMD_LINE(REQUIRED_ARG,X) > diff --git a/storage/connect/ha_connect.cc b/storage/connect/ha_connect.cc > index 849cf7594eb..b6e8f3b3226 100644 > --- a/storage/connect/ha_connect.cc > +++ b/storage/connect/ha_connect.cc > @@ -6099,8 +6099,8 @@ static int connect_assisted_discovery(handlerton *, THD* thd, > case FLD_NAME: > if (ttp == TAB_PRX || > (ttp == TAB_CSV && topt->data_charset && > - (!stricmp(topt->data_charset, "UTF8") || > - !stricmp(topt->data_charset, "UTF-8")))) > + (!stricmp(topt->data_charset, "UTF8MB3") || > + !stricmp(topt->data_charset, "UTF-8MB3")))) UTF-8MB3 again > cnm= crp->Kdata->GetCharValue(i); > else > cnm= encode(g, crp->Kdata->GetCharValue(i)); > diff --git a/storage/mroonga/vendor/groonga/CMakeLists.txt b/storage/mroonga/vendor/groonga/CMakeLists.txt > index d271d4c4eb9..fa5d6f97ddd 100644 > --- a/storage/mroonga/vendor/groonga/CMakeLists.txt > +++ b/storage/mroonga/vendor/groonga/CMakeLists.txt > @@ -95,7 +95,7 @@ set(GRN_LOG_PATH > "${CMAKE_INSTALL_PREFIX}/var/log/${GRN_PROJECT_NAME}/${GRN_PROJECT_NAME}.log" > CACHE FILEPATH "log file path") > set(GRN_DEFAULT_ENCODING > - "utf8" > + "utf8mb3" utf8 is better here > CACHE STRING "Groonga's default encoding") > set(GRN_DEFAULT_MATCH_ESCALATION_THRESHOLD > 0 > diff --git a/storage/perfschema/unittest/pfs_connect_attr-t.cc b/storage/perfschema/unittest/pfs_connect_attr-t.cc > index b57ead3ec26..7e5b511d543 100644 > --- a/storage/perfschema/unittest/pfs_connect_attr-t.cc > +++ b/storage/perfschema/unittest/pfs_connect_attr-t.cc > @@ -343,7 +343,12 @@ int main(int, char **) > { > MY_INIT("pfs_connect_attr-t"); > > - cs_cp1251= get_charset_by_csname("cp1251", MY_CS_PRIMARY, MYF(0)); > + cs_cp1251= thd->get_charset_by_csname("cp1251", > + thd->variables.old_behavior & > + OLD_MODE_UTF8_IS_UTF8M3 ? > + MY_CS_PRIMARY | MY_CS_UTF8_IS_UTF8MB3 : well, here there is surely no reason to use MY_CS_UTF8_IS_UTF8MB3, because the charset is a hard-coded literal and it's not "utf8", is it? > + MY_CS_PRIMARY, > + MYF(0)); > if (!cs_cp1251) > diag("skipping the cp1251 tests : missing character set"); > plan(59 + (cs_cp1251 ? 10 : 0)); > diff --git a/strings/ctype.c b/strings/ctype.c > index 46951c3ae1f..9054cbfc3e6 100644 > --- a/strings/ctype.c > +++ b/strings/ctype.c > @@ -37,7 +37,7 @@ > */ > > const char charset_name_latin2[]= "latin2"; > -const char charset_name_utf8[]= "utf8"; > +const char charset_name_utf8[]= "utf8mb3"; better rename the variable to charset_name_utf8mb3 > const char charset_name_utf16[]= "utf16"; > const char charset_name_utf32[]= "utf32"; > const char charset_name_ucs2[]= "ucs2"; > diff --git a/tests/mysql_client_test.c b/tests/mysql_client_test.c > index 0043786d477..37da9e1f62e 100644 > --- a/tests/mysql_client_test.c > +++ b/tests/mysql_client_test.c > @@ -405,7 +405,7 @@ static void test_prepare_simple() > > /* update */ > strmov(query, "UPDATE test_prepare_simple SET id=? " > - "WHERE id=? AND CONVERT(name USING utf8)= ?"); > + "WHERE id=? AND CONVERT(name USING utf8mb3)= ?"); I suspect all tests should better use "utf8" > stmt= mysql_simple_prepare(mysql, query); > check_stmt(stmt); > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

[Maria-developers] MDEV-8306: Some pieces of input
by Sergey Petrunia 05 Mar '21

05 Mar '21

Hi Varun, I'm still looking at MDEV-8306 code. Sending the input I have so far. There's nothing important but I thought I'd share it now so you can address it while I'm continuing with the review. > propagate_equal_field_for_orderby Please use "order_by" as in all other functions, and also change "field" to "fields". The name becomes propagate_equal_fields_for_order_by. > void JOIN::propagate_equal_field_for_orderby() > { > if (!sort_nest_possible) > return; > ORDER *ord; > for (ord= order; ord; ord= ord->next) > { > if (optimizer_flag(thd, OPTIMIZER_SWITCH_ORDERBY_EQ_PROP) && cond_equal) > { Is there any reason not to check optimizer_switch flag value once at the start of the function? > void JOIN_TAB::find_keys_that_can_achieve_ordering() > { > if (!join->sort_nest_possible) > return; > > table->keys_with_ordering.clear_all(); > for (uint index= 0; index < table->s->keys; index++) > { > if (table->keys_in_use_for_query.is_set(index) && > test_if_order_by_key(join, join->order, table, index)) > table->keys_with_ordering.set_bit(index); > } > table->keys_with_ordering.intersect(table->keys_in_use_for_order_by); > } Is there any reason to call test_if_order_by_key() and then interset with keys_in_use_for_order_by? Why not just check that bitmap first, like it's done for keys_with_ordering? ... in struct JOIN_TAB: > void find_keys_that_can_achieve_ordering(); > bool check_if_index_satisfies_ordering(int index_used); are "achieving ordering" and "satisfying ordering" the same thing? If yes, it's better to use one term for this. > Item* Item_subselect::transform(THD *thd, Item_transformer transformer, > bool transform_subquery, uchar *arg) We've already discussed this, but I'll mention it here to keep track: this should ON expressions. I would also consider adding an assert that the subquery's joins do not have the query plans, yet. > class Field { > ... > void statistics_available_via_keys(); > void statistics_available_via_stat_tables(); These names need to be better. Function names typically start with a verb. This is especially true for functions that return nothing. > bool Item_func_between::predicate_selectivity_checker(void *arg) > bool Item_func_in::predicate_selectivity_checker(void *arg) These functions need a comment. > bool Item_equal::predicate_selectivity_checker(void *arg) > { > ... > while (it++) > { > Field *field= it.get_curr_field(); > field->is_range_statistics_available(); What is the above? Does the function have some side-effect? This needs to be fixed or at least commented. ... later in the same function: > it.rewind(); > Item *item= (it++); > SAME_FIELD *same_field_arg= (SAME_FIELD *) arg; > > if (same_field_arg->item == NULL) > { > item->is_resolved_by_same_column(arg); > return false; > } This looks very confusing, too. Why are we checking the first element in the Item_equal. Are we assuming it is the constant? But the execution reaches this point even when Item_equal doesn't contain a constant? > bool Item_equal::is_statistics_available() Maybe, rename this to is_range_statistics_available() to make it clear with what kind of statistics is it? > void add_nest_tables_to_trace(Mat_join_tab_nest_info* nest_info) rename this to trace_XXX() to be uniform with other tracing functions? e.g. rename to trace_tables_in_sort_nest BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://petrunia.net

1 0

[Maria-developers] MDEV-17399: JSON_TABLE: Code cleanup
by Sergey Petrunia 04 Mar '21

04 Mar '21

Hi Alexey, Please check out this patch: http://lists.askmonty.org/pipermail/commits/2021-March/014492.html Any objections to it? BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://petrunia.net

1 0

[Maria-developers] MDEV-17399: JSON_TABLE: Crash with nested path
by Sergey Petrunia 01 Mar '21

01 Mar '21

Hi Alexey, I was looking at Json_table_nested_path::set_position(), wondering why does it have an assignment np->m_null= TRUE; but doesn't clear the NULL values and trying to come up with an example of this going wrong when I've hit this crash: select * from json_table( '[ {"name": "X", "colors":["blue"], "sizes": [1,2,3,4], "prices" : [10,20]}, {"name": "Y", "colors":["red"], "sizes": [10,11], "prices" : [100,200,300]} ]', '$[*]' columns ( seq0 for ordinality, name varchar(4) path '$.name', nested path '$.colors[*]' columns ( seq1 for ordinality, color text path '$' ), nested path '$.sizes[*]' columns ( seq2 for ordinality, size int path '$' ), nested path '$.prices[*]' columns ( seq3 for ordinality, price int path '$' ) ) ) as T order by seq0, name; Note this==NULL: (gdb) wher #0 0x00005555560edf72 in Json_table_nested_path::set_position (this=0x0, j_start=0x7ffeb0016e68 "[ \n {\"name\": \"X\", \n \"colors\":[\"blue\"], \"sizes\": [1,2,3,4], \"prices\" : [10,20]},\n {\"name\": \"Y\", \n \"colors\":[\"red\"], \"sizes\": [10,11], \"prices\" : [100,200,300]}\n]", j_end=0x7ffeb0016f12 "", pos=0x7ffeb0035e51 "\245\245\245\245\245\245\245\245\006") at /home/psergey/dev-git2/10.6-hf-review6/sql/table_function.cc:239 #1 0x00005555560ee12f in Json_table_nested_path::set_position (this=0x7ffeb0017060, j_start=0x7ffeb0016e68 "[ \n {\"name\": \"X\", \n \"colors\":[\"blue\"], \"sizes\": [1,2,3,4], \"prices\" : [10,20]},\n {\"name\": \"Y\", \n \"colors\":[\"red\"], \"sizes\": [10,11], \"prices\" : [100,200,300]}\n]", j_end=0x7ffeb0016f12 "", pos=0x7ffeb0035e48 "") at /home/psergey/dev-git2/10.6-hf-review6/sql/table_function.cc:262 #2 0x00005555560ee9f0 in ha_json_table::rnd_pos (this=0x7ffeb0014f00, buf=0x7ffeb0025570 "\377", pos=0x7ffeb0035e48 "") at /home/psergey/dev-git2/10.6-hf-review6/sql/table_function.cc:434 #3 0x00005555561ca6a4 in handler::ha_rnd_pos (this=0x7ffeb0014f00, buf=0x7ffeb0025570 "\377", pos=0x7ffeb0035e48 "") at /home/psergey/dev-git2/10.6-hf-review6/sql/handler.cc:3101 #4 0x00005555563852e3 in rr_from_pointers (info=0x7ffeb001f9e0) at /home/psergey/dev-git2/10.6-hf-review6/sql/records.cc:615 #5 0x0000555555da4a75 in READ_RECORD::read_record (this=0x7ffeb001f9e0) at /home/psergey/dev-git2/10.6-hf-review6/sql/records.h:81 #6 0x0000555555ee1876 in join_init_read_record (tab=0x7ffeb001f918) at /home/psergey/dev-git2/10.6-hf-review6/sql/sql_select.cc:21644 #7 0x0000555555edf35a in sub_select (join=0x7ffeb001d948, join_tab=0x7ffeb001f918, end_of_records=false) at /home/psergey/dev-git2/10.6-hf-review6/sql/sql_select.cc:20666 #8 0x0000555555ede8e6 in do_select (join=0x7ffeb001d948, procedure=0x0) at /home/psergey/dev-git2/10.6-hf-review6/sql/sql_select.cc:20216 #9 0x0000555555eb24e7 in JOIN::exec_inner (this=0x7ffeb001d948) at /home/psergey/dev-git2/10.6-hf-review6/sql/sql_select.cc:4484 #10 0x0000555555eb1613 in JOIN::exec (this=0x7ffeb001d948) at /home/psergey/dev-git2/10.6-hf-review6/sql/sql_select.cc:4264 Please fix. BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://petrunia.net

2 2

Re: [Maria-developers] ee538938345: MDEV-21117: refine the server binlog-based recovery for semisync
by Sergei Golubchik 26 Feb '21

26 Feb '21

Hi, Sujatha! The main comment - the logic is so complex, I wasn't able to understand it, unfortunately. I've reviewed almost everything, see comments below. But not the Recovery_context methods. Please explain how it works and how all these truncate_validated, truncate_reset_done, truncate_set_in_1st, etc all work together. On Feb 26, Sujatha wrote: > revision-id: ee538938345 (mariadb-10.3.26-68-gee538938345) > parent(s): 7d04ce6a2d4 > author: Sujatha <sujatha.sivakumar(a)mariadb.com> > committer: Andrei Elkin <andrei.elkin(a)mariadb.com> > timestamp: 2021-02-08 17:58:03 +0200 > message: > > MDEV-21117: refine the server binlog-based recovery for semisync > this should be MDEV subject. But if you like this better you can rename the MDEV instead > > Problem: > ======= > When the semisync master is crashed and restarted as slave it could > recover transactions that former slaves may never have seen. > A known method existed to clear out all prepared transactions > with --tc-heuristic-recover=rollback does not care to adjust > binlog accordingly. > > Fix: > === > The binlog-based recovery is made to concern of the slave semisync role of > post-crash restarted server. > No changes in behaviour is done to the "normal" binloggging server > and the semisync master. > > When the restarted server is configured with > --rpl-semi-sync-slave-enabled=1 > the refined recovery attempts to roll back prepared transactions > and truncate binlog accordingly. > In case of a partically committed (that is committed at least > partially > > in one of the engine participants) such transaction gets committed. > It's guaranteed no (partially as well) committed transactions > exist beyond the truncate position. > In case there exists a non-transactional replication event > (being in a way a committed transaction) past the > computed truncate position the recovery ends with an error. > > To facilite the failover on the slave side facilitate > conditions to accept own events (having been discarded by the above recovery) > are relaxed to let so for the semisync slave that connects to master > in gtid mode. gtid_strict_mode is further recommended to secure > from inadvertent re-applying out of order gtids in general. > Non-gtid mode connected semisync slave would require > --replicate-same-server-id (mind --log-slave-updates must be OFF then). Sorry, I failed to understand this paragraph :( > > diff --git a/mysql-test/suite/binlog/t/binlog_truncate_active_log.test b/mysql-test/suite/binlog/t/binlog_truncate_active_log.test > new file mode 100644 > index 00000000000..cf89525dcac > --- /dev/null > +++ b/mysql-test/suite/binlog/t/binlog_truncate_active_log.test > @@ -0,0 +1,76 @@ > +# ==== Purpose ==== > +# > +# Test verifies the truncation of single binary log file. > +# > +# ==== Implementation ==== > +# > +# Steps: > +# 0 - Create table t1 and insert/commit a row. > +# 1 - Insert an another row such that it gets written to binlog but commit > +# in engine fails as server crashed at this point. > +# 2 - Restart server with --rpl-semi-sync-slave-enabled=1 > +# 3 - Upon server start 'master-bin.000001' will be truncated to contain > +# only the first insert > +# > +# ==== References ==== > +# > +# MDEV-21117: recovery for --rpl-semi-sync-slave-enabled server > + > +--source include/have_innodb.inc > +--source include/have_aria.inc > +--source include/have_log_bin.inc > +--source include/have_debug.inc > +--source include/have_binlog_format_statement.inc have_binlog_format_statement obviously implies have_log_bin. it's redundant to specify it explicitly. and you don't need have_debug here > + > +call mtr.add_suppression("Can.t init tc log"); > +call mtr.add_suppression("Aborting"); > + > +# The following cases are tested: > +# A. 2pc transaction is followed by a blank "zero-engines" one > +# B. 2pc transaction follows the blank one > +# C. Similarly to A, with the XA blank transaction > + > +--connection default it's connection default by default (not surprisingly) :) > +RESET MASTER; > +CREATE TABLE t ( f INT ) ENGINE=INNODB; > +CREATE TABLE t2 ( f INT ) ENGINE=INNODB; > +CREATE TABLE tm ( f INT ) ENGINE=Aria; Why Aria? I mean, the default "standard" non-transactional engine is MyISAM. If you use Aria, it means you thought about it and decided to deviate intentionally from the default. What were your considerations? > + > +--echo # Case A. > +# Both are doomed into truncation. > +--let $this_search_pattern = Successfully truncated.*to remove transactions starting from GTID 0-1-6 why not simply --let SEARCH_PATTERN = Successfully truncated...etc... > +--let $query1 = INSERT INTO t VALUES (20) > +--let $query2 = DELETE FROM t2 WHERE f = 666 /* no such record */ > +--source binlog_truncate_active_log.inc > + > +--echo # Case B. > +# The inverted sequence ends up to truncate only $query2 > +--let $this_search_pattern = Successfully truncated.*to remove transactions starting from GTID 0-1-10 > +--let $query1 = DELETE FROM t2 WHERE f = 0 why not `= 666 /* no such record */` ? is `= 0` important here? > +--let $query2 = INSERT INTO t VALUES (20) > +--source binlog_truncate_active_log.inc > + > + > +delimiter |; > +CREATE PROCEDURE sp_blank_xa() > +BEGIN > + XA START 'blank'; > + DELETE FROM t2 WHERE f = 666 /* no such record */; > + XA END 'blank'; > + XA PREPARE 'blank'; > +END| > +delimiter ;| > + > + > +--echo # Case C. > +--let $this_search_pattern = Successfully truncated.*to remove transactions starting from GTID 0-1-13 > +--let $query1 = INSERT INTO t VALUES (20) > +--let $pre_q2 = CALL sp_blank_xa > +--let $query2 = XA COMMIT 'blank' > +--source binlog_truncate_active_log.inc > +DROP PROCEDURE sp_blank_xa; > + > +--echo # Cleanup > +DROP TABLE t,t2,tm; > + > +--echo # End of the tests > diff --git a/mysql-test/suite/binlog/t/binlog_truncate_active_log.inc b/mysql-test/suite/binlog/t/binlog_truncate_active_log.inc > new file mode 100644 > index 00000000000..b1ffaf18268 > --- /dev/null > +++ b/mysql-test/suite/binlog/t/binlog_truncate_active_log.inc > @@ -0,0 +1,70 @@ you need source include/have_debug_sync.inc here, as this file uses DEBUG_SYNC > +connect(master1,localhost,root,,); > +connect(master2,localhost,root,,); > +connect(master3,localhost,root,,); just FYI (no need to change anything), you know that parentheses and extra commas are optional, right? that is, it could be written as connect master1,localhost,root; connect master2,localhost,root; connect master3,localhost,root; > + > +--connection default > + > +# First to commit few transactions > +INSERT INTO t VALUES (10); > +INSERT INTO tm VALUES (10); > + > +--connection master1 > +# Hold insert after write to binlog and before "run_commit_ordered" in engine > +SET DEBUG_SYNC= "commit_before_get_LOCK_commit_ordered SIGNAL master1_ready WAIT_FOR signal_never_arrives"; > +--send_eval $query1 > + > +--connection master2 > +SET DEBUG_SYNC= "now WAIT_FOR master1_ready"; > +if ($pre_q2) is $pre_q2 a flag or a query string? it's used as a flag, it's set as a query string. > +{ > + CALL sp_blank_xa; > +} > +SET DEBUG_SYNC= "commit_before_get_LOCK_after_binlog_sync SIGNAL master2_ready"; > +# To binlog non-xid transactional group which will be truncated all right > +--send_eval $query2 > + > + > +--connection master3 > +SET DEBUG_SYNC= "now WAIT_FOR master2_ready"; > +SELECT @@global.gtid_binlog_pos as 'Before the crash'; > + > +--write_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect > +wait > +EOF > + > +--source include/kill_mysqld.inc > +--source include/wait_until_disconnected.inc you know that kill_mysqld both writes 'wait' to the expect file and includes wait_until_disconnected ? > +# > +# Server restart > +# > +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect > +restart: --rpl-semi-sync-slave-enabled=1 > +EOF why "append_file" and not "write_file" ? and why wait, you could've written restart right away, couldn't you? > + > +connection default; > +--enable_reconnect > +--source include/wait_until_connected_again.inc > + > +# Check error log for a successful truncate message. > +let $log_error_= `SELECT @@GLOBAL.log_error`; > +if(!$log_error_) > +{ > + # MySQL Server on windows is started with --console and thus > + # does not know the location of its .err log, use default location > + let $log_error_ = $MYSQLTEST_VARDIR/log/mysqld.1.err; isn't it always $MYSQLTEST_VARDIR/log/mysqld.1.err? other tests don't do it conditionally > +} > +--let SEARCH_FILE=$log_error_ > +--let SEARCH_RANGE=-50000 if you're searching in the error log - and you are - it's best not to set the SEARCH_RANGE at all. Then it'll search from the last CURRENT_TEST: label. That is, it'll search in all error log entries generated by the current test and not in the error log that came from earlier tests. > +--let SEARCH_PATTERN=$this_search_pattern > +--replace_regex /FOUND [0-9]+/FOUND #/ can it be found multiple times? Why would binlog be truncated more than once? > +--source include/search_pattern_in_file.inc > + > +SELECT @@global.gtid_binlog_pos as 'After the crash'; > +--echo "One row should be present in table 't'" > +SELECT COUNT(*) as 'One' FROM t; Do you know which one? If yes, why not `SELECT * FROM t` ? > + > +# Local cleanup > +DELETE FROM t; > +--disconnect master1 > +--disconnect master2 > +--disconnect master3 > diff --git a/mysql-test/suite/binlog/t/binlog_truncate_multi_engine.test b/mysql-test/suite/binlog/t/binlog_truncate_multi_engine.test > new file mode 100644 > index 00000000000..fe153e5703c > --- /dev/null > +++ b/mysql-test/suite/binlog/t/binlog_truncate_multi_engine.test > @@ -0,0 +1,78 @@ > +# ==== Purpose ==== > +# > +# Test verifies truncation of multiple binary logs. and "with multiple transactional storage engines" ? > +# > +# ==== Implementation ==== > +# > +# Steps: > +# 0 - Create two tables in innodb and rocksdb engines, > +# > +# In loop for A,B,C cases (below) do 1-5: there's no loop now (which is good, don't add it please :) > +# 1 - execute FLUSH LOGS command to generate a new binary log. > +# Start a transaction inserting rows of sufficient sizes > +# so binary log gets rotated at commit > +# 2 - Using debug simulation make the server crash at a point where > +# the transaction is written to binary log *and* either of > +# A. neither of them commits > +# B. only one commits > +# C. both commit > +# 3 - print the # of binlog files before the transaction starts and after its > +# commit is submitted > +# 4 - Restart server with --tc-heuristic-recover=BINLOG_TRUNCATE outdated comment? > +# 5 - Restart normally to print post recovery status. > +# > +# ==== References ==== > +# > +# MDEV-21117: recovery for --rpl-semi-sync-slave-enabled server > + > + > +--source include/have_innodb.inc > +--source include/have_rocksdb.inc > +--source include/have_log_bin.inc this is redundant > +--source include/have_debug.inc > +--source include/have_debug_sync.inc > +--source include/have_binlog_format_row.inc > + > +--let $old_max_binlog_size= `select @@global.max_binlog_size` > +call mtr.add_suppression("Can.t init tc log"); > +call mtr.add_suppression("Aborting"); > +--let $MYSQLD_DATADIR= `SELECT @@datadir` > + > +CREATE TABLE t1 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; > +CREATE TABLE t2 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=rocksdb; > + > +--let $case = A: neither engine committed => rollback & binlog truncate > +# Hold off engine commits after write to binlog and its rotation. > +# The transaction is killed along with the server after that. > +--let $kill_server=1 > +--let $debug_sync_action = "commit_after_release_LOCK_log SIGNAL con1_ready WAIT_FOR signal_no_signal" > +--let $restart_parameters = --rpl-semi-sync-slave-enabled=1 > +--let $test_outcome= 1 row should be present in both tables; binlog is truncated; number of binlogs at reconnect - 3 > + --source binlog_truncate_multi_engine.inc > +--echo Proof of the truncated binlog file is readable (two transactions must be seen): > +--let $MYSQLD_DATADIR = `select @@datadir` no need to set the $MYSQLD_DATADIR twice > +--exec $MYSQL_BINLOG --short-form --skip-annotate-row-events $MYSQLD_DATADIR/master-bin.000002 > + > +--let $case = B: one engine has committed its transaction branch > +# Hold off after one engine has committed. > +--let $kill_server=1 > +--let $debug_sync_action = "commit_after_run_commit_ordered SIGNAL con1_ready WAIT_FOR signal_no_signal" > +--let $restart_simulate_partial_commit = 1 this variable seems to be unused > +--let $restart_parameters = --rpl-semi-sync-slave-enabled=1 --debug-dbug=d,binlog_truncate_partial_commit this seems to be a rather crude way of faking a partially committed transaction. better to crash after the first engine has committed, that'd be much more natural. > +--let $test_outcome= 2 rows should be present in both tables; no binlog truncation; one extra binlog file compare with A; number of binlogs at reconnect - 4 > + --source binlog_truncate_multi_engine.inc > + > +--let $case = C: both engines have committed its transaction branch you didn't do --let $debug_sync_action = "reset", so the old value is used. Intentional? > +# Hold off after both engines have committed. The server is shut down. > +--let $kill_server=0 > +--let $restart_parameters = --rpl-semi-sync-slave-enabled=1 > +--let $test_outcome= 2 rows should be present in both tables; no binlog truncation; the same # of binlog files as in B; number of binlogs at reconnect - 4 > + --source binlog_truncate_multi_engine.inc > + > + > + > +DROP TABLE t1, t2; > +--replace_result $old_max_binlog_size VALUE_AT_START > +--eval SET @@global.max_binlog_size = $old_max_binlog_size better use evalp instead of eval, then you won't need replace_result but really I think you don't need to save/restore max_binlog_size at all, because you restart the server anyway. > + > +--echo # End of the tests > diff --git a/mysql-test/suite/binlog/t/binlog_truncate_multi_engine.inc b/mysql-test/suite/binlog/t/binlog_truncate_multi_engine.inc > new file mode 100644 > index 00000000000..c260a7987e2 > --- /dev/null > +++ b/mysql-test/suite/binlog/t/binlog_truncate_multi_engine.inc > @@ -0,0 +1,64 @@ > +# > +# Loop body of binlog_truncate_multi_engine.test > +# Parameters: > +# $debug_sync_action describes debug-sync actions > +# $kill_server 1 when to crash, 0 for regular restart > +# $restart_parameters the caller may simulate partial commit at recovery > +# $test_outcome summary of extected results > +# $MYSQLD_DATADIR > + > +--echo # > +--echo # > +--echo # Case $case > +--echo # > +RESET MASTER; > +FLUSH LOGS; > +SET GLOBAL max_binlog_size= 4096; > + > +connect(con1,localhost,root,,); > +#--connection con1 > +--echo List of binary logs before rotation > +--source include/show_binary_logs.inc > +INSERT INTO t1 VALUES (1, REPEAT("x", 1)); > +INSERT INTO t2 VALUES (1, REPEAT("x", 1)); > +BEGIN; > + INSERT INTO t1 VALUES (2, REPEAT("x", 4100)); > + INSERT INTO t2 VALUES (2, REPEAT("x", 4100)); > + > +--eval SET DEBUG_SYNC= $debug_sync_action > +send COMMIT; > + > +--connection default > +SET DEBUG_SYNC= "now WAIT_FOR con1_ready"; > +--echo List of binary logs after rotation > +--source include/show_binary_logs.inc > + > +--echo # restart the server with $restart_parameters > +if ($kill_server) > +{ > + --echo # the server is crashed > + --source include/kill_mysqld.inc > + --source include/start_mysqld.inc > +} > +if (!$kill_server) > +{ > + --echo # the server is restarted > + --source include/restart_mysqld.inc it'd be simpler not to use $kill_server at all. And instead write let $shutdown_timeout=0; instead of $kill_server=1; and let $shutdown_timeout=; instead of $kill_server=0; and above you won't need two if's, you can just do source include/restart_mysqld.inc; > +} > + > +--connection default > +--echo # > +--echo # *** Summary: $test_outcome: > +--echo # > +SELECT COUNT(*) FROM t1; > +SELECT COUNT(*) FROM t2; > +SELECT @@GLOBAL.gtid_binlog_state; > +SELECT @@GLOBAL.gtid_binlog_pos; > +--echo List of binary logs at the end of the tests > +--source include/show_binary_logs.inc > +--echo # *** > +# cleanup > +DELETE FROM t1; > +DELETE FROM t2; > +--disconnect con1 > +--echo # > diff --git a/mysql-test/suite/binlog/t/binlog_truncate_multi_log.test b/mysql-test/suite/binlog/t/binlog_truncate_multi_log.test > new file mode 100644 > index 00000000000..231e90dbdc9 > --- /dev/null > +++ b/mysql-test/suite/binlog/t/binlog_truncate_multi_log.test > @@ -0,0 +1,105 @@ > +# ==== Purpose ==== > +# > +# Test verifies truncation of multiple binary logs. > +# > +# ==== Implementation ==== > +# > +# Steps: > +# 1 - Set max_binlog_size= 4096, to help a series of inserts into a > +# transaction table 'ti' get binlog rotated so many time while the > +# transactions won't be committed, being stopped at > +# a prior to commit debug_sync point > +# 2 - kill and restart the server as semisync slave successfully to > +# end with an expected first binlog and the gtid state. > +# > +# ==== References ==== > +# MDEV-21117: recovery for --rpl-semi-sync-slave-enabled server > + > +--source include/have_innodb.inc > +--source include/have_log_bin.inc > +--source include/have_debug.inc > +--source include/have_debug_sync.inc > +--source include/have_binlog_format_row.inc > + > +call mtr.add_suppression("Can.t init tc log"); > +call mtr.add_suppression("Aborting"); > + > +--let $old_max_binlog_size= `select @@global.max_binlog_size` > +SET @@global.max_binlog_size= 4096; > + > +RESET MASTER; > +FLUSH LOGS; > +CREATE TABLE ti (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; > +CREATE TABLE tm (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=MyISAM; > + > +connect(master1,localhost,root,,); > +--echo "List of binary logs before rotation" > +--source include/show_binary_logs.inc > + > +# Some load to either non- and transactional egines > +# that should not affect the following recovery: > +INSERT INTO ti VALUES(1,"I am gonna survive"); > +INSERT INTO tm VALUES(1,"me too!"); > + > +# hold on near engine commit > +SET DEBUG_SYNC= "commit_after_release_LOCK_after_binlog_sync SIGNAL master1_ready WAIT_FOR con1_go"; > +--send_eval INSERT INTO ti VALUES (2, REPEAT("x", 4100)) > + > +connect(master2,localhost,root,,); > +# The 2nd trx for recovery, it does not rotate binlog > +SET DEBUG_SYNC= "now WAIT_FOR master1_ready"; > +SET DEBUG_SYNC= "commit_before_get_LOCK_commit_ordered SIGNAL master2_ready WAIT_FOR master2_go"; > +--send_eval INSERT INTO ti VALUES (3, "not gonna survive") > + > +--connection default > +SET DEBUG_SYNC= "now WAIT_FOR master2_ready"; > +--echo "List of binary logs before crash" > +--source include/show_binary_logs.inc > +--echo # The gtid binlog state prior the crash will be truncated at the end of the test > +SELECT @@global.gtid_binlog_state; > +--write_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect > +wait > +EOF > + > +--source include/kill_mysqld.inc > +--source include/wait_until_disconnected.inc > + > +# > +# Server restart > +# > +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect > +restart: --rpl-semi-sync-slave-enabled=1 > +EOF > + > +connection default; > +--enable_reconnect > +--source include/wait_until_connected_again.inc > + > +# Check error log for a successful truncate message. > +let $log_error_= `SELECT @@GLOBAL.log_error`; > +if(!$log_error_) > +{ > + # MySQL Server on windows is started with --console and thus > + # does not know the location of its .err log, use default location > + let $log_error_ = $MYSQLTEST_VARDIR/log/mysqld.1.err; > +} > +--let SEARCH_FILE=$log_error_ > +--let SEARCH_RANGE=-50000 > +--let SEARCH_PATTERN=truncated binlog file:.*master.*000002 > +--replace_regex /FOUND [0-9]+/FOUND #/ > +--source include/search_pattern_in_file.inc > + > + > +--echo "One record should be present in table" > +SELECT count(*) FROM ti; > + > +--echo # The truncated gtid binlog state > +SELECT @@global.gtid_binlog_state; > +SELECT @@global.gtid_binlog_pos; > + > +--echo # Cleanup > +--replace_result $old_max_binlog_size VALUE_AT_START > +--eval SET @@global.max_binlog_size = $old_max_binlog_size > +DROP TABLE ti; > + many of my comments above apply to this test too > +--echo # End of the tests > diff --git a/mysql-test/suite/binlog/t/binlog_truncate_multi_log_unsafe.test b/mysql-test/suite/binlog/t/binlog_truncate_multi_log_unsafe.test > new file mode 100644 > index 00000000000..6aba2935fde > --- /dev/null > +++ b/mysql-test/suite/binlog/t/binlog_truncate_multi_log_unsafe.test > @@ -0,0 +1,142 @@ > +# ==== Purpose ==== > +# The test verifies attempt to recover by the semisync slave server whose > +# binlog is unsafe for truncation. > +# > +# ==== Implementation ==== > +# 2 binlog files are created with the 1st one destined to be the binlog > +# checkpoint file for recovery. > +# The final group of events is replication unsafe (myisam INSERT). > +# Therefore the semisync slave recovery may not. > +# > +# Steps: > +# 0 - Set max_binlog_size= 4096, to help an insert into a > +# transaction table 'ti' get binlog rotated while the > +# transaction won't be committed, being stopped at > +# a prior to commit debug_sync point > +# 1 - insert into a non-transactional 'tm' table completes with > +# binary logging as well > +# 2 - kill and attempt to restart the server as semisync slave that > +# must produce an expected unsafe-to-recover error > +# 3 - complete the test with a normal restart that successfully finds and > +# commits the transaction in doubt. > +# > +# ==== References ==== > +# MDEV-21117: recovery for --rpl-semi-sync-slave-enabled server > + > + > +--source include/have_innodb.inc > +--source include/have_log_bin.inc > +--source include/have_debug.inc > +--source include/have_debug_sync.inc > +--source include/have_binlog_format_row.inc > + > +--let $old_max_binlog_size= `select @@global.max_binlog_size` > +SET @@global.max_binlog_size= 4096; > + > +call mtr.add_suppression("Table '.*tm' is marked as crashed and should be repaired"); > +call mtr.add_suppression("Got an error from unknown thread"); > +call mtr.add_suppression("Checking table: '.*tm'"); > +call mtr.add_suppression("Recovering table: '.*tm'"); > +call mtr.add_suppression("Cannot trim the binary log to file"); > +call mtr.add_suppression("Crash recovery failed"); > +call mtr.add_suppression("Can.t init tc log"); > +call mtr.add_suppression("Aborting"); > +call mtr.add_suppression("Found 1 prepared transactions"); > +call mtr.add_suppression("mysqld: Table.*tm.*is marked as crashed"); > +call mtr.add_suppression("Checking table.*tm"); > + > +RESET MASTER; > +FLUSH LOGS; > +CREATE TABLE ti (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; > +CREATE TABLE tm (f INT) ENGINE=MYISAM; > + > +--let $row_count = 5 > +--let $i = 3 > +--disable_query_log > +while ($i) > +{ > + --eval INSERT INTO ti VALUES ($i, REPEAT("x", 1)) > +--dec $i > +} > +--enable_query_log > +INSERT INTO tm VALUES(1); > + > +connect(master1,localhost,root,,); > +connect(master2,localhost,root,,); > +connect(master3,localhost,root,,); > + > +--connection master1 > + > +# The 1st trx binlogs, rotate binlog and hold on before committing at engine > +SET DEBUG_SYNC= "commit_after_release_LOCK_after_binlog_sync SIGNAL master1_ready WAIT_FOR master1_go"; you don't wait for master1_ready anywhere. intentional? > +--send_eval INSERT INTO ti VALUES ($row_count - 1, REPEAT("x", 4100)) if you plan to insert rows 1,2,3,4,5 then your $i=3 and $row_count=5 are not independent, you should use $i=$row_count-2. (probably $i=`select $row_count-2`) or don't pretent this test is tunable and use literals instead of variables. > + > +--connection master2 > + > +# The 2nd trx for recovery, it does not rotate binlog > +SET DEBUG_SYNC= "commit_before_get_LOCK_commit_ordered SIGNAL master2_ready WAIT_FOR master2_go"; > +--send_eval INSERT INTO ti VALUES ($row_count, REPEAT("x", 1)) > + > +--connection master3 > +SET DEBUG_SYNC= "now WAIT_FOR master2_ready"; > +SET DEBUG_SYNC= "commit_before_get_LOCK_after_binlog_sync SIGNAL master3_ready"; > +--send INSERT INTO tm VALUES (2) > + > +--connection default > +SET DEBUG_SYNC= "now WAIT_FOR master3_ready"; > +--echo # The gtid binlog state prior the crash must be restored at the end of the testSELECT @@global.gtid_binlog_state; eh? Was this SELECT supposed to be on a separate line? > + > +--write_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect > +wait > +EOF > + > +SELECT @@global.gtid_binlog_state; > +--source include/kill_mysqld.inc > +--source include/wait_until_disconnected.inc > + > +# > +# Server restarts > +# > +--echo # Failed restart as the semisync slave > +--error 1 > +--exec $MYSQLD_LAST_CMD --rpl-semi-sync-slave-enabled=1 >> $MYSQLTEST_VARDIR/log/mysqld.1.err 2>&1 > + > +--echo # Normal restart > +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect > +restart: Okay. *this* is a reasonable use of wait/restart in the expect file. In other cases, I think, you can just restart, no need to wait, as you don't do anything while the server is down. still, here you don't need to write 'wait' to the expect file and don't need to wait_until_disconnected (kill_mysqld is doing both) > +EOF > + > +connection default; > +--enable_reconnect > +--source include/wait_until_connected_again.inc > + > +# Check error log for correct messages. > +let $log_error_= `SELECT @@GLOBAL.log_error`; > +if(!$log_error_) > +{ > + # MySQL Server on windows is started with --console and thus > + # does not know the location of its .err log, use default location > + let $log_error_ = $MYSQLTEST_VARDIR/log/mysqld.1.err; > +} > +--let SEARCH_FILE=$log_error_ > +--let SEARCH_RANGE=-50000 > +--let SEARCH_PATTERN=Cannot trim the binary log to file > +--replace_regex /FOUND [0-9]+/FOUND #/ > +--source include/search_pattern_in_file.inc > + > +--echo # Proof that the in-doubt transactions are recovered by the 2nd normal server restart > +--eval SELECT COUNT(*) = $row_count as 'True' FROM ti > +# myisam table may require repair (which is not tested here) > +--disable_warnings > +SELECT COUNT(*) <= 1 FROM tm; > +--enable_warnings > + > +--echo # The gtid binlog state prior the crash is restored now > +SELECT @@GLOBAL.gtid_binlog_state; > +SELECT @@GLOBAL.gtid_binlog_pos; > + > +--echo # Cleanup > +--replace_result $old_max_binlog_size VALUE_AT_START > +--eval SET @@global.max_binlog_size = $old_max_binlog_size see comments elsewhere about $old_max_binlog_size, -50000, $log_error_, etc > +DROP TABLE ti, tm; > +--echo End of test > diff --git a/mysql-test/suite/rpl/t/rpl_semi_sync_fail_over.cnf b/mysql-test/suite/rpl/t/rpl_semi_sync_fail_over.cnf > new file mode 100644 > index 00000000000..3518eb95b67 > --- /dev/null > +++ b/mysql-test/suite/rpl/t/rpl_semi_sync_fail_over.cnf > @@ -0,0 +1,13 @@ > +!include suite/rpl/rpl_1slave_base.cnf > +!include include/default_client.cnf > + > + > +[mysqld.1] > +log-slave-updates > +gtid-strict-mode=1 > +max_binlog_size= 4096 > + > +[mysqld.2] > +log-slave-updates > +gtid-strict-mode=1 > +max_binlog_size= 4096 why not an .opt file? why max_binlog_size here and not at run-time like in other tests? > diff --git a/mysql-test/suite/rpl/t/rpl_semi_sync_fail_over.test b/mysql-test/suite/rpl/t/rpl_semi_sync_fail_over.test > new file mode 100644 > index 00000000000..972aaf2c8b4 > --- /dev/null > +++ b/mysql-test/suite/rpl/t/rpl_semi_sync_fail_over.test > @@ -0,0 +1,144 @@ > +# ==== Purpose ==== > +# > +# Test verifies replication failover scenario. > +# > +# ==== Implementation ==== > +# > +# Steps: > +# 0 - Having two servers 1 and 2 enable semi-sync replication with > +# with the master wait 'after_sync'. > +# 1 - Insert a row. While inserting second row simulate > +# a server crash at once the transaction is written to binlog, flushed > +# and synced but the binlog position is not updated. > +# 2 - Post crash-recovery on the old master execute there CHANGE MASTER > +# TO command to connect to server id 2. > +# 3 - The old master new slave server 1 must connect to the new > +# master server 2. > +# 4 - repeat the above to crash the new master and restore in role the old one > +# > +# ==== References ==== > +# > +# MDEV-21117: recovery for --rpl-semi-sync-slave-enabled server > + > + > +--source include/have_innodb.inc > +--source include/have_log_bin.inc > +--source include/have_debug.inc > +--source include/have_debug_sync.inc > +--source include/have_binlog_format_row.inc > +--let $rpl_topology=1->2 > +--source include/rpl_init.inc isn't this a normal master-slave.inc topology? > + > +--connection server_2 > +--source include/stop_slave.inc > + > +--connection server_1 > +RESET MASTER; > + > +--connection server_2 > +RESET MASTER; > +set @@global.rpl_semi_sync_slave_enabled = 1; > +set @@global.gtid_slave_pos = ""; > +CHANGE MASTER TO master_use_gtid= slave_pos; > +--source include/start_slave.inc > + > + > +--connection server_1 > +ALTER TABLE mysql.gtid_slave_pos ENGINE=InnoDB; why? > +set @@global.rpl_semi_sync_master_enabled = 1; > +set @@global.rpl_semi_sync_master_wait_point=AFTER_SYNC; > + > +call mtr.add_suppression("Can.t init tc log"); > +call mtr.add_suppression("Aborting"); > +call mtr.add_suppression("1 client is using or hasn.t closed the table properly"); > +call mtr.add_suppression("Table './mtr/test_suppressions' is marked as crashed and should be repaired"); > + > +CREATE TABLE t1 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; > +INSERT INTO t1 VALUES (1, 'dummy1'); > + > +# > +# CRASH the original master, and FAILOVER to the new > +# > + > +# value 1 for server id 1 -> 2 failover > +--let $failover_to_slave=1 > +--let $query_to_crash= INSERT INTO t1 VALUES (2, REPEAT("x", 4100)) > +--let $log_search_pattern=truncated binlog file:.*master.*000001 > +--source rpl_semi_sync_crash.inc > + > +--connection server_2 > +--let $rows_so_far=3 > +--eval INSERT INTO t1 VALUES ($rows_so_far, 'dummy3') > +--save_master_pos > +--echo # The gtid state on current master must be equal to ... > +SHOW VARIABLES LIKE 'gtid_binlog_pos'; > + > +--connection server_1 > +--sync_with_master > +--eval SELECT COUNT(*) = $rows_so_far as 'true' FROM t1 > +--echo # ... the gtid states on the slave: > +SHOW VARIABLES LIKE 'gtid_slave_pos'; > +SHOW VARIABLES LIKE 'gtid_binlog_pos'; > + > +--connection server_2 > +# > +# CRASH the new master and FAILOVER back to the original > +# > + > +# value 0 for the reverse server id 2 -> 1 failover > +--let $failover_to_slave=0 > +--let $query_to_crash = INSERT INTO t1 VALUES (4, REPEAT("x", 4100)) > +--let $query2_to_crash= INSERT INTO t1 VALUES (5, REPEAT("x", 4100)) > +--let $log_search_pattern=truncated binlog file:.*slave.*000001 > +--source rpl_semi_sync_crash.inc > + > +--connection server_1 > +--let $rows_so_far=6 > +--eval INSERT INTO t1 VALUES ($rows_so_far, 'Done') > +--save_master_pos > +--echo # The gtid state on current master must be equal to ... > +SHOW VARIABLES LIKE 'gtid_binlog_pos'; > + > +--connection server_2 > +--sync_with_master > +--eval SELECT COUNT(*) = $rows_so_far as 'true' FROM t1 > +--echo # ... the gtid states on the slave: > +SHOW VARIABLES LIKE 'gtid_slave_pos'; > +SHOW VARIABLES LIKE 'gtid_binlog_pos'; > + > + > +--let $diff_tables=server_1:t1, server_2:t1 > +--source include/diff_tables.inc > + > +# > +--echo # Cleanup > +# > +--connection server_1 > +DROP TABLE t1; > +--save_master_pos > + > +--connection server_2 > +--sync_with_master > +--source include/stop_slave.inc > + > +--connection server_1 > +set @@global.rpl_semi_sync_master_enabled = 0; > +set @@global.rpl_semi_sync_slave_enabled = 0; > +set @@global.rpl_semi_sync_master_wait_point=default; > +RESET SLAVE; > +RESET MASTER; > + > +--connection server_2 > +set @@global.rpl_semi_sync_master_enabled = 0; > +set @@global.rpl_semi_sync_slave_enabled = 0; > +set @@global.rpl_semi_sync_master_wait_point=default; > + > +--replace_result $SERVER_MYPORT_1 SERVER_MYPORT_1 > +eval CHANGE MASTER TO master_host='127.0.0.1', master_port=$SERVER_MYPORT_1, master_user='root', master_use_gtid=no; evalp > +--source include/start_slave.inc > + > +connection default; > +--enable_reconnect > +--source include/wait_until_connected_again.inc > + > +--source include/rpl_end.inc > diff --git a/mysql-test/suite/rpl/t/rpl_semi_sync_crash.inc b/mysql-test/suite/rpl/t/rpl_semi_sync_crash.inc > new file mode 100644 > index 00000000000..4289df9155f > --- /dev/null > +++ b/mysql-test/suite/rpl/t/rpl_semi_sync_crash.inc > @@ -0,0 +1,87 @@ > +if ($failover_to_slave) > +{ > + --let $server_to_crash=1 > + --let $server_to_promote=2 > + --let $new_master_port=$SERVER_MYPORT_2 > + --let $client_port=$SERVER_MYPORT_1 > + > + --connect (conn_client,127.0.0.1,root,,test,$SERVER_MYPORT_1,) > +} > +if (!$failover_to_slave) > +{ > + --let $server_to_crash=2 > + --let $server_to_promote=1 > + --let $new_master_port=$SERVER_MYPORT_1 > + --let $client_port=$SERVER_MYPORT_2 > + > + --connect (conn_client,127.0.0.1,root,,test,$SERVER_MYPORT_2,) > +} > + > + > +# Hold insert after write to binlog and before "run_commit_ordered" in engine > + > +SET DEBUG_SYNC= "commit_after_release_LOCK_after_binlog_sync SIGNAL con1_ready WAIT_FOR con1_go"; > +--send_eval $query_to_crash > + > +# complicate recovery with an extra binlog file > +if (!$failover_to_slave) > +{ > + --connect (conn_client_2,127.0.0.1,root,,test,$SERVER_MYPORT_2,) > + # use the same signal with $query_to_crash > + SET DEBUG_SYNC= "now WAIT_FOR con1_ready"; > + SET DEBUG_SYNC= "commit_after_release_LOCK_lock SIGNAL con1_ready WAIT_FOR con2_go"; both signal con1_ready? which one will you be waiting below then? > + --send_eval $query2_to_crash > +} > + > +--connection server_$server_to_crash > +SET DEBUG_SYNC= "now WAIT_FOR con1_ready"; > +--source include/kill_mysqld.inc > +--source include/wait_until_disconnected.inc > + > +--connection server_$server_to_promote > +--error 2003 > +--source include/stop_slave.inc > +SELECT @@GLOBAL.gtid_current_pos; > + > +--let $_expect_file_name=$MYSQLTEST_VARDIR/tmp/mysqld.$server_to_crash.expect wasn't $_expect_file_name already set above? > +--let $restart_parameters=--rpl-semi-sync-slave-enabled=1 > +--let $allow_rpl_inited=1 > +--source include/start_mysqld.inc > + > +--connection server_$server_to_crash > +--enable_reconnect > +--source include/wait_until_connected_again.inc > + > +# Check error log for correct messages. > +let $log_error_= `SELECT @@GLOBAL.log_error`; > +if(!$log_error_) > +{ > + # MySQL Server on windows is started with --console and thus > + # does not know the location of its .err log, use default location > + let $log_error_ = $MYSQLTEST_VARDIR/log/mysqld.$server_to_crash.err; > +} > +--let SEARCH_FILE=$log_error_ > +--let SEARCH_RANGE=-50000 > +--let SEARCH_PATTERN=$log_search_pattern > +--source include/search_pattern_in_file.inc > + > +--disconnect conn_client > + > +# > +# FAIL OVER now to new master > +# > +--connection server_$server_to_promote > +set global rpl_semi_sync_master_enabled = 1; > +set global rpl_semi_sync_master_wait_point=AFTER_SYNC; > + > +--connection server_$server_to_crash > +--let $master_port=$SERVER_MYPORT_2 > +if (`select $server_to_crash = 2`) > +{ > + --let $master_port=$SERVER_MYPORT_1 > +} > +--replace_result $SERVER_MYPORT_1 SERVER_MYPORT_1 $SERVER_MYPORT_2 SERVER_MYPORT_2 > +eval CHANGE MASTER TO master_host='127.0.0.1', master_port=$new_master_port, master_user='root', master_use_gtid=SLAVE_POS; > +set global rpl_semi_sync_slave_enabled = 1; > +set @@global.gtid_slave_pos=@@global.gtid_binlog_pos; > +--source include/start_slave.inc see comments about -50000, $log_error_, wait_until_disconnected, evalp, etc > diff --git a/sql/sql_class.h b/sql/sql_class.h > index 140394fefc1..87fa88d4c89 100644 > --- a/sql/sql_class.h > +++ b/sql/sql_class.h > @@ -4632,7 +4632,7 @@ class THD :public Statement, > LF_PINS *tdc_hash_pins; > LF_PINS *xid_hash_pins; > bool fix_xid_hash_pins(); > - > + bool is_1pc_ro_trans; I don't like how it looks, it's in THD (for every single connection and more) and it's only used in some exotic cases. It's set in two places around ha_commit_one_phase(), when it would be better to set it inside ha_commit_one_phase(), in one place only. But I cannot suggest how to get rid of it yet as I don't understand what it's for, I've asked a question about it below, in Gtid_log_event::Gtid_log_event > /* Members related to temporary tables. */ > public: > /* Opened table states. */ > diff --git a/sql/handler.h b/sql/handler.h > index 2a346e8d9d1..bd39f46bf1f 100644 > --- a/sql/handler.h > +++ b/sql/handler.h > @@ -873,6 +873,14 @@ typedef struct xid_t XID; > /* The 'buf' has to have space for at least SQL_XIDSIZE bytes. */ > uint get_sql_xid(XID *xid, char *buf); > > +/* struct for heuristic binlog truncate recovery */ it's not "heuristic binlog truncate recovery" anymore, please don't call it that, it's confusing > +struct xid_recovery_member > +{ > + my_xid xid; > + uint in_engine_prepare; // number of engines that have xid prepared > + bool decided_to_commit; > +}; > + > /* for recover() handlerton call */ > #define MIN_XID_LIST_SIZE 128 > #define MAX_XID_LIST_SIZE (1024*128) > diff --git a/sql/slave.cc b/sql/slave.cc > index 28e08e32346..9b3f73e5341 100644 > --- a/sql/slave.cc > +++ b/sql/slave.cc > @@ -6944,7 +6945,9 @@ static int queue_event(Master_info* mi,const char* buf, ulong event_len) > } > else > if ((s_id == global_system_variables.server_id && > - !mi->rli.replicate_same_server_id) || > + (!mi->rli.replicate_same_server_id && > + !(semisync_recovery= (rpl_semi_sync_slave_enabled && > + mi->using_gtid != Master_info::USE_GTID_NO)))) || How can "semisync recovery" code path reach queue_event()? > event_that_should_be_ignored(buf) || > /* > the following conjunction deals with IGNORE_SERVER_IDS, if set > diff --git a/sql/log_event.h b/sql/log_event.h > index 8a342cb5cd3..1588ab85104 100644 > --- a/sql/log_event.h > +++ b/sql/log_event.h > @@ -482,6 +482,16 @@ class String; > */ > #define LOG_EVENT_IGNORABLE_F 0x80 > > +/** > + @def LOG_EVENT_ACCEPT_OWN_F > + > + Flag sets by the gtid-mode connected semisync slave for > + the same server_id ("own") events which the slave must not have > + in its state. Typically such events were never committed by > + their originator (this server) and discared at its crash recovery sorry, I failed to understand that > +*/ > +#define LOG_EVENT_ACCEPT_OWN_F 0x4000 > + > /** > @def LOG_EVENT_SKIP_REPLICATION_F > > diff --git a/sql/log_event.cc b/sql/log_event.cc > index e344fc8894f..a593db10d16 100644 > --- a/sql/log_event.cc > +++ b/sql/log_event.cc > @@ -7918,6 +7924,7 @@ Gtid_log_event::Gtid_log_event(const char *buf, uint event_len, > domain_id= uint4korr(buf); > buf+= 4; > flags2= *buf; > + ++buf; I'd written flags2= *buf++; :) > if (flags2 & FL_GROUP_COMMIT_ID) > { > if (event_len < (uint)header_size + GTID_HEADER_LEN + 2) > @@ -7925,9 +7932,31 @@ Gtid_log_event::Gtid_log_event(const char *buf, uint event_len, > seq_no= 0; // So is_valid() returns false > return; > } > - ++buf; > commit_id= uint8korr(buf); > + buf+= 8; > + } > + /* the extra flags check and actions */ > + if (static_cast<uint>(buf - buf_0) < event_len) > + { > + flags_extra= *buf; > + ++buf; > + /* extra flags presence is identifed by non-zero byte value at this point */ "extra engines" > + if (flags_extra & FL_EXTRA_MULTI_ENGINE) > + { safety, check that buf+4-buf_0 <= event_len > + extra_engines= uint4korr(buf); > + buf += 4; four bytes for the number of engines participating in a transaction is very generous... and optimistic. I'd say one byte is more than enough. > + > + DBUG_ASSERT(extra_engines > 0); > + } > } > + /* > + the strict '<' part of the assert corresponds to extra zero-padded > + trailing bytes, > + */ > + DBUG_ASSERT(static_cast<uint>(buf - buf_0) <= event_len); > + /* and the last of them is tested. */ > + DBUG_ASSERT(static_cast<uint>(buf - buf_0) == event_len || > + buf_0[event_len - 1] == 0); > } > > > @@ -7958,6 +7990,22 @@ Gtid_log_event::Gtid_log_event(THD *thd_arg, uint64 seq_no_arg, > /* Preserve any DDL or WAITED flag in the slave's binlog. */ > if (thd_arg->rgi_slave) > flags2|= (thd_arg->rgi_slave->gtid_ev_flags2 & (FL_DDL|FL_WAITED)); > + /* count non-zero extra recoverable engines; total = extra + 1 */ > + if (is_transactional) > + { > + if (has_xid) > + { > + extra_engines= > + max<uint>(1, ha_count_rw(thd, thd_arg->in_multi_stmt_transaction_mode())) 1. a bit confusing to use both thd and thd_arg in the same line 2. max() is strange. it means the result is 1, when ha_count_rw() is 0. can ha_count_rw() here even be 0? 3. also it's somewhat an unfounded assumption that only r/w engines will have the transaction prepared. But we can make it a fact, if we won't prepare r/o engines at all (in ha_prepare). > + - 1; > + } > + else if (unlikely(thd_arg->is_1pc_ro_trans)) > + { > + extra_engines= UINT_MAX; // neither extra nor base engine why is that? > + } > + if (extra_engines > 0) > + flags_extra|= FL_EXTRA_MULTI_ENGINE; > + } > } > > > diff --git a/sql/handler.cc b/sql/handler.cc > index 6792e80b8fe..247ab7267bf 100644 > --- a/sql/handler.cc > +++ b/sql/handler.cc > @@ -1245,6 +1245,24 @@ int ha_prepare(THD *thd) > DBUG_RETURN(error); > } > > +/* > + Returns counted number of > + read-write recoverable transaction participants. > +*/ > +uint ha_count_rw(THD *thd, bool all) the name doesn't match the implementation, please, rename > +{ > + unsigned rw_ha_count= 0; > + THD_TRANS *trans=all ? &thd->transaction.all : &thd->transaction.stmt; > + > + for (Ha_trx_info * ha_info= trans->ha_list; ha_info; > + ha_info= ha_info->next()) > + { > + if (ha_info->is_trx_read_write() && ha_info->ht()->recover) > + ++rw_ha_count; > + } > + return rw_ha_count; > +} > + > /** > Check if we can skip the two-phase commit. > > @@ -1960,8 +1982,122 @@ struct xarecover_st > XID *list; > HASH *commit_list; > bool dry_run; > + MEM_ROOT *mem_root; > + bool error; > }; > > +/** > + Inserts a new hash member. > + > + returns a successfully created and inserted @c xid_recovery_member > + into hash @c hash_arg, > + or NULL. > +*/ > +static xid_recovery_member* > +xid_member_insert(HASH *hash_arg, my_xid xid_arg, MEM_ROOT *ptr_mem_root) > +{ > + xid_recovery_member *member= (xid_recovery_member*) > + alloc_root(ptr_mem_root, sizeof(xid_recovery_member)); > + if (!member) > + return NULL; > + > + member->xid= xid_arg; > + member->in_engine_prepare= 1; > + member->decided_to_commit= false; > + > + return my_hash_insert(hash_arg, (uchar*) member) ? NULL : member; > +} > + > +/* > + Inserts a new or updates an existing hash member to increment > + the member's prepare counter. > + > + returns false on success, > + true otherwise. > +*/ > +static bool xid_member_replace(HASH *hash_arg, my_xid xid_arg, > + MEM_ROOT *ptr_mem_root) > +{ > + xid_recovery_member* member; > + if ((member= (xid_recovery_member *) > + my_hash_search(hash_arg, (uchar *)& xid_arg, sizeof(xid_arg)))) > + member->in_engine_prepare++; > + else > + member= xid_member_insert(hash_arg, xid_arg, ptr_mem_root); > + > + return member == NULL; > +} > + > +/* > + Hash iterate function to complete with commit or rollback as decided > + (typically at binlog recovery processing) in member->in_engine_prepare. > +*/ > +static my_bool xarecover_do_commit_or_rollback(void *member_arg, > + void *hton_arg) > +{ > + xid_recovery_member *member= (xid_recovery_member*) member_arg; > + handlerton *hton= (handlerton*) hton_arg; > + xid_t x; > + my_bool rc; > + > + x.set(member->xid); > + rc= member->decided_to_commit ? hton->commit_by_xid(hton, &x) : > + hton->rollback_by_xid(hton, &x); > + > + DBUG_ASSERT(rc || member->in_engine_prepare > 0); > + > + if (!rc) > + { I don't think you can trust rc on that. if it's non-zero, it's an error all right. but it's a bit of a stretch to presume that nonexisting xid is always an error. also commit_by_xid returns int, not my_bool. > + member->in_engine_prepare--; > + if (global_system_variables.log_warnings > 2) > + sql_print_warning("%s transaction with xid %llu", > + member->decided_to_commit ? > + "Committed" : "Rolled back", (ulonglong) member->xid); > + } > + > + return false; > +} > + > +static my_bool xarecover_do_count_in_prepare(void *member_arg, > + void *ptr_count) > +{ > + xid_recovery_member *member= (xid_recovery_member*) member_arg; > + if (member->in_engine_prepare) > + { > + *(uint*) ptr_count += member->in_engine_prepare; This is a rather weird semantics. it's kind of a number of transactions times number of engines. if one transaction wasn't committed in two engines and another transaction wasn't rolled back in one engine, the counter will be 3. how are you going to explain this to users? > + if (global_system_variables.log_warnings > 2) > + sql_print_warning("Found prepared transaction with xid %llu", > + (ulonglong) member->xid); > + } > + > + return false; > +} > + > +static my_bool xarecover_binlog_truncate_handlerton(THD *unused, > + plugin_ref plugin, > + void *arg) > +{ > + handlerton *hton= plugin_hton(plugin); > + > + if (hton->state == SHOW_OPTION_YES && hton->recover) > + { > + my_hash_iterate((HASH*) arg, xarecover_do_commit_or_rollback, hton); Why is the function called xarecover_binlog_truncate_handlerton if it doesn't truncate anything? > + } > + > + return FALSE; > +} > + > +uint ha_recover_complete(HASH *commit_list) > +{ > + uint count= 0; > + > + plugin_foreach(NULL, xarecover_binlog_truncate_handlerton, > + MYSQL_STORAGE_ENGINE_PLUGIN, commit_list); > + my_hash_iterate(commit_list, xarecover_do_count_in_prepare, &count); > + > + return count; > +} > + > static my_bool xarecover_handlerton(THD *unused, plugin_ref plugin, > void *arg) > { > @@ -1969,6 +2105,9 @@ static my_bool xarecover_handlerton(THD *unused, plugin_ref plugin, > struct xarecover_st *info= (struct xarecover_st *) arg; > int got; > > + if (info->error) > + return TRUE; plugin_foreach() aborts as soon as the callback returns true. so, remove info->error, and return TRUE from xarecover_handlerton on error instead. > + > if (hton->state == SHOW_OPTION_YES && hton->recover) > { > while ((got= hton->recover(hton, info->list, info->len)) > 0 ) > @@ -1988,7 +2127,7 @@ static my_bool xarecover_handlerton(THD *unused, plugin_ref plugin, > } > #endif /* WITH_WSREP */ > > - for (int i=0; i < got; i ++) > + for (int i=0; i < got && !info->error; i ++) eh? how can info->error ever be true above? you break out of the loop when setting info->error= true, > { > my_xid x= IF_WSREP(WSREP_ON && wsrep_is_wsrep_xid(&info->list[i]) ? > wsrep_xid_seqno(info->list[i]) : > @@ -2013,7 +2152,20 @@ static my_bool xarecover_handlerton(THD *unused, plugin_ref plugin, > info->found_my_xids++; > continue; > } > - // recovery mode > + > + /* > + Regular and semisync slave server recovery only collects > + xids to make decisions on them later by the caller. > + */ > + if (info->mem_root) > + { > + if (xid_member_replace(info->commit_list, x, info->mem_root)) > + { > + info->error= true; > + sql_print_error("Error in memory allocation at xarecover_handlerton"); > + break; > + } > + } else add into the else branch something like /* this branch is only for the legacy TC_LOG_MMAP */ compile_time_assert(sizeof(TC_LOG_MMAP) > 1 ); > if (IF_WSREP((wsrep_emulate_bin_log && > wsrep_is_wsrep_xid(info->list + i) && > x <= wsrep_limit), false) || > diff --git a/sql/log.cc b/sql/log.cc > index 8073f09ab88..a9808ed8823 100644 > --- a/sql/log.cc > +++ b/sql/log.cc > @@ -7977,6 +7982,7 @@ MYSQL_BIN_LOG::trx_group_commit_leader(group_commit_entry *leader) > #endif > } > > + DEBUG_SYNC(leader->thd, "commit_before_update_end_pos"); this isn't used in any tests > /* > update binlog_end_pos so it can be read by dump thread > Note: must be _after_ the RUN_HOOK(after_flush) or else > @@ -9609,6 +9618,180 @@ int TC_LOG::using_heuristic_recover() > /****** transaction coordinator log for 2pc - binlog() based solution ******/ > #define TC_LOG_BINLOG MYSQL_BIN_LOG > > +/** > + Truncates the current binlog to specified position. Removes the rest of binlogs > + which are present after this binlog file. > + > + @param truncate_file Holds the binlog name to be truncated > + @param truncate_pos Position within binlog from where it needs to > + truncated. > + > + @retval true ok > + @retval false error > + > +*/ > +bool MYSQL_BIN_LOG::truncate_and_remove_binlogs(const char *file_name, > + my_off_t pos, > + rpl_gtid *ptr_gtid, > + enum_binlog_checksum_alg cs_alg) > +{ > + int error= 0; > +#ifdef HAVE_REPLICATION > + LOG_INFO log_info; > + THD *thd= current_thd; > + my_off_t index_file_offset= 0; > + File file= -1; > + IO_CACHE cache; > + MY_STAT s; > + > + if ((error= find_log_pos(&log_info, file_name, 1))) > + { > + sql_print_error("Failed to locate binary log file:%s." > + "Error:%d", file_name, error); > + goto end; > + } > + > + while (!(error= find_next_log(&log_info, 1))) > + { > + if (!index_file_offset) > + { > + index_file_offset= log_info.index_file_start_offset; > + if ((error= open_purge_index_file(TRUE))) > + { > + sql_print_error("Failed to open purge index " > + "file:%s. Error:%d", purge_index_file_name, error); > + goto end; > + } > + } > + if ((error= register_purge_index_entry(log_info.log_file_name))) > + { > + sql_print_error("Failed to copy %s to purge index" > + " file. Error:%d", log_info.log_file_name, error); > + goto end; > + } > + } > + > + if (error != LOG_INFO_EOF) > + { > + sql_print_error("Failed to find the next binlog to " > + "add to purge index register. Error:%d", error); > + goto end; > + } > + > + if (is_inited_purge_index_file()) > + { > + if (!index_file_offset) > + index_file_offset= log_info.index_file_start_offset; > + > + if ((error= sync_purge_index_file())) > + { > + sql_print_error("Failed to flush purge index " > + "file. Error:%d", error); > + goto end; > + } > + > + // Trim index file > + if ((error= > + mysql_file_chsize(index_file.file, index_file_offset, '\n', > + MYF(MY_WME))) || > + (error= > + mysql_file_sync(index_file.file, MYF(MY_WME|MY_SYNC_FILESIZE)))) > + { > + sql_print_error("Failed to trim binlog index " > + "file:%s to offset:%llu. Error:%d", index_file_name, > + index_file_offset, error); > + goto end; > + } > + > + /* Reset data in old index cache */ > + if ((error= reinit_io_cache(&index_file, READ_CACHE, (my_off_t) 0, 0, 1))) > + { > + sql_print_error("Failed to reinit binlog index " > + "file. Error:%d", error); > + goto end; > + } > + > + /* Read each entry from purge_index_file and delete the file. */ > + if ((error= purge_index_entry(thd, NULL, TRUE))) > + { > + sql_print_error("Failed to process registered " > + "files that would be purged."); > + goto end; > + } > + } > + > + DBUG_ASSERT(pos); > + > + if ((file= mysql_file_open(key_file_binlog, file_name, > + O_RDWR | O_BINARY, MYF(MY_WME))) < 0) > + { > + error= 1; > + sql_print_error("Failed to open binlog file:%s for " > + "truncation.", file_name); > + goto end; > + } > + my_stat(file_name, &s, MYF(0)); > + > + /* Change binlog file size to truncate_pos */ > + if ((error= > + mysql_file_chsize(file, pos, 0, MYF(MY_WME))) || > + (error= mysql_file_sync(file, MYF(MY_WME|MY_SYNC_FILESIZE)))) > + { > + sql_print_error("Failed to trim the " > + "binlog file:%s to size:%llu. Error:%d", > + file_name, pos, error); > + goto end; > + } > + else > + { > + char buf[21]; > + > + longlong10_to_str(ptr_gtid->seq_no, buf, 10); > + sql_print_information("Successfully truncated binlog file:%s " > + "to pos:%llu to remove transactions starting from " > + "GTID %u-%u-%s", file_name, pos, > + ptr_gtid->domain_id, ptr_gtid->server_id, buf); > + } > + if (!(error= init_io_cache(&cache, file, IO_SIZE, WRITE_CACHE, > + (my_off_t) pos, 0, MYF(MY_WME|MY_NABP)))) > + { > + /* > + Write Stop_log_event to ensure clean end point for the binary log being > + truncated. > + */ I'm not sure about it. The less you touch the corrupted binlog the better. simply truncating it would be enough, wouldn't it? > + Stop_log_event se; > + se.checksum_alg= cs_alg; > + if ((error= write_event(&se, NULL, &cache))) > + { > + sql_print_error("Failed to write stop event to " > + "binary log. Errno:%d", > + (cache.error == -1) ? my_errno : error); > + goto end; > + } > + clear_inuse_flag_when_closing(cache.file); > + if ((error= flush_io_cache(&cache)) || > + (error= mysql_file_sync(file, MYF(MY_WME|MY_SYNC_FILESIZE)))) > + { > + sql_print_error("Faild to write stop event to " > + "binary log. Errno:%d", > + (cache.error == -1) ? my_errno : error); > + } > + } > + else > + sql_print_error("Failed to initialize binary log " > + "cache for writing stop event. Errno:%d", > + (cache.error == -1) ? my_errno : error); 1. I don't see why you need to sync after every operation. 2. you should clear inuse flag just before closing, otherwise flush_io_cache can overwrite it again if it'll happen to have it in the cache. > + > +end: > + if (file >= 0) > + { > + end_io_cache(&cache); > + mysql_file_close(file, MYF(MY_WME)); > + } > + error= error || close_purge_index_file(); > +#endif > + return error > 0; > +} > int TC_LOG_BINLOG::open(const char *opt_name) > { > int error= 1; > @@ -10040,26 +10223,515 @@ start_binlog_background_thread() > > return 0; > } > +#ifdef HAVE_REPLICATION > +class Recovery_context > +{ > +public: > + my_off_t prev_event_pos; > + rpl_gtid last_gtid; > + bool last_gtid_standalone; > + bool last_gtid_valid; > + bool last_gtid_no2pc; // true when the group does not end with Xid event > + uint last_gtid_engines; > + std::pair<uint, my_off_t> last_gtid_coord; may be a comment at the end // <binlog id, binlog offset> ? > + /* > + When true, it's semisync slave recovery mode > + rolls back transactions in doubt and wipes them off from binlog. > + The rest of declarations deal with this type of recovery. > + */ > + bool do_truncate; > + rpl_gtid binlog_unsafe_gtid, truncate_gtid; > + char binlog_truncate_file_name[FN_REFLEN] ; > + char binlog_unsafe_file_name[FN_REFLEN] ; > + /* > + When do_truncate is true, the truncate position may not be > + found in one round when recovered transactions are multi-engine > + or just on different engines. > + In the single recoverable engine case `truncate_reset_done` and > + therefore `truncate_validated` remain `false' when the last > + binlog is the binlog-checkpoint one. > + The meaning of `truncate_reset_done` is according to the following example: > + Let round = 1, Binlog contains the sequence of replication event groups: > + [g1, G2, g3] > + where `G` (in capital) stands for committed, `g` for prepared. > + g1 is first set as truncation candidate, then G2 reset it to indicate > + the actual truncation is behind (to the right of) it. > + `truncate_validated` is set to true when `binlog_truncate_pos` (as of g3) > + won't change. > + Observe last_gtid_valid is affected, so in the above example g1 > + would have to be discarded from the 1st round binlog state estimate which > + is handled by the 2nd round (see below). I failed to understand that either :( > + */ > + bool truncate_validated; // trued when the truncate position settled > + bool truncate_reset_done; // trued when the position is to reevaluate Nor that :( > + /* Flags the fact of truncate position estimation is done the 1st round */ > + bool truncate_set_in_1st; > + /* > + Monotonically indexes binlog files in the recovery list. > + When the list is "likely" singleton the value is UINT_MAX. > + Otherwise enumeration starts with zero for the first file, increments > + by one for any next file except for the last file in the list, which > + is also the initial binlog file for recovery, > + that is enumberated with UINT_MAX. > + */ > + uint id_binlog; > + enum_binlog_checksum_alg cs_alg; // for Stop_event with do_truncate rename to checksum_alg, please. "charset algorithm" looks very puzzling and "cs" almost everywhere means CHARSET_INFO. > + bool single_binlog; > + std::pair<uint, my_off_t> binlog_truncate_coord; > + std::pair<uint, my_off_t> binlog_unsafe_coord; > + > + Recovery_context(); > + bool decide_or_assess(xid_recovery_member *member, int round, > + Format_description_log_event *fdle, > + LOG_INFO *linfo, my_off_t pos); > + void process_gtid(int round, Gtid_log_event *gev, LOG_INFO *linfo); > + int next_binlog_or_round(int& round, > + const char *last_log_name, > + const char *binlog_checkpoint_name, > + LOG_INFO *linfo, MYSQL_BIN_LOG *log); > + bool is_safe() what do you mean "safe" ? > + { > + return !do_truncate ? true : > + (truncate_gtid.seq_no == 0 || // no truncate > + binlog_unsafe_coord < binlog_truncate_coord); // or unsafe is earlier > + } > + bool complete(MYSQL_BIN_LOG *log, HASH &xids); > + void update_binlog_unsafe_coord_if_needed(LOG_INFO *linfo); > + void reset_truncate_coord(my_off_t pos); > + void set_truncate_coord(LOG_INFO *linfo, int round, > + enum_binlog_checksum_alg fd_cs_alg); > +}; > comments about what every method does? > +bool Recovery_context::complete(MYSQL_BIN_LOG *log, HASH &xids) > +{ > + if (!do_truncate || is_safe()) > + { > + uint count_in_prepare= ha_recover_complete(&xids); > + if (count_in_prepare > 0 && global_system_variables.log_warnings > 2) > + { > + sql_print_warning("Could not complete %u number of transactions in " > + "engines.", count_in_prepare); > + return false; // there's later dry run ha_recover() to error out > + } > + } > + > + /* Truncation is not done when there's no transaction to roll back */ > + if (do_truncate && truncate_gtid.seq_no > 0) > + { > + if (is_safe()) > + { > + if (log->truncate_and_remove_binlogs(binlog_truncate_file_name, > + binlog_truncate_coord.second, > + &truncate_gtid, > + cs_alg)) > + { > + sql_print_error("Failed to trim the binary log to " > + "file:%s pos:%llu.", binlog_truncate_file_name, > + binlog_truncate_coord.second); > + return true; > + } > + } > + else > + { > + sql_print_error("Cannot trim the binary log to file:%s " > + "pos:%llu as unsafe statement " > + "is found at file:%s pos:%llu which is " > + "beyond the truncation position " > + "(the farest in binlog order only is reported); " "farest" doesn't look like a word > + "all transactions in doubt are left intact. ", > + binlog_truncate_file_name, binlog_truncate_coord.second, > + binlog_unsafe_file_name, binlog_unsafe_coord.second); > + return true; will this abort server startup? > + } > + } > + > + return false; > +} > + > +Recovery_context::Recovery_context() : > + prev_event_pos(0), > + last_gtid_standalone(false), last_gtid_valid(false), last_gtid_no2pc(false), > + last_gtid_engines(0), > + do_truncate(rpl_semi_sync_slave_enabled), > + truncate_validated(false), truncate_reset_done(false), > + truncate_set_in_1st(false), id_binlog(UINT_MAX), > + cs_alg(BINLOG_CHECKSUM_ALG_UNDEF), single_binlog(false) > +{ > + last_gtid_coord= std::pair<uint,my_off_t>(0,0); > + binlog_truncate_coord= std::pair<uint,my_off_t>(0,0); > + binlog_unsafe_coord= std::pair<uint,my_off_t>(0,0); > + binlog_truncate_file_name[0]= 0; > + binlog_unsafe_file_name [0]= 0; > + binlog_unsafe_gtid= truncate_gtid= rpl_gtid(); > +} > + > +/** > + Is called when a committed or to-be-committed transaction is detected. > + @c truncate_gtid is set to "nil" with its @c rpl_gtid::seq_no := 0. > + @c truncate_reset_done remembers the fact of that has been done at least > + once in the current round; > + @c binlog_truncate_coord is "suggested" to a next group start to indicate > + the actual settled value must be at most as the last suggested one. > +*/ > +void Recovery_context::reset_truncate_coord(my_off_t pos) > +{ > + DBUG_ASSERT(binlog_truncate_coord.second == 0 || > + last_gtid_coord >= binlog_truncate_coord || > + truncate_set_in_1st); > + > + binlog_truncate_coord= std::pair<uint,my_off_t>(id_binlog, pos); > + truncate_gtid= rpl_gtid(); > + truncate_reset_done= true; > +} > + > + > +/* > + Sets binlog_truncate_pos to the value of the current transaction's gtid. > + In multi-engine case that might be just an assessment to be exacted > + in the current round and confirmed in a next one. > +*/ > +void Recovery_context::set_truncate_coord(LOG_INFO *linfo, int round, > + enum_binlog_checksum_alg fd_cs_alg) > +{ > + binlog_truncate_coord= last_gtid_coord; > + strmake_buf(binlog_truncate_file_name, linfo->log_file_name); > + > + truncate_gtid= last_gtid; > + cs_alg= fd_cs_alg; > + truncate_set_in_1st= (round == 1); > +} > + > +bool Recovery_context::decide_or_assess(xid_recovery_member *member, int round, > + Format_description_log_event *fdle, > + LOG_INFO *linfo, my_off_t pos) > +{ > + if (member) > + { > + DBUG_EXECUTE_IF("binlog_truncate_partial_commit", > + if (last_gtid_engines == 2) > + { > + DBUG_ASSERT(member->in_engine_prepare > 0); > + member->in_engine_prepare= 1; > + }); > + /* > + xid in doubt are resolved as follows: > + in_engine_prepare is compared agaist binlogged info to > + yield the commit-or-rollback decision in the normal case. > + In the semisync-slave recovery the decision may be > + approximate to change in later rounds. > + */ > + if (member->in_engine_prepare > last_gtid_engines) > + { > + char buf[21]; > + longlong10_to_str(last_gtid.seq_no, buf, 10); > + sql_print_error("Error to recovery multi-engine transaction: " > + "the number of engines prepared %u exceeds the " > + "respective number %u in its GTID %u-%u-%s " > + "located at file:%s pos:%llu", > + member->in_engine_prepare, last_gtid_engines, > + last_gtid.domain_id, last_gtid.server_id, buf, > + linfo->log_file_name, last_gtid_coord.second); > + return true; > + } > + else if (member->in_engine_prepare < last_gtid_engines) > + { > + DBUG_EXECUTE_IF("binlog_truncate_partial_commit", > + member->in_engine_prepare= 2;); > + DBUG_ASSERT(member->in_engine_prepare > 0); > + /* > + This is an "unlikely" branch of two or more engines in transaction > + that is partially committed, so to complete. > + */ > + member->decided_to_commit= true; > + if (do_truncate) > + { > + /* > + A validated truncate pos may not change later. > + Any "unlikely" (two or more engines in transaction) reset > + to not-validated yet position ensues correcting early > + estimates in the following round(s). > + */ > + if (!truncate_validated) > + reset_truncate_coord(pos); > + } > + } > + else // member->in_engine_prepare == last_gtid_engines > + { > + if (!do_truncate) // "normal" recovery > + { > + member->decided_to_commit= true; > + } > + else > + { > + /* > + The first time or further estimate the truncate position > + until validation. Already set not validated yet postion > + may change only through previous reset > + unless this xid in doubt is the first in the 2nd round. > + */ > + if (!truncate_validated) > + { > + DBUG_ASSERT(round <= 2); > + /* > + Either truncate was not set or was reset, else > + it gets incremented, otherwise it may only set to an earlier > + offset only at the turn out of the 1st round. sorry, I cannot parse that :( > + */ > + DBUG_ASSERT(truncate_gtid.seq_no == 0 || > + last_gtid_coord >= binlog_truncate_coord || > + (round == 2 && truncate_set_in_1st)); > + > + last_gtid_valid= false; // may still (see gtid) flip later > + if (truncate_gtid.seq_no == 0 /* was reset or never set */ || > + (truncate_set_in_1st && round == 2 /* reevaluted at round turn */)) > + set_truncate_coord(linfo, round, fdle->checksum_alg); > + > + DBUG_ASSERT(member->decided_to_commit == false); // may flip later > + } > + else > + { > + DBUG_ASSERT(!truncate_reset_done); // the position was settled > + /* > + Correct earlier decisions of 1st and/or 2nd round to > + rollback and invalidate last_gtid in binlog state. > + */ > + if (!(member->decided_to_commit= > + last_gtid_coord < binlog_truncate_coord)) > + { > + last_gtid_valid= false; // settled > + if (truncate_gtid.seq_no == 0) > + truncate_gtid= last_gtid; > + } > + } > + } > + } > + } > + else if (do_truncate) // "0" < last_gtid_engines > + { > + /* > + Similar to the multi-engine partial commit of "then" branch above. > + The 2nd condition economizes away an extra (3rd) round in I don't think "economizes" is a word > + expected cases of first xid:s in the binlog checkpoint file > + are actually of committed transactions. So fully committed > + xid sequence is passed in this branch without any action. > + */ > + if (!truncate_validated) > + reset_truncate_coord(pos); > + } > + > + return false; > +} > + > +/* > + Is invoked when a standalone or non-2pc group is detected. > + Both are unsafe to truncate in the semisync-slave recovery so > + the maximum unsafe coordinate may be updated. > + In the non-2pc group case though, *exeptionally*, > + the no-engine group is considered safe, to be invalidated > + to not contribute to binlog state. > +*/ > +void Recovery_context::update_binlog_unsafe_coord_if_needed(LOG_INFO *linfo) > +{ > + if (!do_truncate) > + return; > + > + if (truncate_gtid.seq_no > 0 && // g1,U2, *not* G1,U2 > + last_gtid_coord > binlog_truncate_coord) > + { > + DBUG_ASSERT(binlog_truncate_coord.second > 0); > + /* > + Potentially unsafe when the truncate coordinate is not determined, > + just detected as unsafe when behind the latter. > + */ > + if (last_gtid_engines == 0) > + { > + last_gtid_valid= false; > + } > + else > + { > + binlog_unsafe_gtid= last_gtid; > + binlog_unsafe_coord= last_gtid_coord; > + strmake_buf(binlog_unsafe_file_name, linfo->log_file_name); > + } > + } > +} > + > +/* > + Assigns last_gtid and assesses the maximum (in the binlog offset term) > + unsafe gtid (group of events). > +*/ > +void Recovery_context::process_gtid(int round, Gtid_log_event *gev, > + LOG_INFO *linfo) > +{ > + last_gtid.domain_id= gev->domain_id; > + last_gtid.server_id= gev->server_id; > + last_gtid.seq_no= gev->seq_no; > + last_gtid_engines= gev->extra_engines != UINT_MAX ? > + gev->extra_engines + 1 : 0; > + last_gtid_coord= std::pair<uint,my_off_t>(id_binlog, prev_event_pos); > + if (round == 1 || do_truncate) > + { > + DBUG_ASSERT(!last_gtid_valid); > + > + last_gtid_no2pc= false; > + last_gtid_standalone= > + (gev->flags2 & Gtid_log_event::FL_STANDALONE) ? true : false; > + if (do_truncate && last_gtid_standalone) > + update_binlog_unsafe_coord_if_needed(linfo); > + /* Update the binlog state with any 'valid' GTID logged after Gtid_list. */ > + last_gtid_valid= true; // may flip at Xid when falls to truncate > + } > +} > + > +/* > + At the end of processing of the current binlog compute next action. > + When round increments in the semisync-slave recovery > + truncate_validated, truncate_reset_done > + gets reset/set for the next round, > + as well as id_binlog gets reset either to zero or UINT_MAX > + when recovery deals with the only binlog file. > + > + @param[in,out] round the current round that may increment here > + @param last_log_name the recovery starting binlog file > + @param binlog_checkpoint_name > + binlog checkpoint file > + @param linfo binlog file list struct for next file > + @param log pointer to mysql_bin_log instance > + @return 0 when rounds continue, maybe the current one remains > + 1 when all rounds are done > + -1 on error > +*/ > +int Recovery_context::next_binlog_or_round(int& round, > + const char *last_log_name, > + const char *binlog_checkpoint_name, > + LOG_INFO *linfo, > + MYSQL_BIN_LOG *log) > +{ > + if (!strcmp(linfo->log_file_name, last_log_name)) > + { > + /* Either exit the loop now, or increment round */ > + DBUG_ASSERT(round <= 3); > + > + if ((round <= 2 && likely(!truncate_reset_done)) || round == 3) > + { > + // Exit now as the 1st round ends up with the binlog-checkpoint file > + // is the same as the initial binlog file, so already parsed; or > + // the 2nd round has not made own truncate_reset_done; or > + // at the end of the 3rd "truncate" round. > + DBUG_ASSERT(do_truncate || round == 1); > + // The 3rd round is done only when the 2nd trued truncate_reset_done > + // and consequently truncate_validated. > + // No instances of truncate_reset_done in first 2 rounds allows > + // for exiting now. our (unwritten) coding style says that multi-line comments should use /*...*/ > + DBUG_ASSERT(round < 3 || truncate_validated); > + > + return true; > + } > + else > + { > + DBUG_ASSERT(do_truncate); > + /* > + the last binlog file, having truncate_reset_done to indicate > + needed correction to member->decided_to_commit to reflect > + changed binlog_truncate_pos. > + */ > + truncate_reset_done= false; > + truncate_validated= true; > + rpl_global_gtid_binlog_state.reset_nolock(); > + > + if (round > 1 && > + log->find_log_pos(linfo, binlog_checkpoint_name, 1)) > + { > + sql_print_error("Binlog file '%s' not found in binlog index, needed " > + "for recovery. Aborting.", binlog_checkpoint_name); > + return -1; > + } > + id_binlog= (single_binlog= !strcmp(linfo->log_file_name, last_log_name)) ? > + UINT_MAX : 0; > + round++; > + } > + } > + else if (round == 1) > + { > + if (do_truncate) > + { > + truncate_validated= truncate_reset_done; > + rpl_global_gtid_binlog_state.reset_nolock(); > + truncate_reset_done= false; > + id_binlog= 0; > + > + DBUG_ASSERT(!single_binlog); > + } > + round++; > + } > + else > + { > + /* > + NOTE: reading other binlog's FD is necessary for finding out > + the checksum status of the respective binlog file. > + Round remains in this branch. > + */ > + if (log->find_next_log(linfo, 1)) > + { > + sql_print_error("Error reading binlog files during recovery. " > + "Aborting."); > + return -1; > + } > + if (!strcmp(linfo->log_file_name, last_log_name)) > + { > + if (do_truncate) > + { > + DBUG_ASSERT(!single_binlog); > + id_binlog= UINT_MAX; > + } > + else > + return 1; // regular recovery exit > + } > + else if (do_truncate) > + id_binlog++; > + > + DBUG_ASSERT(id_binlog <= UINT_MAX); // the assert is "practical" > + } > + > + DBUG_ASSERT(!single_binlog || > + !strcmp(linfo->log_file_name, binlog_checkpoint_name)); > + > + return 0; > +} > +#endif > > int TC_LOG_BINLOG::recover(LOG_INFO *linfo, const char *last_log_name, > IO_CACHE *first_log, > - Format_description_log_event *fdle, bool do_xa) > + Format_description_log_event *fdle_arg, bool do_xa) > { > Log_event *ev= NULL; > HASH xids; > MEM_ROOT mem_root; > char binlog_checkpoint_name[FN_REFLEN]; > bool binlog_checkpoint_found; > - bool first_round; > IO_CACHE log; > File file= -1; > const char *errmsg; > + Format_description_log_event *fdle= fdle_arg; > #ifdef HAVE_REPLICATION > - rpl_gtid last_gtid; > - bool last_gtid_standalone= false; > - bool last_gtid_valid= false; > + Recovery_context ctx; > #endif > + /* > + The for-loop variable is updated by the following rule set: > + Initially set to 1. > + After the initial binlog file is processed to identify > + the Binlog-checkpoint file it is incremented when the latter file > + is different from the initial one. Otherwise the only log has been > + fully parsed and the loop exits. > + The 2nd starts with the Binlog-checkpoint file and ends when the initial > + binlog file is reached. It is excluded from yet another processing > + in the "normal" non-semisync-slave configuration, and then the loop is > + done at this point. Otherwise in the semisync slave case it may be parsed > + over again. The 2nd round may turn to a third in "unlikely" condition of the > + semisync-slave is being recovered having multi- or different engine > + transactions in doubt. > + */ I don't understand that. Why semisync-slave case is special? > + int round; > > if (! fdle->is_valid() || > (do_xa && my_hash_init(&xids, &my_charset_bin, TC_LOG_PAGE_SIZE/3, 0, > @@ -10072,39 +10744,63 @@ int TC_LOG_BINLOG::recover(LOG_INFO *linfo, const char *last_log_name, > > fdle->flags&= ~LOG_EVENT_BINLOG_IN_USE_F; // abort on the first error > > + /* finds xids when root is not NULL */ > + if (do_xa && ha_recover(&xids, &mem_root)) > + goto err1; > + > /* > Scan the binlog for XIDs that need to be committed if still in the > prepared stage. > > Start with the latest binlog file, then continue with any other binlog > files if the last found binlog checkpoint indicates it is needed. > + > + In case the semisync slave recovery the 2nd round may include > + the initial file as well as cause a 3rd round when transactions > + with multiple engines are discovered. > + Additionally to find and decide on transactions in doubt that > + the semisync slave may need to roll back, the binlog can be truncated > + in the end to reflect the rolled back decisions. > */ > > binlog_checkpoint_found= false; > - first_round= true; > - for (;;) > + for (round= 1;;) > { > - while ((ev= Log_event::read_log_event(first_round ? first_log : &log, > + while ((ev= Log_event::read_log_event(round == 1 ? first_log : &log, > fdle, opt_master_verify_checksum)) > && ev->is_valid()) > { > enum Log_event_type typ= ev->get_type_code(); > switch (typ) > { > + case FORMAT_DESCRIPTION_EVENT: > + if (round > 1) > + { > + if (fdle != fdle_arg) > + delete fdle; > + fdle= (Format_description_log_event*) ev; why is it needed? all binlog files should have the same Format_description_log_event anyway. > + } > + break; > case XID_EVENT: > + if (do_xa) > { > - if (do_xa) > + xid_recovery_member *member= > + (xid_recovery_member*) > + my_hash_search(&xids, (uchar*) &static_cast<Xid_log_event*>(ev)->xid, > + sizeof(my_xid)); > +#ifndef HAVE_REPLICATION > { > - Xid_log_event *xev=(Xid_log_event *)ev; > - uchar *x= (uchar *) memdup_root(&mem_root, (uchar*) &xev->xid, > - sizeof(xev->xid)); > - if (!x || my_hash_insert(&xids, x)) > - goto err2; > + if (member) > + member->decided_to_commit= true; > } > - break; > +#else > + if (ctx.decide_or_assess(member, round, fdle, linfo, ev->log_pos)) > + goto err2; > +#endif > } > + break; > case BINLOG_CHECKPOINT_EVENT: > - if (first_round && do_xa) > + if (round == 1 && do_xa) > { > size_t dir_len; > Binlog_checkpoint_log_event *cev= (Binlog_checkpoint_log_event *)ev; > @@ -10124,9 +10820,18 @@ int TC_LOG_BINLOG::recover(LOG_INFO *linfo, const char *last_log_name, > } > } > break; > +#ifdef HAVE_REPLICATION > case GTID_LIST_EVENT: > - if (first_round) > + if (round == 1 || (ctx.do_truncate && > + (ctx.id_binlog == 0 || ctx.single_binlog))) > { > + /* > + Unlike the normal case, in do_truncate the initial state is > + in the first binlog file of the recovery list. > + */ don't understand that either :( > + DBUG_ASSERT(!ctx.do_truncate || !ctx.single_binlog || > + ctx.id_binlog == UINT_MAX); > + > Gtid_list_log_event *glev= (Gtid_list_log_event *)ev; > > /* Initialise the binlog state from the Gtid_list event. */ > @@ -10135,19 +10840,16 @@ int TC_LOG_BINLOG::recover(LOG_INFO *linfo, const char *last_log_name, > } > break; > > -#ifdef HAVE_REPLICATION > case GTID_EVENT: > - if (first_round) > + ctx.process_gtid(round, (Gtid_log_event *)ev, linfo); > + break; > + > + case QUERY_EVENT: > + if (((Query_log_event *)ev)->is_commit() || > + ((Query_log_event *)ev)->is_rollback()) > { > - Gtid_log_event *gev= (Gtid_log_event *)ev; > - > - /* Update the binlog state with any GTID logged after Gtid_list. */ > - last_gtid.domain_id= gev->domain_id; > - last_gtid.server_id= gev->server_id; > - last_gtid.seq_no= gev->seq_no; > - last_gtid_standalone= > - ((gev->flags2 & Gtid_log_event::FL_STANDALONE) ? true : false); > - last_gtid_valid= true; > + ctx.last_gtid_no2pc= true; > + ctx.update_binlog_unsafe_coord_if_needed(linfo); what about DML on non-transactonal tables? it won't have COMMIT/ROLLBACK will it? > } > break; > #endif > @@ -10217,35 +10923,43 @@ int TC_LOG_BINLOG::recover(LOG_INFO *linfo, const char *last_log_name, > file= -1; > } > > - if (!strcmp(linfo->log_file_name, last_log_name)) > - break; // No more files to do > +#ifdef HAVE_REPLICATION > + int rc= ctx.next_binlog_or_round(round, last_log_name, > + binlog_checkpoint_name, linfo, this); > + if (rc == -1) > + goto err2; > + else if (rc == 1) > + break; // all rounds done > +#else > + if (!strcmp(linfo->log_file_name, last_log_name)) > + break; // No more files to do indentation > +#endif > + > if ((file= open_binlog(&log, linfo->log_file_name, &errmsg)) < 0) > { > sql_print_error("%s", errmsg); > goto err2; > } > - /* > - We do not need to read the Format_description_log_event of other binlog > - files. It is not possible for a binlog checkpoint to span multiple > - binlog files written by different versions of the server. So we can use > - the first one read for reading from all binlog files. > - */ > - if (find_next_log(linfo, 1)) > - { > - sql_print_error("Error reading binlog files during recovery. Aborting."); > - goto err2; > - } > fdle->reset_crypto(); > - } > + } // end of for > > if (do_xa) > { > - if (ha_recover(&xids)) > - goto err2; > - > + if (binlog_checkpoint_found) > + { > +#ifndef HAVE_REPLICATION > + if (ha_recover_complete(&xids)) > +#else > + if (ctx.complete(this, xids)) > +#endif > + goto err2; > + } > free_root(&mem_root, MYF(0)); > my_hash_free(&xids); > } > + if (fdle != fdle_arg) > + delete fdle; > + > return 0; > > err2: Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

[Maria-developers] Systems Integration and Raising of the Abstraction Level
by Alejandro Sánchez 25 Feb '21

25 Feb '21

We have highly evolved systems such as SQL, HTTP, HTML, file formats or high level programming languages such as Java or PHP that allow us to program many things with little code. Even so a lot of effort is invested in the integration of these systems. To try to reduce this problem libraries and frameworks that help in some ways are created but the integration is not complete. It is well known that most of the time when you try to create something to integrate several incompatible systems what you get in the end is to have another incompatible system :). Still I think the integration between the systems mentioned above is something very important that can mean a great step in the evolution of computing and worth a try. To explore how this integration can be I have created a framework that I have called NextTypes and that its main objective is the integration of data types. For me it is something very illogical that something as basic as a 16 bits integer receives a different name in each of the systems ("smallint", "short", "number") and can also be signed or unsigned. In any moment, due to a mistake from the programmer, the number of the programming language does not fit in the database column. Besides these names are little indicative of its characteristics, it would be clearer for example to use "int16". Whatever names are chosen the most important thing is to use in all systems the same names for types of the same characteristics. Also there is no standard system for defining composite data types of primitive types and other composite types. From an HTML form to a SQL table or from one application to another are required multiple transformations of the data. Lack of integration also lowers the level of abstraction, making it necessary to do lots of low level stuff for systems to fit. NextTypes at this moment is nothing more than another incompatible system with the others. It simply integrates them quite a bit and raises the level of abstraction. But what I would like is that the things that compose NextTypes were included in the systems it integrates. Finally I would like to list some examples of improvements in database managers, SQL, HTTP, HTML, browsers and programming languages that would help the integration and elevation of the level of abstraction. Some of these enhancements are already included in NextTypes and other frameworks. SQL --- - Custom metadata in tables and columns. - Date of creation and modification of the rows. - Date of creation and modification of the definition of the tables. - Use of table and column names in prepared statements. Example: select # from # where # = ?; - Use of arrays in prepared statements. Example: select # from article where id in (?); # = author,title ? = 10,24,45 - Standardization of ranges of valid values and resolution for date, time and datetime types in database managers an HTML time element. PostgreSQL ---------- - Facilitate access to the definition of full text search indexes with a function to parse "pg_index.indexprs" column. Other database managers ----------------------- - Allow transactional DDL, deferrable constraints and composite types. JDBC ---- - High level methods that allow queries with the execution of a single method. Example: Tuple [] tuples = query("select author,title from article where id in (?), ids); - Integration with java.time data types. HTTP - Servers -------------- - Processing of arrays of elements composed of several parameters. fields:0:type = string fields:0:name = title fields:0:parameters = 250 fields:0:not_null = true fields:1:type = string fields:1:name = author fields:1:parameters = 250 fields:1:not_null = true Another possibility is to generate in the browser arrays of JSON objects from the forms. "fields": [ { "type": "string", "name": "title", "parameters": 250, "not_null": true }, { "type": "string", "name": "author", "parameters": 250, "not_null": true } ] XML/HTML - BROWSER ------------------ - Input elements for different types of numbers with min and max values: 16 bits integer, 32 bits integer, 32 bits float and 64 bits float. - Input elements for images, audios and videos with preview. - Timezone input element. - Boolean input element with "true" and "false" values. - Null value in file inputs. - Clear button in file inputs like in date and time inputs. - Show size in file inputs. - Extension of the DOM API with high level and chainable methods. Example: paragraph.appendElement("a").setAttribute("href", "/article"); - Change of the "action" parameter of the forms to "target" to indicate the URL where to execute the action. The "action" parameter is moved to the different buttons on the form and allows executing a different action with each of the the buttons. Example: <form target="/article"> <button action="delete">Delete</button> <button action="export">Export</button> </form> - "select" elements that change a parameter of the current URL. Example: <select url-parameter="lang"/> <option>en</option> <option>es</option> URL = https://demo.nexttypes.com/?lang=en - "content-type" attribute in links and context menu in the browser to open links with external applications using WEBDAV, similar to a file manager. Example: <a href="" content- type="application/vnd.oasis.opendocument.text"> ------------------------------ | Open link with ... | | Open link with LibreOffice | - Background submission of forms without using XMLHttpRequest, display of result in dialog window or file download, and execution of a Javascript function for subsequent actions. Example: <form background show-progress callback=function()> - Dialog with progress indicator of form submission. Must show total size, transmitted data and average speed. Possibility to pause or cancel the submission. - Dynamic datalist with searchable JSON source. Over data source URL is added a "search" parameter with input value. Example: <input list="article-list" name="article" type="text" /> <datalist id="article-list" src="/article?lang=en&view=json&names" /> Example query URL: "/article?lang=en&view=json&names&search=Ne" - Same appearance and operation of inputs with dynamic datalist in all browsers. - Option tags with icons. Example: <option icon="/icons/save.svg">Save</option> - Tabs, Tree, etc widgets - Mechanism to close HTTPS sessions initiated with client certificate authentication. JAVA ---- - Subclasses of String or some system of variants of String that allows assigning a regular expression or class that limits its valid values to avoid code injection or values that will crash the system. Example: String:Type = "[a-z0-9 _] +"; Or String:Type = TypeChecks; String:Type type = "article_language"; -> correct String:Type type = "Article-Language"; -> error We can talk about the characteristics of each system in its mailing list or group. For the general topic of system integration I have created a discussion in the github project. https://github.com/alejsanc/nexttypes/discussions/6 This email has been sent to the following mailing lists and groups: pgsql-hackers(a)lists.postgresql.org pgsql-jdbc(a)lists.postgresql.org mozilla.dev.platform(a)googlegroups.com public-html(a)w3.org jdbc-spec-discuss(a)openjdk.java.net jdk-dev(a)openjdk.java.net maria-developers(a)lists.launchpad.net chromium-dev(a)chromium.org https://mysqlcommunity.slack.com/archives/C8R1336M7 Best regards. Alejandro Sánchez.

1 0

[Maria-developers] Re MDEV-7317: Ignored indexes
by Sergey Petrunia 24 Feb '21

24 Feb '21

Hi Varun, The code for the task is mostly fine. But I would like to question the user experience. == Ignoring an index causes InnoDB table rebuild == First (and foremost), making an index ignored (or not ignored) in an InnoDB table rebuilds the table: MariaDB [test]> set session alter_algorithm='instant'; alter table t1 alter index c ignore; Query OK, 0 rows affected (0.001 sec) ERROR 1845 (0A000): ALGORITHM=INSTANT is not supported for this operation. Try ALGORITHM=COPY This misses the point of the feature. If one needs to rebuild the table anyway, they can just drop (or recreate) the index. Curiously, it works with MyISAM. I think, the ideal solution would be to make ignoring/not ignoring the index an SQL layer change that doesn't concern the storage engine. If this is not possible, lets make InnoDB aware that it doesn't need to rebuild the table in this case. == Backward compatibility for SHOW CREATE TABLE == SHOW CREATE TABLE will show something like: KEY `a` (`a`) IGNORE, MySQL here is more backward-compatible and shows: KEY `a` (`a`) /*!80000 INVISIBLE */, Do we need to do this as well? I have no strong opinion here. == Use of Ignored index in hints == I can use Ignored index in FORCE/IGNORE index hints: MariaDB [test]> explain select * from t1 force index(a) where a<3 and b<3; +------+-------------+-------+------+---------------+------+---------+------+------+-------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +------+-------------+-------+------+---------------+------+---------+------+------+-------------+ | 1 | SIMPLE | t1 | ALL | NULL | NULL | NULL | NULL | 1000 | Using where | +------+-------------+-------+------+---------------+------+---------+------+------+-------------+ This is allowed. I think, this misses the point of the feature. The point of this feature is to check "what will happen if one drops the index". For this particular query, the effect will be like this: MariaDB [test]> explain select * from t1 force index(no_such_index) where a<3 and b<3; ERROR 1176 (42000): Key 'no_such_index' doesn't exist in table 't1' and MySQL agrees with me here (below index a is present but is invisible) : mysql> explain select * from t1 force index(a) where a<3 and b<3; ERROR 1176 (42000): Key 'a' doesn't exist in table 't1' == optimizer_switch flag == Seeing "ignore_indexes=on" in optimizer_switch looks scary. I would say, we don't need an @@optimizer_switch flag. Maybe, Serg disagrees. If we have to have a switch flag, and its name should be short (my opinion), I would use the noun form: ignored_indexes=ON|OFF I'm not 100% happy with this, but ignoreD is much better here than ignore. (Before acting on this review point, lets check with others). BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://petrunia.net

2 1

Re: [Maria-developers] MDEV-17399: JSON_TABLE, commit 85757ecbef44484270e385a8d52998c67f72d11a
by Sergey Petrunia 23 Feb '21

23 Feb '21

Hi Alexey, First reactions to the JSON_TABLE patch: == CI Build is failing == Please check http://buildbot.askmonty.org/buildbot/grid?category=main&branch=bb-10.6-mde… the JSON_TABLE tests are failing. == On testcases == I see now the patch has tests in 3 files: 1. mysql-test/main/json_table_mysql.test 2. mysql-test/main/json_table_mysql2.test 3. mysql-test/suite/json/t/json_table.test #1 and #2 are both fixed copies of MySQL's test. Please remove one of them (the one that has less coverage). Then, can we have both files in the same testsuite? == On Table_function_json_table::ready_for_lookup() == Now that we have end_lateral_table, do we need to use Table_function_json_table::ready_for_lookup? This looks like two solutions for [almost] the same problem? I see that this example from json_table_mysql.test still uses the code path with ready_for_lookup() : --error ER_BAD_FIELD_ERROR UPDATE t1, JSON_TABLE(t1.c1,'$[*]' COLUMNS (a INT PATH '$.a')) AS jt1 SET jt1.a=1; Here, the name resolution for "jt1.a" is done without adjustment for end_lateral_table. Do we need the ready_for_lookup() trick here? (If we do, are we fine with m_setup_done to be set to true but never reset back to false? How will this work with PS?) == On opt_sum.cc == The diff has this: diff --git a/sql/opt_sum.cc b/sql/opt_sum.cc index af2d9ddc2e7..d021c651196 100644 --- a/sql/opt_sum.cc +++ b/sql/opt_sum.cc @@ -299,6 +299,7 @@ int opt_sum_query(THD *thd, tl->schema_table) { maybe_exact_count&= MY_TEST(!tl->schema_table && +//mdev-17399 !tl->table_function && (tl->table->file->ha_table_flags() & HA_HAS_RECORDS)); is_exact_count= FALSE; Please remove this if it is no longer needed. == On table dependencies: table elimination == create table t20 (a int not null); create table t21 (a int not null primary key, js varchar(100)); insert into t20 values (1),(2); insert into t21 values (1, '{"a":100}'); explain select t20.a, jt1.ab from t20 left join t21 on t20.a=t21.a join JSON_TABLE(t21.js,'$' COLUMNS (ab INT PATH '$.a')) AS jt1; +------+-------------+-------+------+---------------+------+---------+------+------+----------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +------+-------------+-------+------+---------------+------+---------+------+------+----------------------------+ | 1 | SIMPLE | t20 | ALL | NULL | NULL | NULL | NULL | 2 | | | 1 | SIMPLE | jt1 | ALL | NULL | NULL | NULL | NULL | 40 | Table function: json_table | +------+-------------+-------+------+---------------+------+---------+------+------+----------------------------+ Here we can see an apparently invalid query plan: table t21 was eleminated even if JSON_TABLE uses it. == On table dependencies: regular subqueries == But it doesn't work for regular subqueries either (and this is unexpected): insert into t21 values (3, '{"a":300}'); explain select a, (select jt1.ab from JSON_TABLE(t21.js,'$' COLUMNS (ab INT PATH '$.a')) as jt1 ) from t21; +------+-------------+-------+------+---------------+------+---------+------+------+----------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +------+-------------+-------+------+---------------+------+---------+------+------+----------------------------+ | 1 | PRIMARY | t21 | ALL | NULL | NULL | NULL | NULL | 2 | | | 2 | SUBQUERY | jt1 | ALL | NULL | NULL | NULL | NULL | 40 | Table function: json_table | +------+-------------+-------+------+---------------+------+---------+------+------+----------------------------+ Note the 'SUBQUERY', that is, the subquery is considered uncorrelated? This needs to be investigated. BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://petrunia.net

2 3

Re: [Maria-developers] MDEV-17399: Columns of JSON datatype?
by Sergey Petrunia 16 Feb '21

16 Feb '21

On Sun, Feb 14, 2021 at 05:14:10PM +0400, Alexey Botchkov wrote: > One question left about the json_table_mysql.text. > > --error ER_WRONG_OUTER_JOIN > SELECT * FROM t1 RIGHT JOIN > (SELECT o FROM t2, JSON_TABLE(j, '$[*]' COLUMNS (o FOR ORDINALITY)) AS jt) > AS t3 > ON (t3.o = t1.x); > > What is exactly wrong with the above join? It should work. The reason it doesn't work was described by me in the email with subject "MDEV-17399: JSON_TABLE's dependencies are not updated after derived merge" dated Feb, 8th. What happens here is: Thread 14 "mysqld" hit Breakpoint 1, Table_function_json_table::setup() (gdb) list 1091 } 1092 } 1093 > 1094 m_dep_tables= m_json->used_tables(); 1095 (gdb) p m_dep_tables $44 = 1 If you go one frame up, you will see that we are in select#2. tables are: (gdb) p join->select_lex->leaf_tables.elem(0)->alias.str $56 = 0x7ffeb0016550 "t2" (gdb) p join->select_lex->leaf_tables.elem(0)->table->map $58 = 1 (gdb) p join->select_lex->leaf_tables.elem(1)->alias.str $60 = 0x7ffeb0017d70 "jt" (gdb) p join->select_lex->leaf_tables.elem(1)->table->map $62 = 2 Ok, jt depends on t2. Then: Thread 14 "mysqld" hit Breakpoint 2, simplify_joins (gdb) p join->select_lex->leaf_tables.elements $65 = 3 (gdb) p join->select_lex->leaf_tables.elem(0)->alias.str $66 = 0x7ffeb0015870 "t1" (gdb) p join->select_lex->leaf_tables.elem(0)->table->map $67 = 1 (gdb) p join->select_lex->leaf_tables.elem(1)->alias.str $68 = 0x7ffeb0016550 "t2" (gdb) p join->select_lex->leaf_tables.elem(1)->table->map $69 = 2 (gdb) p join->select_lex->leaf_tables.elem(2)->alias.str $70 = 0x7ffeb0017d70 "jt" (gdb) p join->select_lex->leaf_tables.elem(2)->table->map $71 = 4 But the m_dep_tables still has the old value: (gdb) p join->select_lex->leaf_tables.elem(2)->table_function->m_dep_tables $72 = 1 BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://petrunia.net

1 0

[Maria-developers] MDEV-8306: EXPLAIN JSON output is not JSON-ish
by Sergey Petrunia 15 Feb '21

15 Feb '21

Hi Varun, (This email is not the complete input, and not the most important part of the input. More to follow. I think it's better to split input into multiple smaller pieces, it's easier to track, and you get to see the first portions of the input sooner) EXPLAIN EXTENDED SELECT t3.b, t2.a, t1.b, t1.a FROM t1,t2,t3 WHERE t1.b=t3.b ORDER BY t1.b DESC, t2.a DESC LIMIT 3; +------+-------------+-------------+------+---------------+------+---------+------+------+----------+------------------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | filtered | Extra | +------+-------------+-------------+------+---------------+------+---------+------+------+----------+------------------------------------+ | 1 | SIMPLE | t2 | ALL | NULL | NULL | NULL | NULL | 10 | 100.00 | | | 1 | SIMPLE | t1 | ALL | NULL | NULL | NULL | NULL | 20 | 100.00 | Using join buffer (flat, BNL join) | | 1 | SIMPLE | <sort-nest> | ALL | NULL | NULL | NULL | NULL | 3 | 100.00 | Using filesort | | 1 | SIMPLE | t3 | ALL | NULL | NULL | NULL | NULL | 100 | 5.00 | Using where | +------+-------------+-------------+------+---------------+------+---------+------+------+----------+------------------------------------+ So, here <sort-nest> includes t2 and t1, and then t3 is joined with it. However in EXPLAIN FORMAT=JSON output they all look like peers: { "query_block": { "select_id": 1, "table": { "table_name": "t2", "access_type": "ALL", "rows": 10, "filtered": 100 }, "block-nl-join": { "table": { "table_name": "t1", "access_type": "ALL", "rows": 20, "filtered": 100 }, "buffer_type": "flat", "buffer_size": "65", "join_type": "BNL" }, "read_sorted_file": { "filesort": { "sort_key": "`sort-nest`.b desc, `sort-nest`.a desc", "table": { "table_name": "<sort-nest>", "access_type": "ALL", "rows": 3, "filtered": 100 } } }, "table": { "table_name": "t3", "access_type": "ALL", "rows": 100, "filtered": 5, "attached_condition": "t3.b = `sort-nest`.b" } } } which is counter-intuitive. Also, `sort-nest` is quoted. I assume it is because of the '-' in it? Maybe we should change the name to avoid the quoting? BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://petrunia.net

2 3

Re: [Maria-developers] MDEV-17399 New patch for JSON_TABLE
by Sergey Petrunia 14 Feb '21

14 Feb '21

Hi Alexey, Input on the latest patch for MDEV-17399. It's good to see the patch taking shape, but have you tried running MySQL's tests for JSON_TABLE on it? I did, and it has caught several crashes, as well as error-no-error discrepancies and different query results. Some of these were expected, but some show that something needs to be fixed. Please find the first few cases below. It seems there is something odd going odd with the name resolution, both in MariaDB and MySQL - I don't have explanation for some of the errors. I intend to study the issue more and elaborate in a later email. Meanwhile, first obvious cases: === Crash in Name resolution == CREATE TABLE t1 (a INT, b INT); CREATE VIEW v2 AS SELECT * FROM t1 LIMIT 2; SELECT b FROM (SELECT * FROM v2) vq1, JSON_TABLE(CONCAT(vq1.b,'[{\"a\":\"3\"}]'), '$[*]' COLUMNS (id FOR ORDINALITY, jpath VARCHAR(100) PATH '$.a', JEXST INT EXISTS PATH '$.b') ) AS dt; === Crash in error reporting === select * from json_table( '[{"a":"3"},{"a":2},{"b":1},{"a":0}]', "!@#$!@#$" columns (id for ordinality, jpath varchar(100) path '$.a', jexst int exists path '$.b') ) as tt; === Item print misses quotes === CREATE VIEW v2 AS SELECT * FROM JSON_TABLE('{}', '$' COLUMNS ( x VARCHAR(10) PATH '$.a' DEFAULT '"isn''t here"' ON EMPTY) ) t; SHOW CREATE VIEW v2; ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 't here"' ON EMPTY)) `t`' at line 1 === Error for outer reference == # JSON_TABLE referring outer scope CREATE TABLE t1(id int, jd JSON); INSERT INTO t1 values (1, '[1,3,5]'),(2,'[2,4,6]'); SELECT * FROM t1 WHERE id IN (SELECT * FROM JSON_TABLE(t1.jd, '$[*]' COLUMNS (id INT PATH '$')) AS jt); Produces: ERROR 1054 (42S22): Unknown column 't1.jd' in 'JSON_TABLE argument' Should not produce it (Note that for some other kinds of outer references it seemed to work) BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

2 15

[Maria-developers] MDEV-17399: Name resolution and handling dependencies
by Sergey Petrunia 14 Feb '21

14 Feb '21

Hi Alexey, At the moment, name resolution of JSON_TABLE's first argument is done "like in the WHERE clause" - one can refer to any table that is defined in the WHERE clause. This allows one to write queries where JSON_TABLE tables have incorrect dependencies - circular - dependency that contradicts the dependency imposed by OUTER JOIN - dependency that contradicts STRAIGHT_JOIN (WRONG-DEPS) the patch checks for these cases but I've reported cases where it fails. I haven't been find any statement in the SQL Standard about this, but it makes a statement about similar constructs, table functions and LATERAL subqueries: https://jira.mariadb.org/browse/MDEV-17399?focusedCommentId=179327&page=com… Other databases seem to apply the same limitation to JSON_TABLE's argument. When they do it, this automatically fixes all the wrong-dependency issues - a query with (WRONG-DEPS) is rejected at the name resolution phase. I think we should follow this and modify name resolution process to work in the same way. The way MySQL did it is described here: https://dev.mysql.com/worklog/task/?id=8867, LLD, grep for end_lateral_table. I'm not fond of having implicit parameters (end_lateral_table) which change they way name resolution works, but I think it is an acceptable solution (and we already have other such parameters). (An alternative option would be to have items in JSON_TABLE's first argument to use their own Name_resolution_context object that would specify the correct first/last table they should look at. This seems to be much harder to do). What do you think? BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://petrunia.net

2 1

[Maria-developers] 4f9c5a8cf54: MDEV-24773: slave_compressed_protocol doesn't work properly with semi-sync replication
by sujatha 08 Feb '21

08 Feb '21

revision-id: 4f9c5a8cf545133c3d29c0de3f7e2e57564607f2 (mariadb-10.3.26-78-g4f9c5a8cf54) parent(s): 59eda73eff1a22ac0373d818bc802c05e82b5449 author: Sujatha committer: Sujatha timestamp: 2021-02-05 16:45:35 +0530 message: MDEV-24773: slave_compressed_protocol doesn't work properly with semi-sync replication Back port upstream fix commit 1800b015a1d487330f7b15f2020b887be348a66b Author: Venkatesh Duggirala <venkatesh.duggirala(a)oracle.com> Date: Fri Sep 8 20:29:22 2017 +0530 Bug#26027024 SLAVE_COMPRESSED_PROTOCOL DOESN'T WORK WITH SEMI-SYNC REPLICATION IN MYSQL-5.7 Analysis: In mysql-5.6, dump thread (the thread that is created on Master after Slave requested for a binlog dump) is also used to receive acknowledgements from the Slave and act on them accordingly. For performance reasons, a special thread called Ack Receiver thread is added in mysql-5.7 Semi synchronous replication plugin. This thread does not have special handling to receive acknowledgements if Slave has enabled compression in the protocol. Hence Master is unable to handle any slave if Slave_compressed_protocol is enabled on it. Fix: Enable compress flag on the communication channels if the Slave has Slave_compressed_protocol ON. --- .../rpl_semi_sync_slave_compressed_protocol.result | 19 ++++++++ ...l_semi_sync_slave_compressed_protocol-slave.opt | 1 + .../t/rpl_semi_sync_slave_compressed_protocol.test | 55 ++++++++++++++++++++++ sql/semisync_master_ack_receiver.cc | 5 ++ 4 files changed, 80 insertions(+) diff --git a/mysql-test/suite/rpl/r/rpl_semi_sync_slave_compressed_protocol.result b/mysql-test/suite/rpl/r/rpl_semi_sync_slave_compressed_protocol.result new file mode 100644 index 00000000000..86816652565 --- /dev/null +++ b/mysql-test/suite/rpl/r/rpl_semi_sync_slave_compressed_protocol.result @@ -0,0 +1,19 @@ +include/master-slave.inc +[connection master] +SET @@GLOBAL.rpl_semi_sync_master_enabled = 1; +connection slave; +include/stop_slave.inc +SET @@GLOBAL.rpl_semi_sync_slave_enabled = 1; +include/start_slave.inc +connection master; +CREATE TABLE t1 (i INT); +DROP TABLE t1; +include/rpl_sync.inc +include/assert_grep.inc [Check that there is no 'Read semi-sync reply magic number error' in error log.] +connection master; +SET @@GLOBAL. rpl_semi_sync_master_enabled = 0; +connection slave; +include/stop_slave.inc +SET @@GLOBAL. rpl_semi_sync_slave_enabled = 0; +include/start_slave.inc +include/rpl_end.inc diff --git a/mysql-test/suite/rpl/t/rpl_semi_sync_slave_compressed_protocol-slave.opt b/mysql-test/suite/rpl/t/rpl_semi_sync_slave_compressed_protocol-slave.opt new file mode 100644 index 00000000000..a1b687d691e --- /dev/null +++ b/mysql-test/suite/rpl/t/rpl_semi_sync_slave_compressed_protocol-slave.opt @@ -0,0 +1 @@ +--slave_compressed_protocol diff --git a/mysql-test/suite/rpl/t/rpl_semi_sync_slave_compressed_protocol.test b/mysql-test/suite/rpl/t/rpl_semi_sync_slave_compressed_protocol.test new file mode 100644 index 00000000000..6343fb29ffe --- /dev/null +++ b/mysql-test/suite/rpl/t/rpl_semi_sync_slave_compressed_protocol.test @@ -0,0 +1,55 @@ +################################################################################ +# Bug#26027024 SLAVE_COMPRESSED_PROTOCOL DOESN'T WORK WITH SEMI-SYNC +# REPLICATION IN MYSQL-5.7 +# +# Steps to reproduce: +# 1) Set slave_compressed_protocol ON on Slave. +# 2) Do some sample work on Master +# 3) After the work is synced on Slave, check that there is no error +# (Read semi-sync reply magic number error) on Slave. +# 4) Cleanup +################################################################################ +# Test is independent of Binlog format. One of the three formats is enough +# for testing. Choosing 'Row' format. +--source include/have_binlog_format_row.inc +--source include/master-slave.inc + +--let $sav_enabled_master=`SELECT @@GLOBAL.rpl_semi_sync_master_enabled ` +SET @@GLOBAL.rpl_semi_sync_master_enabled = 1; + +--connection slave +source include/stop_slave.inc; +--let $sav_enabled_slave=`SELECT @@GLOBAL.rpl_semi_sync_slave_enabled ` +SET @@GLOBAL.rpl_semi_sync_slave_enabled = 1; +source include/start_slave.inc; + +--connection master +# Do some sample work on Master with slave_compressed_protocol ON. +# (slave_compressed_protocol is set to ON in -slave.opt file of this test.) +CREATE TABLE t1 (i INT); +DROP TABLE t1; + +# Make sure sync is done, so that next 'assert' step can be executed without +# any issues. +--source include/rpl_sync.inc + +# Without the fix, the test would have generated few +# errors in the error log. With the fix, test will +# pass without any errors in the error log. +--let $assert_text= Check that there is no 'Read semi-sync reply magic number error' in error log. +--let $assert_select=Read semi-sync reply magic number error +--let $assert_file= $MYSQLTEST_VARDIR/log/mysqld.1.err +--let $assert_count= 0 +--let $assert_only_after = CURRENT_TEST:rpl.rpl_semi_sync_slave_compressed_protocol.test +--source include/assert_grep.inc + +--connection master +--eval SET @@GLOBAL. rpl_semi_sync_master_enabled = $sav_enabled_master + +--connection slave +source include/stop_slave.inc; +--eval SET @@GLOBAL. rpl_semi_sync_slave_enabled = $sav_enabled_slave +source include/start_slave.inc; + +# Cleanup +--source include/rpl_end.inc diff --git a/sql/semisync_master_ack_receiver.cc b/sql/semisync_master_ack_receiver.cc index 81f494c9d34..cc3a56a56b2 100644 --- a/sql/semisync_master_ack_receiver.cc +++ b/sql/semisync_master_ack_receiver.cc @@ -268,6 +268,11 @@ void Ack_receiver::run() net_clear(&net, 0); net.vio= &slave->vio; + /* + Set compress flag. This is needed to support + Slave_compress_protocol flag enabled Slaves + */ + net.compress= slave->thd->net.compress; len= my_net_read(&net); if (likely(len != packet_error))

1 0

[Maria-developers] RFC: incompatible changes to mtr debugging flags
by Sergei Golubchik 27 Jan '21

27 Jan '21

Hi, all, Here's a feature I want to discuss, as it's rather incompatibly removes a bunch of mtr command-line options. It's a unified handling of all debuggers. I wanted to do it for quite a while, but the actual trigger was me trying to add support for --rr (https://rr-project.org/) in embedded. It was supported only for non-embedded server, and making it work for embedded needed way more copy-pasting than I was comfortable with. So, in my branch all debuggers are not haphazardly added using whatever syntax one thought of at the moment and copy-pasted all over. They've handled by a common module, all support the same set of options (using gdb as an example): --gdb, --boot-gdb, --client-gdb, --manual-gdb. Both --gdb and --client-gdb work for embedded. All four of them accept an optional argument that can be a semicolon separated list of gdb commands, like in ./mtr 1st --gdb='b mysql_parse;r' but can start from command-line options too: ./mtr 1st --boot-gdb='-nh -q;b mysql_parse;r' A "debugger" is anything that wraps mysqld or mysqltest execution, the current list of debuggers is: gdb, ddd, dbx, lldb, valgrind, strace, ktrace, rr, devenv, windbg, vc_express, vc, vsjitdebugger. There's also "valgdb" that does what '--valgrind --gdb' was doing before. This removes the following mtr options: --rr-args --rr-dir --manual-debug --debugger --strace-option --stracer --valgrind-all --valgrind-mysqltest --valgrind-mysqld --valgrind-options --valgrind-option --valgrind-path --callgrind, and --valgrind only enables it for the server, not for everything. If pushed, it'll definitely break backward compatibility. But I don't think mtr backward compatibility is all that important, it might only affect a handful of developers who use scripts to start mtr with specific hard-coded settings, those scripts could be easily adjusted. It's much more important to have same mtr features in all branches, so that when switching between branches, one wouldn't need to guess what command line options mtr supports now. That is, if pushed, this should go into 10.2. Opinions? Missing features? Push / not push? for the reference, it's in bb-10.2-serg branch at the moment. Regards, Sergei

5 7

Re: [Maria-developers] 3e01da668d1: MDEV-21785: sequences used as default by other table not dumped in right order by mysqldump
by Sergei Golubchik 21 Jan '21

21 Jan '21

Hi, Oleksandr! On Jan 21, Oleksandr Byelkin wrote: > revision-id: 3e01da668d1 (mariadb-10.3.26-64-g3e01da668d1) > parent(s): 59998d3480f > author: Oleksandr Byelkin <sanja(a)mariadb.com> > committer: Oleksandr Byelkin <sanja(a)mariadb.com> > timestamp: 2021-01-14 10:45:51 +0100 > message: > > MDEV-21785: sequences used as default by other table not dumped in right order by mysqldump > > Dump sequences first. > > This atch made to keep it small and > to keep number of queries to the server the same. > > Order of tables in a dump can not be changed > (except sequences first) because mysql_list_tables > uses SHOW TABLES and I used SHOW FULL TABLES. it looks like a rather overengineered implementation. I'd suggest to simplify it as: > diff --git a/client/mysqldump.c b/client/mysqldump.c > index a964f96437d..17837713c01 100644 > --- a/client/mysqldump.c > +++ b/client/mysqldump.c > @@ -42,6 +42,11 @@ > /* on merge conflict, bump to a higher version again */ > #define DUMP_VERSION "10.19" > > +/** > + First mysql version supporting sequences. > +*/ > +#define FIRST_SEQUENCE_VERSION 100300 seems unnecessary, you implemented SHOW FULL TABLES in 5.0.2-alpha. Oct 2004. You can safely use SHOW FULL TABLES unconditionally. > + > #include <my_global.h> > #include <my_sys.h> > #include <my_user.h> > @@ -92,6 +97,13 @@ > /* Max length GTID position that we will output. */ > #define MAX_GTID_LENGTH 1024 > > + > +/* Dump sequence/tables control */ > +#define DUMP_TABLE_TABLE 1 > +#define DUMP_TABLE_SEQUENCE 2 > +#define DUMP_TABLE_ALL (DUMP_TABLE_TABLE | DUMP_TABLE_SEQUENCE) > + > + > static my_bool ignore_table_data(const uchar *hash_key, size_t len); > static void add_load_option(DYNAMIC_STRING *str, const char *option, > const char *option_value); > @@ -171,6 +183,7 @@ static DYNAMIC_STRING dynamic_where; > static MYSQL_RES *get_table_name_result= NULL; > static MEM_ROOT glob_root; > static MYSQL_RES *routine_res, *routine_list_res; > +static int get_table_name_result_short= 1; that'll be unnecessary > > > #include <sslopt-vars.h> > @@ -3853,13 +3866,16 @@ static char *alloc_query_str(size_t size) > ARGS > table - table name > db - db name > + type_ctrl - dump table or sequences or both > > RETURNS > void > */ > > > -static void dump_table(const char *table, const char *db, const uchar *hash_key, size_t len) > +static void dump_table(const char *table, const char *db, > + const uchar *hash_key, size_t len, > + const uint type_ctrl) > { > char ignore_flag; > char buf[200], table_buff[NAME_LEN+3]; > @@ -3881,9 +3897,11 @@ static void dump_table(const char *table, const char *db, const uchar *hash_key, > */ > if (check_if_ignore_table(table, table_type) & IGNORE_SEQUENCE_TABLE) > { > - get_sequence_structure(table, db); > + if (type_ctrl & DUMP_TABLE_SEQUENCE) > + get_sequence_structure(table, db); > + DBUG_VOID_RETURN; > + } else if (!(type_ctrl & DUMP_TABLE_TABLE)) > DBUG_VOID_RETURN; > - } here I'd remove the whole if(). Let's say the convention is that dump_table() should never be invoked for sequences, it's the caller's job to ensure it. > /* > Make sure you get the create table info before the following check for > --no-data flag below. Otherwise, the create table info won't be printed. > @@ -4368,18 +4386,49 @@ static void dump_table(const char *table, const char *db, const uchar *hash_key, > } /* dump_table */ > > > -static char *getTableName(int reset) > +static char *getTableName(int reset, uint table_control) > { > MYSQL_ROW row; > > if (!get_table_name_result) > { > - if (!(get_table_name_result= mysql_list_tables(mysql,NullS))) > - return(NULL); > + if (mysql_get_server_version(mysql) >= FIRST_SEQUENCE_VERSION) > + { > + const char *query= > + "SHOW FULL TABLES"; > + > + if (mysql_query_with_error_report(mysql, 0, query)) > + return (NULL); > + > + if (!(get_table_name_result= mysql_store_result(mysql))) > + return (NULL); > + > + get_table_name_result_short= 0; > + } > + else > + { > + if (!(get_table_name_result= mysql_list_tables(mysql,NullS))) > + return(NULL); > + get_table_name_result_short= 1; > + } this will be twice as small, if you won't try to support before-5.0 servers. > } > if ((row= mysql_fetch_row(get_table_name_result))) > - return((char*) row[0]); > - > + { > + int sequence; > + if (get_table_name_result_short || > + table_control == DUMP_TABLE_ALL) > + return((char*) row[0]); > + while (row && > + (((sequence= (strcmp(row[1], "SEQUENCE") == 0)) && > + (table_control & DUMP_TABLE_TABLE)) || > + ((!sequence) && > + (table_control & DUMP_TABLE_SEQUENCE)))) I'd rather just do static char *getTableName(int reset, uint want_sequence) and while (row && strcmp(row[1], "SEQUENCE") == want_sequence) row= mysql_fetch_row(get_table_name_result); > + { > + row= mysql_fetch_row(get_table_name_result); > + } > + if (row) > + return((char*) row[0]); > + } > if (reset) > mysql_data_seek(get_table_name_result,0); /* We want to read again */ > else > @@ -4785,17 +4834,17 @@ static int dump_all_stats() > /* EITS added in 10.0.1 */ > if (mysql_get_server_version(mysql) >= 100001) > { > - dump_table("column_stats", "mysql", NULL, 0); > - dump_table("index_stats", "mysql", NULL, 0); > - dump_table("table_stats", "mysql", NULL, 0); > + dump_table("column_stats", "mysql", NULL, 0, DUMP_TABLE_TABLE); > + dump_table("index_stats", "mysql", NULL, 0, DUMP_TABLE_TABLE); > + dump_table("table_stats", "mysql", NULL, 0, DUMP_TABLE_TABLE); this wouldn't be needed if the caller will only use dump_table for tables > } > /* Innodb may be disabled */ > if (!mysql_query(mysql, "show fields from innodb_index_stats")) > { > MYSQL_RES *tableres= mysql_store_result(mysql); > mysql_free_result(tableres); > - dump_table("innodb_index_stats", "mysql", NULL, 0); > - dump_table("innodb_table_stats", "mysql", NULL, 0); > + dump_table("innodb_index_stats", "mysql", NULL, 0, DUMP_TABLE_TABLE); > + dump_table("innodb_table_stats", "mysql", NULL, 0, DUMP_TABLE_TABLE); > } > opt_no_create_info= prev_no_create_info; > return 0; > @@ -4817,11 +4866,11 @@ static int dump_all_timezones() > opt_prev_no_create_info= opt_no_create_info; > opt_no_create_info= 1; > fprintf(md_result_file,"\nUSE mysql;\n"); > - dump_table("time_zone", "mysql", NULL, 0); > - dump_table("time_zone_name", "mysql", NULL, 0); > - dump_table("time_zone_leap_second", "mysql", NULL, 0); > - dump_table("time_zone_transition", "mysql", NULL, 0); > - dump_table("time_zone_transition_type", "mysql", NULL, 0); > + dump_table("time_zone", "mysql", NULL, 0, DUMP_TABLE_TABLE); > + dump_table("time_zone_name", "mysql", NULL, 0, DUMP_TABLE_TABLE); > + dump_table("time_zone_leap_second", "mysql", NULL, 0, DUMP_TABLE_TABLE); > + dump_table("time_zone_transition", "mysql", NULL, 0, DUMP_TABLE_TABLE); > + dump_table("time_zone_transition_type", "mysql", NULL, 0, DUMP_TABLE_TABLE); > opt_no_create_info= opt_prev_no_create_info; > return 0; > } > @@ -5346,12 +5396,29 @@ static int dump_all_tables_in_db(char *database) > DBUG_RETURN(1); > } > } > - while ((table= getTableName(0))) > + > + if (mysql_get_server_version(mysql) >= FIRST_SEQUENCE_VERSION && > + !opt_no_create_info) > + { > + // First process sequences > + while ((table= getTableName(1, DUMP_TABLE_SEQUENCE))) > + { > + char *end= strmov(afterdot, table); > + if (include_table((uchar*) hash_key, end - hash_key)) > + { > + dump_table(table, database, (uchar*) hash_key, end - hash_key, > + DUMP_TABLE_SEQUENCE); > + } > + } > + table_ctrl= DUMP_TABLE_TABLE; // next pass > + } and instead of all the above, you could do just for (int table_ctrl= 1; table_ctrl >= 0; table_ctrl--) > + while ((table= getTableName(0, table_ctrl))) > { > char *end= strmov(afterdot, table); > if (include_table((uchar*) hash_key, end - hash_key)) > { > - dump_table(table, database, (uchar*) hash_key, end - hash_key); > + dump_table(table, database, (uchar*) hash_key, end - hash_key, > + table_ctrl); > my_free(order_by); > order_by= 0; > if (opt_dump_triggers && mysql_get_server_version(mysql) >= 50009) > @@ -5745,11 +5813,24 @@ static int dump_selected_tables(char *db, char **table_names, int tables) > DBUG_RETURN(1); > } > } > + > + if (mysql_get_server_version(mysql) >= FIRST_SEQUENCE_VERSION && > + !opt_no_create_info) > + { > + /* Dump Sequence first */ > + for (pos= dump_tables; pos < end; pos++) > + { > + DBUG_PRINT("info",("Dumping sequence(?) %s", *pos)); > + dump_table(*pos, db, NULL, 0, DUMP_TABLE_SEQUENCE); > + } > + } > + else > + table_ctrl|= DUMP_TABLE_SEQUENCE; // dump all on next pass and here, in dump_selected_tables(), you'd need to do check_if_ignore_table() == IGNORE_SEQUENCE_TABLE. > /* Dump each selected table */ > for (pos= dump_tables; pos < end; pos++) > { > DBUG_PRINT("info",("Dumping table %s", *pos)); > - dump_table(*pos, db, NULL, 0); > + dump_table(*pos, db, NULL, 0, table_ctrl); > if (opt_dump_triggers && > mysql_get_server_version(mysql) >= 50009) > { Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

[Maria-developers] eb75e8705d9: MDEV-8134: The relay-log is not flushed after the slave-relay-log.999999 showed
by sujatha 21 Jan '21

21 Jan '21

revision-id: eb75e8705d9a444e10057967eaebf947b1115ff8 (mariadb-10.2.31-695-geb75e8705d9) parent(s): 53acd1c1d88be82190c56af3e4cc11fb2770a169 author: Sujatha committer: Sujatha timestamp: 2021-01-21 13:00:02 +0530 message: MDEV-8134: The relay-log is not flushed after the slave-relay-log.999999 showed Problem: ======== Auto purge of relaylogs stops when relay-log-file is 'slave-relay-log.999999' and slave_parallel_threads is enabled. Analysis: ========= The problem is that in Relay_log_info::inc_group_relay_log_pos() function, when two log names are compared via strcmp() function, it gives correct result, when log name sequence numbers are of same digits(6 digits), But when the number goes to 7 digits, a 999999 compares greater than 1000000, which is wrong, hence the bug. Fix: ==== Extract the numeric extension part of the file name, convert it into unsigned long and compare. Thanks to David Zhao for the contribution. --- .../suite/rpl/r/rpl_relay_max_extension.result | 37 +++++++ .../suite/rpl/t/rpl_relay_max_extension.test | 109 +++++++++++++++++++++ sql/rpl_parallel.cc | 5 +- sql/rpl_rli.cc | 4 +- sql/sql_repl.cc | 17 ++++ sql/sql_repl.h | 1 + 6 files changed, 169 insertions(+), 4 deletions(-) diff --git a/mysql-test/suite/rpl/r/rpl_relay_max_extension.result b/mysql-test/suite/rpl/r/rpl_relay_max_extension.result new file mode 100644 index 00000000000..4444398203e --- /dev/null +++ b/mysql-test/suite/rpl/r/rpl_relay_max_extension.result @@ -0,0 +1,37 @@ +include/rpl_init.inc [topology=1->2] +connection server_2; +include/stop_slave.inc +RESET SLAVE; +include/start_slave.inc +include/stop_slave.inc +# +# Stop slave server +# +# +# Simulate file number get close to 999997 +# by renaming relay logs and modifying index/info files +# +# Restart slave server +# +SET @save_slave_parallel_threads= @@GLOBAL.slave_parallel_threads; +SET @save_max_relay_log_size= @@GLOBAL.max_relay_log_size; +SET GLOBAL slave_parallel_threads=1; +SET GLOBAL max_relay_log_size=100 * 1024; +include/start_slave.inc +connection server_1; +create table t1 (i int, c varchar(1024)); +# +# Insert some data to generate enough amount of binary logs +# +connection server_2; +# +# Assert that 'slave-relay-bin.999999' is purged. +# +NOT FOUND /slave-relay-bin.999999/ in slave-relay-bin.index +include/stop_slave.inc +SET GLOBAL slave_parallel_threads= @save_slave_parallel_threads; +SET GLOBAL max_relay_log_size= @save_max_relay_log_size; +include/start_slave.inc +connection server_1; +DROP TABLE t1; +include/rpl_end.inc diff --git a/mysql-test/suite/rpl/t/rpl_relay_max_extension.test b/mysql-test/suite/rpl/t/rpl_relay_max_extension.test new file mode 100644 index 00000000000..e1e087f2e0e --- /dev/null +++ b/mysql-test/suite/rpl/t/rpl_relay_max_extension.test @@ -0,0 +1,109 @@ +# ==== Purpose ==== +# +# Test verifies that auto purging mechanism of relay logs works fine when the +# file extension grows beyond 999999. +# +# ==== Implementation ==== +# +# Steps: +# 0 - In master-slave setup clear all the relay logs on the slave server. +# 1 - Start the slave so that new relay logs starting from +# 'slave-relay-bin.000001' are created. +# 2 - Get the active relay-log file name by using SHOW SLAVE STATUS. +# Shutdown the slave server. +# 3 - Rename active relay log to '999997' in both 'relay-log.info' and +# 'slave-relay-bin.index' files. +# 4 - Restart the slave server by configuring 'slave_parallel_threads=1' +# and 'max_relay_log_size=100K'. +# 5 - Generate load on master such that few relay logs are generated on +# slave. The relay log sequence number will change to 7 digits. +# 6 - Sync slave with master to ensure that relay logs are applied on +# slave. They should have been automatically purged. +# 7 - Assert that there is no 'slave-relay-bin.999999' file in +# 'relay-log.info'. +# +# ==== References ==== +# +# MDEV-8134: The relay-log is not flushed after the slave-relay-log.999999 +# showed +# + +--source include/have_innodb.inc +--source include/have_binlog_format_row.inc +--let $rpl_topology=1->2 +--source include/rpl_init.inc + +--connection server_2 +--source include/stop_slave.inc +RESET SLAVE; +--source include/start_slave.inc +--source include/stop_slave.inc +--let $relay_log=query_get_value(SHOW SLAVE STATUS, Relay_Log_File, 1) + +--echo # +--echo # Stop slave server +--echo # + +--let $datadir = `select @@datadir` +--exec echo "wait" > $MYSQLTEST_VARDIR/tmp/mysqld.2.expect +--shutdown_server 10 +--source include/wait_until_disconnected.inc + +--exec sed -i "s/$relay_log/slave-relay-bin.999997/g" $datadir/relay-log.info +--exec sed -i "s/$relay_log/slave-relay-bin.999997/g" $datadir/slave-relay-bin.index + +--echo # +--echo # Simulate file number get close to 999997 +--echo # by renaming relay logs and modifying index/info files + +--move_file $datadir/$relay_log $datadir/slave-relay-bin.999997 + +--echo # +--echo # Restart slave server +--echo # + +--exec echo "restart" > $MYSQLTEST_VARDIR/tmp/mysqld.2.expect +--enable_reconnect +--source include/wait_until_connected_again.inc +SET @save_slave_parallel_threads= @@GLOBAL.slave_parallel_threads; +SET @save_max_relay_log_size= @@GLOBAL.max_relay_log_size; + +SET GLOBAL slave_parallel_threads=1; +SET GLOBAL max_relay_log_size=100 * 1024; +--source include/start_slave.inc + +--connection server_1 +create table t1 (i int, c varchar(1024)); +--echo # +--echo # Insert some data to generate enough amount of binary logs +--echo # +--let $count = 1000 +--disable_query_log +while ($count) +{ + eval insert into t1 values (1001 - $count, repeat('a',1000)); + dec $count; +} +--enable_query_log +--save_master_pos + +--connection server_2 +--sync_with_master + +--let $relay_log=query_get_value(SHOW SLAVE STATUS, Relay_Log_File, 1) + +--echo # +--echo # Assert that 'slave-relay-bin.999999' is purged. +--echo # +let SEARCH_FILE=$datadir/slave-relay-bin.index; +let SEARCH_PATTERN=slave-relay-bin.999999; +source include/search_pattern_in_file.inc; + +--source include/stop_slave.inc +SET GLOBAL slave_parallel_threads= @save_slave_parallel_threads; +SET GLOBAL max_relay_log_size= @save_max_relay_log_size; +--source include/start_slave.inc + +--connection server_1 +DROP TABLE t1; +--source include/rpl_end.inc diff --git a/sql/rpl_parallel.cc b/sql/rpl_parallel.cc index 4cf87ba73b7..869640fd46f 100644 --- a/sql/rpl_parallel.cc +++ b/sql/rpl_parallel.cc @@ -4,6 +4,7 @@ #include "rpl_mi.h" #include "sql_parse.h" #include "debug_sync.h" +#include "sql_repl.h" /* Code for optional parallel execution of replicated events on the slave. @@ -82,7 +83,7 @@ handle_queued_pos_update(THD *thd, rpl_parallel_thread::queued_event *qev) return; mysql_mutex_lock(&rli->data_lock); - cmp= strcmp(rli->group_relay_log_name, qev->event_relay_log_name); + cmp= compare_log_name(rli->group_relay_log_name, qev->event_relay_log_name); if (cmp < 0) { rli->group_relay_log_pos= qev->future_event_relay_log_pos; @@ -91,7 +92,7 @@ handle_queued_pos_update(THD *thd, rpl_parallel_thread::queued_event *qev) rli->group_relay_log_pos < qev->future_event_relay_log_pos) rli->group_relay_log_pos= qev->future_event_relay_log_pos; - cmp= strcmp(rli->group_master_log_name, qev->future_event_master_log_name); + cmp= compare_log_name(rli->group_master_log_name, qev->future_event_master_log_name); if (cmp < 0) { strcpy(rli->group_master_log_name, qev->future_event_master_log_name); diff --git a/sql/rpl_rli.cc b/sql/rpl_rli.cc index 40ab375571a..5273b33c728 100644 --- a/sql/rpl_rli.cc +++ b/sql/rpl_rli.cc @@ -989,7 +989,7 @@ void Relay_log_info::inc_group_relay_log_pos(ulonglong log_pos, if (rgi->is_parallel_exec) { /* In case of parallel replication, do not update the position backwards. */ - int cmp= strcmp(group_relay_log_name, rgi->event_relay_log_name); + int cmp= compare_log_name(group_relay_log_name, rgi->event_relay_log_name); if (cmp < 0) { group_relay_log_pos= rgi->future_event_relay_log_pos; @@ -1001,7 +1001,7 @@ void Relay_log_info::inc_group_relay_log_pos(ulonglong log_pos, In the parallel case we need to update the master_log_name here, rather than in Rotate_log_event::do_update_pos(). */ - cmp= strcmp(group_master_log_name, rgi->future_event_master_log_name); + cmp= compare_log_name(group_master_log_name, rgi->future_event_master_log_name); if (cmp <= 0) { if (cmp < 0) diff --git a/sql/sql_repl.cc b/sql/sql_repl.cc index 4af8ebc2dd8..59a3f686e45 100644 --- a/sql/sql_repl.cc +++ b/sql/sql_repl.cc @@ -4541,5 +4541,22 @@ rpl_gtid_pos_update(THD *thd, char *str, size_t len) return false; } +int compare_log_name(const char *log_1, const char *log_2) { + int res= 1; + const char *ext1_str= strrchr(log_1, '.'); + const char *ext2_str= strrchr(log_2, '.'); + char file_name_1[255], file_name_2[255]; + strmake(file_name_1, log_1, (ext1_str - log_1)); + strmake(file_name_2, log_2, (ext2_str - log_2)); + char *endptr = NULL; + res= strcmp(file_name_1, file_name_2); + if (!res) + { + ulong ext1= strtoul(++ext1_str, &endptr, 10); + ulong ext2= strtoul(++ext2_str, &endptr, 10); + res= (ext1 > ext2 ? 1 : ((ext1 == ext2) ? 0 : -1)); + } + return res; +} #endif /* HAVE_REPLICATION */ diff --git a/sql/sql_repl.h b/sql/sql_repl.h index 8ddfa9239f6..9129aaeed5e 100644 --- a/sql/sql_repl.h +++ b/sql/sql_repl.h @@ -56,6 +56,7 @@ bool show_binlogs(THD* thd); extern int init_master_info(Master_info* mi); void kill_zombie_dump_threads(uint32 slave_server_id); int check_binlog_magic(IO_CACHE* log, const char** errmsg); +int compare_log_name(const char *log_1, const char *log_2); struct LOAD_FILE_IO_CACHE : public IO_CACHE {

1 0

[Maria-developers] How to change max key length for MyISAM?
by Witold Filipczyk 15 Jan '21

15 Jan '21

Sorry, for private question, but I did not know where to ask. mariadb-10.3.27. I changed in include/myisam.h: #define MI_MAX_KEY_LENGTH 3072 /* Max length in bytes */ but, it did not help, still this ERROR 1071 (42000): Specified key was too long; max key length is 1000 bytes. What more to change to get key length 3072 for MyISAM tables? And what can be negative impact of such a change? Witold Filipczyk

5 8

[Maria-developers] MDEV-13467 review.
by Alexey Botchkov 09 Jan '21

09 Jan '21

Hi, Anel! Sorry for that long delay with this review. Had real difficult weeks of my life :( Now it seems to be over, so getting back to normal. Below is my opinions of the code i see in the branch bb-10.1-anel-MDEV-13467-gis-feature-st_sphere-v2 Hope it's the recent version. + Adapter for functions that takes two or three arguments. +*/ + +class Create_func_arg2_or_3 : public Create_func +{ Do you think we need the separate Create_fund_arg2_or_3 class? If you don't see any other class to inherit from it, i'd recommend to leave the Create_func_distance_sphere alone. No need to have two constructors in the Item_func_sphere_distance too. Just use the list argument as it's done for the Item_func_json_contains (in 10.2). Anyway i belive we should get rid of duplicating code about the 'param1' and 'param2'. They can be popped/checked with no if (arg_count == 2) condition. + double distance= 0.0; + null_value= (args[0]->null_value || args[1]->null_value); + if (null_value) + { + // Returned item must be set to Null + DBUG_ASSERT(maybe_null); don't think it's needed. + return 0; Should be return 0.0 i belive. Or goto handle_errors. Thet goes to all but one 'return' statements in this function. + } > ... > if (!arg1 || !arg2) This last condition should never happen since we have not-null_value. So i'd remove that 'if' section completely. And anyway it should do the 'return' in that branch. + null_value= args[2]->null_value; + if (null_value) + { + return 0; + } seems nicer to me this way if (args[2]->null_value) { null_value= true; goto handle_errors; } + if (!(g1= Geometry::construct(&buffer1, arg1->ptr(), arg1->length())) || + !(g2= Geometry::construct(&buffer2, arg2->ptr(), arg2->length()))) + { + goto handle_errors; we should launch the ER_GIS_INVALID_DATA here. + // Generate error message in case different geometry is used? Yes, i think the explainig message here is good to have. +double Item_func_sphere_distance::spherical_distance(Geometry *g1, + Geometry *g2, + const double sphere_radius) +double Item_func_sphere_distance::spherical_distance_points(Geometry *g1, + Geometry *g2, + const double r) Why these two are member static functions, not simply static? If you plan any 'external' use for them, please add comments about what exacly they do. +double Item_func_sphere_distance::spherical_distance(Geometry *g1, + Geometry *g2, + const double sphere_radius) +{ + double res= 0.0; + switch (g1->get_class_info()->m_type_id) + { + case Geometry::wkb_point: + res= spherical_distance_points(g1, g2, sphere_radius); + break; + + case Geometry::wkb_multipoint: + res= spherical_distance_points(g1, g2, sphere_radius); + break; Why do we need this function? We confirmed already that the type is either wkb_point or wkb_multipolygon, so just can call spherical_distance_points. No? >>>>> +class Item_func_sphere_distance: public Item_real_func +{ + double sphere_radius= 6370986.0; // Default radius equals Earth radius Some compilers don't like this syntax, and we didn't use it yet, so i'd recomment to move this to the constructor or even better to the ::val_real(). +double Gis_point::calculate_haversine(const Geometry *g, + const double sphere_radius) +{ + DBUG_ASSERT(sphere_radius > 0); + double x1r= 0.0; + double x2r= 0.0; + double y1r= 0.0; + double y2r= 0.0; That lot of unnecessary initializations seems excessive. + if (g->get_class_info()->m_type_id == Geometry::wkb_multipoint) + { + const char point_size= 4 + WKB_HEADER_SIZE + POINT_DATA_SIZE+1; //1 for the type + char point_temp[point_size]; + memset(point_temp+4, Geometry::wkb_point, 1); + memcpy(point_temp+5, static_cast<const Gis_multi_point *>(g)->get_data_ptr()+5, 4); + memcpy(point_temp+4+WKB_HEADER_SIZE, g->get_data_ptr()+4+WKB_HEADER_SIZE, + POINT_DATA_SIZE); + point_temp[point_size-1]= '\0'; + Geometry_buffer gbuff; + Geometry *gg= Geometry::construct(&gbuff, point_temp, point_size-1); There is the Gis_multi_point::geometry_n() function, should be used here to get the point. I don't like that lot of static_cast-s. Isn't it nicer to have just the function of double calculate_harvesine_points(Geometry *p1, Geometry *p2, double radius) double calculate_harvesine_multipoint_point(Geometry *mp, Geometry *p, double radius) double calculate_harvesine_multipoint_multipoint(Geometry *mp1, Geometry *mp2, double radius) which would call each other? The ::get_xy_radian() method looks unneeded to me. It's enough to have the to_radian(double d) function. BTW it's possible to do the multipoint/multipoint calculations faster than n^2. Not necessary to do it right now, just not to remember. Best regards. HF

1 0

[Maria-developers] 608b0ee52ef: MDEV-23033: All slaves crash once in ~24 hours and loop restart with signal 11
by sujatha 04 Jan '21

04 Jan '21

revision-id: 608b0ee52ef3e854ce14a407e64e936adbbeba23 (mariadb-10.2.31-648-g608b0ee52ef) parent(s): 25db9ffa8bdab8a2f2af3c7f154343dd6c6d238f author: Sujatha committer: Sujatha timestamp: 2021-01-04 15:06:12 +0530 message: MDEV-23033: All slaves crash once in ~24 hours and loop restart with signal 11 Problem: ======= Upon deleting or updating a row in a parent table (with primary key), if the child table has virtual column and an associated key with ON UPDATE CASCADE/ON DELETE CASCADE, it will result in slave crash. Analysis: ======== Tables which are related through foreign key require prelocking similar to triggers. i.e If a table has triggers/foreign keys we should add all tables and routines used by them to the prelocking set. This prelocking happens during 'open_and_lock_tables' call. Each table being opened is checked for foreign key references. If foreign key reference exists then the child table is opened and it is linked to the table_list. Upon any modification to parent table its corresponding child tables are retried from table_list and they are updated accordingly. This prelocking work fine on master. On slave prelocking works for following cases. - Statement/mixed based replication - In row based replication when trigger execution is enabled through 'slave_run_triggers_for_rbr=YES/LOGGING/ENFORCE' Otherwise it results in an assert/crash, as the parent table will not find the corresponding child table and it will be NULL. Dereferencing NULL pointer leads to slave server exit. Fix: === Introduce a new 'slave_fk_event_map' flag similar to 'trg_event_map'. This flag will ensure that when foreign key is enabled in row based replication all the parent and child tables are prelocked, so that parent is able to locate the child table. Note: This issue is specific to slave, hence only slave needs to be upgraded. --- mysql-test/suite/rpl/r/rpl_row_vcol_crash.result | 380 ++++++++++++++++++++ mysql-test/suite/rpl/t/rpl_row_vcol_crash.test | 425 +++++++++++++++++++++++ sql/log_event.cc | 42 +-- sql/sql_base.cc | 118 ++++--- sql/table.h | 6 +- 5 files changed, 906 insertions(+), 65 deletions(-) diff --git a/mysql-test/suite/rpl/r/rpl_row_vcol_crash.result b/mysql-test/suite/rpl/r/rpl_row_vcol_crash.result new file mode 100644 index 00000000000..f76d8935fa8 --- /dev/null +++ b/mysql-test/suite/rpl/r/rpl_row_vcol_crash.result @@ -0,0 +1,380 @@ +include/master-slave.inc +[connection master] +# +# Test case 1: KEY on a virtual column with ON DELETE CASCADE +# +CREATE TABLE t1 (id INT NOT NULL PRIMARY KEY) ENGINE=InnoDB; +INSERT INTO t1 VALUES (1),(2),(3); +CREATE TABLE t2 (id INT NOT NULL PRIMARY KEY, +t1_id INT NOT NULL, +v_col INT AS (t1_id+1) VIRTUAL, KEY (v_col), KEY (t1_id), +CONSTRAINT a FOREIGN KEY (t1_id) REFERENCES t1 (id) ON DELETE CASCADE +) ENGINE=InnoDB; +INSERT INTO t2 VALUES (90,1,NULL); +INSERT INTO t2 VALUES (91,2,default); +DELETE FROM t1 WHERE id=1; +connection slave; +# +# Verify data consistency on slave +# +include/diff_tables.inc [master:test.t1, slave:test.t1] +include/diff_tables.inc [master:test.t2, slave:test.t2] +connection master; +DROP TABLE t2,t1; +connection slave; +# +# Test case 2: Verify "ON DELETE CASCADE" for parent->child->child scenario +# Parent table: users +# Child tables: matchmaking_groups, matchmaking_group_users +# Parent table: matchmaking_groups +# Child tables: matchmaking_group_users, matchmaking_group_maps +# +# Deleting a row from parent table should be reflected in +# child tables. +# matchmaking_groups->matchmaking_group_users->matchmaking_group_maps +# users->matchmaking_group_users->matchmaking_group_maps +# +connection master; +CREATE TABLE users (id INT UNSIGNED AUTO_INCREMENT PRIMARY KEY, +name VARCHAR(32) NOT NULL DEFAULT '' +) ENGINE=InnoDB DEFAULT CHARSET=utf8; +CREATE TABLE matchmaking_groups ( +id BIGINT UNSIGNED AUTO_INCREMENT PRIMARY KEY, +host_user_id INT UNSIGNED NOT NULL UNIQUE, +v_col INT AS (host_user_id+1) VIRTUAL, KEY (v_col), +CONSTRAINT FOREIGN KEY (host_user_id) REFERENCES users (id) +ON DELETE CASCADE ON UPDATE CASCADE +) ENGINE=InnoDB DEFAULT CHARSET=utf8; +CREATE TABLE matchmaking_group_users ( +matchmaking_group_id BIGINT UNSIGNED NOT NULL, +user_id INT UNSIGNED NOT NULL, +v_col1 int as (user_id+1) virtual, KEY (v_col1), +PRIMARY KEY (matchmaking_group_id,user_id), +UNIQUE KEY user_id (user_id), +CONSTRAINT FOREIGN KEY (matchmaking_group_id) +REFERENCES matchmaking_groups (id) ON DELETE CASCADE ON UPDATE CASCADE, +CONSTRAINT FOREIGN KEY (user_id) +REFERENCES users (id) ON DELETE CASCADE ON UPDATE CASCADE +) ENGINE=InnoDB DEFAULT CHARSET=utf8; +CREATE TABLE matchmaking_group_maps ( +matchmaking_group_id BIGINT UNSIGNED NOT NULL, +map_id TINYINT UNSIGNED NOT NULL, +v_col2 INT AS (map_id+1) VIRTUAL, KEY (v_col2), +PRIMARY KEY (matchmaking_group_id,map_id), +CONSTRAINT FOREIGN KEY (matchmaking_group_id) +REFERENCES matchmaking_groups (id) ON DELETE CASCADE ON UPDATE CASCADE +) ENGINE=InnoDB DEFAULT CHARSET=utf8; +connection slave; +connection master; +INSERT INTO users VALUES (NULL,'foo'),(NULL,'bar'); +INSERT INTO matchmaking_groups VALUES (10,1,default),(11,2,default); +INSERT INTO matchmaking_group_users VALUES (10,1,default),(11,2,default); +INSERT INTO matchmaking_group_maps VALUES (10,55,default),(11,66,default); +DELETE FROM matchmaking_groups WHERE id = 10; +connection slave; +# +# No rows should be returned as ON DELETE CASCASE should have removed +# corresponding rows from child tables. There should not any mismatch +# of 'id' field between parent->child. +# +SELECT * FROM matchmaking_group_users WHERE matchmaking_group_id NOT IN (SELECT id FROM matchmaking_groups); +matchmaking_group_id user_id v_col1 +SELECT * FROM matchmaking_group_maps WHERE matchmaking_group_id NOT IN (SELECT id FROM matchmaking_groups); +matchmaking_group_id map_id v_col2 +# +# Rows with id=11 should be present +# +SELECT * FROM matchmaking_group_users; +matchmaking_group_id user_id v_col1 +11 2 3 +SELECT * FROM matchmaking_group_maps; +matchmaking_group_id map_id v_col2 +11 66 67 +connection master; +DELETE FROM users WHERE id = 2; +connection slave; +# +# No rows should be present in both the child tables +# +SELECT * FROM matchmaking_group_users; +matchmaking_group_id user_id v_col1 +SELECT * FROM matchmaking_group_maps; +matchmaking_group_id map_id v_col2 +connection master; +DROP TABLE matchmaking_group_maps, matchmaking_group_users, matchmaking_groups, users; +connection slave; +# +# Test case 3: KEY on a virtual column with ON UPDATE CASCADE +# +connection master; +CREATE TABLE t1 (a INT NOT NULL PRIMARY KEY, b INT NOT NULL) ENGINE=InnoDB; +INSERT INTO t1 VALUES (1, 80); +CREATE TABLE t2 (a INT KEY, b INT, +v_col int as (b+1) virtual, KEY (v_col), +CONSTRAINT b FOREIGN KEY (b) REFERENCES t1(a) ON UPDATE CASCADE +) ENGINE=InnoDB; +INSERT INTO t2 VALUES (51, 1, default); +connection slave; +connection master; +UPDATE t1 SET a = 50 WHERE a = 1; +# +# Master: Verify that ON UPDATE CASCADE works fine +# old_row: (51, 1, 2) ON UPDATE New_row: (51, 50, 51) +# +SELECT * FROM t2 WHERE b=50; +a b v_col +51 50 51 +connection slave; +# +# Slave: Verify that ON UPDATE CASCADE works fine +# old_row: (51, 1, 2) ON UPDATE New_row: (51, 50, 51) +# +SELECT * FROM t2 WHERE b=50; +a b v_col +51 50 51 +connection master; +DROP TABLE t2, t1; +connection slave; +# +# Test case 4: Define triggers on master, their results should be +# replicated as part of row events and they should be +# applied on slave with the default +# slave_run_triggers_for_rbr=NO +# +connection master; +CREATE TABLE t1 (id INT NOT NULL PRIMARY KEY) ENGINE=InnoDB; +CREATE TABLE t2 (count INT NOT NULL) ENGINE=InnoDB; +CREATE TRIGGER trg AFTER INSERT ON t1 FOR EACH ROW INSERT INTO t2 VALUES (1); +INSERT INTO t1 VALUES (2),(3); +connection slave; +SHOW GLOBAL VARIABLES LIKE 'slave_run_triggers_for_rbr'; +Variable_name Value +slave_run_triggers_for_rbr NO +# +# As two rows are inserted in table 't1', two rows should get inserted +# into table 't2' as part of trigger. +# +include/assert.inc [Table t2 should have two rows.] +connection master; +DROP TABLE t1,t2; +connection slave; +# +# Test case 5: Define triggers + Foreign Keys on master, their results +# should be replicated as part of row events and master +# and slave should be in sync. +# +connection master; +CREATE TABLE t1 (id INT NOT NULL PRIMARY KEY) ENGINE=InnoDB; +CREATE TABLE t2 (t1_id INT NOT NULL, +v_col INT AS (t1_id+1) VIRTUAL, KEY (v_col), KEY (t1_id), +CONSTRAINT a FOREIGN KEY (t1_id) REFERENCES t1 (id) ON DELETE CASCADE +) ENGINE=InnoDB; +CREATE TABLE t3 (count INT NOT NULL) ENGINE=InnoDB; +CREATE TRIGGER trg AFTER INSERT ON t1 FOR EACH ROW INSERT INTO t3 VALUES (1); +INSERT INTO t1 VALUES (2),(3); +INSERT INTO t2 VALUES (2, default), (3, default); +connection slave; +# +# As two rows are inserted in table 't1', two rows should get inserted +# into table 't3' as part of trigger. +# +include/assert.inc [Table t3 should have two rows.] +# +# Verify ON DELETE CASCASE correctness +# +connection master; +DELETE FROM t1 WHERE id=2; +connection slave; +connection master; +include/diff_tables.inc [master:test.t1, slave:test.t1] +include/diff_tables.inc [master:test.t2, slave:test.t2] +include/diff_tables.inc [master:test.t3, slave:test.t3] +DROP TABLE t3,t2,t1; +connection slave; +# +# Test case 6: Triggers are present only on slave and +# 'slave_run_triggers_for_rbr=NO' +# +connection slave; +SET @save_slave_run_triggers_for_rbr= @@GLOBAL.slave_run_triggers_for_rbr; +SET GLOBAL slave_run_triggers_for_rbr= NO;; +SHOW GLOBAL VARIABLES LIKE '%slave_run_triggers_for_rbr%'; +Variable_name Value +slave_run_triggers_for_rbr NO +connection master; +CREATE TABLE t1 (id INT NOT NULL PRIMARY KEY) ENGINE=InnoDB; +CREATE TABLE t2 (t1_id INT NOT NULL, +v_col INT AS (t1_id+1) VIRTUAL, KEY (v_col), +KEY (t1_id), CONSTRAINT a FOREIGN KEY (t1_id) REFERENCES t1 (id) ON DELETE CASCADE +) ENGINE=InnoDB; +CREATE TABLE t3 (count INT NOT NULL) ENGINE=InnoDB; +connection slave; +CREATE TRIGGER trg AFTER INSERT ON t2 FOR EACH ROW INSERT INTO t3 VALUES (1); +connection master; +INSERT INTO t1 VALUES (2),(3); +INSERT INTO t2 VALUES (2, default), (3, default); +connection slave; +# +# Count must be 0 +# +include/assert.inc [Table t3 should have zero rows.] +connection master; +DELETE FROM t1 WHERE id=2; +connection slave; +SET GLOBAL slave_run_triggers_for_rbr= @save_slave_run_triggers_for_rbr; +# +# Verify t1, t2 are consistent on slave. +# +include/diff_tables.inc [master:test.t1, slave:test.t1] +include/diff_tables.inc [master:test.t2, slave:test.t2] +connection master; +DROP TABLE t3,t2,t1; +connection slave; +# +# Test case 7: Triggers are present only on slave and +# 'slave_run_triggers_for_rbr=YES' +# +connection slave; +SET @save_slave_run_triggers_for_rbr= @@GLOBAL.slave_run_triggers_for_rbr; +SET GLOBAL slave_run_triggers_for_rbr= YES;; +SHOW GLOBAL VARIABLES LIKE '%slave_run_triggers_for_rbr%'; +Variable_name Value +slave_run_triggers_for_rbr YES +connection master; +CREATE TABLE t1 (id INT NOT NULL PRIMARY KEY) ENGINE=InnoDB; +CREATE TABLE t2 (t1_id INT NOT NULL, +v_col INT AS (t1_id+1) VIRTUAL, KEY (v_col), +KEY (t1_id), CONSTRAINT a FOREIGN KEY (t1_id) REFERENCES t1 (id) ON DELETE CASCADE +) ENGINE=InnoDB; +CREATE TABLE t3 (count INT NOT NULL) ENGINE=InnoDB; +connection slave; +CREATE TRIGGER trg AFTER INSERT ON t2 FOR EACH ROW INSERT INTO t3 VALUES (1); +connection master; +INSERT INTO t1 VALUES (2),(3); +INSERT INTO t2 VALUES (2, default), (3, default); +connection slave; +# +# Count must be 2 +# +include/assert.inc [Table t3 should have two rows.] +connection master; +DELETE FROM t1 WHERE id=2; +connection slave; +SET GLOBAL slave_run_triggers_for_rbr= @save_slave_run_triggers_for_rbr; +# +# Verify t1, t2 are consistent on slave. +# +include/diff_tables.inc [master:test.t1, slave:test.t1] +include/diff_tables.inc [master:test.t2, slave:test.t2] +connection master; +DROP TABLE t3,t2,t1; +connection slave; +# +# Test case 8: Triggers and Foreign Keys are present only on slave and +# 'slave_run_triggers_for_rbr=NO' +# +connection slave; +SET @save_slave_run_triggers_for_rbr= @@GLOBAL.slave_run_triggers_for_rbr; +SET GLOBAL slave_run_triggers_for_rbr= NO;; +SHOW GLOBAL VARIABLES LIKE '%slave_run_triggers_for_rbr%'; +Variable_name Value +slave_run_triggers_for_rbr NO +connection master; +CREATE TABLE t1 (id INT NOT NULL PRIMARY KEY) ENGINE=InnoDB; +SET sql_log_bin=0; +CREATE TABLE t2 (t1_id INT NOT NULL,v_col INT AS (t1_id+1) VIRTUAL) ENGINE=INNODB; +SET sql_log_bin=1; +CREATE TABLE t3 (count INT NOT NULL) ENGINE=InnoDB; +connection slave; +CREATE TABLE t2 (t1_id INT NOT NULL, +v_col INT AS (t1_id+1) VIRTUAL, KEY (v_col), KEY (t1_id), +CONSTRAINT a FOREIGN KEY (t1_id) REFERENCES t1 (id) ON DELETE CASCADE +) ENGINE=InnoDB; +CREATE TRIGGER trg AFTER INSERT ON t2 FOR EACH ROW INSERT INTO t3 VALUES (1); +connection master; +INSERT INTO t1 VALUES (2),(3); +INSERT INTO t2 VALUES (2, default), (3, default); +connection slave; +# +# Count must be 0 +# +include/assert.inc [Table t3 should have zero rows.] +connection master; +DELETE FROM t1 WHERE id=2; +# t1: Should have one row +SELECT * FROM t1; +id +3 +# t2: Should have two rows +SELECT * FROM t2; +t1_id v_col +2 3 +3 4 +connection slave; +# t1: Should have one row +SELECT * FROM t1; +id +3 +# t2: Should have one row on slave due to ON DELETE CASCASE +SELECT * FROM t2; +t1_id v_col +3 4 +SET GLOBAL slave_run_triggers_for_rbr= @save_slave_run_triggers_for_rbr; +connection master; +DROP TABLE t3,t2,t1; +connection slave; +# +# Test case 9: Triggers are Foreign Keys are present only on slave and +# 'slave_run_triggers_for_rbr=YES' +# +connection slave; +SET @save_slave_run_triggers_for_rbr= @@GLOBAL.slave_run_triggers_for_rbr; +SET GLOBAL slave_run_triggers_for_rbr= YES;; +SHOW GLOBAL VARIABLES LIKE '%slave_run_triggers_for_rbr%'; +Variable_name Value +slave_run_triggers_for_rbr YES +connection master; +CREATE TABLE t1 (id INT NOT NULL PRIMARY KEY) ENGINE=InnoDB; +SET sql_log_bin=0; +CREATE TABLE t2 (t1_id INT NOT NULL,v_col INT AS (t1_id+1) VIRTUAL) ENGINE=INNODB; +SET sql_log_bin=1; +CREATE TABLE t3 (count INT NOT NULL) ENGINE=InnoDB; +connection slave; +CREATE TABLE t2 (t1_id INT NOT NULL, +v_col INT AS (t1_id+1) VIRTUAL, KEY (v_col), KEY (t1_id), +CONSTRAINT a FOREIGN KEY (t1_id) REFERENCES t1 (id) ON DELETE CASCADE +) ENGINE=InnoDB; +CREATE TRIGGER trg AFTER INSERT ON t2 FOR EACH ROW INSERT INTO t3 VALUES (1); +connection master; +INSERT INTO t1 VALUES (2),(3); +INSERT INTO t2 VALUES (2, default), (3, default); +connection slave; +# +# Count must be 2 +# +include/assert.inc [Table t3 should have two rows.] +connection master; +DELETE FROM t1 WHERE id=2; +# t1: Should have one row +SELECT * FROM t1; +id +3 +# t2: Should have two rows +SELECT * FROM t2; +t1_id v_col +2 3 +3 4 +connection slave; +# t1: Should have one row +SELECT * FROM t1; +id +3 +# t2: Should have one row on slave due to ON DELETE CASCASE +SELECT * FROM t2; +t1_id v_col +3 4 +SET GLOBAL slave_run_triggers_for_rbr= @save_slave_run_triggers_for_rbr; +connection master; +DROP TABLE t3,t2,t1; +connection slave; +include/rpl_end.inc diff --git a/mysql-test/suite/rpl/t/rpl_row_vcol_crash.test b/mysql-test/suite/rpl/t/rpl_row_vcol_crash.test new file mode 100644 index 00000000000..84ee14977f3 --- /dev/null +++ b/mysql-test/suite/rpl/t/rpl_row_vcol_crash.test @@ -0,0 +1,425 @@ +# ==== Purpose ==== +# +# Test verifies that, slave doesn't report any assert on UPDATE or DELETE of +# row which tries to update the virtual columns with associated KEYs. +# +# Test scenarios are listed below. +# 1) KEY on a virtual column with ON DELETE CASCADE +# 2) Verify "ON DELETE CASCADE" for parent->child->child scenario +# 3) KEY on a virtual column with ON UPDATE CASCADE +# 4) Define triggers on master, their results should be replicated +# as part of row events and they should be applied on slave with +# the default slave_run_triggers_for_rbr=NO +# 5) Define triggers + Foreign Keys on master, their results should be +# replicated as part of row events and master and slave should be in sync. +# 6) Triggers are present only on slave and 'slave_run_triggers_for_rbr=NO' +# 7) Triggers are present only on slave and 'slave_run_triggers_for_rbr=YES' +# 8) Triggers and Foreign Keys are present only on slave and +# 'slave_run_triggers_for_rbr=NO' +# 9) Triggers are Foreign Keys are present only on slave and +# 'slave_run_triggers_for_rbr=YES' +# +# ==== References ==== +# +# MDEV-23033: All slaves crash once in ~24 hours and loop restart with signal 11 +# + +--source include/have_binlog_format_row.inc +--source include/have_innodb.inc +--source include/master-slave.inc + + +--echo # +--echo # Test case 1: KEY on a virtual column with ON DELETE CASCADE +--echo # +CREATE TABLE t1 (id INT NOT NULL PRIMARY KEY) ENGINE=InnoDB; +INSERT INTO t1 VALUES (1),(2),(3); + +CREATE TABLE t2 (id INT NOT NULL PRIMARY KEY, + t1_id INT NOT NULL, + v_col INT AS (t1_id+1) VIRTUAL, KEY (v_col), KEY (t1_id), + CONSTRAINT a FOREIGN KEY (t1_id) REFERENCES t1 (id) ON DELETE CASCADE +) ENGINE=InnoDB; + +INSERT INTO t2 VALUES (90,1,NULL); +INSERT INTO t2 VALUES (91,2,default); + +# Following query results in an assert on slave +DELETE FROM t1 WHERE id=1; +--sync_slave_with_master + +--echo # +--echo # Verify data consistency on slave +--echo # +--let $diff_tables= master:test.t1, slave:test.t1 +--source include/diff_tables.inc +--let $diff_tables= master:test.t2, slave:test.t2 +--source include/diff_tables.inc + +--connection master +DROP TABLE t2,t1; +--sync_slave_with_master + +--echo # +--echo # Test case 2: Verify "ON DELETE CASCADE" for parent->child->child scenario +--echo # Parent table: users +--echo # Child tables: matchmaking_groups, matchmaking_group_users +--echo # Parent table: matchmaking_groups +--echo # Child tables: matchmaking_group_users, matchmaking_group_maps +--echo # +--echo # Deleting a row from parent table should be reflected in +--echo # child tables. +--echo # matchmaking_groups->matchmaking_group_users->matchmaking_group_maps +--echo # users->matchmaking_group_users->matchmaking_group_maps +--echo # + +--connection master +CREATE TABLE users (id INT UNSIGNED AUTO_INCREMENT PRIMARY KEY, + name VARCHAR(32) NOT NULL DEFAULT '' +) ENGINE=InnoDB DEFAULT CHARSET=utf8; + +CREATE TABLE matchmaking_groups ( + id BIGINT UNSIGNED AUTO_INCREMENT PRIMARY KEY, + host_user_id INT UNSIGNED NOT NULL UNIQUE, + v_col INT AS (host_user_id+1) VIRTUAL, KEY (v_col), + CONSTRAINT FOREIGN KEY (host_user_id) REFERENCES users (id) + ON DELETE CASCADE ON UPDATE CASCADE +) ENGINE=InnoDB DEFAULT CHARSET=utf8; + +CREATE TABLE matchmaking_group_users ( + matchmaking_group_id BIGINT UNSIGNED NOT NULL, + user_id INT UNSIGNED NOT NULL, + v_col1 int as (user_id+1) virtual, KEY (v_col1), + PRIMARY KEY (matchmaking_group_id,user_id), + UNIQUE KEY user_id (user_id), + CONSTRAINT FOREIGN KEY (matchmaking_group_id) + REFERENCES matchmaking_groups (id) ON DELETE CASCADE ON UPDATE CASCADE, + CONSTRAINT FOREIGN KEY (user_id) + REFERENCES users (id) ON DELETE CASCADE ON UPDATE CASCADE +) ENGINE=InnoDB DEFAULT CHARSET=utf8; + +CREATE TABLE matchmaking_group_maps ( + matchmaking_group_id BIGINT UNSIGNED NOT NULL, + map_id TINYINT UNSIGNED NOT NULL, + v_col2 INT AS (map_id+1) VIRTUAL, KEY (v_col2), + PRIMARY KEY (matchmaking_group_id,map_id), + CONSTRAINT FOREIGN KEY (matchmaking_group_id) + REFERENCES matchmaking_groups (id) ON DELETE CASCADE ON UPDATE CASCADE +) ENGINE=InnoDB DEFAULT CHARSET=utf8; +--sync_slave_with_master + +--connection master +INSERT INTO users VALUES (NULL,'foo'),(NULL,'bar'); +INSERT INTO matchmaking_groups VALUES (10,1,default),(11,2,default); +INSERT INTO matchmaking_group_users VALUES (10,1,default),(11,2,default); +INSERT INTO matchmaking_group_maps VALUES (10,55,default),(11,66,default); + +DELETE FROM matchmaking_groups WHERE id = 10; +--sync_slave_with_master + +--echo # +--echo # No rows should be returned as ON DELETE CASCASE should have removed +--echo # corresponding rows from child tables. There should not any mismatch +--echo # of 'id' field between parent->child. +--echo # +SELECT * FROM matchmaking_group_users WHERE matchmaking_group_id NOT IN (SELECT id FROM matchmaking_groups); +SELECT * FROM matchmaking_group_maps WHERE matchmaking_group_id NOT IN (SELECT id FROM matchmaking_groups); + +--echo # +--echo # Rows with id=11 should be present +--echo # +SELECT * FROM matchmaking_group_users; +SELECT * FROM matchmaking_group_maps; + +--connection master +DELETE FROM users WHERE id = 2; +--sync_slave_with_master + +--echo # +--echo # No rows should be present in both the child tables +--echo # +SELECT * FROM matchmaking_group_users; +SELECT * FROM matchmaking_group_maps; + +--connection master +DROP TABLE matchmaking_group_maps, matchmaking_group_users, matchmaking_groups, users; +--sync_slave_with_master + +--echo # +--echo # Test case 3: KEY on a virtual column with ON UPDATE CASCADE +--echo # + +--connection master +CREATE TABLE t1 (a INT NOT NULL PRIMARY KEY, b INT NOT NULL) ENGINE=InnoDB; +INSERT INTO t1 VALUES (1, 80); + +CREATE TABLE t2 (a INT KEY, b INT, + v_col int as (b+1) virtual, KEY (v_col), + CONSTRAINT b FOREIGN KEY (b) REFERENCES t1(a) ON UPDATE CASCADE +) ENGINE=InnoDB; +INSERT INTO t2 VALUES (51, 1, default); +--sync_slave_with_master + +--connection master +UPDATE t1 SET a = 50 WHERE a = 1; + +--echo # +--echo # Master: Verify that ON UPDATE CASCADE works fine +--echo # old_row: (51, 1, 2) ON UPDATE New_row: (51, 50, 51) +--echo # +SELECT * FROM t2 WHERE b=50; +--sync_slave_with_master + +--echo # +--echo # Slave: Verify that ON UPDATE CASCADE works fine +--echo # old_row: (51, 1, 2) ON UPDATE New_row: (51, 50, 51) +--echo # +SELECT * FROM t2 WHERE b=50; + +--connection master +DROP TABLE t2, t1; +--sync_slave_with_master + +--echo # +--echo # Test case 4: Define triggers on master, their results should be +--echo # replicated as part of row events and they should be +--echo # applied on slave with the default +--echo # slave_run_triggers_for_rbr=NO +--echo # + +# In row-based replication, the binary log contains row changes. It will have +# both the changes made by the statement itself, and the changes made by the +# triggers that were invoked by the statement. Slave server(s) do not need to +# run triggers for row changes they are applying. Hence verify that this +# property remains the same and data should be available as if trigger was +# executed. Please note by default slave_run_triggers_for_rbr=NO. + +--connection master +CREATE TABLE t1 (id INT NOT NULL PRIMARY KEY) ENGINE=InnoDB; +CREATE TABLE t2 (count INT NOT NULL) ENGINE=InnoDB; +CREATE TRIGGER trg AFTER INSERT ON t1 FOR EACH ROW INSERT INTO t2 VALUES (1); +INSERT INTO t1 VALUES (2),(3); +--sync_slave_with_master + +SHOW GLOBAL VARIABLES LIKE 'slave_run_triggers_for_rbr'; +--echo # +--echo # As two rows are inserted in table 't1', two rows should get inserted +--echo # into table 't2' as part of trigger. +--echo # +--let $assert_cond= COUNT(*) = 2 FROM t2 +--let $assert_text= Table t2 should have two rows. +--source include/assert.inc + +--connection master +DROP TABLE t1,t2; +--sync_slave_with_master + +--echo # +--echo # Test case 5: Define triggers + Foreign Keys on master, their results +--echo # should be replicated as part of row events and master +--echo # and slave should be in sync. +--echo # +--connection master +CREATE TABLE t1 (id INT NOT NULL PRIMARY KEY) ENGINE=InnoDB; +CREATE TABLE t2 (t1_id INT NOT NULL, + v_col INT AS (t1_id+1) VIRTUAL, KEY (v_col), KEY (t1_id), + CONSTRAINT a FOREIGN KEY (t1_id) REFERENCES t1 (id) ON DELETE CASCADE +) ENGINE=InnoDB; +CREATE TABLE t3 (count INT NOT NULL) ENGINE=InnoDB; +CREATE TRIGGER trg AFTER INSERT ON t1 FOR EACH ROW INSERT INTO t3 VALUES (1); + +INSERT INTO t1 VALUES (2),(3); +INSERT INTO t2 VALUES (2, default), (3, default); +--sync_slave_with_master + +--echo # +--echo # As two rows are inserted in table 't1', two rows should get inserted +--echo # into table 't3' as part of trigger. +--echo # +--let $assert_cond= COUNT(*) = 2 FROM t3 +--let $assert_text= Table t3 should have two rows. +--source include/assert.inc + +--echo # +--echo # Verify ON DELETE CASCASE correctness +--echo # +--connection master +DELETE FROM t1 WHERE id=2; +--sync_slave_with_master + +--connection master +--let $diff_tables= master:test.t1, slave:test.t1 +--source include/diff_tables.inc +--let $diff_tables= master:test.t2, slave:test.t2 +--source include/diff_tables.inc +--let $diff_tables= master:test.t3, slave:test.t3 +--source include/diff_tables.inc + +DROP TABLE t3,t2,t1; +--sync_slave_with_master + +# +# Test case: Triggers only on slave +# +--write_file $MYSQLTEST_VARDIR/tmp/trig_on_slave.inc PROCEDURE + if ($slave_run_triggers_for_rbr == '') { + --die !!!ERROR IN TEST: you must set $slave_run_triggers_for_rbr + } + +--connection slave +SET @save_slave_run_triggers_for_rbr= @@GLOBAL.slave_run_triggers_for_rbr; +--eval SET GLOBAL slave_run_triggers_for_rbr= $slave_run_triggers_for_rbr; +SHOW GLOBAL VARIABLES LIKE '%slave_run_triggers_for_rbr%'; + +--connection master +CREATE TABLE t1 (id INT NOT NULL PRIMARY KEY) ENGINE=InnoDB; +CREATE TABLE t2 (t1_id INT NOT NULL, + v_col INT AS (t1_id+1) VIRTUAL, KEY (v_col), + KEY (t1_id), CONSTRAINT a FOREIGN KEY (t1_id) REFERENCES t1 (id) ON DELETE CASCADE +) ENGINE=InnoDB; +CREATE TABLE t3 (count INT NOT NULL) ENGINE=InnoDB; +--sync_slave_with_master + +CREATE TRIGGER trg AFTER INSERT ON t2 FOR EACH ROW INSERT INTO t3 VALUES (1); + +--connection master +INSERT INTO t1 VALUES (2),(3); +INSERT INTO t2 VALUES (2, default), (3, default); +--sync_slave_with_master + +if ($slave_run_triggers_for_rbr == 'NO') { +--echo # +--echo # Count must be 0 +--echo # +--let $assert_cond= COUNT(*) = 0 FROM t3 +--let $assert_text= Table t3 should have zero rows. +--source include/assert.inc +} +if ($slave_run_triggers_for_rbr == 'YES') { +--echo # +--echo # Count must be 2 +--echo # +--let $assert_cond= COUNT(*) = 2 FROM t3 +--let $assert_text= Table t3 should have two rows. +--source include/assert.inc +} + +--connection master +DELETE FROM t1 WHERE id=2; +--sync_slave_with_master +SET GLOBAL slave_run_triggers_for_rbr= @save_slave_run_triggers_for_rbr; + +--echo # +--echo # Verify t1, t2 are consistent on slave. +--echo # +--let $diff_tables= master:test.t1, slave:test.t1 +--source include/diff_tables.inc +--let $diff_tables= master:test.t2, slave:test.t2 +--source include/diff_tables.inc + +--connection master +DROP TABLE t3,t2,t1; +--sync_slave_with_master +#END OF +PROCEDURE + +--echo # +--echo # Test case 6: Triggers are present only on slave and +--echo # 'slave_run_triggers_for_rbr=NO' +--echo # +--let $slave_run_triggers_for_rbr=NO +--source $MYSQLTEST_VARDIR/tmp/trig_on_slave.inc + +--echo # +--echo # Test case 7: Triggers are present only on slave and +--echo # 'slave_run_triggers_for_rbr=YES' +--echo # +--let $slave_run_triggers_for_rbr=YES +--source $MYSQLTEST_VARDIR/tmp/trig_on_slave.inc +--remove_file $MYSQLTEST_VARDIR/tmp/trig_on_slave.inc + +# +# Test case: Trigger and Foreign Key are present only on slave +# +--write_file $MYSQLTEST_VARDIR/tmp/trig_fk_on_slave.inc PROCEDURE + if ($slave_run_triggers_for_rbr == '') { + --die !!!ERROR IN TEST: you must set $slave_run_triggers_for_rbr + } + +--connection slave +SET @save_slave_run_triggers_for_rbr= @@GLOBAL.slave_run_triggers_for_rbr; +--eval SET GLOBAL slave_run_triggers_for_rbr= $slave_run_triggers_for_rbr; +SHOW GLOBAL VARIABLES LIKE '%slave_run_triggers_for_rbr%'; + +--connection master +CREATE TABLE t1 (id INT NOT NULL PRIMARY KEY) ENGINE=InnoDB; +SET sql_log_bin=0; +CREATE TABLE t2 (t1_id INT NOT NULL,v_col INT AS (t1_id+1) VIRTUAL) ENGINE=INNODB; +SET sql_log_bin=1; +CREATE TABLE t3 (count INT NOT NULL) ENGINE=InnoDB; +--sync_slave_with_master + +# Have foreign key and trigger on slave. +CREATE TABLE t2 (t1_id INT NOT NULL, + v_col INT AS (t1_id+1) VIRTUAL, KEY (v_col), KEY (t1_id), + CONSTRAINT a FOREIGN KEY (t1_id) REFERENCES t1 (id) ON DELETE CASCADE +) ENGINE=InnoDB; +CREATE TRIGGER trg AFTER INSERT ON t2 FOR EACH ROW INSERT INTO t3 VALUES (1); + +--connection master +INSERT INTO t1 VALUES (2),(3); +INSERT INTO t2 VALUES (2, default), (3, default); +--sync_slave_with_master + +if ($slave_run_triggers_for_rbr == 'NO') { +--echo # +--echo # Count must be 0 +--echo # +--let $assert_cond= COUNT(*) = 0 FROM t3 +--let $assert_text= Table t3 should have zero rows. +--source include/assert.inc +} +if ($slave_run_triggers_for_rbr == 'YES') { +--echo # +--echo # Count must be 2 +--echo # +--let $assert_cond= COUNT(*) = 2 FROM t3 +--let $assert_text= Table t3 should have two rows. +--source include/assert.inc +} + +--connection master +DELETE FROM t1 WHERE id=2; +--echo # t1: Should have one row +SELECT * FROM t1; +--echo # t2: Should have two rows +SELECT * FROM t2; +--sync_slave_with_master +--echo # t1: Should have one row +SELECT * FROM t1; +--echo # t2: Should have one row on slave due to ON DELETE CASCASE +SELECT * FROM t2; +SET GLOBAL slave_run_triggers_for_rbr= @save_slave_run_triggers_for_rbr; + +--connection master +DROP TABLE t3,t2,t1; +--sync_slave_with_master +#END OF +PROCEDURE + +--echo # +--echo # Test case 8: Triggers and Foreign Keys are present only on slave and +--echo # 'slave_run_triggers_for_rbr=NO' +--echo # +--let $slave_run_triggers_for_rbr=NO +--source $MYSQLTEST_VARDIR/tmp/trig_fk_on_slave.inc + +--echo # +--echo # Test case 9: Triggers are Foreign Keys are present only on slave and +--echo # 'slave_run_triggers_for_rbr=YES' +--echo # +--let $slave_run_triggers_for_rbr=YES +--source $MYSQLTEST_VARDIR/tmp/trig_fk_on_slave.inc +--remove_file $MYSQLTEST_VARDIR/tmp/trig_fk_on_slave.inc + +--source include/rpl_end.inc diff --git a/sql/log_event.cc b/sql/log_event.cc index c649e1f64fa..c32f31db1f6 100644 --- a/sql/log_event.cc +++ b/sql/log_event.cc @@ -10718,7 +10718,7 @@ int Rows_log_event::do_add_row_data(uchar *row_data, size_t length) There was the same problem with MERGE MYISAM tables and so here we try to go the same way. */ -static void restore_empty_query_table_list(LEX *lex) +inline void restore_empty_query_table_list(LEX *lex) { if (lex->first_not_own_table()) (*lex->first_not_own_table()->prev_global)= NULL; @@ -10733,6 +10733,8 @@ int Rows_log_event::do_apply_event(rpl_group_info *rgi) TABLE* table; DBUG_ENTER("Rows_log_event::do_apply_event(Relay_log_info*)"); int error= 0; + LEX *lex= thd->lex; + uint8 new_trg_event_map= get_trg_event_map(); /* If m_table_id == ~0ULL, then we have a dummy event that does not contain any data. In that case, we just remove all tables in the @@ -10823,27 +10825,29 @@ int Rows_log_event::do_apply_event(rpl_group_info *rgi) DBUG_ASSERT(!debug_sync_set_action(thd, STRING_WITH_LEN(action))); };); - if (slave_run_triggers_for_rbr) - { - LEX *lex= thd->lex; - uint8 new_trg_event_map= get_trg_event_map(); - - /* - Trigger's procedures work with global table list. So we have to add - rgi->tables_to_lock content there to get trigger's in the list. + /* + Trigger's procedures work with global table list. So we have to add + rgi->tables_to_lock content there to get trigger's in the list. - Then restore_empty_query_table_list() restore the list as it was - */ - DBUG_ASSERT(lex->query_tables == NULL); - if ((lex->query_tables= rgi->tables_to_lock)) - rgi->tables_to_lock->prev_global= &lex->query_tables; + Then restore_empty_query_table_list() restore the list as it was + */ + DBUG_ASSERT(lex->query_tables == NULL); + if ((lex->query_tables= rgi->tables_to_lock)) + rgi->tables_to_lock->prev_global= &lex->query_tables; - for (TABLE_LIST *tables= rgi->tables_to_lock; tables; - tables= tables->next_global) + for (TABLE_LIST *tables= rgi->tables_to_lock; tables; + tables= tables->next_global) + { + if (slave_run_triggers_for_rbr) { tables->trg_event_map= new_trg_event_map; lex->query_tables_last= &tables->next_global; } + else if (!WSREP_ON) + { + tables->slave_fk_event_map= new_trg_event_map; + lex->query_tables_last= &tables->next_global; + } } if (open_and_lock_tables(thd, rgi->tables_to_lock, FALSE, 0)) { @@ -11193,8 +11197,7 @@ int Rows_log_event::do_apply_event(rpl_group_info *rgi) } /* remove trigger's tables */ - if (slave_run_triggers_for_rbr) - restore_empty_query_table_list(thd->lex); + restore_empty_query_table_list(thd->lex); #if defined(WITH_WSREP) && defined(HAVE_QUERY_CACHE) if (WSREP(thd) && thd->wsrep_exec_mode == REPL_RECV) @@ -11212,8 +11215,7 @@ int Rows_log_event::do_apply_event(rpl_group_info *rgi) DBUG_RETURN(error); err: - if (slave_run_triggers_for_rbr) - restore_empty_query_table_list(thd->lex); + restore_empty_query_table_list(thd->lex); rgi->slave_close_thread_tables(thd); DBUG_RETURN(error); } diff --git a/sql/sql_base.cc b/sql/sql_base.cc index 8e57ea437b6..b8d18abb50c 100644 --- a/sql/sql_base.cc +++ b/sql/sql_base.cc @@ -4341,6 +4341,70 @@ bool table_already_fk_prelocked(TABLE_LIST *tl, LEX_STRING *db, return false; } +/** + Extend the table_list to include foreign tables for prelocking. + + @param[in] thd Thread context. + @param[in] prelocking_ctx Prelocking context of the statement. + @param[in] table_list Table list element for table. + @param[in] sp Routine body. + @param[out] need_prelocking Set to TRUE if method detects that prelocking + required, not changed otherwise. + + @retval FALSE Success. + @retval TRUE Failure (OOM). +*/ +inline bool +prepare_fk_prelocking_list(THD *thd, Query_tables_list *prelocking_ctx, + TABLE_LIST *table_list, bool *need_prelocking, + uint8 op) +{ + List <FOREIGN_KEY_INFO> fk_list; + List_iterator<FOREIGN_KEY_INFO> fk_list_it(fk_list); + FOREIGN_KEY_INFO *fk; + Query_arena *arena, backup; + + arena= thd->activate_stmt_arena_if_needed(&backup); + + table_list->table->file->get_parent_foreign_key_list(thd, &fk_list); + if (thd->is_error()) + { + if (arena) + thd->restore_active_arena(arena, &backup); + return TRUE; + } + + *need_prelocking= TRUE; + + while ((fk= fk_list_it++)) + { + // FK_OPTION_RESTRICT and FK_OPTION_NO_ACTION only need read access + static bool can_write[]= { true, false, true, true, false, true }; + thr_lock_type lock_type; + + if ((op & (1 << TRG_EVENT_DELETE) && can_write[fk->delete_method]) + || (op & (1 << TRG_EVENT_UPDATE) && can_write[fk->update_method])) + lock_type= TL_WRITE_ALLOW_WRITE; + else + lock_type= TL_READ; + + if (table_already_fk_prelocked(prelocking_ctx->query_tables, + fk->foreign_db, fk->foreign_table, + lock_type)) + continue; + + TABLE_LIST *tl= (TABLE_LIST *) thd->alloc(sizeof(TABLE_LIST)); + tl->init_one_table_for_prelocking(fk->foreign_db->str, fk->foreign_db->length, + fk->foreign_table->str, fk->foreign_table->length, + NULL, lock_type, false, table_list->belong_to_view, + op, &prelocking_ctx->query_tables_last); + } + if (arena) + thd->restore_active_arena(arena, &backup); + + return FALSE; +} + /** Defines how prelocking algorithm for DML statements should handle table list @@ -4381,55 +4445,21 @@ handle_table(THD *thd, Query_tables_list *prelocking_ctx, add_tables_and_routines_for_triggers(thd, prelocking_ctx, table_list)) return TRUE; } - if (table_list->table->file->referenced_by_foreign_key()) { - List <FOREIGN_KEY_INFO> fk_list; - List_iterator<FOREIGN_KEY_INFO> fk_list_it(fk_list); - FOREIGN_KEY_INFO *fk; - Query_arena *arena, backup; - - arena= thd->activate_stmt_arena_if_needed(&backup); - - table_list->table->file->get_parent_foreign_key_list(thd, &fk_list); - if (thd->is_error()) - { - if (arena) - thd->restore_active_arena(arena, &backup); - return TRUE; - } - - *need_prelocking= TRUE; - - while ((fk= fk_list_it++)) - { - // FK_OPTION_RESTRICT and FK_OPTION_NO_ACTION only need read access - static bool can_write[]= { true, false, true, true, false, true }; - uint8 op= table_list->trg_event_map; - thr_lock_type lock_type; - - if ((op & (1 << TRG_EVENT_DELETE) && can_write[fk->delete_method]) - || (op & (1 << TRG_EVENT_UPDATE) && can_write[fk->update_method])) - lock_type= TL_WRITE_ALLOW_WRITE; - else - lock_type= TL_READ; - - if (table_already_fk_prelocked(prelocking_ctx->query_tables, - fk->foreign_db, fk->foreign_table, - lock_type)) - continue; - - TABLE_LIST *tl= (TABLE_LIST *) thd->alloc(sizeof(TABLE_LIST)); - tl->init_one_table_for_prelocking(fk->foreign_db->str, fk->foreign_db->length, - fk->foreign_table->str, fk->foreign_table->length, - NULL, lock_type, false, table_list->belong_to_view, - op, &prelocking_ctx->query_tables_last); - } - if (arena) - thd->restore_active_arena(arena, &backup); + return (prepare_fk_prelocking_list(thd, prelocking_ctx, table_list, + need_prelocking, + table_list->trg_event_map)); } } + else if (table_list->slave_fk_event_map && + table_list->table->file->referenced_by_foreign_key()) + { + return (prepare_fk_prelocking_list(thd, prelocking_ctx, + table_list, need_prelocking, + table_list->slave_fk_event_map)); + } return FALSE; } diff --git a/sql/table.h b/sql/table.h index 9a864f7ce9f..57706655d9b 100644 --- a/sql/table.h +++ b/sql/table.h @@ -2277,8 +2277,12 @@ struct TABLE_LIST Indicates what triggers we need to pre-load for this TABLE_LIST when opening an associated TABLE. This is filled after the parsed tree is created. + + slave_fk_event_map is filled on the slave side with bitmaps value + representing row-based event operation to help find and prelock + possible FK constrain-related child tables. */ - uint8 trg_event_map; + uint8 trg_event_map, slave_fk_event_map; /* TRUE <=> this table is a const one and was optimized away. */ bool optimized_away;

2 2

[Maria-developers] 3f48bc08b92: MDEV-23033: All slaves crash once in ~24 hours and loop restart with signal 11
by sujatha 31 Dec '20

31 Dec '20

revision-id: 3f48bc08b92b2734c2af7408c246b96338675205 (mariadb-10.2.31-646-g3f48bc08b92) parent(s): 78292047a4747ccd9210ba36a185a1dbe825de89 author: Sujatha committer: Sujatha timestamp: 2020-12-31 15:43:53 +0530 message: MDEV-23033: All slaves crash once in ~24 hours and loop restart with signal 11 Problem: ======= Upon deleting or updating a row in a parent table (with primary key), if the child table has virtual column and an associated key with ON UPDATE CASCADE/ON DELETE CASCADE, it will result in slave crash. Analysis: ======== Tables which are related through foreign key require prelocking similar to triggers. i.e If a table has triggers/foreign keys we should add all tables and routines used by them to the prelocking set. This prelocking happens during 'open_and_lock_tables' call. Each table being opened is checked for foreign key references. If foreign key reference exists then the child table is opened and it is linked to the table_list. Upon any modification to parent table its corresponding child tables are retried from table_list and they are updated accordingly. This prelocking work fine on master. On slave prelocking works for following cases. - Statement/mixed based replication - In row based replication when trigger execution is enabled through 'slave_run_triggers_for_rbr=YES/LOGGING/ENFORCE' Otherwise it results in an assert/crash, as the parent table will not find the corresponding child table and it will be NULL. Dereferencing NULL pointer leads to slave server exit. Fix: === Introduce a new 'slave_fk_event_map' flag similar to 'trg_event_map'. This flag will ensure that when foreign key is enabled in row based replication all the parent and child tables are prelocked, so that parent is able to locate the child table. --- mysql-test/suite/rpl/r/rpl_row_vcol_crash.result | 381 +++++++++++++++++++++ mysql-test/suite/rpl/t/rpl_row_vcol_crash.test | 410 +++++++++++++++++++++++ sql/log_event.cc | 44 ++- sql/sql_base.cc | 6 +- sql/table.h | 6 +- 5 files changed, 821 insertions(+), 26 deletions(-) diff --git a/mysql-test/suite/rpl/r/rpl_row_vcol_crash.result b/mysql-test/suite/rpl/r/rpl_row_vcol_crash.result new file mode 100644 index 00000000000..e2a5d9b65e5 --- /dev/null +++ b/mysql-test/suite/rpl/r/rpl_row_vcol_crash.result @@ -0,0 +1,381 @@ +include/master-slave.inc +[connection master] +# +# Test case: KEY on a virtual column with ON DELETE CASCADE +# +CREATE TABLE t1 (id INT NOT NULL PRIMARY KEY) ENGINE=InnoDB; +INSERT INTO t1 VALUES (1),(2),(3); +CREATE TABLE t2 (id INT NOT NULL PRIMARY KEY, +t1_id INT NOT NULL, +v_col INT AS (t1_id+1) VIRTUAL, KEY (v_col), KEY (t1_id), +CONSTRAINT a FOREIGN KEY (t1_id) REFERENCES t1 (id) ON DELETE CASCADE +) ENGINE=InnoDB; +INSERT INTO t2 VALUES (90,1,NULL); +INSERT INTO t2 VALUES (91,2,default); +DELETE FROM t1 WHERE id=1; +connection slave; +# +# Verify data consistency on slave +# +include/diff_tables.inc [master:test.t1, slave:test.t1] +include/diff_tables.inc [master:test.t2, slave:test.t2] +connection master; +DROP TABLE t2,t1; +connection slave; +# +# Test case: Verify cascading on delete with multiple levels of child +# tables works fine. +# Parent table: users +# Child tables: matchmaking_groups, matchmaking_group_users +# Parent table: matchmaking_groups +# Child tables: matchmaking_group_users, matchmaking_group_maps +# +# Verify ON DELETE CASCASE works fine when a row is deleted +# from parent table users +# matchmaking_groups->matchmaking_group_users->matchmaking_group_maps +# users->matchmaking_group_users->matchmaking_group_maps +# +connection master; +CREATE TABLE users (id INT UNSIGNED AUTO_INCREMENT PRIMARY KEY, +name VARCHAR(32) NOT NULL DEFAULT '' +) ENGINE=InnoDB DEFAULT CHARSET=utf8; +CREATE TABLE matchmaking_groups ( +id BIGINT UNSIGNED AUTO_INCREMENT PRIMARY KEY, +host_user_id INT UNSIGNED NOT NULL UNIQUE, +v_col INT AS (host_user_id+1) VIRTUAL, KEY (v_col), +CONSTRAINT FOREIGN KEY (host_user_id) REFERENCES users (id) +ON DELETE CASCADE ON UPDATE CASCADE +) ENGINE=InnoDB DEFAULT CHARSET=utf8; +CREATE TABLE matchmaking_group_users ( +matchmaking_group_id BIGINT UNSIGNED NOT NULL, +user_id INT UNSIGNED NOT NULL, +v_col1 int as (user_id+1) virtual, KEY (v_col1), +PRIMARY KEY (matchmaking_group_id,user_id), +UNIQUE KEY user_id (user_id), +CONSTRAINT FOREIGN KEY (matchmaking_group_id) +REFERENCES matchmaking_groups (id) ON DELETE CASCADE ON UPDATE CASCADE, +CONSTRAINT FOREIGN KEY (user_id) +REFERENCES users (id) ON DELETE CASCADE ON UPDATE CASCADE +) ENGINE=InnoDB DEFAULT CHARSET=utf8; +CREATE TABLE matchmaking_group_maps ( +matchmaking_group_id BIGINT UNSIGNED NOT NULL, +map_id TINYINT UNSIGNED NOT NULL, +v_col2 INT AS (map_id+1) VIRTUAL, KEY (v_col2), +PRIMARY KEY (matchmaking_group_id,map_id), +CONSTRAINT FOREIGN KEY (matchmaking_group_id) +REFERENCES matchmaking_groups (id) ON DELETE CASCADE ON UPDATE CASCADE +) ENGINE=InnoDB DEFAULT CHARSET=utf8; +connection slave; +connection master; +INSERT INTO users VALUES (NULL,'foo'),(NULL,'bar'); +INSERT INTO matchmaking_groups VALUES (10,1,default),(11,2,default); +INSERT INTO matchmaking_group_users VALUES (10,1,default),(11,2,default); +INSERT INTO matchmaking_group_maps VALUES (10,55,default),(11,66,default); +DELETE FROM matchmaking_groups WHERE id = 10; +connection slave; +# +# No rows should be returned as ON DELETE CASCASE should have removed +# corresponding rows from child tables. There should not any mismatch +# of 'id' field between parent->child. +# +SELECT * FROM matchmaking_group_users WHERE matchmaking_group_id NOT IN (SELECT id FROM matchmaking_groups); +matchmaking_group_id user_id v_col1 +SELECT * FROM matchmaking_group_maps WHERE matchmaking_group_id NOT IN (SELECT id FROM matchmaking_groups); +matchmaking_group_id map_id v_col2 +# +# Rows with id=11 should be present +# +SELECT * FROM matchmaking_group_users; +matchmaking_group_id user_id v_col1 +11 2 3 +SELECT * FROM matchmaking_group_maps; +matchmaking_group_id map_id v_col2 +11 66 67 +connection master; +DELETE FROM users WHERE id = 2; +connection slave; +# +# No rows should be present in both the child tables +# +SELECT * FROM matchmaking_group_users; +matchmaking_group_id user_id v_col1 +SELECT * FROM matchmaking_group_maps; +matchmaking_group_id map_id v_col2 +connection master; +DROP TABLE matchmaking_group_maps, matchmaking_group_users, matchmaking_groups, users; +connection slave; +# +# Test case: ON UPDATE CASCADE scenario +# +connection master; +CREATE TABLE t1 (a INT NOT NULL PRIMARY KEY, b INT NOT NULL) ENGINE=InnoDB; +INSERT INTO t1 VALUES (1, 80); +CREATE TABLE t2 (a INT KEY, b INT, +v_col int as (b+1) virtual, KEY (v_col), +CONSTRAINT b FOREIGN KEY (b) REFERENCES t1(a) ON UPDATE CASCADE +) ENGINE=InnoDB; +INSERT INTO t2 VALUES (51, 1, default); +connection slave; +connection master; +UPDATE t1 SET a = 50 WHERE a = 1; +# +# Master: Verify that ON UPDATE CASCADE works fine +# old_row: (51, 1, 2) ON UPDATE New_row: (51, 50, 51) +# +SELECT * FROM t2 WHERE b=50; +a b v_col +51 50 51 +connection slave; +# +# Slave: Verify that ON UPDATE CASCADE works fine +# old_row: (51, 1, 2) ON UPDATE New_row: (51, 50, 51) +# +SELECT * FROM t2 WHERE b=50; +a b v_col +51 50 51 +connection master; +DROP TABLE t2, t1; +connection slave; +# +# Test case: When triggers are defined on master they should be +# replicated as part of row events and they should be +# applied on slave with the default +# slave_run_tiggers_for_rbr=NO +# +connection master; +CREATE TABLE t1 (id INT NOT NULL PRIMARY KEY) ENGINE=InnoDB; +CREATE TABLE t2 (count INT NOT NULL) ENGINE=InnoDB; +CREATE TRIGGER trg AFTER INSERT ON t1 FOR EACH ROW INSERT INTO t2 VALUES (1); +INSERT INTO t1 VALUES (2),(3); +connection slave; +SHOW GLOBAL VARIABLES LIKE 'slave_run_triggers_for_rbr'; +Variable_name Value +slave_run_triggers_for_rbr NO +# +# As two rows are inserted in table 't1', two rows should get inserted +# into table 't2' as part of trigger. +# +include/assert.inc [Table t2 should have two rows.] +connection master; +DROP TABLE t1,t2; +connection slave; +# +# Test case: On master create triggers and tables with foreign key +# relation. Upon replication to slave verify their results +# are fine. Master and Slave should be in sync. +# +connection master; +CREATE TABLE t1 (id INT NOT NULL PRIMARY KEY) ENGINE=InnoDB; +CREATE TABLE t2 (t1_id INT NOT NULL, +v_col INT AS (t1_id+1) VIRTUAL, KEY (v_col), KEY (t1_id), +CONSTRAINT a FOREIGN KEY (t1_id) REFERENCES t1 (id) ON DELETE CASCADE +) ENGINE=InnoDB; +CREATE TABLE t3 (count INT NOT NULL) ENGINE=InnoDB; +CREATE TRIGGER trg AFTER INSERT ON t1 FOR EACH ROW INSERT INTO t3 VALUES (1); +INSERT INTO t1 VALUES (2),(3); +INSERT INTO t2 VALUES (2, default), (3, default); +connection slave; +# +# As two rows are inserted in table 't1', two rows should get inserted +# into table 't3' as part of trigger. +# +include/assert.inc [Table t3 should have two rows.] +# +# Verify ON DELETE CASCASE correctness +# +connection master; +DELETE FROM t1 WHERE id=2; +connection slave; +connection master; +include/diff_tables.inc [master:test.t1, slave:test.t1] +include/diff_tables.inc [master:test.t2, slave:test.t2] +include/diff_tables.inc [master:test.t3, slave:test.t3] +DROP TABLE t3,t2,t1; +connection slave; +# +# Test case: Triggers are present only on slave and +# 'slave_run_triggers_for_rbr=NO' +# +connection slave; +SET @save_slave_run_triggers_for_rbr= @@GLOBAL.slave_run_triggers_for_rbr; +SET GLOBAL slave_run_triggers_for_rbr= NO;; +SHOW GLOBAL VARIABLES LIKE '%slave_run_triggers_for_rbr%'; +Variable_name Value +slave_run_triggers_for_rbr NO +connection master; +CREATE TABLE t1 (id INT NOT NULL PRIMARY KEY) ENGINE=InnoDB; +CREATE TABLE t2 (t1_id INT NOT NULL, +v_col INT AS (t1_id+1) VIRTUAL, KEY (v_col), +KEY (t1_id), CONSTRAINT a FOREIGN KEY (t1_id) REFERENCES t1 (id) ON DELETE CASCADE +) ENGINE=InnoDB; +CREATE TABLE t3 (count INT NOT NULL) ENGINE=InnoDB; +connection slave; +CREATE TRIGGER trg AFTER INSERT ON t2 FOR EACH ROW INSERT INTO t3 VALUES (1); +connection master; +INSERT INTO t1 VALUES (2),(3); +INSERT INTO t2 VALUES (2, default), (3, default); +connection slave; +# +# Count must be 0 +# +include/assert.inc [Table t3 should have zero rows.] +connection master; +DELETE FROM t1 WHERE id=2; +connection slave; +SET GLOBAL slave_run_triggers_for_rbr= @save_slave_run_triggers_for_rbr; +# +# Verify t1,t2 are consistent on slave. +# +include/diff_tables.inc [master:test.t1, slave:test.t1] +include/diff_tables.inc [master:test.t2, slave:test.t2] +connection master; +DROP TABLE t3,t2,t1; +connection slave; +# +# Test case: Triggers are present only on slave and +# 'slave_run_triggers_for_rbr=YES' +# +connection slave; +SET @save_slave_run_triggers_for_rbr= @@GLOBAL.slave_run_triggers_for_rbr; +SET GLOBAL slave_run_triggers_for_rbr= YES;; +SHOW GLOBAL VARIABLES LIKE '%slave_run_triggers_for_rbr%'; +Variable_name Value +slave_run_triggers_for_rbr YES +connection master; +CREATE TABLE t1 (id INT NOT NULL PRIMARY KEY) ENGINE=InnoDB; +CREATE TABLE t2 (t1_id INT NOT NULL, +v_col INT AS (t1_id+1) VIRTUAL, KEY (v_col), +KEY (t1_id), CONSTRAINT a FOREIGN KEY (t1_id) REFERENCES t1 (id) ON DELETE CASCADE +) ENGINE=InnoDB; +CREATE TABLE t3 (count INT NOT NULL) ENGINE=InnoDB; +connection slave; +CREATE TRIGGER trg AFTER INSERT ON t2 FOR EACH ROW INSERT INTO t3 VALUES (1); +connection master; +INSERT INTO t1 VALUES (2),(3); +INSERT INTO t2 VALUES (2, default), (3, default); +connection slave; +# +# Count must be 2 +# +include/assert.inc [Table t3 should have two rows.] +connection master; +DELETE FROM t1 WHERE id=2; +connection slave; +SET GLOBAL slave_run_triggers_for_rbr= @save_slave_run_triggers_for_rbr; +# +# Verify t1,t2 are consistent on slave. +# +include/diff_tables.inc [master:test.t1, slave:test.t1] +include/diff_tables.inc [master:test.t2, slave:test.t2] +connection master; +DROP TABLE t3,t2,t1; +connection slave; +# +# Test case: Triggers and Foreign Keys are present only on slave and +# 'slave_run_triggers_for_rbr=NO' +# +connection slave; +SET @save_slave_run_triggers_for_rbr= @@GLOBAL.slave_run_triggers_for_rbr; +SET GLOBAL slave_run_triggers_for_rbr= NO;; +SHOW GLOBAL VARIABLES LIKE '%slave_run_triggers_for_rbr%'; +Variable_name Value +slave_run_triggers_for_rbr NO +connection master; +CREATE TABLE t1 (id INT NOT NULL PRIMARY KEY) ENGINE=InnoDB; +SET sql_log_bin=0; +CREATE TABLE t2 (t1_id INT NOT NULL,v_col INT AS (t1_id+1) VIRTUAL) ENGINE=INNODB; +SET sql_log_bin=1; +CREATE TABLE t3 (count INT NOT NULL) ENGINE=InnoDB; +connection slave; +CREATE TABLE t2 (t1_id INT NOT NULL, +v_col INT AS (t1_id+1) VIRTUAL, KEY (v_col), KEY (t1_id), +CONSTRAINT a FOREIGN KEY (t1_id) REFERENCES t1 (id) ON DELETE CASCADE +) ENGINE=InnoDB; +CREATE TRIGGER trg AFTER INSERT ON t2 FOR EACH ROW INSERT INTO t3 VALUES (1); +connection master; +INSERT INTO t1 VALUES (2),(3); +INSERT INTO t2 VALUES (2, default), (3, default); +connection slave; +# +# Count must be 0 +# +include/assert.inc [Table t3 should have zero rows.] +connection master; +DELETE FROM t1 WHERE id=2; +# t1: Should have one row +SELECT * FROM t1; +id +3 +# t2: Should have two rows +SELECT * FROM t2; +t1_id v_col +2 3 +3 4 +connection slave; +# t1: Should have one row +SELECT * FROM t1; +id +3 +# t2: Should have one row on slave due to ON DELETE CASCASE +SELECT * FROM t2; +t1_id v_col +3 4 +SET GLOBAL slave_run_triggers_for_rbr= @save_slave_run_triggers_for_rbr; +connection master; +DROP TABLE t3,t2,t1; +connection slave; +# +# Test case: Triggers are Foreign Keys are present only on slave and +# 'slave_run_triggers_for_rbr=YES' +# +connection slave; +SET @save_slave_run_triggers_for_rbr= @@GLOBAL.slave_run_triggers_for_rbr; +SET GLOBAL slave_run_triggers_for_rbr= YES;; +SHOW GLOBAL VARIABLES LIKE '%slave_run_triggers_for_rbr%'; +Variable_name Value +slave_run_triggers_for_rbr YES +connection master; +CREATE TABLE t1 (id INT NOT NULL PRIMARY KEY) ENGINE=InnoDB; +SET sql_log_bin=0; +CREATE TABLE t2 (t1_id INT NOT NULL,v_col INT AS (t1_id+1) VIRTUAL) ENGINE=INNODB; +SET sql_log_bin=1; +CREATE TABLE t3 (count INT NOT NULL) ENGINE=InnoDB; +connection slave; +CREATE TABLE t2 (t1_id INT NOT NULL, +v_col INT AS (t1_id+1) VIRTUAL, KEY (v_col), KEY (t1_id), +CONSTRAINT a FOREIGN KEY (t1_id) REFERENCES t1 (id) ON DELETE CASCADE +) ENGINE=InnoDB; +CREATE TRIGGER trg AFTER INSERT ON t2 FOR EACH ROW INSERT INTO t3 VALUES (1); +connection master; +INSERT INTO t1 VALUES (2),(3); +INSERT INTO t2 VALUES (2, default), (3, default); +connection slave; +# +# Count must be 2 +# +include/assert.inc [Table t3 should have two rows.] +connection master; +DELETE FROM t1 WHERE id=2; +# t1: Should have one row +SELECT * FROM t1; +id +3 +# t2: Should have two rows +SELECT * FROM t2; +t1_id v_col +2 3 +3 4 +connection slave; +# t1: Should have one row +SELECT * FROM t1; +id +3 +# t2: Should have one row on slave due to ON DELETE CASCASE +SELECT * FROM t2; +t1_id v_col +3 4 +SET GLOBAL slave_run_triggers_for_rbr= @save_slave_run_triggers_for_rbr; +connection master; +DROP TABLE t3,t2,t1; +connection slave; +include/rpl_end.inc diff --git a/mysql-test/suite/rpl/t/rpl_row_vcol_crash.test b/mysql-test/suite/rpl/t/rpl_row_vcol_crash.test new file mode 100644 index 00000000000..2ed7c219aeb --- /dev/null +++ b/mysql-test/suite/rpl/t/rpl_row_vcol_crash.test @@ -0,0 +1,410 @@ +# ==== Purpose ==== +# +# Test verifies that, slave doesn't report any assert on UPDATE or DELETE of +# row which tries to update the virtual columns with associated KEYs. +# +# ==== References ==== +# +# MDEV-23033: All slaves crash once in ~24 hours and loop restart with signal 11 +# + +--source include/have_binlog_format_row.inc +--source include/have_innodb.inc +--source include/master-slave.inc + + +--echo # +--echo # Test case: KEY on a virtual column with ON DELETE CASCADE +--echo # +CREATE TABLE t1 (id INT NOT NULL PRIMARY KEY) ENGINE=InnoDB; +INSERT INTO t1 VALUES (1),(2),(3); + +CREATE TABLE t2 (id INT NOT NULL PRIMARY KEY, + t1_id INT NOT NULL, + v_col INT AS (t1_id+1) VIRTUAL, KEY (v_col), KEY (t1_id), + CONSTRAINT a FOREIGN KEY (t1_id) REFERENCES t1 (id) ON DELETE CASCADE +) ENGINE=InnoDB; + +INSERT INTO t2 VALUES (90,1,NULL); +INSERT INTO t2 VALUES (91,2,default); + +# Following query results in an assert on slave +DELETE FROM t1 WHERE id=1; +--sync_slave_with_master + +--echo # +--echo # Verify data consistency on slave +--echo # +--let $diff_tables= master:test.t1, slave:test.t1 +--source include/diff_tables.inc +--let $diff_tables= master:test.t2, slave:test.t2 +--source include/diff_tables.inc + +--connection master +DROP TABLE t2,t1; +--sync_slave_with_master + +--echo # +--echo # Test case: Verify cascading on delete with multiple levels of child +--echo # tables works fine. +--echo # Parent table: users +--echo # Child tables: matchmaking_groups, matchmaking_group_users +--echo # Parent table: matchmaking_groups +--echo # Child tables: matchmaking_group_users, matchmaking_group_maps +--echo # +--echo # Verify ON DELETE CASCASE works fine when a row is deleted +--echo # from parent table users +--echo # matchmaking_groups->matchmaking_group_users->matchmaking_group_maps +--echo # users->matchmaking_group_users->matchmaking_group_maps +--echo # + +--connection master +CREATE TABLE users (id INT UNSIGNED AUTO_INCREMENT PRIMARY KEY, + name VARCHAR(32) NOT NULL DEFAULT '' +) ENGINE=InnoDB DEFAULT CHARSET=utf8; + +CREATE TABLE matchmaking_groups ( + id BIGINT UNSIGNED AUTO_INCREMENT PRIMARY KEY, + host_user_id INT UNSIGNED NOT NULL UNIQUE, + v_col INT AS (host_user_id+1) VIRTUAL, KEY (v_col), + CONSTRAINT FOREIGN KEY (host_user_id) REFERENCES users (id) + ON DELETE CASCADE ON UPDATE CASCADE +) ENGINE=InnoDB DEFAULT CHARSET=utf8; + +CREATE TABLE matchmaking_group_users ( + matchmaking_group_id BIGINT UNSIGNED NOT NULL, + user_id INT UNSIGNED NOT NULL, + v_col1 int as (user_id+1) virtual, KEY (v_col1), + PRIMARY KEY (matchmaking_group_id,user_id), + UNIQUE KEY user_id (user_id), + CONSTRAINT FOREIGN KEY (matchmaking_group_id) + REFERENCES matchmaking_groups (id) ON DELETE CASCADE ON UPDATE CASCADE, + CONSTRAINT FOREIGN KEY (user_id) + REFERENCES users (id) ON DELETE CASCADE ON UPDATE CASCADE +) ENGINE=InnoDB DEFAULT CHARSET=utf8; + +CREATE TABLE matchmaking_group_maps ( + matchmaking_group_id BIGINT UNSIGNED NOT NULL, + map_id TINYINT UNSIGNED NOT NULL, + v_col2 INT AS (map_id+1) VIRTUAL, KEY (v_col2), + PRIMARY KEY (matchmaking_group_id,map_id), + CONSTRAINT FOREIGN KEY (matchmaking_group_id) + REFERENCES matchmaking_groups (id) ON DELETE CASCADE ON UPDATE CASCADE +) ENGINE=InnoDB DEFAULT CHARSET=utf8; +--sync_slave_with_master + +--connection master +INSERT INTO users VALUES (NULL,'foo'),(NULL,'bar'); +INSERT INTO matchmaking_groups VALUES (10,1,default),(11,2,default); +INSERT INTO matchmaking_group_users VALUES (10,1,default),(11,2,default); +INSERT INTO matchmaking_group_maps VALUES (10,55,default),(11,66,default); + +DELETE FROM matchmaking_groups WHERE id = 10; +--sync_slave_with_master + +--echo # +--echo # No rows should be returned as ON DELETE CASCASE should have removed +--echo # corresponding rows from child tables. There should not any mismatch +--echo # of 'id' field between parent->child. +--echo # +SELECT * FROM matchmaking_group_users WHERE matchmaking_group_id NOT IN (SELECT id FROM matchmaking_groups); +SELECT * FROM matchmaking_group_maps WHERE matchmaking_group_id NOT IN (SELECT id FROM matchmaking_groups); + +--echo # +--echo # Rows with id=11 should be present +--echo # +SELECT * FROM matchmaking_group_users; +SELECT * FROM matchmaking_group_maps; + +--connection master +DELETE FROM users WHERE id = 2; +--sync_slave_with_master + +--echo # +--echo # No rows should be present in both the child tables +--echo # +SELECT * FROM matchmaking_group_users; +SELECT * FROM matchmaking_group_maps; + +--connection master +DROP TABLE matchmaking_group_maps, matchmaking_group_users, matchmaking_groups, users; +--sync_slave_with_master + +--echo # +--echo # Test case: ON UPDATE CASCADE scenario +--echo # + +--connection master +CREATE TABLE t1 (a INT NOT NULL PRIMARY KEY, b INT NOT NULL) ENGINE=InnoDB; +INSERT INTO t1 VALUES (1, 80); + +CREATE TABLE t2 (a INT KEY, b INT, + v_col int as (b+1) virtual, KEY (v_col), + CONSTRAINT b FOREIGN KEY (b) REFERENCES t1(a) ON UPDATE CASCADE +) ENGINE=InnoDB; +INSERT INTO t2 VALUES (51, 1, default); +--sync_slave_with_master + +--connection master +UPDATE t1 SET a = 50 WHERE a = 1; + +--echo # +--echo # Master: Verify that ON UPDATE CASCADE works fine +--echo # old_row: (51, 1, 2) ON UPDATE New_row: (51, 50, 51) +--echo # +SELECT * FROM t2 WHERE b=50; +--sync_slave_with_master + +--echo # +--echo # Slave: Verify that ON UPDATE CASCADE works fine +--echo # old_row: (51, 1, 2) ON UPDATE New_row: (51, 50, 51) +--echo # +SELECT * FROM t2 WHERE b=50; + +--connection master +DROP TABLE t2, t1; +--sync_slave_with_master + +--echo # +--echo # Test case: When triggers are defined on master they should be +--echo # replicated as part of row events and they should be +--echo # applied on slave with the default +--echo # slave_run_tiggers_for_rbr=NO +--echo # + +# In row-based replication, the binary log contains row changes. It will have +# both the changes made by the statement itself, and the changes made by the +# triggers that were invoked by the statement. Slave server(s) do not need to +# run triggers for row changes they are applying. Hence verify that this +# property remains the same and data should be available as if trigger was +# executed. Please note by default slave_run_tiggers_for_rbr=NO. + +--connection master +CREATE TABLE t1 (id INT NOT NULL PRIMARY KEY) ENGINE=InnoDB; +CREATE TABLE t2 (count INT NOT NULL) ENGINE=InnoDB; +CREATE TRIGGER trg AFTER INSERT ON t1 FOR EACH ROW INSERT INTO t2 VALUES (1); +INSERT INTO t1 VALUES (2),(3); +--sync_slave_with_master + +SHOW GLOBAL VARIABLES LIKE 'slave_run_triggers_for_rbr'; +--echo # +--echo # As two rows are inserted in table 't1', two rows should get inserted +--echo # into table 't2' as part of trigger. +--echo # +--let $assert_cond= COUNT(*) = 2 FROM t2 +--let $assert_text= Table t2 should have two rows. +--source include/assert.inc + +--connection master +DROP TABLE t1,t2; +--sync_slave_with_master + +--echo # +--echo # Test case: On master create triggers and tables with foreign key +--echo # relation. Upon replication to slave verify their results +--echo # are fine. Master and Slave should be in sync. +--echo # +--connection master +CREATE TABLE t1 (id INT NOT NULL PRIMARY KEY) ENGINE=InnoDB; +CREATE TABLE t2 (t1_id INT NOT NULL, + v_col INT AS (t1_id+1) VIRTUAL, KEY (v_col), KEY (t1_id), + CONSTRAINT a FOREIGN KEY (t1_id) REFERENCES t1 (id) ON DELETE CASCADE +) ENGINE=InnoDB; +CREATE TABLE t3 (count INT NOT NULL) ENGINE=InnoDB; +CREATE TRIGGER trg AFTER INSERT ON t1 FOR EACH ROW INSERT INTO t3 VALUES (1); + +INSERT INTO t1 VALUES (2),(3); +INSERT INTO t2 VALUES (2, default), (3, default); +--sync_slave_with_master + +--echo # +--echo # As two rows are inserted in table 't1', two rows should get inserted +--echo # into table 't3' as part of trigger. +--echo # +--let $assert_cond= COUNT(*) = 2 FROM t3 +--let $assert_text= Table t3 should have two rows. +--source include/assert.inc + +--echo # +--echo # Verify ON DELETE CASCASE correctness +--echo # +--connection master +DELETE FROM t1 WHERE id=2; +--sync_slave_with_master + +--connection master +--let $diff_tables= master:test.t1, slave:test.t1 +--source include/diff_tables.inc +--let $diff_tables= master:test.t2, slave:test.t2 +--source include/diff_tables.inc +--let $diff_tables= master:test.t3, slave:test.t3 +--source include/diff_tables.inc + +DROP TABLE t3,t2,t1; +--sync_slave_with_master + +# +# Test case: Triggers only on slave +# +--write_file $MYSQLTEST_VARDIR/tmp/trig_on_slave.inc PROCEDURE + if ($slave_run_triggers_for_rbr == '') { + --die !!!ERROR IN TEST: you must set $slave_run_triggers_for_rbr + } + +--connection slave +SET @save_slave_run_triggers_for_rbr= @@GLOBAL.slave_run_triggers_for_rbr; +--eval SET GLOBAL slave_run_triggers_for_rbr= $slave_run_triggers_for_rbr; +SHOW GLOBAL VARIABLES LIKE '%slave_run_triggers_for_rbr%'; + +--connection master +CREATE TABLE t1 (id INT NOT NULL PRIMARY KEY) ENGINE=InnoDB; +CREATE TABLE t2 (t1_id INT NOT NULL, + v_col INT AS (t1_id+1) VIRTUAL, KEY (v_col), + KEY (t1_id), CONSTRAINT a FOREIGN KEY (t1_id) REFERENCES t1 (id) ON DELETE CASCADE +) ENGINE=InnoDB; +CREATE TABLE t3 (count INT NOT NULL) ENGINE=InnoDB; +--sync_slave_with_master + +CREATE TRIGGER trg AFTER INSERT ON t2 FOR EACH ROW INSERT INTO t3 VALUES (1); + +--connection master +INSERT INTO t1 VALUES (2),(3); +INSERT INTO t2 VALUES (2, default), (3, default); +--sync_slave_with_master + +if ($slave_run_triggers_for_rbr == 'NO') { +--echo # +--echo # Count must be 0 +--echo # +--let $assert_cond= COUNT(*) = 0 FROM t3 +--let $assert_text= Table t3 should have zero rows. +--source include/assert.inc +} +if ($slave_run_triggers_for_rbr == 'YES') { +--echo # +--echo # Count must be 2 +--echo # +--let $assert_cond= COUNT(*) = 2 FROM t3 +--let $assert_text= Table t3 should have two rows. +--source include/assert.inc +} + +--connection master +DELETE FROM t1 WHERE id=2; +--sync_slave_with_master +SET GLOBAL slave_run_triggers_for_rbr= @save_slave_run_triggers_for_rbr; + +--echo # +--echo # Verify t1,t2 are consistent on slave. +--echo # +--let $diff_tables= master:test.t1, slave:test.t1 +--source include/diff_tables.inc +--let $diff_tables= master:test.t2, slave:test.t2 +--source include/diff_tables.inc + +--connection master +DROP TABLE t3,t2,t1; +--sync_slave_with_master +#END OF +PROCEDURE + +--echo # +--echo # Test case: Triggers are present only on slave and +--echo # 'slave_run_triggers_for_rbr=NO' +--echo # +--let $slave_run_triggers_for_rbr=NO +--source $MYSQLTEST_VARDIR/tmp/trig_on_slave.inc + +--echo # +--echo # Test case: Triggers are present only on slave and +--echo # 'slave_run_triggers_for_rbr=YES' +--echo # +--let $slave_run_triggers_for_rbr=YES +--source $MYSQLTEST_VARDIR/tmp/trig_on_slave.inc +--remove_file $MYSQLTEST_VARDIR/tmp/trig_on_slave.inc + +# +# Test case: Trigger and Foreign Key are present only on slave +# +--write_file $MYSQLTEST_VARDIR/tmp/trig_fk_on_slave.inc PROCEDURE + if ($slave_run_triggers_for_rbr == '') { + --die !!!ERROR IN TEST: you must set $slave_run_triggers_for_rbr + } + +--connection slave +SET @save_slave_run_triggers_for_rbr= @@GLOBAL.slave_run_triggers_for_rbr; +--eval SET GLOBAL slave_run_triggers_for_rbr= $slave_run_triggers_for_rbr; +SHOW GLOBAL VARIABLES LIKE '%slave_run_triggers_for_rbr%'; + +--connection master +CREATE TABLE t1 (id INT NOT NULL PRIMARY KEY) ENGINE=InnoDB; +SET sql_log_bin=0; +CREATE TABLE t2 (t1_id INT NOT NULL,v_col INT AS (t1_id+1) VIRTUAL) ENGINE=INNODB; +SET sql_log_bin=1; +CREATE TABLE t3 (count INT NOT NULL) ENGINE=InnoDB; +--sync_slave_with_master + +# Have foreign key and trigger on slave. +CREATE TABLE t2 (t1_id INT NOT NULL, + v_col INT AS (t1_id+1) VIRTUAL, KEY (v_col), KEY (t1_id), + CONSTRAINT a FOREIGN KEY (t1_id) REFERENCES t1 (id) ON DELETE CASCADE +) ENGINE=InnoDB; +CREATE TRIGGER trg AFTER INSERT ON t2 FOR EACH ROW INSERT INTO t3 VALUES (1); + +--connection master +INSERT INTO t1 VALUES (2),(3); +INSERT INTO t2 VALUES (2, default), (3, default); +--sync_slave_with_master + +if ($slave_run_triggers_for_rbr == 'NO') { +--echo # +--echo # Count must be 0 +--echo # +--let $assert_cond= COUNT(*) = 0 FROM t3 +--let $assert_text= Table t3 should have zero rows. +--source include/assert.inc +} +if ($slave_run_triggers_for_rbr == 'YES') { +--echo # +--echo # Count must be 2 +--echo # +--let $assert_cond= COUNT(*) = 2 FROM t3 +--let $assert_text= Table t3 should have two rows. +--source include/assert.inc +} + +--connection master +DELETE FROM t1 WHERE id=2; +--echo # t1: Should have one row +SELECT * FROM t1; +--echo # t2: Should have two rows +SELECT * FROM t2; +--sync_slave_with_master +--echo # t1: Should have one row +SELECT * FROM t1; +--echo # t2: Should have one row on slave due to ON DELETE CASCASE +SELECT * FROM t2; +SET GLOBAL slave_run_triggers_for_rbr= @save_slave_run_triggers_for_rbr; + +--connection master +DROP TABLE t3,t2,t1; +--sync_slave_with_master +#END OF +PROCEDURE + +--echo # +--echo # Test case: Triggers and Foreign Keys are present only on slave and +--echo # 'slave_run_triggers_for_rbr=NO' +--echo # +--let $slave_run_triggers_for_rbr=NO +--source $MYSQLTEST_VARDIR/tmp/trig_fk_on_slave.inc + +--echo # +--echo # Test case: Triggers are Foreign Keys are present only on slave and +--echo # 'slave_run_triggers_for_rbr=YES' +--echo # +--let $slave_run_triggers_for_rbr=YES +--source $MYSQLTEST_VARDIR/tmp/trig_fk_on_slave.inc +--remove_file $MYSQLTEST_VARDIR/tmp/trig_fk_on_slave.inc + +--source include/rpl_end.inc diff --git a/sql/log_event.cc b/sql/log_event.cc index c649e1f64fa..10aa0afbad8 100644 --- a/sql/log_event.cc +++ b/sql/log_event.cc @@ -10718,7 +10718,7 @@ int Rows_log_event::do_add_row_data(uchar *row_data, size_t length) There was the same problem with MERGE MYISAM tables and so here we try to go the same way. */ -static void restore_empty_query_table_list(LEX *lex) +inline void restore_empty_query_table_list(LEX *lex) { if (lex->first_not_own_table()) (*lex->first_not_own_table()->prev_global)= NULL; @@ -10733,6 +10733,8 @@ int Rows_log_event::do_apply_event(rpl_group_info *rgi) TABLE* table; DBUG_ENTER("Rows_log_event::do_apply_event(Relay_log_info*)"); int error= 0; + LEX *lex= thd->lex; + uint8 new_trg_event_map= get_trg_event_map(); /* If m_table_id == ~0ULL, then we have a dummy event that does not contain any data. In that case, we just remove all tables in the @@ -10823,27 +10825,25 @@ int Rows_log_event::do_apply_event(rpl_group_info *rgi) DBUG_ASSERT(!debug_sync_set_action(thd, STRING_WITH_LEN(action))); };); - if (slave_run_triggers_for_rbr) - { - LEX *lex= thd->lex; - uint8 new_trg_event_map= get_trg_event_map(); - - /* - Trigger's procedures work with global table list. So we have to add - rgi->tables_to_lock content there to get trigger's in the list. + /* + Trigger's procedures work with global table list. So we have to add + rgi->tables_to_lock content there to get trigger's in the list. - Then restore_empty_query_table_list() restore the list as it was - */ - DBUG_ASSERT(lex->query_tables == NULL); - if ((lex->query_tables= rgi->tables_to_lock)) - rgi->tables_to_lock->prev_global= &lex->query_tables; + Then restore_empty_query_table_list() restore the list as it was + */ + DBUG_ASSERT(lex->query_tables == NULL); + if ((lex->query_tables= rgi->tables_to_lock)) + rgi->tables_to_lock->prev_global= &lex->query_tables; - for (TABLE_LIST *tables= rgi->tables_to_lock; tables; - tables= tables->next_global) - { + for (TABLE_LIST *tables= rgi->tables_to_lock; tables; + tables= tables->next_global) + { + if (slave_run_triggers_for_rbr) tables->trg_event_map= new_trg_event_map; - lex->query_tables_last= &tables->next_global; - } + else + tables->slave_fk_event_map= new_trg_event_map; + + lex->query_tables_last= &tables->next_global; } if (open_and_lock_tables(thd, rgi->tables_to_lock, FALSE, 0)) { @@ -11193,8 +11193,7 @@ int Rows_log_event::do_apply_event(rpl_group_info *rgi) } /* remove trigger's tables */ - if (slave_run_triggers_for_rbr) - restore_empty_query_table_list(thd->lex); + restore_empty_query_table_list(thd->lex); #if defined(WITH_WSREP) && defined(HAVE_QUERY_CACHE) if (WSREP(thd) && thd->wsrep_exec_mode == REPL_RECV) @@ -11212,8 +11211,7 @@ int Rows_log_event::do_apply_event(rpl_group_info *rgi) DBUG_RETURN(error); err: - if (slave_run_triggers_for_rbr) - restore_empty_query_table_list(thd->lex); + restore_empty_query_table_list(thd->lex); rgi->slave_close_thread_tables(thd); DBUG_RETURN(error); } diff --git a/sql/sql_base.cc b/sql/sql_base.cc index 8e57ea437b6..a7cddbca16a 100644 --- a/sql/sql_base.cc +++ b/sql/sql_base.cc @@ -4371,7 +4371,8 @@ handle_table(THD *thd, Query_tables_list *prelocking_ctx, /* We rely on a caller to check that table is going to be changed. */ DBUG_ASSERT(table_list->lock_type >= TL_WRITE_ALLOW_WRITE); - if (table_list->trg_event_map) + if (table_list->trg_event_map || + (thd->rgi_slave && thd->is_current_stmt_binlog_format_row())) { if (table_list->table->triggers) { @@ -4405,7 +4406,8 @@ handle_table(THD *thd, Query_tables_list *prelocking_ctx, { // FK_OPTION_RESTRICT and FK_OPTION_NO_ACTION only need read access static bool can_write[]= { true, false, true, true, false, true }; - uint8 op= table_list->trg_event_map; + uint8 op= ((table_list->trg_event_map) ? table_list->trg_event_map : + table_list->slave_fk_event_map); thr_lock_type lock_type; if ((op & (1 << TRG_EVENT_DELETE) && can_write[fk->delete_method]) diff --git a/sql/table.h b/sql/table.h index 9a864f7ce9f..57706655d9b 100644 --- a/sql/table.h +++ b/sql/table.h @@ -2277,8 +2277,12 @@ struct TABLE_LIST Indicates what triggers we need to pre-load for this TABLE_LIST when opening an associated TABLE. This is filled after the parsed tree is created. + + slave_fk_event_map is filled on the slave side with bitmaps value + representing row-based event operation to help find and prelock + possible FK constrain-related child tables. */ - uint8 trg_event_map; + uint8 trg_event_map, slave_fk_event_map; /* TRUE <=> this table is a const one and was optimized away. */ bool optimized_away;

1 0

Re: [Maria-developers] [Commits] cdc305c8dd8: MDEV-19620: Changing join_buffer_size causes different results
by Sergey Petrunia 29 Dec '20

29 Dec '20

Hi Varun, Please find input below. On Mon, Dec 28, 2020 at 02:34:40PM +0530, varun wrote: > revision-id: cdc305c8dd89a726e09e5fe70ff890d06609cbfb (mariadb-10.3.21-309-gcdc305c8dd8) > parent(s): 043bd85a574a88856ab9c6d497e682ed06fe45e9 > author: Varun Gupta > committer: Varun Gupta > timestamp: 2020-12-28 14:12:14 +0530 > message: > > MDEV-19620: Changing join_buffer_size causes different results > > The scenario here is that query refinement phase decides to use a hash join. > When the join buffers are allocated in the JOIN::init_join_caches, for a table > the size exceeds the value for join_buffer_space_limit (which is the limit of the > space available for all join buffers). When this happens then we disallow join > buffering for the table, this is done in the revise_cache_usage and set_join_cache_denial. > > In this issue the hash key is created on an index for which ref access is possible, so > when we disallow hash join then instead of switching to REF access we switch to a table > scan. This is a problem because the equijoin conditions for which a lookup can be made > are not attached to the table(or are not evaluated for the table). This leads to incorrect > results. > > The fix here would be to switch to using a lookup because it was picked by the join planner > to be more efficient than the table scan. > > --- > mysql-test/main/join_cache.result | 138 ++++++++++++++++++++++++++++++++++++++ > mysql-test/main/join_cache.test | 105 +++++++++++++++++++++++++++++ > sql/sql_select.cc | 69 +++++++++++++++---- > sql/sql_select.h | 6 ++ > 4 files changed, 304 insertions(+), 14 deletions(-) > > diff --git a/mysql-test/main/join_cache.result b/mysql-test/main/join_cache.result > index 3d1d91df997..e58503f422f 100644 > --- a/mysql-test/main/join_cache.result > +++ b/mysql-test/main/join_cache.result > @@ -6128,4 +6128,142 @@ EXPLAIN > } > } > drop table t1,t2,t3; > +# > +# MDEV-19620: Changing join_buffer_size causes different results > +# > +SET @save_join_cache_level= @@join_cache_level; > +SET @save_join_buffer_size= @@join_buffer_size; > +SET @save_join_buffer_space_limit= @@join_buffer_space_limit; > +SET join_cache_level = 3; > +CREATE TABLE t1 (i2 VARCHAR(500), c2 VARCHAR(500)) ENGINE=MyISAM; > +INSERT INTO t1 VALUES ('1','v'),('7','s'); > +CREATE TABLE t2 (pk3 INT PRIMARY KEY, i3 VARCHAR(300), c3 VARCHAR(500)) ENGINE=MyISAM; > +INSERT INTO t2 VALUES (1,'7','g'),(2,'4','p'),(3,'1','q'); > +INSERT INTO t2 VALUES (4,'7','g'),(5,'4','p'),(6,'1','q'); > +INSERT INTO t2 VALUES (16,'7','g'),(17,'4','p'),(28,'1','q'); > +# > +# Hash join + table Scan on t2 > +# > +set join_buffer_size=1024; > +set join_buffer_space_limit=2048; > +EXPLAIN SELECT * FROM t1 LEFT JOIN t2 ON (c3=c2); > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 ALL NULL NULL NULL NULL 2 > +1 SIMPLE t2 ALL NULL NULL NULL NULL 9 Using where > +SELECT * FROM t1 LEFT JOIN t2 ON (c3=c2); > +i2 c2 pk3 i3 c3 > +1 v NULL NULL NULL > +7 s NULL NULL NULL > +set join_buffer_space_limit=262144; > +EXPLAIN SELECT * FROM t1 LEFT JOIN t2 ON (c3=c2); > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 ALL NULL NULL NULL NULL 2 > +1 SIMPLE t2 hash_ALL NULL #hash#$hj 503 test.t1.c2 9 Using where; Using join buffer (flat, BNLH join) > +SELECT * FROM t1 LEFT JOIN t2 ON (c3=c2); > +i2 c2 pk3 i3 c3 > +1 v NULL NULL NULL > +7 s NULL NULL NULL > +# > +# HASH join + ref access on t2 > +# > +ALTER TABLE t2 ADD KEY k1(c3); > +set join_buffer_size=1024; > +set join_buffer_space_limit=2048; > +EXPLAIN SELECT * FROM t1 LEFT JOIN t2 ON (c3=c2); > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 ALL NULL NULL NULL NULL 2 > +1 SIMPLE t2 ref k1 k1 503 test.t1.c2 2 Using where > +SELECT * FROM t1 LEFT JOIN t2 ON (c3=c2); > +i2 c2 pk3 i3 c3 > +1 v NULL NULL NULL > +7 s NULL NULL NULL > +set join_buffer_space_limit=262144; > +EXPLAIN SELECT * FROM t1 LEFT JOIN t2 ON (c3=c2); > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 ALL NULL NULL NULL NULL 2 > +1 SIMPLE t2 hash_ALL k1 #hash#k1 503 test.t1.c2 9 Using where; Using join buffer (flat, BNLH join) > +SELECT * FROM t1 LEFT JOIN t2 ON (c3=c2); > +i2 c2 pk3 i3 c3 > +1 v NULL NULL NULL > +7 s NULL NULL NULL > +# > +# Hash join + index scan on t2 > +# > +ALTER TABLE t2 DROP KEY k1; > +ALTER TABLE t2 ADD KEY k1(i3,c3); > +set join_buffer_size=1024; > +set join_buffer_space_limit=2048; > +EXPLAIN SELECT c3,i3, i2,c2 FROM t1 LEFT JOIN t2 ON (c3 = c2); > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 ALL NULL NULL NULL NULL 2 > +1 SIMPLE t2 index NULL k1 806 NULL 9 Using where; Using index > +SELECT c3,i3, i2,c2 FROM t1 LEFT JOIN t2 ON (c3 = c2); > +c3 i3 i2 c2 > +NULL NULL 1 v > +NULL NULL 7 s > +set join_buffer_space_limit=262144; > +EXPLAIN SELECT c3,i3, i2,c2 FROM t1 LEFT JOIN t2 ON (c3 = c2); > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 ALL NULL NULL NULL NULL 2 > +1 SIMPLE t2 hash_index NULL #hash#$hj:k1 503:806 test.t1.c2 9 Using where; Using index; Using join buffer (flat, BNLH join) > +SELECT c3,i3, i2,c2 FROM t1 LEFT JOIN t2 ON (c3 = c2); > +c3 i3 i2 c2 > +NULL NULL 1 v > +NULL NULL 7 s > +DROP TABLE t1,t2; > +# > +# Hash join + range scan on t2 > +# > +CREATE TABLE t1 (i2 VARCHAR(500), c2 VARCHAR(500)); > +INSERT INTO t1 VALUES ('1','v'),('7','s'); > +CREATE TABLE t2 (pk3 INT PRIMARY KEY, i3 VARCHAR(300), c3 VARCHAR(500), INDEX(i3,c3)); > +INSERT INTO t2 VALUES (1,'7','g'),(2,'4','s'),(3,'1','q'); > +set join_buffer_size=1024; > +set join_buffer_space_limit=2048; > +EXPLAIN SELECT * FROM t1 LEFT JOIN t2 ON (c3 = c2) WHERE i3 >= '4'; > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 ALL NULL NULL NULL NULL 2 Using where > +1 SIMPLE t2 range i3 i3 303 NULL 2 Using index condition; Using where > +SELECT * FROM t1 LEFT JOIN t2 ON (c3 = c2) WHERE i3 >= '4'; > +i2 c2 pk3 i3 c3 > +7 s 2 4 s > +set join_buffer_space_limit=262144; > +EXPLAIN SELECT * FROM t1 LEFT JOIN t2 ON (c3 = c2) WHERE i3 >= '4'; > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 ALL NULL NULL NULL NULL 2 Using where > +1 SIMPLE t2 hash_range i3 #hash#$hj:i3 503:303 test.t1.c2 2 Using where; Using join buffer (flat, BNLH join) > +SELECT * FROM t1 LEFT JOIN t2 ON (c3 = c2) WHERE i3 >= '4'; > +i2 c2 pk3 i3 c3 > +7 s 2 4 s > +DROP TABLE t1,t2; > +# > +# Hash join + eq ref access on t2 > +# > +CREATE TABLE t1 (i2 VARCHAR(500), c2 VARCHAR(500)); > +INSERT INTO t1 VALUES ('1','v'),('7','s'); > +CREATE TABLE t2 (pk3 INT, i3 VARCHAR(300), c3 VARCHAR(500) PRIMARY KEY); > +INSERT INTO t2 VALUES (1,'7','g'),(2,'4','s'),(3,'1','q'); > +set join_buffer_size=1024; > +set join_buffer_space_limit=2048; > +EXPLAIN SELECT c3,i3, i2,c2 FROM t1 LEFT JOIN t2 ON (c3 = c2); > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 ALL NULL NULL NULL NULL 2 > +1 SIMPLE t2 eq_ref PRIMARY PRIMARY 502 test.t1.c2 1 Using where > +SELECT c3,i3, i2,c2 FROM t1 LEFT JOIN t2 ON (c3 = c2); > +c3 i3 i2 c2 > +NULL NULL 1 v > +s 4 7 s > +set join_buffer_space_limit=262144; > +EXPLAIN SELECT c3,i3, i2,c2 FROM t1 LEFT JOIN t2 ON (c3 = c2); > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 ALL NULL NULL NULL NULL 2 > +1 SIMPLE t2 hash_ALL PRIMARY #hash#PRIMARY 502 test.t1.c2 3 Using where; Using join buffer (flat, BNLH join) > +SELECT c3,i3, i2,c2 FROM t1 LEFT JOIN t2 ON (c3 = c2); > +c3 i3 i2 c2 > +NULL NULL 1 v > +s 4 7 s > +DROP TABLE t1,t2; > +SET @save_join_cache_level= @@join_cache_level; > +SET @save_join_buffer_size= @@join_buffer_size; > +SET @save_join_buffer_space_limit= @@join_buffer_space_limit; > set @@optimizer_switch=@save_optimizer_switch; > diff --git a/mysql-test/main/join_cache.test b/mysql-test/main/join_cache.test > index 91339c2cb21..6670c62516b 100644 > --- a/mysql-test/main/join_cache.test > +++ b/mysql-test/main/join_cache.test > @@ -4054,5 +4054,110 @@ where > > drop table t1,t2,t3; > > +--echo # > +--echo # MDEV-19620: Changing join_buffer_size causes different results > +--echo # > + > +SET @save_join_cache_level= @@join_cache_level; > +SET @save_join_buffer_size= @@join_buffer_size; > +SET @save_join_buffer_space_limit= @@join_buffer_space_limit; > +SET join_cache_level = 3; > + > +CREATE TABLE t1 (i2 VARCHAR(500), c2 VARCHAR(500)) ENGINE=MyISAM; > +INSERT INTO t1 VALUES ('1','v'),('7','s'); > +CREATE TABLE t2 (pk3 INT PRIMARY KEY, i3 VARCHAR(300), c3 VARCHAR(500)) ENGINE=MyISAM; > +INSERT INTO t2 VALUES (1,'7','g'),(2,'4','p'),(3,'1','q'); > +INSERT INTO t2 VALUES (4,'7','g'),(5,'4','p'),(6,'1','q'); > +INSERT INTO t2 VALUES (16,'7','g'),(17,'4','p'),(28,'1','q'); > + > +--echo # > +--echo # Hash join + table Scan on t2 > +--echo # > + > +set join_buffer_size=1024; > +set join_buffer_space_limit=2048; > +EXPLAIN SELECT * FROM t1 LEFT JOIN t2 ON (c3=c2); > +SELECT * FROM t1 LEFT JOIN t2 ON (c3=c2); > + > +set join_buffer_space_limit=262144; > +EXPLAIN SELECT * FROM t1 LEFT JOIN t2 ON (c3=c2); > +SELECT * FROM t1 LEFT JOIN t2 ON (c3=c2); > + > +--echo # > +--echo # HASH join + ref access on t2 > +--echo # > + > +ALTER TABLE t2 ADD KEY k1(c3); > +set join_buffer_size=1024; > +set join_buffer_space_limit=2048; > +EXPLAIN SELECT * FROM t1 LEFT JOIN t2 ON (c3=c2); > +SELECT * FROM t1 LEFT JOIN t2 ON (c3=c2); > + > +set join_buffer_space_limit=262144; > +EXPLAIN SELECT * FROM t1 LEFT JOIN t2 ON (c3=c2); > +SELECT * FROM t1 LEFT JOIN t2 ON (c3=c2); > + > +--echo # > +--echo # Hash join + index scan on t2 > +--echo # > +ALTER TABLE t2 DROP KEY k1; > +ALTER TABLE t2 ADD KEY k1(i3,c3); > + > +set join_buffer_size=1024; > +set join_buffer_space_limit=2048; > + > +EXPLAIN SELECT c3,i3, i2,c2 FROM t1 LEFT JOIN t2 ON (c3 = c2); > +SELECT c3,i3, i2,c2 FROM t1 LEFT JOIN t2 ON (c3 = c2); > + > +set join_buffer_space_limit=262144; > +EXPLAIN SELECT c3,i3, i2,c2 FROM t1 LEFT JOIN t2 ON (c3 = c2); > +SELECT c3,i3, i2,c2 FROM t1 LEFT JOIN t2 ON (c3 = c2); > +DROP TABLE t1,t2; > + > +--echo # > +--echo # Hash join + range scan on t2 > +--echo # > + > +CREATE TABLE t1 (i2 VARCHAR(500), c2 VARCHAR(500)); > +INSERT INTO t1 VALUES ('1','v'),('7','s'); > +CREATE TABLE t2 (pk3 INT PRIMARY KEY, i3 VARCHAR(300), c3 VARCHAR(500), INDEX(i3,c3)); > +INSERT INTO t2 VALUES (1,'7','g'),(2,'4','s'),(3,'1','q'); > + > +set join_buffer_size=1024; > +set join_buffer_space_limit=2048; > + > +EXPLAIN SELECT * FROM t1 LEFT JOIN t2 ON (c3 = c2) WHERE i3 >= '4'; > +SELECT * FROM t1 LEFT JOIN t2 ON (c3 = c2) WHERE i3 >= '4'; > + > +set join_buffer_space_limit=262144; > +EXPLAIN SELECT * FROM t1 LEFT JOIN t2 ON (c3 = c2) WHERE i3 >= '4'; > +SELECT * FROM t1 LEFT JOIN t2 ON (c3 = c2) WHERE i3 >= '4'; > + > +DROP TABLE t1,t2; > + > +--echo # > +--echo # Hash join + eq ref access on t2 > +--echo # > + > +CREATE TABLE t1 (i2 VARCHAR(500), c2 VARCHAR(500)); > +INSERT INTO t1 VALUES ('1','v'),('7','s'); > +CREATE TABLE t2 (pk3 INT, i3 VARCHAR(300), c3 VARCHAR(500) PRIMARY KEY); > +INSERT INTO t2 VALUES (1,'7','g'),(2,'4','s'),(3,'1','q'); > + > +set join_buffer_size=1024; > +set join_buffer_space_limit=2048; > + > +EXPLAIN SELECT c3,i3, i2,c2 FROM t1 LEFT JOIN t2 ON (c3 = c2); > +SELECT c3,i3, i2,c2 FROM t1 LEFT JOIN t2 ON (c3 = c2); > + > +set join_buffer_space_limit=262144; > +EXPLAIN SELECT c3,i3, i2,c2 FROM t1 LEFT JOIN t2 ON (c3 = c2); > +SELECT c3,i3, i2,c2 FROM t1 LEFT JOIN t2 ON (c3 = c2); > +DROP TABLE t1,t2; > + > +SET @save_join_cache_level= @@join_cache_level; > +SET @save_join_buffer_size= @@join_buffer_size; > +SET @save_join_buffer_space_limit= @@join_buffer_space_limit; > + > # The following command must be the last one in the file > set @@optimizer_switch=@save_optimizer_switch; > diff --git a/sql/sql_select.cc b/sql/sql_select.cc > index 203422f0b43..8b7deb98dc6 100644 > --- a/sql/sql_select.cc > +++ b/sql/sql_select.cc > @@ -111,6 +111,7 @@ static bool best_extension_by_limited_search(JOIN *join, > uint prune_level, > uint use_cond_selectivity); > static uint determine_search_depth(JOIN* join); > +static void pick_table_access_method(JOIN_TAB *tab); > C_MODE_START > static int join_tab_cmp(const void *dummy, const void* ptr1, const void* ptr2); > static int join_tab_cmp_straight(const void *dummy, const void* ptr1, const void* ptr2); > @@ -10081,6 +10082,7 @@ static bool create_ref_for_key(JOIN *join, JOIN_TAB *j, > j->ref.disable_cache= FALSE; > j->ref.null_ref_part= NO_REF_PART; > j->ref.const_ref_part_map= 0; > + j->ref.not_null_keyparts= 0; > keyuse=org_keyuse; > > store_key **ref_key= j->ref.key_copy; > @@ -10173,23 +10175,12 @@ static bool create_ref_for_key(JOIN *join, JOIN_TAB *j, > } > } /* not ftkey */ > *ref_key=0; // end_marker > + j->ref.not_null_keyparts= not_null_keyparts; > if (j->type == JT_FT) > DBUG_RETURN(0); > - ulong key_flags= j->table->actual_key_flags(keyinfo); > if (j->type == JT_CONST) > j->table->const_table= 1; > - else if (!((keyparts == keyinfo->user_defined_key_parts && > - ( > - (key_flags & (HA_NOSAME | HA_NULL_PART_KEY)) == HA_NOSAME || > - /* Unique key and all keyparts are NULL rejecting */ > - ((key_flags & HA_NOSAME) && keyparts == not_null_keyparts) > - )) || > - /* true only for extended keys */ > - (keyparts > keyinfo->user_defined_key_parts && > - MY_TEST(key_flags & HA_EXT_NOSAME) && > - keyparts == keyinfo->ext_key_parts) > - ) || > - null_ref_key) > + else if (!j->is_eq_ref_access()|| null_ref_key) > { > /* Must read with repeat */ > j->type= null_ref_key ? JT_REF_OR_NULL : JT_REF; > @@ -11582,11 +11573,25 @@ void set_join_cache_denial(JOIN_TAB *join_tab) > don't do join buffering for the first table in sjm nest. > */ > join_tab[-1].next_select= sub_select; > - if (join_tab->type == JT_REF && join_tab->is_ref_for_hash_join()) > + if ((join_tab->type == JT_REF || join_tab->type == JT_HASH) && > + join_tab->is_ref_for_hash_join()) > { > join_tab->type= JT_ALL; > join_tab->ref.key_parts= 0; Do we ever get here if join_tab->type == JT_REF? I think we don't. If this is so, please remove it. > } > + > + if (join_tab->type == JT_HASH && !join_tab->is_ref_for_hash_join()) > + { > + join_tab->type= join_tab->is_eq_ref_access() ? JT_EQ_REF : JT_REF; > + pick_table_access_method(join_tab); > + } Also please add a comment explaining that the join optimizer's choice was ref access and we fall back to that. > + > + if (join_tab->type == JT_HASH_NEXT) > + { > + join_tab->type = JT_NEXT; > + DBUG_ASSERT(join_tab->ref.key_parts == 0); > + } > + > join_tab->join->return_tab= join_tab; > } > } > @@ -27954,6 +27959,42 @@ void JOIN_TAB::partial_cleanup() > } > > > +/* > + @brief > + Check if the access method for the table is EQ_REF access or not > + > + @retval > + TRUE EQ_REF access > + FALSE Otherwise > +*/ > +bool JOIN_TAB::is_eq_ref_access() > +{ > + > + KEY *keyinfo; > + if (!is_hash_join_key_no(ref.key)) > + keyinfo= table->key_info + ref.key; > + else > + keyinfo= hj_key; This code now looks as if it was possible that hash join is used and also eq_ref is used. Is this really possible? I've tried catching an example of this with MTR and I wasn't able to do so. Maybe we should just return false if hash join is used and also a comment about this? > + > + uint keyparts= ref.key_parts; > + ulong key_flags= table->actual_key_flags(keyinfo); > + if ( (keyparts == keyinfo->user_defined_key_parts && > + ( > + (key_flags & (HA_NOSAME | HA_NULL_PART_KEY)) == HA_NOSAME || > + /* Unique key and all keyparts are NULL rejecting */ > + ((key_flags & HA_NOSAME) && keyparts == ref.not_null_keyparts) > + ) > + ) || > + /* true only for extended keys */ > + (keyparts > keyinfo->user_defined_key_parts && > + MY_TEST(key_flags & HA_EXT_NOSAME) && > + keyparts == keyinfo->ext_key_parts) > + ) > + return true; > + return false; > +} > + > + > /** > @} (end of group Query_Optimizer) > */ BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://petrunia.net

2 1

Re: [Maria-developers] 33787ea47a3: MDEV-6899: Database Dump has Syntax Error When Importing
by Sergei Golubchik 23 Dec '20

23 Dec '20

Hi, Anel! On Dec 23, Anel Husakovic wrote: > revision-id: 33787ea47a3 (mariadb-10.2.31-588-g33787ea47a3) > parent(s): ceef26cf866 > author: Anel Husakovic <anel(a)mariadb.org> > committer: Anel Husakovic <anel(a)mariadb.org> > timestamp: 2020-11-17 15:11:14 +0100 > message: > > MDEV-6899: Database Dump has Syntax Error When Importing > > --- > mysql-test/r/mysqldump.result | 32 ++++++++++++++++++++++++++++++++ > mysql-test/t/mysqldump.test | 26 ++++++++++++++++++++++++++ > sql/events.cc | 10 ++++++++-- > sql/sql_show.cc | 4 +++- > 4 files changed, 69 insertions(+), 3 deletions(-) > > diff --git a/sql/events.cc b/sql/events.cc > index abac2833833..2880113bc75 100644 > --- a/sql/events.cc > +++ b/sql/events.cc > @@ -680,6 +680,12 @@ send_show_create_event(THD *thd, Event_timed *et, Protocol *protocol) > if (et->get_create_event(thd, &show_str)) > DBUG_RETURN(TRUE); > > + LEX_STRING def_show_str= show_str.lex_string(); > + if (!strcmp(&def_show_str.str[def_show_str.length -1], ";")) > + { > + --def_show_str.length; > + def_show_str.str[def_show_str.length]='\0'; > + } No, this is clearly wrong. The bug is that semicolon even gets into the event definition, see other commands in the bug report. You should fix that, not strip it away in SHOW CREATE, which will only hide the bug. Why is the semicolon parsed as a part of the event anyway? > field_list.push_back(new (mem_root) > Item_empty_string(thd, "Event", NAME_CHAR_LEN), > mem_root); Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] ab7092a415f: MDEV-24179: Assertion `m_status == DA_ERROR || m_status == DA_OK ||
by Sergei Golubchik 21 Dec '20

21 Dec '20

Hi, Rucha! On Dec 21, Rucha Deodhar wrote: > revision-id: ab7092a415f (mariadb-10.2.31-584-gab7092a415f) > parent(s): 9b30212f15e > author: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > committer: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > timestamp: 2020-12-18 10:20:20 +0530 > message: > > MDEV-24179: Assertion `m_status == DA_ERROR || m_status == DA_OK || > m_status == DA_OK_BULK' failed in Diagnostics_area::message() > > Analysis: Assertion failure happens because we reach the maximum limit to > examine rows. > Fix: Return the error state. > > diff --git a/sql/sql_show.cc b/sql/sql_show.cc > index 5544c765775..73b7a84fae1 100644 > --- a/sql/sql_show.cc > +++ b/sql/sql_show.cc > @@ -5046,6 +5046,12 @@ int get_all_tables(THD *thd, TABLE_LIST *tables, COND *cond) > continue; > } > > + if (thd->killed == ABORT_QUERY) > + { > + error= 0; > + goto err; > + } > + you already have exactly the same if() few lines below. may be it shouldn't be duplicated, but moved down to the err: label? > DEBUG_SYNC(thd, "before_open_in_get_all_tables"); > if (fill_schema_table_by_open(thd, FALSE, > table, schema_table, Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 889f90365ef: MDEV-23536 : Race condition between KILL and transaction commit
by Sergei Golubchik 20 Dec '20

20 Dec '20

Hi, Jan! On Dec 19, Jan LindstrÃ¶m wrote: > revision-id: 889f90365ef (mariadb-10.2.31-623-g889f90365ef) > parent(s): 74223c33d1a > author: Jan Lindström <jan.lindstrom(a)mariadb.com> > committer: Jan Lindström <jan.lindstrom(a)mariadb.com> > timestamp: 2020-12-18 16:10:39 +0200 > message: > > MDEV-23536 : Race condition between KILL and transaction commit ... > If you look carefully into the above, you can conclude that > thd->free_connection() can be called concurrently with > KILL/thd->awake(). Which is the bug. And it is partially fixed in > THD::~THD(), that is destructor waits for KILL completion: > > diff --git a/sql/sql_class.cc b/sql/sql_class.cc > index 7eaafbd9044..46aeb4f2bb3 100644 > --- a/sql/sql_class.cc > +++ b/sql/sql_class.cc > @@ -1437,6 +1437,31 @@ void THD::cleanup(void) > void THD::free_connection() > { > DBUG_ASSERT(free_connection_done == 0); > +#ifdef WITH_WSREP > + /* > + wsrep_rgi should be protected with LOCK_thd_data mutex and > + mutexing ordering rules state that LOCK_thd_data must be taken > + before LOCK_thd_kill. > + */ > + if (wsrep_rgi && WSREP(this)) mysql_mutex_lock(&LOCK_thd_data); > +#endif > + /* > + Other threads may have a lock on LOCK_thd_kill to ensure that this > + THD is not deleted while they access it. The following mutex_lock > + ensures that no one else is using this THD and it's now safe to > + continue. > + */ > + mysql_mutex_lock(&LOCK_thd_kill); > + mysql_mutex_unlock(&LOCK_thd_kill); Note that THD destructor starts from `assert_not_linked()` - it's important, because if the THD is linked in the list of threads, then someone might start using it after you released LOCK_thd_kill. With the new logic you need to move assert_not_linked to free_connection(). otherwise ok to push > + > +#ifdef WITH_WSREP > + if (rpl_group_info *rgi= wsrep_rgi) > + { > + wsrep_rgi= 0; > + if (WSREP(this)) mysql_mutex_unlock(&LOCK_thd_data); > + delete rgi; > + } > +#endif > my_free(db); > db= NULL; Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

3 4

Re: [Maria-developers] a36f27e200e: Fixes that enables my_new.cc (new wrapper using my_malloc)
by Sergei Golubchik 11 Dec '20

11 Dec '20

Hi, Michael! On Dec 11, Michael Widenius wrote: > revision-id: a36f27e200e (mariadb-10.5.2-275-ga36f27e200e) > parent(s): 9f2bbf84e8d > author: Michael Widenius <michael.widenius(a)gmail.com> > committer: Michael Widenius <michael.widenius(a)gmail.com> > timestamp: 2020-09-24 17:26:40 +0300 > message: > > Fixes that enables my_new.cc (new wrapper using my_malloc) > > This is not enabled by default as there are leaks in the > server that needs to be fixed first. One can compile > with -DREALLY_USE_MYSYS_NEW -DSF_REMEMBER_FRAMES=14 to find the > memory leaks from 'new' The change in my_new.cc is ok. The change in my_global.h and CMakeLists.txt is confusing and redundant. You unconditionaly define USE_MYSYS_NEW then conditionally undefine it again? Everything already worked just fine. By default cxx new is used, if one wants to use mysys version, it's done with -DHAVE_CXX_NEW=0 this works for me in 10.5 and fails to compile my_new.cc, as it should. Please, only keed your my_new.cc changes and say in the commit message "One can compile with -DHAVE_CXX_NEW=0 to find memory leaks from 'new'" As for -DSF_REMEMBER_FRAMES=14, perhaps it's better to do it automatically? Like #ifdef USE_MYSYS_NEW #define SF_REMEMBER_FRAMES 14 #else #define SF_REMEMBER_FRAMES 8 #endif or may be even just always make it 14. It should be easy to enable this feature, right? > diff --git a/CMakeLists.txt b/CMakeLists.txt > index 80fac4b2ccc..f97a6354f19 100644 > --- a/CMakeLists.txt > +++ b/CMakeLists.txt > @@ -362,10 +362,8 @@ ENDIF() > # Run platform tests > INCLUDE(configure.cmake) > > -# force -DUSE_MYSYS_NEW unless already done by HAVE_CXX_NEW > -IF(NOT HAVE_CXX_NEW) > - ADD_DEFINITIONS(-DUSE_MYSYS_NEW) > -ENDIF() > +# Always use USE_MYSYS_NEW > +ADD_DEFINITIONS(-DUSE_MYSYS_NEW) > > # Find header files from the bundled libraries > # (wolfssl, readline, pcre2, etc) > diff --git a/include/my_global.h b/include/my_global.h > index 86ef5f882f6..8862a397f73 100644 > --- a/include/my_global.h > +++ b/include/my_global.h > @@ -527,6 +527,9 @@ typedef unsigned short ushort; > duplicate declaration of __cxa_pure_virtual, solved by declaring it a > weak symbol. > */ > +#ifndef REALLY_USE_MYSYS_NEW > +#undef USE_MYSYS_NEW > +#endif > #if defined(USE_MYSYS_NEW) && ! defined(DONT_DECLARE_CXA_PURE_VIRTUAL) > C_MODE_START > int __cxa_pure_virtual () __attribute__ ((weak)); Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 9f2bbf84e8d1: Fixed my_addr_resolve
by Sergei Golubchik 10 Dec '20

10 Dec '20

Hi, Michael! On Dec 03, Michael Widenius wrote: > revision-id: 9f2bbf84e8d1 (mariadb-10.5.2-274-g9f2bbf84e8d1) > parent(s): 173116c3a573 > author: Michael Widenius <michael.widenius(a)gmail.com> > committer: Michael Widenius <michael.widenius(a)gmail.com> > timestamp: 2020-09-24 17:26:40 +0300 > message: > > Fixed my_addr_resolve > > At least on openSUSE Leap 15.1 one should not use the dli_fbase offset > provided by dladdr() for addresses in the main program. > Without this patch the stack traces from sf_report_leaked_memory() > are not usable. Sorry, I was not able to repeat it. I built on openSUSE Leap and the stack trace was perfectly usable without your patch. I tried both `kill -ABRT` and an artificially introduced memory leak. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 53a88b2b7032: Optimize usage of c_ptr(), c_ptr_quick() and String::alloc()
by Sergei Golubchik 04 Dec '20

04 Dec '20

Hi, Michael! On Dec 03, Michael Widenius wrote: > revision-id: 53a88b2b7032 (mariadb-10.5.2-272-g53a88b2b7032) > parent(s): 0aab153e05be > author: Michael Widenius <michael.widenius(a)gmail.com> > committer: Michael Widenius <michael.widenius(a)gmail.com> > timestamp: 2020-09-24 17:26:40 +0300 > message: > > Optimize usage of c_ptr(), c_ptr_quick() and String::alloc() > > The proble was that hen using String::alloc() to allocate a string, "hen" ? after a minute of staring I realized that you probably meant "when". (and "problem") > the String ensures that there is space for an extra NULL byte in the > buffer and if not, reallocates the string. This is a problem with the > String::set_int() that calls alloc(21), which forces > an extra malloc/free to happen. > > - We do not anymore re-allocate String if alloc() is called with the > Allocated_length. This reduces number of malloc() allocations, > especially one big re-allocation in Protocol::send_result_Set_metadata() > for almost every query that produced a result to the connnected client. > - Avoid extra mallocs when using LONGLONG_BUFFER_SIZE > This can now be done as alloc() doesn't increase buffers if new length is > not bigger than old one. > - c_ptr() is redesigned to be safer (but a bit longer) than before. > - Remove wrong usage of c_ptr_quick() > c_ptr_quick() where used in many cases to get the pointer to the used > buffer, even when it didn't need to be \0 terminated. In this case > ptr() is a better substitute. > Another problem with c_ptr_quick() is that it didn't guarantee that > the string would be \0 terminated. > - item_val_str(), an API function not used currently by the server, > now always returns a null terminated string (before it didn't always > do that). > - Ensure that all String allocations uses STRING_PSI_MEMORY_KEY. The old > mixed usage of performance keys caused assert's when String buffers > where shrunk. > - Binary_string::shrink() is simplifed > > diff --git a/sql/item.cc b/sql/item.cc > index 4a26cb6c140c..328fe96018a5 100644 > --- a/sql/item.cc > +++ b/sql/item.cc > @@ -4574,7 +4574,7 @@ const String *Item_param::value_query_val_str(THD *thd, String *str) const > break; > } > DBUG_ASSERT(str->length() <= typelen); > - buf= str->c_ptr_quick(); > + buf= (char*) str->ptr(); > ptr= buf + str->length(); > *ptr++= '\''; > ptr+= (uint) my_TIME_to_str(&value.time, ptr, decimals); this is just wrong. One cannot simply write data at the String's end pointer. One must alloc() the space first. > diff --git a/sql/partition_info.cc b/sql/partition_info.cc > index 8ad14ca260c4..ce72a7af051d 100644 > --- a/sql/partition_info.cc > +++ b/sql/partition_info.cc > @@ -126,7 +126,7 @@ partition_info *partition_info::get_clone(THD *thd) > /** > Mark named [sub]partition to be used/locked. > > - @param part_name Partition name to match. > + @param part_name Partition name to match. Must be \0 terminated! > @param length Partition name length. > > @return Success if partition found > @@ -172,9 +172,9 @@ bool partition_info::add_named_partition(const char *part_name, size_t length) > else > bitmap_set_bit(&read_partitions, part_def->part_id); > } > - DBUG_PRINT("info", ("Found partition %u is_subpart %d for name %s", > + DBUG_PRINT("info", ("Found partition %u is_subpart %d for name %.*s", > part_def->part_id, part_def->is_subpart, > - part_name)); > + length, part_name)); why did you specify a length if the name "must be \0 terminated!" ? (and I agree that it must be) > DBUG_RETURN(false); > } > > diff --git a/sql/sql_plugin.cc b/sql/sql_plugin.cc > index 633d969b3dec..2b603380aa6c 100644 > --- a/sql/sql_plugin.cc > +++ b/sql/sql_plugin.cc > @@ -404,12 +404,16 @@ static int item_value_type(struct st_mysql_value *value) > static const char *item_val_str(struct st_mysql_value *value, > char *buffer, int *length) > { > + size_t org_length= *length; > - String str(buffer, *length, system_charset_info), *res; > + String str(buffer, org_length, system_charset_info), *res; > if (!(res= ((st_item_value_holder*)value)->item->val_str(&str))) > return NULL; > *length= res->length(); > - if (res->c_ptr_quick() == buffer) > + if (res->ptr() == buffer && res->length() < org_length) > + { > + buffer[res->length()]= 0; > return buffer; > + } Okay. This looked like the *only* correct usage of c_ptr_quick(). Agree, better to remove c_ptr_quick(). > > /* > Lets be nice and create a temporary string since the > diff --git a/sql/sql_string.cc b/sql/sql_string.cc > index e2defba434dd..8ab156d0e556 100644 > --- a/sql/sql_string.cc > +++ b/sql/sql_string.cc > @@ -1251,21 +1259,16 @@ bool String::append_semi_hex(const char *s, uint len, CHARSET_INFO *cs) > // Shrink the buffer, but only if it is allocated on the heap. > void Binary_string::shrink(size_t arg_length) > { > - if (!is_alloced()) > - return; > - if (ALIGN_SIZE(arg_length + 1) < Alloced_length) > + if (is_alloced() && ALIGN_SIZE(arg_length + 1) < Alloced_length) > + { > + char *new_ptr; > + /* my_realloc() can't fail as new buffer is less than the original one */ > + if ((new_ptr= (char*) my_realloc(STRING_PSI_MEMORY_KEY, Ptr, arg_length, > + MYF(thread_specific ? > + MY_THREAD_SPECIFIC : 0)))) why do you need an if() if "my_realloc() can't fail" ? (and it really cannot, you're right) > { > - char* new_ptr; > - if (!(new_ptr = (char*)my_realloc(STRING_PSI_MEMORY_KEY, Ptr, arg_length, > - MYF(thread_specific ? MY_THREAD_SPECIFIC : 0)))) > - { > - Alloced_length = 0; > - real_alloc(arg_length); > - } > - else > - { > - Ptr = new_ptr; > - Alloced_length = (uint32)arg_length; > - } > + Ptr= new_ptr; > + Alloced_length= (uint32) arg_length; > } > + } > } > diff --git a/sql/sql_string.h b/sql/sql_string.h > index 2ef817ea0adc..01d5211fe850 100644 > --- a/sql/sql_string.h > +++ b/sql/sql_string.h > @@ -599,25 +599,34 @@ class Binary_string: public Static_binary_string > > inline char *c_ptr() > { > - DBUG_ASSERT(!alloced || !Ptr || !Alloced_length || > - (Alloced_length >= (str_length + 1))); > - > - if (!Ptr || Ptr[str_length]) // Should be safe > - (void) realloc(str_length); > + if (unlikely(!Ptr)) > + return (char*) ""; > + /* > + Here we assume that any buffer used to initalize String has > + an end \0 or have at least an accessable character at end. > + This is to handle the case of String("Hello",5) efficently. > + */ > + if (unlikely(!alloced && !Ptr[str_length])) No, this is wrong. Note the difference between String a("Hello", 5) and char hello[5]; String a(buf, 5); Your assumption should only apply to the first case, not to the second. In the first case alloced==Alloced_length==0, in the second case only alloced==0 and Alloced_length==5. So in the if() above you need to look at Alloced_length. > + return Ptr; > + if (str_length < Alloced_length) > + { > + Ptr[str_length]=0; > + return Ptr; > + } > + (void) realloc(str_length+1); /* This will add end \0 */ > return Ptr; > } > + /* Use c_ptr() instead. This will be deleted soon, kept for compatiblity */ > inline char *c_ptr_quick() > { > - if (Ptr && str_length < Alloced_length) > - Ptr[str_length]=0; > - return Ptr; > + return c_ptr_safe(); > } > inline char *c_ptr_safe() > { > if (Ptr && str_length < Alloced_length) > Ptr[str_length]=0; > else > - (void) realloc(str_length); > + (void) realloc(str_length + 1); what's the difference between c_ptr_safe and c_ptr, again? they're very similar, so they'd need a good comment explaining when to use what. > return Ptr; > } > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 994ea2af3973: Optimize size of lex structures
by Sergei Golubchik 04 Dec '20

04 Dec '20

Hi, Michael! On Dec 03, Michael Widenius wrote: > revision-id: 994ea2af3973 (mariadb-10.5.2-269-g994ea2af3973) > parent(s): 77500a1a1107 > author: Michael Widenius <michael.widenius(a)gmail.com> > committer: Michael Widenius <michael.widenius(a)gmail.com> > timestamp: 2020-09-17 12:24:58 +0300 > message: > > Optimize size of lex structures > > LEX, st_select_lex, st_select_unit optimized for space: > - Use bit fields for bool variables > - Ensure that all bit fields are initialized (improves > performance for init functions as all bit fields can be > initalized with one memory access) > - Move members around in above structures to remove alignment > gaps I did not get any reply to my email From: Sergei Golubchik <serg(a)mariadb.org> To: Michael Widenius <michael.widenius(a)gmail.com> Cc: maria-developers(a)lists.launchpad.net Subject: Re: c3ecf0d6243: Optimize size of lex structures Date: Wed, 9 Sep 2020 15:57:23 +0200 and I don't see any related changes in this commit. Please, do at least one of the two. Regards, Sergei

1 0

Re: [Maria-developers] 77500a1a1107: Added typedef decimal_digits_t (uint16) for number of decimals
by Sergei Golubchik 04 Dec '20

04 Dec '20

Hi, Michael! I did not get any reply to my email From: Sergei Golubchik <serg(a)mariadb.org> To: Michael Widenius <michael.widenius(a)gmail.com> Cc: maria-developers(a)lists.launchpad.net Subject: Re: c79c6f78205: Added typedef decimal_digits_t (uint16) for number of decimals Date: Wed, 9 Sep 2020 13:34:57 +0200 and I don't see any related changes in this commit. Please, do at least one of the two. Regards, Sergei On Dec 03, Michael Widenius wrote: > revision-id: 77500a1a1107 (mariadb-10.5.2-268-g77500a1a1107) > parent(s): 32104b969513 > author: Michael Widenius <michael.widenius(a)gmail.com> > committer: Michael Widenius <michael.widenius(a)gmail.com> > timestamp: 2020-09-17 12:24:58 +0300 > message: > > Added typedef decimal_digits_t (uint16) for number of decimals > > For fields and Item's uint8 should be good enough. After > discussions with Alexander Barkov we choose uint16 (for now) > as some format functions may accept +256 digits. > > The reason for this patch was to make the storage of decimal > digits simlar. Before this patch decimals was stored/used as > uint8, int and uint. > > Changed most decimal variables and functions to use the new > typedef. >

1 0

Re: [Maria-developers] 32104b969513: MDEV-23001 Precreate static Item_bool() to simplify code
by Sergei Golubchik 04 Dec '20

04 Dec '20

Hi, Michael! I did not get any reply to my email From: Sergei Golubchik <serg(a)mariadb.org> To: Michael Widenius <michael.widenius(a)gmail.com> Cc: maria-developers(a)lists.launchpad.net Subject: Re: bf8bd057344: MDEV-23001 Precreate static Item_bool() to simplify code Date: Tue, 8 Sep 2020 20:39:35 +0200 and I don't see any related changes in this commit. Please, do at least one of the two. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org On Dec 03, Michael Widenius wrote: > revision-id: 32104b969513 (mariadb-10.5.2-267-g32104b969513) > parent(s): 32a29afea777 > author: Michael Widenius <michael.widenius(a)gmail.com> > committer: Michael Widenius <michael.widenius(a)gmail.com> > timestamp: 2020-09-17 12:24:45 +0300 > message: > > MDEV-23001 Precreate static Item_bool() to simplify code > > The following changes where done: > - Create global Item: Item_false and Item_true > - Replace all creation if 'FALSE' and 'TRUE' top level items used for > WHERE/HAVING/ON clauses to use Item_false and Item_true. > > The benefit are: > - Less and faster code > - No test needed if we where able to create the new item. > - Fixed possible errors if 'new' would have failed for the Item_bool's >

1 0

[Maria-developers] MDEV-21287 dev community opinion wanted
by Pierre-Jean Clement 02 Dec '20

02 Dec '20

Hi all, I created the Jira ticket MDEV-21287 10 months ago and didn't get much feedback. I tried to take a shot at it myself, but found out that the resolution MDEV-20403 introduced this behavior by purposefully evaluating 'update functions' before the BEFORE UPDATE trigger. I couldn't really understand why this was necessary, and I'd like to get feedback from the dev community to try and understand what is really supposed to happen on update. I kinda hate getting my trigger called with a new value that's not going to be persisted anyway, what do you think about that? Regards, Pierre-Jean

2 3

Re: [Maria-developers] MDEV-21265: IN predicate conversion to IN subquery should be allowed for a broader set of datatype comparison
by Sergey Petrunia 26 Nov '20

26 Nov '20

Hi Varun, Please find some cosmetic input below. Ok to push after addresed. > diff --git a/mysql-test/main/subselect4.test b/mysql-test/main/subselect4.test > index 6f5eb1f2985f..e9da028618f6 100644 > --- a/mysql-test/main/subselect4.test > +++ b/mysql-test/main/subselect4.test > @@ -2238,3 +2238,38 @@ SET join_cache_level= @save_join_cache_level; > DROP TABLE t1,t2,t3,t4; > > --echo # End of 10.2 tests > + > +--echo # > +--echo # MDEV-21265: IN predicate conversion to IN subquery should be allowed for a broader set of datatype comparison > +--echo # > + > +CREATE TABLE t1(a VARCHAR(50) collate utf8_general_ci, b INT); > +INSERT INTO t1 VALUES ('abc',1), ('def', 2), ('ghi', 3), ('jkl', 4), ('mno', 5); > + > +CREATE TABLE t2(a VARCHAR(50) collate utf8mb4_general_ci, b INT); > +INSERT INTO t2 VALUES ('abc',1), ('def', 2), ('ghi', 3), ('jkl', 4), ('mno', 5); > + > +set in_predicate_conversion_threshold=2; > + ... > +EXPLAIN > +SELECT * FROM t2 WHERE (t2.a,t2.b) IN (('abx',1),('def',2), ('abc', 3)); > + > +set names default; > +DROP TABLE t1,t2; > + > +--echo # End of 10.3 tests Please restore the value of in_predicate_conversion_threshold. > diff --git a/sql/item_subselect.h b/sql/item_subselect.h > index 1cea7291c9ef..d1b2f86088d9 100644 > --- a/sql/item_subselect.h > +++ b/sql/item_subselect.h > @@ -573,6 +573,13 @@ class Item_in_subselect :public Item_exists_subselect > TRUE<=>registered in the list of semijoins in outer select > */ > bool is_registered_semijoin; > + > + Item_func_not_all *upper_item; // point on NOT/NOP before ALL/SOME subquery Is this necessary to touch the above member? I think it's better leave it as is. > + > + /* > + SET to TRUE if IN subququery is converted from an IN rredicate > + */ > + bool converted_from_in_predicate; ... > diff --git a/sql/sql_type.h b/sql/sql_type.h > index 907225b7c505..a1403165fe38 100644 > --- a/sql/sql_type.h > +++ b/sql/sql_type.h > @@ -1291,9 +1291,21 @@ class Type_handler > Item *target_expr, Item *target_value, > Item_bool_func2 *source, > Item *source_expr, Item *source_const) const= 0; > + > + /* > + @brief > + Check if an IN subquery allows materialization or not > + @param > + inner expression on the inner side of the IN subquery > + outer expression on the inner side of the IN subquery Typo. Should it should be 'outer' ? > + is_in_predicate SET to true if IN subquery was converted from an > + IN predicate or we are checking if materialization > + strategy can be used for an IN predicate > + */ > virtual bool > subquery_type_allows_materialization(const Item *inner, > - const Item *outer) const= 0; > + const Item *outer, > + bool is_in_predicate) const= 0; > /** > Make a simple constant replacement item for a constant "src", > so the new item can futher be used for comparison with "cmp", e.g.: BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] Question about JSONPath and '**' wildcard
by Sergey Petrunia 20 Nov '20

20 Nov '20

Hi Alexey, I've found this discrepancy in JSONPath evaluation: set @json_doc3 = ' { "root": { "child1" : { "child2" : { "child1" : { "x":124 } } } } } '; select json_extract(@json_doc3, '$**.child1**.x'); MariaDB [test]> select json_extract(@json_doc3, '$**.child1**.x'); +--------------------------------------------+ | json_extract(@json_doc3, '$**.child1**.x') | +--------------------------------------------+ | NULL | +--------------------------------------------+ 1 row in set (0.001 sec) MySQL-8> select json_extract(@json_doc3, '$**.child1**.x'); +--------------------------------------------+ | json_extract(@json_doc3, '$**.child1**.x') | +--------------------------------------------+ | [124] | +--------------------------------------------+ 1 row in set (0.00 sec) Which one is right? My opinion is that MariaDB's answer is definitely incorrect. As for MySQL's answer, it depends on how the result of JSONPath expression is defined. If it is "a set of nodes in the JSON document which match the pattern", MySQL's result is correct. Coming from XPath world, I was expecting a semantics in form find the set of nodes matching the search step#1 ( $**, then apply step #2 ( .child1) then apply step #3 ( any children of those) ... in which case "124" would be in the result twice as it is reachable via two possible paths. Any thoughts about this? BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

2 2

Re: [Maria-developers] 255e5f269d80: MDEV-23610: Slave user can't run "SHOW SLAVE STATUS" anymore after upgrade to 10.5, mysql_upgrade should take of that
by Sergei Golubchik 13 Nov '20

13 Nov '20

Hi, Sujatha! Looks good, thanks! One detail to fix - I think you've mistakenly committed an old test from the previous version of the patch. And two minor suggestions, you can change or not, as you like. otherwise ok to push On Nov 13, Sujatha wrote: > revision-id: 255e5f269d80 (mariadb-10.5.4-319-g255e5f269d80) > parent(s): 97569d3c37a6 > author: Sujatha <sujatha.sivakumar(a)mariadb.com> > committer: Sujatha <sujatha.sivakumar(a)mariadb.com> > timestamp: 2020-11-13 19:08:02 +0530 > message: > > MDEV-23610: Slave user can't run "SHOW SLAVE STATUS" anymore after upgrade to 10.5, mysql_upgrade should take of that > > Add a new privilege "SLAVE MONITOR" which will grant user the permission > to execute "SHOW SLAVE STATUS" and "SHOW RELAYLOG EVENTS" commands. > > SHOW SLAVE STATUS requires either SLAVE MONITOR/SUPER > SHOW RELAYLOG EVENTS requires SLAVE MONITOR privilege. > > diff --git a/mysql-test/main/grant_slave_monitor.test b/mysql-test/main/grant_slave_monitor.test > new file mode 100644 > index 000000000000..a4f21085e537 > --- /dev/null > +++ b/mysql-test/main/grant_slave_monitor.test > @@ -0,0 +1,101 @@ > +# ==== Purpose ==== > +# > +# SLAVE MONITOR privilege is required to execute following commands. > +# SHOW SLAVE STATUS > +# SHOW RELAYLOG EVENTS > +# > +# ==== Implementation ==== > +# > +# Step1: GRANT ALL privileges for a new user 'user1' and then REVOKE > +# SLAVE MONITOR and SUPER privileges. > +# Step2: Execute SHOW SLAVE STAUTS/SHOW RELAYLOG EVENTS commands and expect > +# ER_SPECIFIC_ACCESS_DENIED_ERROR. This also verifies that REPLICATION > +# SLAVE ADMIN privilege is not required for these two commands. > +# Step3: GRANT SLAVE MONITOR privilege and observe that both commands are > +# allowd to execute. > +# Step4: GRANT SUPER privilege and observe that only SHOW SLAVE STATUS command > +# is allowed. > +# > +# ==== References ==== > +# > +# MDEV-23610: Slave user can't run "SHOW SLAVE STATUS" anymore after upgrade > +# to 10.5, mysql_upgrade should take of that > +# MDEV-23918: admin privlege required to view contents of relay logs in 10.5 > +# > + > +--source include/not_embedded.inc > + > +CREATE USER user1@localhost IDENTIFIED BY ''; > +GRANT ALL PRIVILEGES ON *.* TO user1@localhost; > +REVOKE SLAVE MONITOR, SUPER ON *.* FROM user1@localhost; > +FLUSH PRIVILEGES; > + > +--connect(con1,localhost,user1,,) > +--connection con1 > +SHOW GRANTS; > + > +--echo # > +--echo # Verify that having REPLICATION SLAVE ADMIN doesn't allow SHOW SLAVE STATUS > +--echo # Expected error: Access denied; you need (at least one of) the SUPER, SLAVE > +--echo # MONITOR privilege(s) for this operation > +--echo # > +--error ER_SPECIFIC_ACCESS_DENIED_ERROR > +SHOW SLAVE STATUS; > + > +--echo # > +--echo # Verify that having REPLICATION SLAVE ADMIN doesn't allow SHOW RELAYLOG EVENTS > +--echo # Expected error: Access denied; you need (at least one of) the REPLICA MONITOR > +--echo # privilege(s) for this operation > +--echo # > +--disable_ps_protocol > +--error ER_SPECIFIC_ACCESS_DENIED_ERROR > +SHOW RELAYLOG EVENTS; > +--enable_ps_protocol > +--disconnect con1 > + > +--echo # > +--echo # SHOW SLAVE STATUS and SHOW RELAYLOG EVENTS are allowed with SLAVE MONITOR privilege > +--echo # > + > +--connection default > +GRANT SLAVE MONITOR ON *.* TO user1@localhost; > +FLUSH PRIVILEGES; > + > +--connect(con1,localhost,user1,,) > +--connection con1 > +SHOW GRANTS; > + > +--disable_result_log > +SHOW SLAVE STATUS; > +#--disable_ps_protocol > +SHOW RELAYLOG EVENTS; > +#--enable_ps_protocol > +--enable_result_log > +--disconnect con1 > + > +--connection default > +DROP USER user1@localhost; > + > +--echo # > +--echo # SHOW SLAVE STATUS command is allowed with SUPER privilege > +--echo # > +CREATE USER user1@localhost IDENTIFIED BY ''; > +GRANT SUPER ON *.* TO user1@localhost; > + > +--connect(con1,localhost,user1,,) > +--disable_result_log > +SHOW SLAVE STATUS; > +--enable_result_log > + > +--echo # > +--echo # SHOW RELAYLOG EVENTS is not allowed with SUPER privilege, it requires SLAVE MONITOR > +--echo # > + > +--disable_ps_protocol > +--error ER_SPECIFIC_ACCESS_DENIED_ERROR > +SHOW RELAYLOG EVENTS; > +--enable_ps_protocol > +--disconnect con1 > + > +--connection default > +DROP USER user1@localhost; > diff --git a/mysql-test/main/rpl_replica_monitor_priv.test b/mysql-test/main/rpl_replica_monitor_priv.test > --- /dev/null > +++ b/mysql-test/main/rpl_replica_monitor_priv.test > @@ -0,0 +1,90 @@ Did you check in this file by mistake? It's a replication test (source include/master-slave.inc) but not in the rpl suite. It tests the feature that does not need master-slave.inc. And it's duplicated by grant_slave_monitor.test > diff --git a/sql/privilege.h b/sql/privilege.h > index 37cdf4da01ad..d0e6e532b4b5 100644 > --- a/sql/privilege.h > +++ b/sql/privilege.h > @@ -93,13 +94,18 @@ enum privilege_t: unsigned long long > */ > }; > > +constexpr static inline privilege_t ALL_KNOWN_BITS(privilege_t x) > +{ > + return (privilege_t)((x << 1) - 1); Generally it's safer to write x | (x - 1) it won't overflow if x is already the highest possible bit. In privileges we're quite far from it at the moment, so it's not a big deal, just a style issue and setting a good example to others. > +} > > // Version markers > constexpr privilege_t LAST_100304_ACL= DELETE_HISTORY_ACL; > constexpr privilege_t LAST_100502_ACL= BINLOG_REPLAY_ACL; > +constexpr privilege_t LAST_100509_ACL= SLAVE_MONITOR_ACL; > > diff --git a/sql/sql_acl.cc b/sql/sql_acl.cc > index ef96e8f24843..b5b1472f004f 100644 > --- a/sql/sql_acl.cc > +++ b/sql/sql_acl.cc > @@ -1015,7 +1015,7 @@ class User_table_tabular: public User_table > access|= LOCK_TABLES_ACL | CREATE_TMP_ACL | SHOW_DB_ACL; > if (access & FILE_ACL) > access|= BINLOG_MONITOR_ACL | REPL_SLAVE_ACL | BINLOG_ADMIN_ACL | > - BINLOG_REPLAY_ACL; > + BINLOG_REPLAY_ACL | SLAVE_MONITOR_ACL; this does not hurt, but it's rather redundant, you set SLAVE_MONITOR_ACL below anyway. > if (access & PROCESS_ACL) > access|= SUPER_ACL | EXECUTE_ACL; > } > @@ -1057,6 +1057,9 @@ class User_table_tabular: public User_table > if (access & REPL_SLAVE_ACL) > access|= REPL_MASTER_ADMIN_ACL; > > + if (access & REPL_SLAVE_ACL) > + access|= SLAVE_MONITOR_ACL; > + > return access & GLOBAL_ACLS; > } > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] d66d5457589d: MDEV-23610: Slave user can't run "SHOW SLAVE STATUS" anymore after upgrade to 10.5, mysql_upgrade should take of that
by Sergei Golubchik 12 Nov '20

12 Nov '20

Hi, Sujatha! On Nov 12, Sujatha wrote: > revision-id: d66d5457589d (mariadb-10.5.4-315-gd66d5457589d) > parent(s): 10b2d5726fa2 > author: Sujatha <sujatha.sivakumar(a)mariadb.com> > committer: Sujatha <sujatha.sivakumar(a)mariadb.com> > timestamp: 2020-11-11 18:41:44 +0530 > message: > > MDEV-23610: Slave user can't run "SHOW SLAVE STATUS" anymore after upgrade to 10.5, mysql_upgrade should take of that > > Add a new privilege "REPLICA MONITOR" which will grant user the permission > to execute "SHOW SLAVE STATUS" and "SHOW RELAYLOG EVENTS" commands. > > SHOW SLAVE STATUS requires either REPLICA MONITOR/SUPER > SHOW RELAYLOG EVENTS requires REPLICA MONITOR privilege. > > diff --git a/mysql-test/main/grant.result b/mysql-test/main/grant.result > index 01daf0271867..a41b38aaa0ba 100644 > --- a/mysql-test/main/grant.result > +++ b/mysql-test/main/grant.result > @@ -1966,10 +1967,11 @@ GRANT REPLICATION SLAVE ON *.* TO mysqltest_u1@localhost; > GRANT SHOW DATABASES ON *.* TO mysqltest_u1@localhost; > GRANT SHUTDOWN ON *.* TO mysqltest_u1@localhost; > GRANT USAGE ON *.* TO mysqltest_u1@localhost; > +GRANT REPLICA MONITOR ON *.* TO mysqltest_u1@localhost; I don't think we should have a mix of REPLICA and SLAVE terms. Call it SLAVE MONITOR and when we flip the switch on grants, we'll do it for all grants at the same time. > SHOW GRANTS FOR mysqltest_u1@localhost; > Grants for mysqltest_u1@localhost > -GRANT RELOAD, SHUTDOWN, PROCESS, FILE, SHOW DATABASES, REPLICATION SLAVE, BINLOG MONITOR, CREATE USER ON *.* TO `mysqltest_u1`@`localhost` > +GRANT RELOAD, SHUTDOWN, PROCESS, FILE, SHOW DATABASES, REPLICATION SLAVE, BINLOG MONITOR, CREATE USER, REPLICA MONITOR ON *.* TO `mysqltest_u1`@`localhost` > GRANT CREATE TEMPORARY TABLES, LOCK TABLES, EXECUTE, CREATE ROUTINE, ALTER ROUTINE, EVENT ON `mysqltest_db1`.* TO `mysqltest_u1`@`localhost` > connect con1,localhost,mysqltest_u1,,mysqltest_db1; > connection con1; > diff --git a/mysql-test/main/grant.test b/mysql-test/main/grant.test > index 38baf6738255..63be18ecb292 100644 > --- a/mysql-test/main/grant.test > +++ b/mysql-test/main/grant.test > @@ -1833,6 +1833,7 @@ GRANT REPLICATION SLAVE ON *.* TO mysqltest_u1@localhost; > GRANT SHOW DATABASES ON *.* TO mysqltest_u1@localhost; > GRANT SHUTDOWN ON *.* TO mysqltest_u1@localhost; > GRANT USAGE ON *.* TO mysqltest_u1@localhost; > +GRANT REPLICA MONITOR ON *.* TO mysqltest_u1@localhost; But the parser should, of course, accept both, as everywhere. > diff --git a/mysql-test/suite/rpl/t/rpl_replica_monitor_priv.test b/mysql-test/suite/rpl/t/rpl_replica_monitor_priv.test > new file mode 100644 > index 000000000000..f7b2af2e0c51 > --- /dev/null > +++ b/mysql-test/suite/rpl/t/rpl_replica_monitor_priv.test > @@ -0,0 +1,90 @@ > +# ==== Purpose ==== > +# > +# REPLICA MONITOR privilege is required to execute following commands. > +# SHOW SLAVE STATUS > +# SHOW RELAYLOG EVENTS > +# > +# ==== Implementation ==== > +# > +# Step1: GRANT ALL privileges for a new user 'user1' and then REVOKE REPLICA > +# MONITOR and SUPER privileges. > +# Step2: Execute SHOW SLAVE STAUTS/SHOW RELAYLOG commands and expect > +# ER_SPECIFIC_ACCESS_DENIED_ERROR. This also verifies that REPLICATION > +# SLAVE ADMIN privilege is not required for these two commands. > +# Step3: GRANT REPLICA MONITOR privilege and observe that both commands are > +# allowd to execute. > +# Step4: GRANT SUPER privilege and observe that only SHOW SLAVE STATUS command > +# is allowed. why do you need a full master-slave replication setup for that? > +# ==== References ==== > +# > +# MDEV-23610: Slave user can't run "SHOW SLAVE STATUS" anymore after upgrade > +# to 10.5, mysql_upgrade should take of that > +# MDEV-23918: admin privlege required to view contents of relay logs in 10.5 > +# > + > +--source include/not_embedded.inc > +--source include/have_binlog_format_mixed.inc > +--source include/master-slave.inc > + > +--connection master > +CREATE USER user1@localhost IDENTIFIED BY ''; > +GRANT ALL PRIVILEGES ON *.* TO user1@localhost; > +REVOKE REPLICA MONITOR, SUPER ON *.* FROM user1@localhost; > +FLUSH PRIVILEGES; > +--sync_slave_with_master > +--connect(con1,127.0.0.1,user1,,test,$SLAVE_MYPORT) > +--connection con1 > +SELECT CURRENT_USER(); > +SHOW GRANTS; > +--echo "Verify that having REPLICATION SLAVE ADMIN doesn't allow SHOW SLAVE STATUS" > +--echo "Expected error:Access denied; you need (at least one of) the SUPER, REPLICA MONITOR privilege(s) for this operation" better use not quotes, but #, like elsewhere. > +--error ER_SPECIFIC_ACCESS_DENIED_ERROR > +SHOW SLAVE STATUS; > +--echo "Verify that having REPLICATION SLAVE ADMIN doesn't allow SHOW RELAYLOG EVENTS" > +--echo "Expected error: Access denied; you need (at least one of) the REPLICA MONITOR privilege(s) for this operation" > +--disable_ps_protocol why? > +--error ER_SPECIFIC_ACCESS_DENIED_ERROR > +SHOW RELAYLOG EVENTS; > +--enable_ps_protocol > +--disconnect con1 > + > diff --git a/sql/privilege.h b/sql/privilege.h > index 37cdf4da01ad..e7686a206b84 100644 > --- a/sql/privilege.h > +++ b/sql/privilege.h > @@ -115,8 +117,12 @@ constexpr privilege_t ALL_KNOWN_ACL_100304 = > constexpr privilege_t ALL_KNOWN_ACL_100502= > (privilege_t) ((LAST_100502_ACL << 1) - 1); > > +// A combination of all bits defined in 10.5.8 > +constexpr privilege_t ALL_KNOWN_ACL_100508= > + (privilege_t) ((LAST_100508_ACL << 1) - 1); > + > // A combination of all bits defined as of the current version > -constexpr privilege_t ALL_KNOWN_ACL= ALL_KNOWN_ACL_100502; > +constexpr privilege_t ALL_KNOWN_ACL= ALL_KNOWN_ACL_100508; may be (privilege_t) ((LAST_CURRENT_ACL << 1) - 1) and then we won't need to change it anymore? Also, I think this deserves a function, like static inline privilege_t ALL_ACLS_TO(privilege_t x) { return (x << 1) - 1; } or, rather, a bit safer return x | (x - 1) > // Unary operators > @@ -505,9 +511,11 @@ constexpr privilege_t PRIV_STMT_STOP_SLAVE= REPL_SLAVE_ADMIN_ACL | SUPER_ACL; > // Was SUPER_ACL prior to 10.5.2 > constexpr privilege_t PRIV_STMT_CHANGE_MASTER= REPL_SLAVE_ADMIN_ACL | SUPER_ACL; > // Was (SUPER_ACL | REPL_CLIENT_ACL) prior to 10.5.2 > -constexpr privilege_t PRIV_STMT_SHOW_SLAVE_STATUS= REPL_SLAVE_ADMIN_ACL | SUPER_ACL; > +// Was REPL_SLAVE_ADMIN_ACL from 10.5.2 to 10.5.7 // Was SUPER_ACL | REPL_SLAVE_ADMIN_ACL from 10.5.2 to 10.5.7 otherwise it's unclear why you have SUPER_ACL below and not just REPLICA_MONITOR_ACL > +constexpr privilege_t PRIV_STMT_SHOW_SLAVE_STATUS= REPLICA_MONITOR_ACL | SUPER_ACL; > // Was REPL_SLAVE_ACL prior to 10.5.2 > -constexpr privilege_t PRIV_STMT_SHOW_RELAYLOG_EVENTS= REPL_SLAVE_ADMIN_ACL; > +// Was REPL_SLAVE_ADMIN_ACL from 10.5.2 to 10.5.7 > +constexpr privilege_t PRIV_STMT_SHOW_RELAYLOG_EVENTS= REPLICA_MONITOR_ACL; > > /* > Privileges related to binlog replying. > diff --git a/sql/sql_acl.cc b/sql/sql_acl.cc > index ef96e8f24843..8a3551c14f53 100644 > --- a/sql/sql_acl.cc > +++ b/sql/sql_acl.cc > @@ -1057,6 +1057,9 @@ class User_table_tabular: public User_table > if (access & REPL_SLAVE_ACL) > access|= REPL_MASTER_ADMIN_ACL; > > + if (access & FILE_ACL) > + access|= REPLICA_MONITOR_ACL; Why is that? > + > return access & GLOBAL_ACLS; > } > > @@ -1541,19 +1548,22 @@ class User_table_json: public User_table > */ > if (access & SUPER_ACL) > { > - if (access & REPL_SLAVE_ACL) > - { > - /* > - The user could do both before the upgrade: > - - set global variables (because of SUPER_ACL) > - - execute "SHOW SLAVE HOSTS" (because of REPL_SLAVE_ACL) > - Grant all new privileges that were splitted from SUPER (in 10.5.2), > - and REPLICATION MASTER ADMIN, so it still can do "SHOW SLAVE HOSTS". > - */ > - access|= REPL_MASTER_ADMIN_ACL; > - } > access|= GLOBAL_SUPER_ADDED_SINCE_USER_TABLE_ACLS; > } > + if (access & REPL_SLAVE_ACL) > + { > + /* > + The user could do both before the upgrade: > + - set global variables (because of SUPER_ACL) this looked fine where it was before, but not anymore after you moved it out of `if (access & SUPER_ACL)` > + - execute "SHOW SLAVE HOSTS" (because of REPL_SLAVE_ACL) > + Grant all new privileges that were splitted from SUPER (in 10.5.2), > + and REPLICATION MASTER ADMIN, so it still can do "SHOW SLAVE HOSTS". > + - execute "SHOW RELAYLOG EVENTS" (because of REPL_SLAVE_ACL) > + */ > + access|= (REPL_MASTER_ADMIN_ACL | REPLICA_MONITOR_ACL); and REPL_MASTER_ADMIN_ACL allows to modify global variables was was earlier possible only with SUPER_ACL. So, indeed, old code was correct, that if() should be inside SUPER_ACL, one should only get REPL_MASTER_ADMIN_ACL if one had both SUPER_ACL and REPL_SLAVE_ACL. > + } > + if (access & BINLOG_MONITOR_ACL) > + access|= REPLICA_MONITOR_ACL; > } > > if (orig_access & ~mask) > @@ -8974,7 +8984,7 @@ static const char *command_array[]= > "CREATE USER", "EVENT", "TRIGGER", "CREATE TABLESPACE", "DELETE HISTORY", > "SET USER", "FEDERATED ADMIN", "CONNECTION ADMIN", "READ_ONLY ADMIN", > "REPLICATION SLAVE ADMIN", "REPLICATION MASTER ADMIN", "BINLOG ADMIN", > - "BINLOG REPLAY" > + "BINLOG REPLAY", "REPLICA MONITOR" SLAVE MONITOR > }; > > static uint command_lengths[]= Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 2

Re: [Maria-developers] e3f45b2f9ea: MDEV-10267 Add ngram fulltext parser plugin
by Sergei Golubchik 11 Nov '20

11 Nov '20

Hi, Rinat! On Nov 02, Rinat Ibragimov wrote: > > > I can't decide. From my point of view, the current approach is > > > fine. Please pick a variant, and I'll try to implement that. > > > > No, I cannot guess which approach will produce more relevant > > searches. Implement something and then we test what works better > > Variable-length n-grams approach is too innovative, and hard to reason > about. I've never heard about such an approach, and it doesn't look > good to me. So I'll stick with a simple slicer. If you mean that variant where it splits "n-grams approach" to "n-gr", "gra", "ram", "ams", "ms a", "s ap", "app", ... then it's just "n letters in every chunk" very easy to explain. But ok, let's start simple and benchmark. > > Of course, it can. Note that fts_get_word() doesn't generate n-grams > > either, it gets the whole word and the n-gram plugin later splits it > > into n-grams. Similarly param->mysql_parse() will extract words for > > you and you'll split them into n-grams. > > Changed to use param->mysql_parse(). > > Turns out that in Aria, MyISAM, and InnoDB, param->mysql_parser() does > call back param->mysql_add_word(). Is it part of the plugin API? Yes, it is. E.g. slide 12 from my old presentation: http://conferences.oreillynet.com/presentations/mysql06/golubchik_sergei.pdf shows that there are three points where a plugin can add functionality. * It can extract the text and then call param->mysql_parser(), this allows to parse, say, gzip-ed texts or EXIF comments in images. * It can replace param->mysql_parser(), to use different rules for spliting the text into words. This is what the n-gram plugin normally does * It can replace param->mysql_add_word() to post-process every word after the built-in parser did the splitting. For example, stemming or soundex plugin can do that. > Comments in include/mysql/plugin_ftparser.h do not mention that at > all. That's why I initially thought that param->mysql_parse() will > parse the string like the default parser do, without any ways to > interact with the process. I've edited the comment to mention this possibility. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 0cd5377e9ff: MDEV-18734 ASAN heap-use-after-free in my_strnxfrm_simple_internal upon update on versioned partitioned table
by Sergei Golubchik 02 Nov '20

02 Nov '20

Hi, Aleksey! On Oct 20, Aleksey Midenkov wrote: > revision-id: 0cd5377e9ff (mariadb-10.2.31-357-g0cd5377e9ff) > parent(s): dc716da4571 > author: Aleksey Midenkov <midenok(a)gmail.com> > committer: Aleksey Midenkov <midenok(a)gmail.com> > timestamp: 2020-08-17 00:52:35 +0300 > message: > > MDEV-18734 ASAN heap-use-after-free in my_strnxfrm_simple_internal upon update on versioned partitioned table > > update_virtual_fields() may reallocate Field_blob::value buffer due to > different reasons (f.ex. different types, see > Field_str::memcpy_field_possible()). But the buffer address may be > already in m_ordered_rec_buffer which is then used in ordered index > scan queue. > > The patch updates blob virtual fields when the ordered index scan > queue is sorted as well as the top record is restored from the queue. > > New VCOL_UPDATE_BLOBS mode is introduced to update only blobs. Swap is > not needed as it was already done if it was needed. > > Related to ea1b2504 No, I think we need to go deeper here. The bug is not only not caused by system versioning, it is also not caused by virtual columns. The problem is that partitioning copies record[0] out and then back to save and restore values. This does not work for blobs, because record[0] may store the pointer to the memory that Field_blob owns and can free or reallocate as needed. Normally it's not the problem because typically after row reads the engine owns the memory, not Field_blob. When a virtual column is set to a calculated value - then Field_blob owns the memory and we have this bug. But there are engines that use field->store() to set values - in this case Field_blob will own the memory too. Here's a test case with FederatedX: ============================================================= source include/have_partition.inc; source include/have_sequence.inc; source have_federatedx.inc; source include/federated.inc; connection slave; use federated; create table t1_1 (x int, b text, key(x)); create table t1_2 (x int, b text, key(x)); connection master; eval create table t1 (x int, b text, key(x)) engine=federated partition by range columns (x) ( partition p1 values less than (40) connection='mysql://root@127.0.0.1:$SLAVE_MYPORT/federated/t1_1', partition pn values less than (maxvalue) connection='mysql://root@127.0.0.1:$SLAVE_MYPORT/federated/t1_2' ); insert t1 select seq, seq from seq_1_to_35; insert t1 select seq, repeat(seq, 100) from seq_50_to_90; flush tables; select x, b from t1 where x > 30 and x < 60 order by x; select x, b from t1 where x > 30 and x < 60 order by b; drop table t1; connection slave; drop table t1_1, t1_2; source include/federated_cleanup.inc; ============================================================= first select doesn't trigger ASAN error, but it produces incorrect results (I'm showing it here, so that you could create a non-ASAN test that could be run on all builders). the second select gets the same ASAN error as your test below. > diff --git a/mysql-test/suite/vcol/r/partition.result b/mysql-test/suite/vcol/r/partition.result > index bd1353fa145..74c3c3394dc 100644 > --- a/mysql-test/suite/vcol/r/partition.result > +++ b/mysql-test/suite/vcol/r/partition.result > @@ -28,3 +28,26 @@ set statement sql_mode= '' for update t1 set i= 1, v= 2; > Warnings: > Warning 1906 The value specified for generated column 'v' in table 't1' has been ignored > drop table t1; > +# > +# MDEV-18734 ASAN heap-use-after-free in my_strnxfrm_simple_internal upon update on versioned partitioned table > +# > +# Cover queue_fix() in ha_partition::handle_ordered_index_scan() > +create table t1 ( > +x int auto_increment primary key, > +b text, v mediumtext as (b) virtual, > +index (v(10)) > +) partition by range columns (x) ( > +partition p1 values less than (3), > +partition p2 values less than (6), > +partition pn values less than (maxvalue)); > +insert into t1 (b) values ('q'), ('a'), ('b'); > +update t1 set b= 'bar' where v > 'a'; > +drop table t1; this test above doesn't cause ASAN failures for me without your fix. the test below fails all right. > +# Cover return_top_record() in ha_partition::handle_ordered_index_scan() > +create table t1 (x int primary key, b tinytext, v text as (b) virtual) > +partition by range columns (x) ( > +partition p1 values less than (4), > +partition pn values less than (maxvalue)); > +insert into t1 (x, b) values (1, ''), (2, ''), (3, 'a'), (4, 'b'); > +update t1 set b= 'bar' where x > 0 order by v limit 2; > +drop table t1; Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 3

Re: [Maria-developers] 7807d9b6939: MDEV-22974: mysql_native_password make "invalid" valid
by Sergei Golubchik 30 Oct '20

30 Oct '20

Hi, Daniel! On Oct 22, Daniel Black wrote: > revision-id: 7807d9b6939 (mariadb-10.5.4-62-g7807d9b6939) > parent(s): 054f10365c4 > author: Daniel Black <daniel(a)mariadb.org> > committer: Daniel Black <daniel(a)mariadb.org> > timestamp: 2020-07-16 17:06:36 +1000 > message: > > MDEV-22974: mysql_native_password make "invalid" valid > > Per b9f3f06857ac, mysql_system_tables_data.sql creates > a mysql_native_password with a salted hash of "invalid" so that `set password` > will detect a native password can be pplied:. "applied" > > SHOW CREATE USER; dilignently uses this value in its output > generating the SQL: > > MariaDB [(none)]> show create user; > > +---------------------------------------------------------------------------------------------------+ > | CREATE USER for dan@localhost | > +---------------------------------------------------------------------------------------------------+ > | CREATE USER `dan`@`localhost` IDENTIFIED VIA mysql_native_password USING 'invalid' OR unix_socket | > +---------------------------------------------------------------------------------------------------+ > > Attempting to execute this before this patch resutls in: > > MariaDB [(none)]> CREATE USER `dan2`@`localhost` IDENTIFIED VIA mysql_native_password USING 'invalid' OR unix_socket; > ERROR 1372 (HY000): Password hash should be a 41-digit hexadecimal number > > As such deep the the implementation of mysql_native_password we make > "invalid" valid (pun intended) such that the above create user will > succeed. > > native_password_get_salt is only called in the context of > set_user_salt, so all setting of native passwords to hashed content of > 'invalid', quite literally create an invalid password. > > So other forms of "invalid" are valid SQL in creating invalid passwords: > > MariaDB [(none)]> set password = 'invalid'; > Query OK, 0 rows affected (0.001 sec) > > MariaDB [(none)]> alter user dan@localhost IDENTIFIED BY PASSWORD 'invalid'; > Query OK, 0 rows affected (0.000 sec) > > diff --git a/sql/sql_acl.cc b/sql/sql_acl.cc > index d94016b7815..3cd7a67ae1a 100644 > --- a/sql/sql_acl.cc > +++ b/sql/sql_acl.cc > @@ -14539,6 +14539,12 @@ static int native_password_get_salt(const char *hash, size_t hash_length, > > if (hash_length != SCRAMBLED_PASSWORD_CHAR_LENGTH) > { > + if (hash_length == 7 && strcmp(hash, "invalid") == 0) > + { > + memcpy(out, "invalid", 7); > + *out_length= 7; > + return 0; > + } okay. After you said ASAN, I think I can see why this could be problematic. You can, of course, pad it to SCRAMBLED_PASSWORD_CHAR_LENGTH, but then you'll create a *valid* scramble that would correspond to some actual password. One option would be to allow "invalid" literal in set_user_auth(), before even any plugin checks. But I'm unsure of the implications. > my_error(ER_PASSWD_LENGTH, MYF(0), SCRAMBLED_PASSWORD_CHAR_LENGTH); > return 1; > } > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 3

Re: [Maria-developers] f112c4a07475: MDEV-20787: Script dgcov.pl does not work
by Sergei Golubchik 30 Oct '20

30 Oct '20

Hi, Anel! On Oct 30, Anel Husakovic wrote: > revision-id: f112c4a07475 (mariadb-10.2.31-541-gf112c4a07475) > parent(s): 784473b98662 > author: Anel Husakovic <anel(a)mariadb.org> > committer: Anel Husakovic <anel(a)mariadb.org> > timestamp: 2020-10-26 14:25:41 +0100 > message: > > MDEV-20787: Script dgcov.pl does not work > > - Patch is changing CMakeList with `--coverage` flag as an alias for > `-fprofile-arcs -ftest-coverage -lgcov` in addition. > > - About the usage of `dgcov`: > When the server is compiled with `-DENABLE_GCOV=ON`, from object files are generated > `.gcno` and `.gcda` files. > `./mtr --gcov is_check_constraint` is invoking the following script calls: > - `./dgcov.pl --purge`, `./mtr is_check_constraint` and > - `./dgcov.pl --generate>/var/last_changes.dgcov`. > The `purge` flag is clearing `.gcda` files (and others extensions), > while running the test (which follows after the purge) new `.gcda` files are generated. > With dgcov `generate` flag, `gcov -i` (`intermediate format`) is called > on generated `<source-files-name>.gcda` files (`dbug.c.gcda` e.g.). > The patch is tested on `gcov 6.3` and `gcov 7.4` versions > and can be seen that resulting `.gcov` file for `6.3` creates > `<full path>.gcov` (`dbug.c.gcda.gcov` e.g.) file, > where `gcov 7.4+` is still creating `source-file-names.gcov`(`dbug.c.gcov`) files > as `gcov` in general is doing. > Results are stored in `./mysql-test/var/last_changes.dgcov`. > Note: for the newer versions of `gcov` intermediate format is > deprecated `-i` and should be used `--json-format`. does it mean that eventually we'll have to rewrite dgcov to support json format? > > - This patch doesn't take gcov versions in account since is extracting > filename of a `.gcov` file, which is generated with `gcov -i > <source-file>.gcda` during `dgcov.pl --generate`. > > - There is a check if file exists. > Reason for that is because some generated `gcov` files are outliers from above > rule (that a file generated in intermediate format(gcov) has the same source name). > Example 1: `aestables.cpp.gcda` is not generating `aestables.cpp.gcov`, > but only the files used in #include `modes.hpp.gcov runtime.hpp.gcov` > Example 2: `my_sha256.cc.gcda` is generating `my_sha.ic.gcov` > > - Gcov with out-of-source is not working, this patch will solve that > (probably?) > Make sure to test `MTR_BINDIR` which gets set during out-of-source > build and generate an error when running `./mysql-test/mtr --gcov alias`. > > diff --git a/mysql-test/dgcov.pl b/mysql-test/dgcov.pl > index fbc5540e6973..f2bbdad4d20e 100755 > --- a/mysql-test/dgcov.pl > +++ b/mysql-test/dgcov.pl > @@ -161,8 +161,14 @@ sub gcov_one_file { > system($cmd)==0 or die "system($cmd): $? $!"; > } I don't see any changes since the last review. Is it the correct commit? > - # now, read the generated file > - open FH, '<', "$_.gcov" or die "open(<$_.gcov): $!"; > + (my $filename = $_)=~ s/\.[^.]+$//; # remove extension I think this contradicts your explanation in the commit comment. You wrote that the file name can be either dbug.c.gcda.gcov or dbug.c.gcov. Here $_ is dbug.c.gcda, you unconditionally remove .gcda and then below you open $filename.gcov, that is dbug.c.gcov Which means this won't work with an older gcov. Is this your intention? I think it's fine not to support old gcov. But this should be a conscious decision, e.g. you can write in the commit comment "Now dgcov.pl only supports gcov 7.4+" - if that's indeed what you want. Also, if you do it this way, then there's no need to use s///, it's simpler to do, like sub gcov_one_file { return unless /\.gcda$/; + my $filename= $`; unless ($opt_skip_gcov) { or, may be even sub gcov_one_file { return unless /\.gcda$/; + my $gcov_file_path= "$File::Find::dir$`.gcov"; unless ($opt_skip_gcov) { > + my $gcov_file_path= $File::Find::dir."/$filename.gcov"; still an extra slash > + if (! -f $gcov_file_path) > + { > + return; > + } still not return unless -f $gcov_file_path; > + open FH, '<', "$gcov_file_path" or die "open(<$gcov_file_path): $!"; still extra quotes > + > my $fname; > while (<FH>) { > chomp; > diff --git a/mysql-test/mysql-test-run.pl b/mysql-test/mysql-test-run.pl > index 900ef736a455..2189529d82a7 100755 > --- a/mysql-test/mysql-test-run.pl > +++ b/mysql-test/mysql-test-run.pl > @@ -1791,7 +1791,7 @@ sub command_line_setup { > # -------------------------------------------------------------------------- > # Gcov flag > # -------------------------------------------------------------------------- > - if ( ($opt_gcov or $opt_gprof) and ! $source_dist ) > + if ( ($opt_gcov or $opt_gprof) and (! $source_dist or -d $ENV{MTR_BINDIR})) > { > mtr_error("Coverage test needs the source - please use source dist"); > } > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] MDEV-24015: SQL Error (1038): Out of sort memory when enough memory for the sort buffer is provided
by Sergey Petrunia 27 Oct '20

27 Oct '20

Hi Varun, I have only cosmetic comments, please find them below. Ok to push after they are addressed. > The issue here was with the re-execution of a subquey, after few executions > the subquery execution returned error that not enough space was provided inside the sort > buffer. > This is happening because with the introduction of packed sort keys, during the phase > where sort length is calculated the parameters of the sort keys are not reset and > we keep up adding the sort length to the previous calculated sort length for the > earlier executions of the subquery. This happens because the sort keys structure is > allocated only once and so if their is re-execution of a subquery involving sorting > then we should be resetting the parameters of the sort keys. The comment is hard to read. How about being more specific and using shorter sentences: The error was: filesort was done in a subquery which was executed multiple times. During each execution, sortlength() computed total sort key length in Sort_keys::sort_length, without resetting it first. Eventually Sort_keys::sort_length got larger than @@sort_buffer_size, which caused filesort() to be aborted with error. Fixed by making sortlength() to compute lengths only during the first invocation. Subsequent invocations return pre-computed values. > @@ -2518,7 +2526,7 @@ void Sort_param::try_to_pack_sortkeys() > return; > > const uint sz= Sort_keys::size_of_length_field; > - uint sort_len= sort_keys->get_sort_length(); > + uint sort_len= sort_keys->get_sort_length_with_original_values(); > > /* > Heuristic introduced, skip packing sort keys if saving less than 128 bytes > diff --git a/sql/sql_sort.h b/sql/sql_sort.h > index 40f0c5ede5f..6951d4d190a 100644 > --- a/sql/sql_sort.h > +++ b/sql/sql_sort.h > @@ -258,7 +258,9 @@ class Sort_keys :public Sql_alloc, > Sort_keys_array(arr, count), > m_using_packed_sortkeys(false), > size_of_packable_fields(0), > - sort_length(0) > + sort_length_with_original_values(0), > + sort_length_with_memcmp_values(0), > + first_execution(true) This violates the coding style. Please ident the member initialization (see e.g. item.h for examples) > { > DBUG_ASSERT(!is_null()); > }a ... > @@ -307,9 +319,12 @@ class Sort_keys :public Sql_alloc, > > void increment_original_sort_length(uint len) > { > - sort_length+= len; > + sort_length_with_original_values+= len; > } > > + bool is_first_execution() { return first_execution; } > + void set_first_execution(bool val) { first_execution= val; } A comment just about the naming: I would not drag "execution" into here. "Sort_keys" are not really executed. How about changing the functions to be bool is_parameters_computed(); bool set_parameters_computed(); and the variable to be "bool parameters_computed" ? > + > static const uint size_of_length_field= 4; > > private: BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] 5150dfd6ab3: MDEV-17891 Assertion failures in select_insert::abort_result_set and mysql_load upon attempt to replace into a full table
by Sergei Golubchik 25 Oct '20

25 Oct '20

Hi, Nikita! On Apr 27, Nikita Malyavin wrote: > revision-id: 5150dfd6ab3 (mariadb-10.3.12-230-g5150dfd6ab3) > parent(s): cf78b8c699d > author: Nikita Malyavin <nikitamalyavin(a)gmail.com> > committer: Nikita Malyavin <nikitamalyavin(a)gmail.com> > timestamp: 2019-06-12 22:11:45 +1000 > message: > > MDEV-17891 Assertion failures in select_insert::abort_result_set and mysql_load upon attempt to replace into a full table > > * set modified_non_trans_table in one missed place This commit needs way more text than that. This is just too misterious now. What asserts fails? What has the test case to do with the fix? Why modified_non_trans_table even matters without replication? Why it's specific to versioning and partitioning? Where trans_safe becomes false? > diff --git a/mysql-test/suite/versioning/r/partition.result b/mysql-test/suite/versioning/r/partition.result > index 3fcb59bdb40..99a297417d5 100644 > --- a/mysql-test/suite/versioning/r/partition.result > +++ b/mysql-test/suite/versioning/r/partition.result > @@ -541,6 +541,29 @@ t1 CREATE TABLE `t1` ( > PARTITION BY SYSTEM_TIME INTERVAL 7 SECOND > (PARTITION `ver_p1` HISTORY ENGINE = DEFAULT_ENGINE, > PARTITION `ver_pn` CURRENT ENGINE = DEFAULT_ENGINE) > +set timestamp= default; > +# MDEV-17891 Assertion failures in select_insert::abort_result_set and > +# mysql_load upon attempt to replace into a full table > +set @@max_heap_table_size= 1024*1024; > +create or replace table t1 ( > +pk integer auto_increment, > +primary key (pk), > +f varchar(45000) > +) with system versioning engine=memory > +partition by system_time interval 1 year (partition p1 history, > +partition pn current); > +# fill the table until full > +insert into t1 () values (),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(); > +insert into t1 (f) select f from t1; > +ERROR HY000: The table 't1' is full > +# leave space for exactly one record in current partition > +delete from t1 where pk = 1; > +# copy all data into history partition > +replace into t1 select * from t1; > +replace into t1 select * from t1; > +ERROR HY000: The table 't1' is full > +drop table t1; > +set @@max_heap_table_size= 1048576; > # Test cleanup > drop database test; > create database test; > diff --git a/mysql-test/suite/versioning/t/partition.test b/mysql-test/suite/versioning/t/partition.test > index d5c83b4d3bb..23768836efc 100644 > --- a/mysql-test/suite/versioning/t/partition.test > +++ b/mysql-test/suite/versioning/t/partition.test > @@ -474,6 +474,7 @@ set timestamp=1523466002.799571; > insert into t1 values (11),(12); > set timestamp=1523466004.169435; > delete from t1 where pk in (11, 12); > +set timestamp= default; > > --echo # > --echo # MDEV-18136 Server crashes in Item_func_dyncol_create::prepare_arguments > @@ -489,6 +490,34 @@ partition by system_time interval column_get(column_create(7,7), 7 as int) secon > --replace_result $default_engine DEFAULT_ENGINE > show create table t1; > > +--echo # MDEV-17891 Assertion failures in select_insert::abort_result_set and > +--echo # mysql_load upon attempt to replace into a full table > + > +--let $max_heap_table_size_orig= `select @@max_heap_table_size;` > +set @@max_heap_table_size= 1024*1024; > +create or replace table t1 ( > + pk integer auto_increment, > + primary key (pk), > + f varchar(45000) > +) with system versioning engine=memory > + partition by system_time interval 1 year (partition p1 history, > + partition pn current); > + > +--echo # fill the table until full > +insert into t1 () values (),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(); > +--error ER_RECORD_FILE_FULL > +insert into t1 (f) select f from t1; > +--echo # leave space for exactly one record in current partition > +delete from t1 where pk = 1; > +--echo # copy all data into history partition > +replace into t1 select * from t1; > +--error ER_RECORD_FILE_FULL > +replace into t1 select * from t1; > + > +# cleanup > +drop table t1; > +eval set @@max_heap_table_size= $max_heap_table_size_orig; > + > --echo # Test cleanup > drop database test; > create database test; > diff --git a/sql/sql_insert.cc b/sql/sql_insert.cc > index 1f3a70721fc..44502ea6704 100644 > --- a/sql/sql_insert.cc > +++ b/sql/sql_insert.cc > @@ -1953,6 +1953,8 @@ int write_record(THD *thd, TABLE *table,COPY_INFO *info) > if (likely(!error)) > { > info->deleted++; > + if (!table->file->has_transactions()) > + thd->transaction.stmt.modified_non_trans_table= TRUE; > if (table->versioned(VERS_TIMESTAMP)) > { > store_record(table, record[2]); Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] 05968211699: MDEV-16026: Global system_versioning_asof must not be used if client sessions can have non-default time zone
by Sergei Golubchik 25 Oct '20

25 Oct '20

Hi, Nikita! Okay. This reverts my "ok to push" on MDEV-16481 :) If you want to store the value in unix timestamp, you don't need MYSQL_TIME in save_result. You should calculate the value as a unix timestamp in ::check and store that in save_result. Now you still have timezone conversion in ::update and that can fail. Everything that can fail should be done in ::check, that's the contract. May be also I reapplied your patches to 10.3 incorrectly when I was reviewing. Perhaps you could rebase them yourself - to be sure I'm looking at the correct patch? On Oct 22, Nikita Malyavin wrote: > revision-id: 05968211699 (mariadb-10.4.11-291-g05968211699) > parent(s): efb1023b6f4 > author: Nikita Malyavin <nikitamalyavin(a)gmail.com> > committer: Sergei Golubchik <serg(a)mariadb.com> > timestamp: 2020-10-22 17:07:03 +0200 > message: > > MDEV-16026: Global system_versioning_asof must not be used if client sessions can have non-default time zone > > * store `system_versioning_asof` in unix time; > * both session and global vars are processed in session timezone; > * setting `default` does not copy global varibale abymore. Instead, it sets system_time to SYSTEM_TIME_UNSPECIFIED, which means that no 'AS OF' time is applied and `now()` can be assumed > As a regression, we cannot assign values below 1970 anymore > > --- > mysql-test/suite/versioning/r/sysvars.result | 67 +++++++++++++++++++++++ > mysql-test/suite/versioning/t/sysvars.test | 80 ++++++++++++++++++++++++---- > sql/mysqld.h | 3 +- > sql/sql_select.cc | 6 ++- > sql/sys_vars.ic | 27 ++++++---- > 5 files changed, 162 insertions(+), 21 deletions(-) > > diff --git a/sql/mysqld.h b/sql/mysqld.h > index bd45ff7b798..884e5a06066 100644 > --- a/sql/mysqld.h > +++ b/sql/mysqld.h > @@ -194,7 +194,8 @@ enum vers_system_time_t > struct vers_asof_timestamp_t > { > ulong type; > - MYSQL_TIME ltime; > + my_time_t unix_time; > + ulong second_part; > }; > > enum vers_alter_history_enum > diff --git a/sql/sql_select.cc b/sql/sql_select.cc > index d0bb0c816ec..4d62a9829eb 100644 > --- a/sql/sql_select.cc > +++ b/sql/sql_select.cc > @@ -722,8 +722,12 @@ bool vers_select_conds_t::init_from_sysvar(THD *thd) > if (type != SYSTEM_TIME_UNSPECIFIED && type != SYSTEM_TIME_ALL) > { > DBUG_ASSERT(type == SYSTEM_TIME_AS_OF); > + MYSQL_TIME ltime; > + thd->variables.time_zone->gmt_sec_to_TIME(&ltime, in.unix_time); > + ltime.second_part = in.second_part; > + > start.item= new (thd->mem_root) > - Item_datetime_literal(thd, &in.ltime, TIME_SECOND_PART_DIGITS); > + Item_datetime_literal(thd, &ltime, TIME_SECOND_PART_DIGITS); > if (!start.item) > return true; > } > diff --git a/sql/sys_vars.ic b/sql/sys_vars.ic > index f33f469b160..417fa7842c2 100644 > --- a/sql/sys_vars.ic > +++ b/sql/sys_vars.ic > @@ -2646,9 +2646,11 @@ class Sys_var_vers_asof: public Sys_var_enum > } > > private: > - bool update(set_var *var, vers_asof_timestamp_t &out) > + bool update(set_var *var, vers_asof_timestamp_t &out, system_variables& pool) > { > bool res= false; > + uint error; > + MYSQL_TIME ltime; > out.type= static_cast<enum_var_type>(var->save_result.ulonglong_value); > if (out.type == SYSTEM_TIME_AS_OF) > { > @@ -2658,11 +2660,14 @@ class Sys_var_vers_asof: public Sys_var_enum > Datetime::Options opt(TIME_CONV_NONE | > TIME_NO_ZERO_IN_DATE | > TIME_NO_ZERO_DATE, thd); > - res= var->value->get_date(thd, &out.ltime, opt); > + res= var->value->get_date(thd, &ltime, opt); > + out.unix_time= pool.time_zone->TIME_to_gmt_sec(&ltime, &error); > + out.second_part= ltime.second_part; > + res|= (error != 0); > } > else // set DEFAULT from global var > { > - out= global_var(vers_asof_timestamp_t); > + out.type= SYSTEM_TIME_UNSPECIFIED; > res= false; > } > } > @@ -2672,11 +2677,11 @@ class Sys_var_vers_asof: public Sys_var_enum > public: > virtual bool global_update(THD *thd, set_var *var) > { > - return update(var, global_var(vers_asof_timestamp_t)); > + return update(var, global_var(vers_asof_timestamp_t), thd->variables); > } > virtual bool session_update(THD *thd, set_var *var) > { > - return update(var, session_var(thd, vers_asof_timestamp_t)); > + return update(var, session_var(thd, vers_asof_timestamp_t), thd->variables); > } > > private: > @@ -2690,10 +2695,14 @@ class Sys_var_vers_asof: public Sys_var_enum > case SYSTEM_TIME_AS_OF: > { > uchar *buf= (uchar*) thd->alloc(MAX_DATE_STRING_REP_LENGTH); > - if (buf &&!my_datetime_to_str(&val.ltime, (char*) buf, 6)) > + MYSQL_TIME ltime; > + > + thd->variables.time_zone->gmt_sec_to_TIME(&ltime, val.unix_time); > + ltime.second_part= val.second_part; > + > + if (buf && !my_datetime_to_str(&ltime, (char*) buf, 6)) > { > - // TODO: figure out variable name > - my_error(ER_WRONG_VALUE_FOR_VAR, MYF(0), "system_versioning_asof_timestamp", "NULL (wrong datetime)"); > + my_error(ER_WRONG_VALUE_FOR_VAR, MYF(0), name.str, "NULL (wrong datetime)"); > return (uchar*) thd->strdup("Error: wrong datetime"); > } > return buf; > @@ -2701,7 +2710,7 @@ class Sys_var_vers_asof: public Sys_var_enum > default: > break; > } > - my_error(ER_WRONG_VALUE_FOR_VAR, MYF(0), "system_versioning_asof_timestamp", "NULL (wrong range type)"); > + my_error(ER_WRONG_VALUE_FOR_VAR, MYF(0), name.str, "NULL (wrong range type)"); > return (uchar*) thd->strdup("Error: wrong range type"); > } > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] caec6b4b3d7: MDEV-17573 Assertion in federatedx on multi-update
by Sergei Golubchik 23 Oct '20

23 Oct '20

Hi, Aleksey! Looks reasonable to me. MYSQL_RES is stored in the handler. Of course, MYSQL_ROWS *current, which is a pointer into MYSQL_RES, has to be in the handler, it cannot be shared. Ok to push, thanks! On Oct 23, Aleksey Midenkov wrote: > revision-id: caec6b4b3d7 (mariadb-10.3.21-250-gcaec6b4b3d7) > parent(s): a00c668ddf2 > author: Aleksey Midenkov <midenok(a)gmail.com> > committer: Aleksey Midenkov <midenok(a)gmail.com> > timestamp: 2020-10-21 13:38:37 +0300 > message: > > MDEV-17573 Assertion in federatedx on multi-update > > Cause: shared federatedx_io cannot store table-specific data. > > Fix: move current row reference `federatedx_io_mysql::current` to > ha_federatedx. > > FederatedX connection (represented by federatedx_io) is stored into > federatedx_txn::txn_list of per-server connections (see > federatedx_txn::acquire()). federatedx_txn object is stored into THD > (see ha_federatedx::external_lock()). When multiple handlers acquire > FederatedX connection they get single federatedx_io instance. Multiple > handlers do their operation via federatedx_io_mysql::mark_position() > and federatedx_io_mysql::fetch_row() in arbitrarty manner. They access > the same federatedx_io_mysql instance and same MYSQL_ROWS *current > pointer, so one handler disrupts the work of the other. > > Related to "MDEV-14551 Can't find record in table on multi-table update > with ORDER BY". Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 2e0bac4688a: MDEV-22313: Neither SHOW CREATE USER nor SHOW GRANTS prints a user's default role
by Sergei Golubchik 23 Oct '20

23 Oct '20

Hi, Anel! On Oct 23, Anel Husakovic wrote: > revision-id: 2e0bac4688a (mariadb-10.2.31-517-g2e0bac4688a) > parent(s): 985ede92034 > author: Anel Husakovic <anel(a)mariadb.org> > committer: Anel Husakovic <anel(a)mariadb.org> > timestamp: 2020-10-23 12:38:11 +0200 > message: > > MDEV-22313: Neither SHOW CREATE USER nor SHOW GRANTS prints a user's default role > > diff --git a/mysql-test/r/grant5.result b/mysql-test/r/grant5.result > index 338bbd36e14..a69070e8399 100644 > --- a/mysql-test/r/grant5.result > +++ b/mysql-test/r/grant5.result ... > +SHOW GRANTS; > +Grants for root@localhost > +GRANT test_role TO 'root'@'localhost' WITH ADMIN OPTION > +GRANT ALL PRIVILEGES ON *.* TO 'root'@'localhost' WITH GRANT OPTION > +GRANT PROXY ON ''@'%' TO 'root'@'localhost' WITH GRANT OPTION > +GRANT USAGE ON *.* TO 'test_role' > +SET DEFAULT ROLE test_role FOR 'root'@'localhost' ... Yes, that's what I mean. I don't have a strong opinion on that, it seems that I very slightly prefer to have SET DEFAULT ROLE after GRANT PROXY, that is to have all statements for root, and then statements for roles. But having SET DEFAULT ROLE at the end makes sense too - all GRANTs first, then SET DEFAULT ROLE. Just make it the way you want. But either way it's good to have a test for it, a test that shows we've considered this issue and intentionally implemented that particular order. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 6d881a271a9: MDEV-23178: Qualified asterisk not supported in INSERT .. RETURNING
by Sergei Golubchik 23 Oct '20

23 Oct '20

Hi, Rucha! On Oct 23, Rucha Deodhar wrote: > revision-id: 6d881a271a9 (mariadb-10.5.4-108-g6d881a271a9) > parent(s): bbd70fcc43c > author: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > committer: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > timestamp: 2020-10-23 12:57:02 +0530 > message: > > MDEV-23178: Qualified asterisk not supported in INSERT .. RETURNING > > diff --git a/sql/sql_base.cc b/sql/sql_base.cc > index 45ce4be3eb5..4ce7b034029 100644 > --- a/sql/sql_base.cc > +++ b/sql/sql_base.cc > @@ -7959,7 +7959,7 @@ insert_fields(THD *thd, Name_resolution_context *context, const char *db_name, > bool any_privileges, uint *hidden_bit_fields) > { > Field_iterator_table_ref field_iterator; > - bool found; > + bool found, is_insert_or_replace_returning= false; > char name_buff[SAFE_NAME_LEN+1]; > DBUG_ENTER("insert_fields"); > DBUG_PRINT("arena", ("stmt arena: %p",thd->stmt_arena)); > @@ -7978,13 +7978,31 @@ insert_fields(THD *thd, Name_resolution_context *context, const char *db_name, > > found= FALSE; > > + if ((thd->lex->sql_command == SQLCOM_INSERT_SELECT || > + thd->lex->sql_command == SQLCOM_REPLACE_SELECT || > + thd->lex->sql_command == SQLCOM_INSERT || > + thd->lex->sql_command == SQLCOM_REPLACE) && > + thd->lex->has_returning()) > + is_insert_or_replace_returning= true; > + > /* > If table names are qualified, then loop over all tables used in the query, > else treat natural joins as leaves and do not iterate over their underlying > tables. > + Also, When we have INSERT/REPLACE...RETURNING and have qualified asterisk, > + table_name is not NULL and context->table_list is either NULL or has > + incorrect reference because context->table_list has tables from the > + FROM clause. > + context->table_list has incorrect reference (has table from the FROM clause > + instead of table we are inserting into) for > + INSERT/REPLACE...SELECT...RETURNING because we have a FROM clause from the > + SELECT statement. For INSERT/REPLACE...RETURNING it is NULL because > + there is no FROM clause. > */ > - for (TABLE_LIST *tables= (table_name ? context->table_list : > - context->first_name_resolution_table); > + for (TABLE_LIST *tables= (table_name ? (is_insert_or_replace_returning ? > + context->first_name_resolution_table : > + context->table_list) : > + (context->first_name_resolution_table)); ok, I've compiled it to run few queries, now I see that this is wrong. you need to understand the old code and the old comment before modifying them. one way to do it is to change the condition from table_name ? context->table_list : context->first_name_resolution_table to simply context->first_name_resolution_table and see what breaks. You'll see that main.type_varchar will break with At line 319: query 'SELECT t1.* FROM t1 LEFT JOIN t2 USING (c1)' failed: 1051: Unknown table 'test.t1' indeed, t1 is only present in context->table_list, but not in context->first_name_resolution_table. So after your fix the following statement will no longer work: INSERT INTO t2 SELECT t1.* FROM t1 JOIN t1 AS t USING (id1) WHERE id1=1 RETURNING *; it's because you set is_insert_or_replace_returning both for the SELECT and for the RETURNING clauses. > tables; > tables= (table_name ? tables->next_local : > tables->next_name_resolution_table) also this looks suspicious. Normally one starts from table_list and follows the next_local pointer or one starts from first_name_resolution_table and follows the next_name_resolution_table pointer. Your change creates a strange mix of both, it looks like it would be wrong. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 581857bd999: MDEV-22313: Neither SHOW CREATE USER nor SHOW GRANTS prints a user's default role
by Sergei Golubchik 23 Oct '20

23 Oct '20

Hi, Anel! Looks good, thanks! A couple of small comments below. On Oct 23, Anel Husakovic wrote: > revision-id: 581857bd999 (mariadb-10.1.43-314-g581857bd999) let's fix it in 10.2 and up, we're doing the very last 10.1 release, so better be extra-super-conservative about it. > parent(s): 43ec9370b32 > author: Anel Husakovic <anel(a)mariadb.org> > committer: Anel Husakovic <anel(a)mariadb.org> > timestamp: 2020-10-23 00:20:53 +0200 > message: > > MDEV-22313: Neither SHOW CREATE USER nor SHOW GRANTS prints a user's default role > > diff --git a/mysql-test/t/grant5.test b/mysql-test/t/grant5.test > index 74a69952124..bf97a930097 100644 > --- a/mysql-test/t/grant5.test > +++ b/mysql-test/t/grant5.test > @@ -52,6 +52,23 @@ disconnect u1; > drop user u1@localhost; > drop database mysqltest1; > > +# > +# MDEV-22313: Neither SHOW CREATE USER nor SHOW GRANTS prints a user's default role > +# > +CREATE ROLE test_role; > +CREATE USER test_user; > +GRANT test_role TO test_user; > +SET DEFAULT ROLE test_role FOR test_user; > +SHOW GRANTS FOR test_user; > +SET DEFAULT ROLE NONE for test_user; > +SHOW GRANTS FOR test_user; > +SET DEFAULT ROLE test_role; oops, sorry, my mistake. Please, add `SET ROLE test_role` before SHOW GRANTS. The goal is to see that SET DEFAULT ROLE is shown *after* all grants to root@localhost, but before any grants to test_role. > +SHOW GRANTS; > +SET DEFAULT ROLE NONE; > +SHOW GRANTS; > +DROP USER test_user; > +DROP ROLE test_role; > + > # > # End of 10.1 tests > # > diff --git a/sql/sql_acl.cc b/sql/sql_acl.cc > index cf0b1d87bd7..6c852976061 100644 > --- a/sql/sql_acl.cc > +++ b/sql/sql_acl.cc > @@ -8001,15 +8007,44 @@ static ROLE_GRANT_PAIR *find_role_grant_pair(const LEX_STRING *u, > my_hash_search(&acl_roles_mappings, (uchar*)pair_key.ptr(), key_length); > } > > +static bool show_default_role(THD *thd, const char *hostname, > + ACL_USER *acl_entry, char *buff, size_t buffsize) > +{ > + Protocol *protocol= thd->protocol; > + LEX_STRING def_rolename= acl_entry->default_rolename; > + > + if (def_rolename.length) > + { > + String def_str(buff, buffsize, system_charset_info); > + def_str.length(0); > + def_str.append(STRING_WITH_LEN("SET DEFAULT ROLE ")); > + def_str.append(&def_rolename); > + def_str.append(" FOR '"); > + def_str.append(&acl_entry->user); > + DBUG_ASSERT(!(acl_entry->flags & IS_ROLE)); > + def_str.append(STRING_WITH_LEN("'@'")); > + def_str.append(acl_entry->host.hostname, acl_entry->hostname_length, > + system_charset_info); > + def_str.append('\''); > + protocol->prepare_for_resend(); > + protocol->store(def_str.ptr(),def_str.length(),def_str.charset()); > + if (protocol->write()) > + { > + return TRUE; > + } > + } > + return FALSE; looks good. The hostname argument seems to be unused now, doesn't it? > +} > + > -static bool show_role_grants(THD *thd, const char *username, > - const char *hostname, ACL_USER_BASE *acl_entry, > +static bool show_role_grants(THD *thd, const char *hostname, > + ACL_USER_BASE *acl_entry, > char *buff, size_t buffsize) > { > uint counter; > Protocol *protocol= thd->protocol; > LEX_STRING host= {const_cast<char*>(hostname), strlen(hostname)}; > > - String grant(buff,sizeof(buff),system_charset_info); > + String grant(buff, buffsize, system_charset_info); ouch, indeed. thanks! > for (counter= 0; counter < acl_entry->role_grants.elements; counter++) > { > grant.length(0); Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 01612f594b3: MDEV-23769: MTR can abort before it prints the test result summary
by Sergei Golubchik 22 Oct '20

22 Oct '20

Hi, Anel! On Oct 22, Anel Husakovic wrote: > Hi Sergei, > thanks for the review. > > Having warning/error is a necessary condition. > Sufficient condition is to call at least 2 tests. > Incorrect behaviour can be seen if: > - Both tests are correct but there is error generated (incorrect option > passed): > So this `./mysql-test/mtr --mysqld=--lock-wait-timeout=-xx 1st` will > generate and write statistic at the end, > but this `./mysql-test/mtr --mysqld=--lock-wait-timeout=-xx 1st 1st > --force` will not generate the statistic. > Note: warning is generated (--mysqld=--lock-wait-timeout=-1) statistic > will be generated Okay, thanks. This would be useful to show in the commit comment, that one can use ./mtr --mysqld=--lock-wait-timeout=-xx 1st 1st --force --parallel 1 to repeat the error. But then I don't understand why this happens. Why one failure is not enough? Why --parallel 2 destroys the effect? Apparently, it's check_warnings_post_shutdown() that causes that, not exit(1). One can change to exit(0) or remove exit(1) completely - the bug will still happen. Why check_warnings_post_shutdown() writing to the master server causes this bug? It looks rather fishy that in your fix it's the worker process that prints the statistics, not the master. Btw, note that check_warnings_post_shutdown() doesn't have any return statement, it very implicitly return "number of bytes sent" - it would be easier to read if it'd return something explicitly. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 99c14476251: MDEV-22313: Neither SHOW CREATE USER nor SHOW GRANTS prints a user's default role
by Sergei Golubchik 22 Oct '20

22 Oct '20

Hi, Anel! On Oct 22, Anel Husakovic wrote: > revision-id: 99c14476251 (mariadb-10.1.43-314-g99c14476251) > parent(s): 43ec9370b32 > author: Anel Husakovic <anel(a)mariadb.org> > committer: Anel Husakovic <anel(a)mariadb.org> > timestamp: 2020-10-22 16:49:28 +0200 > message: > > MDEV-22313: Neither SHOW CREATE USER nor SHOW GRANTS prints a user's default role > > diff --git a/mysql-test/t/grant5.test b/mysql-test/t/grant5.test > index 74a69952124..8756648c3e1 100644 > --- a/mysql-test/t/grant5.test > +++ b/mysql-test/t/grant5.test > @@ -53,5 +53,19 @@ drop user u1@localhost; > drop database mysqltest1; > > # > -# End of 10.1 tests > +# MDEV-22313: Neither SHOW CREATE USER nor SHOW GRANTS prints a user's default role > +# > +CREATE ROLE test_role; > +CREATE USER test_user; > +GRANT test_role TO test_user; > +SET DEFAULT ROLE test_role FOR test_user; > +SHOW GRANTS FOR test_user; > +SET DEFAULT ROLE NONE for test_user; > +SHOW GRANTS FOR test_user; > +SHOW GRANTS; The point is to run `SHOW GRANTS` for a user with a default role. E.g. SET DEFAULT ROLE test_role; SHOW GRANTS; SET DEFAULT ROLE NONE; > +DROP USER test_user; > +DROP ROLE test_role; > + > # > +# End of 10.1 tests > +# > \ No newline at end of file new line at the end of file? > diff --git a/sql/sql_acl.cc b/sql/sql_acl.cc > index cf0b1d87bd7..f42f7bdcfaf 100644 > --- a/sql/sql_acl.cc > +++ b/sql/sql_acl.cc > @@ -8001,6 +8007,36 @@ static ROLE_GRANT_PAIR *find_role_grant_pair(const LEX_STRING *u, > my_hash_search(&acl_roles_mappings, (uchar*)pair_key.ptr(), key_length); > } > > +static bool show_default_role(THD *thd, const char *hostname, > + ACL_USER *acl_entry, char *buff) > +{ > + Protocol *protocol= thd->protocol; > + LEX_STRING host= {const_cast<char*>(hostname), strlen(hostname)}; > + > + LEX_STRING def_rolename= acl_entry->default_rolename; > + if (def_rolename.length) > + { > + String def_str(buff,sizeof(buff),system_charset_info); eh? sizeof(buff) is sizeof(char*) that is usually 8 on 64-bit, 4 on 32-bit architecture. You want the size of the buffer here, not the size of the pointer to the buffer. > + def_str.length(0); > + def_str.append(STRING_WITH_LEN("SET DEFAULT ROLE ")); > + def_str.append(&def_rolename); > + def_str.append(" FOR '"); > + def_str.append(acl_entry->user.str, acl_entry->user.length, > + system_charset_info); again, just as above, you can use def_str.append(&acl_entry->user); > + DBUG_ASSERT(!(acl_entry->flags & IS_ROLE)); > + def_str.append(STRING_WITH_LEN("'@'")); > + def_str.append(&host); why not to use def_str.append(acl_entry->host.hostname, acl_entry->hostname_length, system_charset_info); > + def_str.append('\''); > + protocol->prepare_for_resend(); > + protocol->store(def_str.ptr(),def_str.length(),def_str.charset()); > + if (protocol->write()) > + { > + return TRUE; > + } > + } > + return FALSE; > +} Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 6729bf5c031: MDEV-22313: Neither SHOW CREATE USER nor SHOW GRANTS prints a user's default role
by Sergei Golubchik 22 Oct '20

22 Oct '20

Hi, Anel! On Oct 22, Anel Husakovic wrote: > Hi Sergei, > thank you for your review. > How is easier for you (or to ask a general question: how does the review > process go?) regarding the commits? > Is it better to add a new commit as a review commit and if the patch gets > accepted to squash after, or to rebase commits and send only a single > commit again for the review? One single commit is better, thanks. At least for small patches. > Addressed in the new patch: > https://github.com/MariaDB/server/commit/99c144762514d4dea3b6ec395de96aaf10… Thanks! I'll reply to the commit directly. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 83614913d9a: MDEV-16243 [Warning] "Versioned table ...: partition ... is full, add more HISTORY partitions" floods the error log
by Sergei Golubchik 22 Oct '20

22 Oct '20

Hi, Aleksey! On Oct 22, Aleksey Midenkov wrote: > revision-id: 83614913d9a (mariadb-10.4.11-291-g83614913d9a) > parent(s): efb1023b6f4 > author: Aleksey Midenkov <midenok(a)gmail.com> > committer: Sergei Golubchik <serg(a)mariadb.com> > timestamp: 2020-10-22 16:44:13 +0200 > message: > > MDEV-16243 [Warning] "Versioned table ...: partition ... is full, add more HISTORY partitions" floods the error log > > diff --git a/sql/partition_info.cc b/sql/partition_info.cc > index 8989a918c0c..f287cfddcd2 100644 > --- a/sql/partition_info.cc > +++ b/sql/partition_info.cc > @@ -879,9 +879,12 @@ void partition_info::vers_set_hist_part(THD *thd) > } > return; > warn: > - my_error(WARN_VERS_PART_FULL, MYF(ME_WARNING|ME_ERROR_LOG), > - table->s->db.str, table->s->table_name.str, > - vers_info->hist_part->partition_name); > + if (vers_info->hist_part->warn_vers_part_full) > + { > + my_error(WARN_VERS_PART_FULL, MYF(ME_WARNING|ME_ERROR_LOG), > + table->s->db.str, table->s->table_name.str, > + vers_info->hist_part->partition_name); > + } would it make sense to put warn_vers_part_full into vers_info, instead of storing it per-partition? > } > > > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 2830b9c290e: MDEV-23926: Fix warnings generated during compilation of plugin/auth_pam/mapper/pam_user_map.c on MacOS
by Sergei Golubchik 22 Oct '20

22 Oct '20

Hi, Dmitry! On Oct 22, Dmitry Shulga wrote: > revision-id: 2830b9c290e (mariadb-10.2.31-489-g2830b9c290e) > parent(s): 8984d779103 > author: Dmitry Shulga <dmitry.shulga(a)mariadb.com> > committer: Dmitry Shulga <dmitry.shulga(a)mariadb.com> > timestamp: 2020-10-22 00:42:10 +0700 > message: > > MDEV-23926: Fix warnings generated during compilation of plugin/auth_pam/mapper/pam_user_map.c on MacOS > > Compiler warnings like one listed below are generated during server build on MacOS: > > > --- > plugin/auth_pam/CMakeLists.txt | 26 ++++++++++++++++++++++++++ > plugin/auth_pam/config.h.cmake | 1 + > plugin/auth_pam/mapper/pam_user_map.c | 30 +++++++++++++++++++----------- > 3 files changed, 46 insertions(+), 11 deletions(-) Nice! Could you also - in a second follow-up commit, not to confuse with the pure bug fix - do a small cleanup and move all other checks from plugin/auth_pam/CMakeLists.txt into your new config.h.cmake ? Currently they're added to the command line, like IF(HAVE_PAM_EXT_H) ADD_DEFINITIONS(-DHAVE_PAM_EXT_H) ENDIF() but as you've introduced config.h.cmake, there's no need to put them all on the command line anymore. You'll need to include config_auth_pam.h into more source files. And, I suppose. add config_auth_pam.h to .gitignore too. ok to push these two commits when done. Thanks! Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 65af1d07771: MDEV-23178: Qualified asterisk not supported in INSERT .. RETURNING
by Sergei Golubchik 22 Oct '20

22 Oct '20

Hi, Rucha! On Oct 22, Rucha Deodhar wrote: > revision-id: 65af1d07771 (mariadb-10.5.4-108-g65af1d07771) > parent(s): bbd70fcc43c > author: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > committer: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > timestamp: 2020-08-06 17:23:49 +0530 > message: > > MDEV-23178: Qualified asterisk not supported in INSERT .. RETURNING > > Anaylsis: When we have INSERT/REPLACE...RETURNING and have qualified asterisk, > table_name is not NULL and context->table_list is either NULL or has > incorrect reference because context->table_list has tables from the FROM clause. > context->table_list has incorrect reference (has table from the FROM clause > instead of table we are inserting into) for INSERT/REPLACE...SELECT...RETURNING > because we have a FROM clause from the SELECT statement. > For INSERT/REPLACE...RETURNING it is NULL because there is no FROM clause. > Fix: If table_name is not null, check if we have INSERT/REPLACE...RETURNING. > If so, the reference should be the table we are inserting into and not > the table in the FROM clause. > > diff --git a/mysql-test/main/insert_returning.result b/mysql-test/main/insert_returning.result > index e664e02bedc..e1bab6f0329 100644 > --- a/mysql-test/main/insert_returning.result > +++ b/mysql-test/main/insert_returning.result > @@ -89,6 +89,9 @@ total val1 id1 && id1 id1 UPPER(val1) f(id1) good tests! > diff --git a/sql/sql_base.cc b/sql/sql_base.cc > index 45ce4be3eb5..4ce7b034029 100644 > --- a/sql/sql_base.cc > +++ b/sql/sql_base.cc > @@ -7959,7 +7959,7 @@ insert_fields(THD *thd, Name_resolution_context *context, const char *db_name, > bool any_privileges, uint *hidden_bit_fields) > { > Field_iterator_table_ref field_iterator; > - bool found; > + bool found, is_insert_or_replace_returning= false; > char name_buff[SAFE_NAME_LEN+1]; > DBUG_ENTER("insert_fields"); > DBUG_PRINT("arena", ("stmt arena: %p",thd->stmt_arena)); > @@ -7978,13 +7978,31 @@ insert_fields(THD *thd, Name_resolution_context *context, const char *db_name, > > found= FALSE; > > + if ((thd->lex->sql_command == SQLCOM_INSERT_SELECT || > + thd->lex->sql_command == SQLCOM_REPLACE_SELECT || > + thd->lex->sql_command == SQLCOM_INSERT || > + thd->lex->sql_command == SQLCOM_REPLACE) && > + thd->lex->has_returning()) > + is_insert_or_replace_returning= true; What about INSERT t1 SELECT t2.* FROM t2 RETURNING t1.*; can your is_insert_or_replace_returning distinguish between the first and the second asterisk? Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 01612f594b3: MDEV-23769: MTR can abort before it prints the test result summary
by Sergei Golubchik 22 Oct '20

22 Oct '20

Hi, Anel! On Oct 22, Anel Husakovic wrote: > revision-id: 01612f594b3 (mariadb-10.1.43-300-g01612f594b3) > parent(s): 82301aea4f2 > author: Anel Husakovic <anel(a)mariadb.org> > committer: Anel Husakovic <anel(a)mariadb.org> > timestamp: 2020-10-03 02:34:03 +0200 > message: > > MDEV-23769: MTR can abort before it prints the test result summary > > - Patch is solving generating report on warning > - Patch is doing cosmetic fix of `current_test` log file which holds the old log value of test `CURRENT TEST:..` in `mark_log()` in case of `unknown option` and as such > the logic which is using it's content doesn't output valid log content and doesn't generate valid `$test->{'comment'}` message. more is not always better. please remove everything in the commit comment below this line - 400 lines that don't explain anything beyond what your *one line* of the comment already did. And now you have to press PageDown many times to get to my main comment about the fix :) > Here are the test cases observed. Note that only in 6. scenario there is `print $child "HELLO\n";` problem in printing (to delete?). > > 1. Valid test case, but warning in server log > ``` > $ ./mtr MDEV-23769 --force > Logging: ./mtr MDEV-23769 --force > vardir: /home/anel/mariadb/10.1-mtr-in-source/mysql-test/var > Checking leftover processes... > Removing old var directory... > Creating var directory '/home/anel/mariadb/10.1-mtr-in-source/mysql-test/var'... > Checking supported features... > MariaDB Version 10.1.47-MariaDB-debug > - SSL connections supported > - binaries are debug compiled > Sphinx 'indexer' binary not found, sphinx suite will be skipped > Collecting tests... > Installing system database... > > ============================================================================== > > TEST RESULT TIME (ms) or COMMENT > -------------------------------------------------------------------------- > > worker[1] Using MTR_BUILD_THREAD 300, with reserved ports 16000..16019 > show databases; > Database > information_schema > mtr > mysql > performance_schema > test > main.MDEV-23769 [ fail ] Found warnings/errors in server log file! > Test ended at 2020-10-03 02:13:00 > line > 2020-10-03 2:13:00 140309996242496 [Warning] option 'lock_wait_timeout': unsigned value 18446744073709551615 adjusted to 31536000 > ^ Found warnings in /home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/log/mysqld.1.err > ok > > - saving '/home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/log/main.MDEV-23769/' to '/home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/log/main.MDEV-23769/' > -------------------------------------------------------------------------- > The servers were restarted 0 times > Spent 0.000 of 4 seconds executing testcases > mysql-test-run: WARNING: Got errors/warnings while running tests, please examine '/home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/log/warnings' for details. > > Completed: Failed 1/1 tests, 0.00% were successful. > > Failing test(s): main.MDEV-23769 > > The log files in var/log may give you some hint of what went wrong. > > If you want to report this error, please read first the documentation > at http://dev.mysql.com/doc/mysql/en/mysql-test-suite.html > > mysql-test-run: *** ERROR: there were failing test cases > ``` > > 2. Valid test cases (same), but warning in server log (Note reported single test case) > ``` > $ ./mtr MDEV-23769 MDEV-23769 --force > Logging: ./mtr MDEV-23769 MDEV-23769 --force > vardir: /home/anel/mariadb/10.1-mtr-in-source/mysql-test/var > Checking leftover processes... > Removing old var directory... > Creating var directory '/home/anel/mariadb/10.1-mtr-in-source/mysql-test/var'... > Checking supported features... > MariaDB Version 10.1.47-MariaDB-debug > - SSL connections supported > - binaries are debug compiled > Sphinx 'indexer' binary not found, sphinx suite will be skipped > Collecting tests... > Installing system database... > > ============================================================================== > > TEST RESULT TIME (ms) or COMMENT > -------------------------------------------------------------------------- > > worker[1] Using MTR_BUILD_THREAD 300, with reserved ports 16000..16019 > show databases; > Database > information_schema > mtr > mysql > performance_schema > test > main.MDEV-23769 [ fail ] Found warnings/errors in server log file! > Test ended at 2020-10-03 02:15:05 > line > 2020-10-03 2:15:04 139868136609344 [Warning] option 'lock_wait_timeout': unsigned value 18446744073709551615 adjusted to 31536000 > ^ Found warnings in /home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/log/mysqld.1.err > ok > > - saving '/home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/log/main.MDEV-23769/' to '/home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/log/main.MDEV-23769/' > show databases; > Database > information_schema > mtr > mysql > performance_schema > test > main.MDEV-23769 [ fail ] Found warnings/errors in server log file! > Test ended at 2020-10-03 02:15:05 > line > 2020-10-03 2:15:05 140513782085184 [Warning] option 'lock_wait_timeout': unsigned value 18446744073709551615 adjusted to 31536000 > ^ Found warnings in /home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/log/mysqld.1.err > ok > > - saving '/home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/log/main.MDEV-23769/' to '/home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/log/main.MDEV-23769/' > -------------------------------------------------------------------------- > The servers were restarted 1 times > Spent 0.000 of 5 seconds executing testcases > mysql-test-run: WARNING: Got errors/warnings while running tests, please examine '/home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/log/warnings' for details. > > Completed: Failed 2/2 tests, 0.00% were successful. > > Failing test(s): main.MDEV-23769 > > The log files in var/log may give you some hint of what went wrong. > > If you want to report this error, please read first the documentation > at http://dev.mysql.com/doc/mysql/en/mysql-test-suite.html > > mysql-test-run: *** ERROR: there were failing test cases > ``` > > 3. Valid test cases (different test cases), but warning in server log (Note one passed one failed) > > ``` > $ ./mtr MDEV-23769 1st --force > Logging: ./mtr MDEV-23769 1st --force > vardir: /home/anel/mariadb/10.1-mtr-in-source/mysql-test/var > Checking leftover processes... > Removing old var directory... > Creating var directory '/home/anel/mariadb/10.1-mtr-in-source/mysql-test/var'... > Checking supported features... > MariaDB Version 10.1.47-MariaDB-debug > - SSL connections supported > - binaries are debug compiled > Sphinx 'indexer' binary not found, sphinx suite will be skipped > Collecting tests... > Installing system database... > > ============================================================================== > > TEST RESULT TIME (ms) or COMMENT > -------------------------------------------------------------------------- > > worker[1] Using MTR_BUILD_THREAD 300, with reserved ports 16000..16019 > show databases; > Database > information_schema > mtr > mysql > performance_schema > test > main.MDEV-23769 [ fail ] Found warnings/errors in server log file! > Test ended at 2020-10-03 02:17:09 > line > 2020-10-03 2:17:08 140658805515840 [Warning] option 'lock_wait_timeout': unsigned value 18446744073709551615 adjusted to 31536000 > ^ Found warnings in /home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/log/mysqld.1.err > ok > > - saving '/home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/log/main.MDEV-23769/' to '/home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/log/main.MDEV-23769/' > main.1st [ pass ] 1 > -------------------------------------------------------------------------- > The servers were restarted 1 times > Spent 0.001 of 4 seconds executing testcases > mysql-test-run: WARNING: Got errors/warnings while running tests, please examine '/home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/log/warnings' for details. > > Completed: Failed 1/2 tests, 50.00% were successful. > > Failing test(s): main.MDEV-23769 > > The log files in var/log may give you some hint of what went wrong. > > If you want to report this error, please read first the documentation > at http://dev.mysql.com/doc/mysql/en/mysql-test-suite.html > > mysql-test-run: *** ERROR: there were failing test cases > > ``` > > 4. Now for example 3. create artificial failure with warnings in server log (note, both test failed) > > ``` > > $ ./mtr MDEV-23769 1st --force > Logging: ./mtr MDEV-23769 1st --force > vardir: /home/anel/mariadb/10.1-mtr-in-source/mysql-test/var > Checking leftover processes... > Removing old var directory... > Creating var directory '/home/anel/mariadb/10.1-mtr-in-source/mysql-test/var'... > Checking supported features... > MariaDB Version 10.1.47-MariaDB-debug > - SSL connections supported > - binaries are debug compiled > Sphinx 'indexer' binary not found, sphinx suite will be skipped > Collecting tests... > Installing system database... > > ============================================================================== > > TEST RESULT TIME (ms) or COMMENT > -------------------------------------------------------------------------- > > worker[1] Using MTR_BUILD_THREAD 300, with reserved ports 16000..16019 > show databases; > Database > information_schema > mtr > mysql > performance_schema > test > main.MDEV-23769 [ fail ] Found warnings/errors in server log file! > Test ended at 2020-10-03 02:17:52 > line > 2020-10-03 2:17:51 140260493149760 [Warning] option 'lock_wait_timeout': unsigned value 18446744073709551615 adjusted to 31536000 > ^ Found warnings in /home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/log/mysqld.1.err > ok > > - saving '/home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/log/main.MDEV-23769/' to '/home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/log/main.MDEV-23769/' > main.1st [ fail ] > Test ended at 2020-10-03 02:17:52 > > CURRENT_TEST: main.1st > mysqltest: At line 5: End of line junk detected: "show tables in mysql > " > > The result from queries just before the failure was: > show databases; > Database > information_schema > mtr > mysql > performance_schema > test > > - saving '/home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/log/main.1st/' to '/home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/log/main.1st/' > -------------------------------------------------------------------------- > The servers were restarted 1 times > Spent 0.000 of 4 seconds executing testcases > mysql-test-run: WARNING: Got errors/warnings while running tests, please examine '/home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/log/warnings' for details. > > Completed: Failed 2/2 tests, 0.00% were successful. > > Failing test(s): main.MDEV-23769 main.1st > > The log files in var/log may give you some hint of what went wrong. > > If you want to report this error, please read first the documentation > at http://dev.mysql.com/doc/mysql/en/mysql-test-suite.html > > mysql-test-run: *** ERROR: there were failing test cases > > ``` > > 5. Generate unknown option, for the same example as 3. (note: both failed) > ``` > > $ ./mtr MDEV-23769 1st --mysqld=--xx --force > Logging: ./mtr MDEV-23769 1st --mysqld=--xx --force > vardir: /home/anel/mariadb/10.1-mtr-in-source/mysql-test/var > Checking leftover processes... > Removing old var directory... > Creating var directory '/home/anel/mariadb/10.1-mtr-in-source/mysql-test/var'... > Checking supported features... > MariaDB Version 10.1.47-MariaDB-debug > - SSL connections supported > - binaries are debug compiled > Sphinx 'indexer' binary not found, sphinx suite will be skipped > Collecting tests... > Installing system database... > > ============================================================================== > > TEST RESULT TIME (ms) or COMMENT > -------------------------------------------------------------------------- > > worker[1] Using MTR_BUILD_THREAD 300, with reserved ports 16000..16019 > worker[1] mysql-test-run: WARNING: Process [mysqld.1 - pid: 52531, winpid: 52531, exit: 512] died after mysql-test-run waited 0.3 seconds for /home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/run/mysqld.1.pid to be created. > main.MDEV-23769 [ fail ] > Test ended at 2020-10-03 02:19:01 > > CURRENT_TEST: main.MDEV-23769 > > Failed to start mysqld.1 > mysqltest failed but provided no output > > - saving '/home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/log/main.MDEV-23769/' to '/home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/log/main.MDEV-23769/' > ***Warnings generated in error logs during shutdown after running tests: main.MDEV-23769 > > 2020-10-03 2:19:01 140238937720384 [Warning] option 'lock_wait_timeout': unsigned value 18446744073709551615 adjusted to 31536000 > 2020-10-03 2:19:01 140238937720384 [ERROR] /home/anel/mariadb/10.1-mtr-in-source/sql/mysqld: unknown option '--xx' > 2020-10-03 2:19:01 140238937720384 [ERROR] Aborting > > worker[1] mysql-test-run: WARNING: Process [mysqld.1 - pid: 52537, winpid: 52537, exit: 512] died after mysql-test-run waited 0.3 seconds for /home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/run/mysqld.1.pid to be created. > main.1st [ fail ] > Test ended at 2020-10-03 02:19:02 > > CURRENT_TEST: main.1st > > Failed to start mysqld.1 > mysqltest failed but provided no output > > - saving '/home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/log/main.1st/' to '/home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/log/main.1st/' > ***Warnings generated in error logs during shutdown after running tests: main.1st > > 2020-10-03 2:19:02 140048201700928 [ERROR] /home/anel/mariadb/10.1-mtr-in-source/sql/mysqld: unknown option '--xx' > 2020-10-03 2:19:02 140048201700928 [ERROR] Aborting > > Can't use an undefined value as a symbol reference at ./mtr line 657. > -------------------------------------------------------------------------- > The servers were restarted 1 times > Spent 0.000 of 5 seconds executing testcases > > Completed: Failed 2/2 tests, 0.00% were successful. > > Failing test(s): main.MDEV-23769 main.1st > > The log files in var/log may give you some hint of what went wrong. > > If you want to report this error, please read first the documentation > at http://dev.mysql.com/doc/mysql/en/mysql-test-suite.html > > Errors/warnings were found in logfiles during server shutdown after running the > following sequence(s) of tests: > main.MDEV-23769 > main.1st > mysql-test-run: *** ERROR: there were failing test cases > > ``` > > 6. Again the same option, but with 2 same tests -> not perl error: `Can't use an undefined value as a symbol reference at ./mtr line 657. > ` ` print $child "HELLO\n";` (solution delete the line). > > ``` > $ ./mtr MDEV-23769 MDEV-23769 --mysqld=--xx --force > Logging: ./mtr MDEV-23769 MDEV-23769 --mysqld=--xx --force > vardir: /home/anel/mariadb/10.1-mtr-in-source/mysql-test/var > Checking leftover processes... > Removing old var directory... > Creating var directory '/home/anel/mariadb/10.1-mtr-in-source/mysql-test/var'... > Checking supported features... > MariaDB Version 10.1.47-MariaDB-debug > - SSL connections supported > - binaries are debug compiled > Sphinx 'indexer' binary not found, sphinx suite will be skipped > Collecting tests... > Installing system database... > > ============================================================================== > > TEST RESULT TIME (ms) or COMMENT > -------------------------------------------------------------------------- > > worker[1] Using MTR_BUILD_THREAD 300, with reserved ports 16000..16019 > worker[1] mysql-test-run: WARNING: Process [mysqld.1 - pid: 52595, winpid: 52595, exit: 512] died after mysql-test-run waited 0.3 seconds for /home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/run/mysqld.1.pid to be created. > main.MDEV-23769 [ fail ] > Test ended at 2020-10-03 02:20:12 > > CURRENT_TEST: main.MDEV-23769 > > Failed to start mysqld.1 > mysqltest failed but provided no output > > - saving '/home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/log/main.MDEV-23769/' to '/home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/log/main.MDEV-23769/' > ***Warnings generated in error logs during shutdown after running tests: main.MDEV-23769 > > 2020-10-03 2:20:12 139815724283456 [Warning] option 'lock_wait_timeout': unsigned value 18446744073709551615 adjusted to 31536000 > 2020-10-03 2:20:12 139815724283456 [ERROR] /home/anel/mariadb/10.1-mtr-in-source/sql/mysqld: unknown option '--xx' > 2020-10-03 2:20:12 139815724283456 [ERROR] Aborting > > worker[1] mysql-test-run: WARNING: Process [mysqld.1 - pid: 52601, winpid: 52601, exit: 512] died after mysql-test-run waited 0.3 seconds for /home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/run/mysqld.1.pid to be created. > main.MDEV-23769 [ fail ] > Test ended at 2020-10-03 02:20:12 > > CURRENT_TEST: main.MDEV-23769 > > Failed to start mysqld.1 > mysqltest failed but provided no output > > - saving '/home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/log/main.MDEV-23769/' to '/home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/log/main.MDEV-23769/' > ***Warnings generated in error logs during shutdown after running tests: main.MDEV-23769 > > 2020-10-03 2:20:12 140650797745728 [Warning] option 'lock_wait_timeout': unsigned value 18446744073709551615 adjusted to 31536000 > 2020-10-03 2:20:12 140650797745728 [ERROR] /home/anel/mariadb/10.1-mtr-in-source/sql/mysqld: unknown option '--xx' > 2020-10-03 2:20:12 140650797745728 [ERROR] Aborting > > Can't use an undefined value as a symbol reference at ./mtr line 657. > -------------------------------------------------------------------------- > The servers were restarted 1 times > Spent 0.000 of 5 seconds executing testcases > > Completed: Failed 2/2 tests, 0.00% were successful. > > Failing test(s): main.MDEV-23769 > > The log files in var/log may give you some hint of what went wrong. > > If you want to report this error, please read first the documentation > at http://dev.mysql.com/doc/mysql/en/mysql-test-suite.html > > Errors/warnings were found in logfiles during server shutdown after running the > following sequence(s) of tests: > main.MDEV-23769 > main.MDEV-23769 > mysql-test-run: *** ERROR: there were failing test cases > > ``` > > diff --git a/mysql-test/disabled.def b/mysql-test/disabled.def > index ee30c4f4d3c..fdb62a7e5f5 100644 > --- a/mysql-test/disabled.def > +++ b/mysql-test/disabled.def > @@ -20,3 +20,4 @@ ssl_crl_clrpath : broken upstream > file_contents : MDEV-6526 these files are not installed anymore > max_statement_time : cannot possibly work, depends on timing > partition_open_files_limit : open_files_limit check broken by MDEV-18360 > +MDEV-23769 : Intentionally disabled, for validating purposes. I don't know if it makes any sense. A dead test that nobody will ever run. May be it's better not to add it at all? In the past I've added tests like that too, the "mtr" and "mtr2" suites. I regret it now :) Tests only help if they're run. > diff --git a/mysql-test/mysql-test-run.pl b/mysql-test/mysql-test-run.pl > index 52920530132..8d761fd70cf 100755 > --- a/mysql-test/mysql-test-run.pl > +++ b/mysql-test/mysql-test-run.pl > @@ -1005,6 +1005,13 @@ sub run_worker ($) { > stop_servers(reverse all_servers()); > if(check_warnings_post_shutdown($server)) { > # Warnings appeared in log file(s) during final server shutdown. > + # Make sure first to run statistics > + my $tests= collect_test_cases($opt_reorder, $opt_suites, \@opt_cases, \@opt_skip_test_list); > + my ($prefix, $fail, $completed, $extra_warnings)= > + run_test_server($server, $tests, $opt_parallel); > + mtr_print_line(); > + print_total_times($opt_parallel) if $opt_report_times; > + mtr_report_stats($prefix, $fail, $completed, $extra_warnings); I don't understand the fix. You're saying that if a warning happens on shutdown then mtr won't print the result summary. This doesn't seem to be the case, I've tried few times to repeat it - modified mtr to always pretend there's a warning on shutdown, and modified mysqld to always issue a warning on shutdown - and in all cases I saw the result summary all right. > exit(1); > } > } > @@ -3971,6 +3978,7 @@ sub run_testcase ($$) { > if (start_servers($tinfo)) > { > report_failure_and_restart($tinfo); > + unlink $path_current_testlog; > return 1; > } > } > diff --git a/mysql-test/t/MDEV-23769.opt b/mysql-test/t/MDEV-23769.opt > new file mode 100644 > index 00000000000..fd8dbcab53f > --- /dev/null > +++ b/mysql-test/t/MDEV-23769.opt > @@ -0,0 +1 @@ > +--lock-wait-timeout=-1 > diff --git a/mysql-test/t/MDEV-23769.test b/mysql-test/t/MDEV-23769.test > new file mode 100644 > index 00000000000..9938c7cf644 > --- /dev/null > +++ b/mysql-test/t/MDEV-23769.test > @@ -0,0 +1,4 @@ > +--echo # > +--echo # MDEV-23769: MTR can abort before it prints the test result summary > +--echo # > +show databases; > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 8c745467560: MDEV-23836: Assertion `! is_set() || m_can_overwrite_status' in
by Sergei Golubchik 21 Oct '20

21 Oct '20

Hi, Rucha! what happens in this bug? Why does the server need to open_table() after my_ok()? On Oct 21, Rucha Deodhar wrote: > revision-id: 8c745467560 (mariadb-10.5.4-226-g8c745467560) > parent(s): 308f8350c7b > author: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > committer: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > timestamp: 2020-10-20 11:53:22 +0530 > message: > > MDEV-23836: Assertion `! is_set() || m_can_overwrite_status' in > Diagnostics_area::set_error_status (interrupted ALTER TABLE under LOCK) > > Analysis: Assertion failure happens because we don't ignore kill query > even if 'OK' is already sent. > Fix: Check if we already sent 'OK'. If sent, ignore kill query. > > diff --git a/mysql-test/main/alter_table.result b/mysql-test/main/alter_table.result > index 197e52b35ca..204bfa5249c 100644 > --- a/mysql-test/main/alter_table.result > +++ b/mysql-test/main/alter_table.result > @@ -3322,5 +3322,23 @@ Note 1176 Key 'x' doesn't exist in table 't1' > unlock tables; > drop table t1; > # > +# MDEV-23836: Assertion `! is_set() || m_can_overwrite_status' in > +# Diagnostics_area::set_error_status (interrupted ALTER TABLE under LOCK) > +# > +set @max_session_mem_used_save= @@max_session_mem_used; > +CREATE TABLE t1 (a INT); > +SELECT * FROM t1; > +a > +ALTER TABLE x MODIFY xx INT; > +ERROR 42S02: Table 'test.x' doesn't exist > +SET SESSION max_session_mem_used= 8192; > +LOCK TABLE t1 WRITE; > +ALTER TABLE t1 CHANGE COLUMN IF EXISTS b c INT; > +Warnings: > +Note 1054 Unknown column 'b' in 't1' > +set SESSION max_session_mem_used = @max_session_mem_used_save; > +UNLOCK TABLES; > +DROP TABLE t1; > +# > # End of 10.5 tests > # > diff --git a/mysql-test/main/alter_table.test b/mysql-test/main/alter_table.test > index 6fe61a3222a..d09c4378339 100644 > --- a/mysql-test/main/alter_table.test > +++ b/mysql-test/main/alter_table.test > @@ -2529,6 +2529,27 @@ alter table t1 rename key if exists x to xx; > unlock tables; > drop table t1; > > +--echo # > +--echo # MDEV-23836: Assertion `! is_set() || m_can_overwrite_status' in > +--echo # Diagnostics_area::set_error_status (interrupted ALTER TABLE under LOCK) > +--echo # > + > +set @max_session_mem_used_save= @@max_session_mem_used; > + > +CREATE TABLE t1 (a INT); > +SELECT * FROM t1; > + > +--error ER_NO_SUCH_TABLE > +ALTER TABLE x MODIFY xx INT; > + > +SET SESSION max_session_mem_used= 8192; > +LOCK TABLE t1 WRITE; > +ALTER TABLE t1 CHANGE COLUMN IF EXISTS b c INT; > + > +set SESSION max_session_mem_used = @max_session_mem_used_save; > +UNLOCK TABLES; > +DROP TABLE t1; > + > --echo # > --echo # End of 10.5 tests > --echo # > diff --git a/sql/sql_base.cc b/sql/sql_base.cc > index e6d0ab13e75..e8d39917ba6 100644 > --- a/sql/sql_base.cc > +++ b/sql/sql_base.cc > @@ -1682,7 +1682,9 @@ bool open_table(THD *thd, TABLE_LIST *table_list, Open_table_context *ot_ctx) > > if (!(flags & MYSQL_OPEN_IGNORE_KILLED) && thd->killed) > { > - thd->send_kill_message(); > + /* If we already sent 'ok', we can ignore any kill query statements */ > + if (! thd->get_stmt_da()->is_set()) > + thd->send_kill_message(); > DBUG_RETURN(TRUE); > } > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 1060cc4235a: MDEV-23500: Assertion `!is_set() || (m_status == DA_OK_BULK && is_bulk_op())'
by Sergei Golubchik 21 Oct '20

21 Oct '20

Hi, Rucha! On Oct 21, Rucha Deodhar wrote: > revision-id: 1060cc4235a (mariadb-10.2.31-438-g1060cc4235a) > parent(s): 80075ba011f > author: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > committer: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > timestamp: 2020-09-22 17:12:57 +0530 > message: > > MDEV-23500: Assertion `!is_set() || (m_status == DA_OK_BULK && is_bulk_op())' > failed in Diagnostics_area::set_ok_status upon multi-table update > > Analysis: When NULL field is copied to NOT-NULL so there is error but > this error is not reported so we get the failure. > Fix: Return the error state. > > diff --git a/mysql-test/r/update.result b/mysql-test/r/update.result > index 9e19abc4e9c..c1d37fc36e7 100644 > --- a/mysql-test/r/update.result > +++ b/mysql-test/r/update.result > @@ -731,4 +731,14 @@ INSERT INTO t2 VALUES > UPDATE t1,t2 SET t1.i1 = -39 WHERE t2.d1 <> t1.i1 AND t2.d1 = t1.d2; > ERROR 22007: Incorrect datetime value: '19' for column `test`.`t1`.`i1` at row 1 > DROP TABLE t1,t2; > +# > +# MDEV-23500: Assertion `!is_set() || (m_status == DA_OK_BULK && > +# is_bulk_op())' failed in Diagnostics_area::set_ok_status upon > +# multi-table update > +# > +CREATE TABLE t (f INT NOT NULL); > +INSERT INTO t VALUES (0),(0); > +UPDATE t AS t1 LEFT JOIN t AS t2 ON t1.f <=> t2.f SET t2.f = NULL; > +ERROR 01000: Data truncated for column 'f' at row 4 with a single-table update I get MariaDB [test]> update t set f=NULL; ERROR 1048 (23000): Column 'f' cannot be null should multi-table update produce the same error? > +DROP TABLE t; Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 9c2570e6c42: MDEV-22422: Assertion `! is_set()' failed in Diagnostics_area::set_eof_status
by Sergei Golubchik 21 Oct '20

21 Oct '20

Hi, Rucha! On Oct 21, Rucha Deodhar wrote: > revision-id: 9c2570e6c42 (mariadb-10.1.43-279-g9c2570e6c42) > parent(s): 5768f57d243 > author: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > committer: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > timestamp: 2020-09-22 17:18:37 +0530 > message: > > MDEV-22422: Assertion `! is_set()' failed in Diagnostics_area::set_eof_status > > Analysis: The error is not returned when the statement can't be used. > And so we still go on to search for keywords. > Fix: Return the error state. > > diff --git a/mysql-test/t/errors.test b/mysql-test/t/errors.test > index 55461002fd4..e2b10559299 100644 > --- a/mysql-test/t/errors.test > +++ b/mysql-test/t/errors.test > @@ -213,3 +213,22 @@ select * from seq_1_to_1000; > --enable_result_log > # We may not be able to execute any more queries with this connection > # because of too little memory# > + > +--echo # > +--echo # 10.1 Test > +--echo # > +--echo # MDEV-22422: Assertion `! is_set()' failed in > +--echo # Diagnostics_area::set_eof_status > +--echo # > + > +SET @saved_max_session_mem_used = @@max_session_mem_used; > + > +SET max_session_mem_used = 50000; > +--error ER_OPTION_PREVENTS_STATEMENT > +help 'it is going to crash'; > + > +SET @@max_session_mem_used=@saved_max_session_mem_used; > + > +--echo # > +--echo # End of 10.1 Test > +--echo # > diff --git a/sql/sql_help.cc b/sql/sql_help.cc > index cd9da566e51..654a3915b29 100644 > --- a/sql/sql_help.cc > +++ b/sql/sql_help.cc > @@ -736,6 +736,9 @@ static bool mysqld_help_internal(THD *thd, const char *mask) > &name, &description, &example); > delete select; > > + if (thd->is_error()) > + goto error; does the error happen during the first count_topics= search_topics() ? There are lots of places below that look kind of fragile too. May be they're safe. Or may be by increasing max_session_mem_used somewhat you can get mysqld_help_internal to crash below again and again? Could you look into that please? > if (count_topics == 0) > { > int UNINIT_VAR(key_id); > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 6729bf5c031: MDEV-22313: Neither SHOW CREATE USER nor SHOW GRANTS prints a user's default role
by Sergei Golubchik 21 Oct '20

21 Oct '20

Hi, Anel! On Oct 21, Anel Husakovic wrote: > revision-id: 6729bf5c031 (mariadb-10.1.43-112-g6729bf5c031) > parent(s): ad4b70562bb > author: Anel Husakovic <anel(a)mariadb.org> > committer: Anel Husakovic <anel(a)mariadb.org> > timestamp: 2020-04-22 20:48:55 +0200 > message: > > MDEV-22313: Neither SHOW CREATE USER nor SHOW GRANTS prints a user's default role > > diff --git a/mysql-test/t/grant5.test b/mysql-test/t/grant5.test > index 649bba7d1ca..5eefd9b9185 100644 > --- a/mysql-test/t/grant5.test > +++ b/mysql-test/t/grant5.test > @@ -33,3 +33,16 @@ REVOKE EXECUTE ON PROCEDURE sp FROM u; > --error ER_TABLE_NOT_LOCKED > REVOKE PROCESS ON *.* FROM u; > DROP TABLE t1; > + > +# > +# MDEV-22313: Neither SHOW CREATE USER nor SHOW GRANTS prints a user's default role > +# > +CREATE ROLE test_role; > +CREATE USER test_user; > +GRANT test_role TO test_user; > +SET DEFAULT ROLE test_role FOR test_user; > +SHOW GRANTS FOR test_user; > +SET DEFAULT ROLE NONE for test_user; > +SHOW GRANTS FOR test_user; add a test for SHOW GRANTS without FOR username > +DROP USER test_user; > +DROP ROLE test_role; > diff --git a/sql/sql_acl.cc b/sql/sql_acl.cc > index b6a6f806e50..90dd552ec43 100644 > --- a/sql/sql_acl.cc > @@ -7891,6 +7893,10 @@ bool mysql_show_grants(THD *thd, LEX_USER *lex_user) > if (show_role_grants(thd, username, hostname, acl_user, buff, sizeof(buff))) > goto end; > > + /* Show default role to acl_user */ > + if (show_default_role(thd, username, hostname, acl_user, buff, sizeof(buff))) > + goto end; Please, print SET DEFAULT ROLE after all grants for this user. > + > /* Add first global access grants */ > if (show_global_privileges(thd, acl_user, FALSE, buff, sizeof(buff))) > goto end; > @@ -7963,6 +7969,42 @@ static ROLE_GRANT_PAIR *find_role_grant_pair(const LEX_STRING *u, > my_hash_search(&acl_roles_mappings, (uchar*)pair_key.ptr(), key_length); > } > > +static bool show_default_role(THD *thd, const char *username, > + const char *hostname, ACL_USER_BASE *acl_entry, why did you cast acl_entry (which you know is ACL_USER) to the base class ACL_USER_BASE? why did you pass the username if you don't use it? > + char *buff, size_t buffsize) > +{ > + Protocol *protocol= thd->protocol; > + LEX_STRING host= {const_cast<char*>(hostname), strlen(hostname)}; This doesn't make much sense. You only create this LEX_STRING to use it for String::append(LEX_STRING *). But you can directly use String::append(const char*) and you won't need any LEX_STRING here. Furthermore, you can print acl_entry->host.hostname just as you print acl_entry->user, so you don't need the above hostname argument at all. > + String def_role(buff,sizeof(buff),system_charset_info); > + def_role.length(0); you don't use def_role anywhere. > + LEX_STRING def_rolename= ((ACL_USER *)acl_entry)->default_rolename; > + if (def_rolename.length) > + { > + String def_str(buff,sizeof(buff),system_charset_info); > + def_str.length(0); > + def_str.append("SET DEFAULT ROLE "); prefer to use STRING_WITH_LEN() for string literals. E.g. here it should be def_str.append(STRING_WITH_LEN("SET DEFAULT ROLE ")); > + def_str.append(def_rolename.str, def_rolename.length, > + system_charset_info); here you can do def_str.append(&def_rolename); > + def_str.append(" FOR '"); > + def_str.append(acl_entry->user.str, acl_entry->user.length, > + system_charset_info); > + if (!(acl_entry->flags & IS_ROLE)) better put an assert here not an if > + { > + def_str.append(STRING_WITH_LEN("'@'")); > + def_str.append(&host); > + } > + def_str.append('\''); > + protocol->prepare_for_resend(); > + protocol->store(def_str.ptr(),def_str.length(),def_str.charset()); > + if (protocol->write()) > + { > + return TRUE; > + } > + } > + return FALSE; > +} > + > static bool show_role_grants(THD *thd, const char *username, > const char *hostname, ACL_USER_BASE *acl_entry, > char *buff, size_t buffsize) > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 8c34a242aa2: MDEV-20787: Script dgcov.pl does not work
by Sergei Golubchik 21 Oct '20

21 Oct '20

Hi, Anel! On Oct 21, Anel Husakovic wrote: > revision-id: 8c34a242aa2 (mariadb-10.2.31-478-g8c34a242aa2) > parent(s): 01ffccd6a4e > author: Anel Husakovic <anel(a)mariadb.org> > committer: Anel Husakovic <anel(a)mariadb.org> > timestamp: 2020-10-06 00:00:58 +0200 > message: > > MDEV-20787: Script dgcov.pl does not work > > - This patch is done after doing test on 00f9df29224ada9cdbb796c99f4 > where difference was found between versions of gcov. don't refer to commits that are not pushed and are not going to be pushed. Because if you push this one you'll have a commit forever in the history that refers to an non-existing commit. It isn't helpful. > - This patch doesn't take gcov versions in account since is extracting > filename of a `.gcov` file, which is generated with `gcov -i > <source-file>.gcda` during `dgcov.pl --generate`. > - There is a check if file exists. > Reason for that is because some generated `gcov` files are outliers from above > rule (that a file generated in intermediate format(gcov) has the same source name). > Example 1: `aestables.cpp.gcda` is not generating `aestables.cpp.gcov`, > but only the files used in #include `modes.hpp.gcov runtime.hpp.gcov` > Example 2: `my_sha256.cc.gcda` is generating `my_sha.ic.gcov` > - Make sure to test `MTR_BINDIR` which gets set during out-of-source > build and generate error when running `./mysql-test/mtr --gcov alias` What is a problem with out of source builds? If they don't work, perhaps it would be better to fix dgcov to work with out of source builds? > - During the testing: > 10.1 results are stored in > - `<basedir>//mysql-test-gcov.msg` and errors in > - `<basedir>//mysql-test-gcov.err` > 10.2 results should be stored in > `./mysql-test/var/last_changes.dgcov`, but only empty results are > obtained (tests `1st` and `alias`). I don't understand this ^^^ paragraph. And also because your patch is 10.2+ so who cares what it does on 10.1? > diff --git a/mysql-test/dgcov.pl b/mysql-test/dgcov.pl > index fbc5540e697..f2bbdad4d20 100755 > --- a/mysql-test/dgcov.pl > +++ b/mysql-test/dgcov.pl > @@ -161,8 +161,14 @@ sub gcov_one_file { > system($cmd)==0 or die "system($cmd): $? $!"; > } > > - # now, read the generated file > - open FH, '<', "$_.gcov" or die "open(<$_.gcov): $!"; > + (my $filename = $_)=~ s/\.[^.]+$//; # remove extension In your previous patch you either removed the extension or not based on gcov version. Why did you abandon this logic? > + my $gcov_file_path= $File::Find::dir."/$filename.gcov"; File::Find documentation seems to be saying that $File::Find::dir ends with a slash (/) so you don't need to prefix $filename.gcov with it. > + if (! -f $gcov_file_path) > + { > + return; > + } may be return unless -f $gcov_file_path it's perl after all :) > + open FH, '<', "$gcov_file_path" or die "open(<$gcov_file_path): $!"; you don't need double quotes around "$gcov_file_path" - they don't serve any purpose. > + > my $fname; > while (<FH>) { > chomp; Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] f83c1615d97: MDEV-20207: Assertion `! is_set()' failed in Diagnostics_area::set_eof_status
by Sergei Golubchik 21 Oct '20

21 Oct '20

Hi, Rucha! On Oct 21, Rucha Deodhar wrote: > revision-id: f83c1615d97 (mariadb-10.1.43-240-gf83c1615d97) > parent(s): 7e9ffc69ecd > author: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > committer: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > timestamp: 2020-08-05 00:33:03 +0530 > message: > > MDEV-20207: Assertion `! is_set()' failed in Diagnostics_area::set_eof_status > upon HANDLER READ > > Analysis: The error state is not stored while checking condition and key name. > Fix: Return true while checking condition and key name if error is reported > because geometry object can't be created from the data in the index value for > HANDLER READ. > > diff --git a/mysql-test/suite/handler/aria.test b/mysql-test/suite/handler/aria.test > index 1913d2b791c..50ef43ce5b2 100644 > --- a/mysql-test/suite/handler/aria.test > +++ b/mysql-test/suite/handler/aria.test > @@ -80,3 +80,23 @@ HANDLER t1 CLOSE; > DROP TABLE t1; > > --echo End of 5.1 tests > + > +--echo # > +--echo # 10.1 Test > +--echo # > +--echo # MDEV-20207: Assertion `! is_set()' failed in > +--echo # Diagnostics_area::set_eof_status upon HANDLER READ > +--echo # > + > +DROP TABLE IF EXISTS t1; there's no need to start the test from DROP TABLE IF EXISTS t1; historically people customarily did that. before mtr had after-test checks. but now all tests are supposed to clean up after itself, you do not need to worry about tests leaving tables around. > + > +CREATE TABLE t1 (a POINT, KEY(a)); > +HANDLER t1 OPEN h; > + > +--error ER_CANT_CREATE_GEOMETRY_OBJECT > +HANDLER h READ a = (0); I don't think it should be an error. SELECT * FROM t1 WHERE a=(0) -- this does not fail, it just doesn't find anything. I think HANDLER should do the same. > + > +HANDLER h CLOSE; > +DROP TABLE t1; note that all handler test files source handler.inc. that is, if you want a test to run for all engines, you do not need to add it to all individual test files, add it to handler.inc Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] e3f45b2f9ea: MDEV-10267 Add ngram fulltext parser plugin
by Sergei Golubchik 21 Oct '20

21 Oct '20

Hi, Rinat! On Oct 21, Rinat Ibragimov wrote: > > > + > > > + // Avoid creating partial n-grams by stopping at word boundaries. > > > + // Underscore is treated as a word character too, since identifiers in > > > + // programming languages often contain them. > > > + if (!is_alnum(char_type) && !is_underscore(cs, next, end)) { > > > + start= next + char_len; > > > + next= start; > > > + n_chars= 0; > > > + continue; > > > > I don't think it's a good idea. On the opposite, I would rather create > > ngrams over word boundaries, so that, say, I'd split "n-gram plugin" to > > > > "n-g", "-gr", "gra", "ram", "am ", "m p", " pl", "plu", "lug", "ugi", "gin". > > I see how this can increase index size: more different strings are > included. Implementation will become a tiny bit simpler. But can it > make searching better somehow? Yes, the idea is that it'll rank "One can use n-gram plugin in MariaDB fulltext search" higher than "Plugins are useful for extending functionality of a program" because some ngrams encode that "n-gram" and "plugin" words must be together as a phrase. > I can't decide. From my point of view, the current approach is fine. > Please pick a variant, and I'll try to implement that. No, I cannot guess which approach will produce more relevant searches. Implement something and then we test what works better > > > + } > > > + > > > + next += char_len; > > > + n_chars++; > > > + > > > + if (n_chars == ngram_token_size) { > > > + bool_info->position= static_cast<uint>(start - doc); > > > > nope, we don't have MYSQL_FTPARSER_FULL_BOOLEAN_INFO::position and this > > plugin is definitely not the reason to break the API and add it. > > There is a patch in the patchset that adds the "position" field. Without > that field, > it's possible to trigger an assertion in InnoDB's FTS code by executing: > > INSTALL PLUGIN simple_parser SONAME 'mypluglib'; > CREATE TABLE articles( > a TEXT DEFAULT NULL, > b TEXT DEFAULT NULL, > FULLTEXT (a, b) WITH PARSER simple_parser > ) ENGINE=InnoDB; > INSERT INTO articles VALUES ('111', '1234 1234 1234'); > DROP TABLE articles; > UNINSTALL PLUGIN simple_parser; > > Same assertion may be triggered by any fts plugin. Are you sure you > want that patch to be removed? I don't think InnoDB assert is a reason to extend the fulltext plugin API. What about this fix instead: diff --git a/storage/innobase/fts/fts0fts.cc b/storage/innobase/fts/fts0fts.cc --- a/storage/innobase/fts/fts0fts.cc +++ b/storage/innobase/fts/fts0fts.cc @@ -4669,7 +4669,7 @@ fts_tokenize_add_word_for_parser( ut_ad(boolean_info->position >= 0); position = boolean_info->position + fts_param->add_pos; */ - position = fts_param->add_pos; + position = fts_param->add_pos++; fts_add_token(result_doc, str, position); this fixed the crash for me. > > > + n_chars= ngram_token_size - 1; > > > + is_first= false; > > > + } > > > + } > > > + > > > + // If nothing was emitted yet, emit the remainder. > > > > what reminder, what is it for? > > It's "remainder", with "a" before "i". Remainder of the processed > string. That way strings that are too short for having even a single > n-gram, will still generate some index entries and will be > discoverable by exact queries. Perhaps the comment should say it? That if the string too short, it'll be emitted here? > > > + case MYSQL_FTPARSER_FULL_BOOLEAN_INFO: > > > + // Reusing InnoDB's boolean mode parser to handle all special characters. > > > + while (fts_get_word(cs, &start, end, &word, &bool_info)) { > > > > nope. InnoDB might not even be compiled it. Use param->mysql_parse() > > Copied fts_get_word() implementation into the plugin code. > > As far as I understand, param->mysql_parse() cannot be used here as it > splits a string into words like default fulltext search does. This > plugin however must generate n-grams. Of course, it can. Note that fts_get_word() doesn't generate n-grams either, it gets the whole word and the n-gram plugin later splits it into n-grams. Similarly param->mysql_parse() will extract words for you and you'll split them into n-grams. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

[Maria-developers] Please assign PRs that modify debian/ contents for Otto to review
by Otto Kekäläinen 20 Oct '20

20 Oct '20

Hello! While working on putting MariaDB 10.5.5 into Debian I noticed that there were a couple of commits that landed more or less on the same day as the release was made and "broke" it just before release. Now there are surprises in a stable release I need to fix and the situation puts extra strain on me. It is good that many people participate in maintaining the contents of debian/ in the MariaDB Server, because I don't have time to do everything. I would however appreciate that when changes are made, I would be given a chance to review them. I will review them post-release anyway and it would save a lot of trouble, if we can fix stuff during the development cycle and not having to patch a post-release in order to be able to put it forward to Debian/Ubuntu. So, please make PRs of any debian/ changes you make, and feel free to add me as the reviewer (or one of the reviewers). This policy can also be automated using a .github/CODEOWNERS file, so I made a PR about it at https://github.com/MariaDB/server/pull/1650 Thanks! PS. I will deliver a MariaDB 10.5 and MariaDB in Debian status talk at https://debconf20.debconf.org/talks/98-whats-new-in-mariadb-server-105-and-… in case anybody is interested to listen in. -- Follow me at @ottokekalainen

2 3

Re: [Maria-developers] db1fbafbead: MDEV-23935: Fix warnings generated during compilation of plugin/auth_pam/testing/pam_mariadb_mtr.c on MacOS
by Sergei Golubchik 19 Oct '20

19 Oct '20

Hi, Dmitry! ok to push On Oct 19, Dmitry Shulga wrote: > revision-id: db1fbafbead (mariadb-10.4.11-420-gdb1fbafbead) > parent(s): c2ac0ce1f02 > author: Dmitry Shulga <dmitry.shulga(a)mariadb.com> > committer: Dmitry Shulga <dmitry.shulga(a)mariadb.com> > timestamp: 2020-10-11 21:51:45 +0700 > message: > > MDEV-23935: Fix warnings generated during compilation of plugin/auth_pam/testing/pam_mariadb_mtr.c on MacOS > > During build server 10.4 on MacOS warnings like the following one > are generated on compiling the file plugin/auth_pam/testing/pam_mariadb_mtr.c > > server-10.4/plugin/auth_pam/testing/pam_mariadb_mtr.c:25:22: error: > initializing 'char *' with an expression of type 'const char [23]' > discards qualifiers [-Werror,-Wincompatible-pointer-types-discards-qualifiers] > { PAM_TEXT_INFO, "Challenge input first." } > > The reason of the warnings is that the data member pam_message::msg is declared > as 'char *' on MacOS but initializer of the data member is 'const char *'. > > To eliminate warnings the compiler option > -Wno-incompatible-pointer-types-discards-qualifiers > has been added to the compiler flags used for compiling the file > pam_mariadb_mtr.c. > > --- > plugin/auth_pam/testing/CMakeLists.txt | 6 ++++++ > 1 file changed, 6 insertions(+) > > diff --git a/plugin/auth_pam/testing/CMakeLists.txt b/plugin/auth_pam/testing/CMakeLists.txt > index 889cb524402..c8d2e3cbb97 100644 > --- a/plugin/auth_pam/testing/CMakeLists.txt > +++ b/plugin/auth_pam/testing/CMakeLists.txt > @@ -4,6 +4,12 @@ ADD_LIBRARY(pam_mariadb_mtr MODULE pam_mariadb_mtr.c) > SET_TARGET_PROPERTIES (pam_mariadb_mtr PROPERTIES PREFIX "") > TARGET_LINK_LIBRARIES(pam_mariadb_mtr pam) > > +IF(APPLE) > + SET_SOURCE_FILES_PROPERTIES( > + pam_mariadb_mtr.c > + PROPERTY COMPILE_FLAGS "-Wno-incompatible-pointer-types-discards-qualifiers") > +ENDIF() > + > SET(dest DESTINATION "${INSTALL_MYSQLTESTDIR}/suite/plugins/pam" COMPONENT Test) > INSTALL(TARGETS pam_mariadb_mtr ${dest}) > INSTALL(FILES mariadb_mtr.conf RENAME mariadb_mtr ${dest}) > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 727d97feb99: MDEV-23926: Fix warnings generated during compilation of plugin/auth_pam/mapper/pam_user_map.c on MacOS
by Sergei Golubchik 19 Oct '20

19 Oct '20

Hi, Dmitry! plugin code should stay in the plugin directory. please move the check to plugin/auth_pam/CMakeLists.txt Also, what about an alternative fix: #ifdef HAVE_POSIX_GETGROUPLIST typedef gid_t my_gid_t; #else typedef int my_gid_t; #end would that help but with less ifdefs? On Oct 19, Dmitry Shulga wrote: > revision-id: 727d97feb99 (mariadb-10.2.31-489-g727d97feb99) > parent(s): 8984d779103 > author: Dmitry Shulga <dmitry.shulga(a)mariadb.com> > committer: Dmitry Shulga <dmitry.shulga(a)mariadb.com> > timestamp: 2020-10-09 20:35:48 +0700 > message: > > MDEV-23926: Fix warnings generated during compilation of plugin/auth_pam/mapper/pam_user_map.c on MacOS > > Compiler warnings listed below are generated during server build on MacOS: > > 88%] Building C object plugin/auth_pam/CMakeFiles/pam_user_map.dir/mapper/pam_user_map.c.o > /Users/shulga/projects/mariadb/server-10.2/plugin/auth_pam/mapper/pam_user_map.c:87:41: error: passing > 'gid_t *' (aka 'unsigned int *') to parameter of type 'int *' converts between pointers to integer types > with different sign [-Werror,-Wpointer-sign] > if (getgrouplist(user, user_group_id, loc_groups, &ng) < 0) > ^~~~~~~~~~ > /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/include/unistd.h:650:43: note: > passing argument to parameter here > int getgrouplist(const char *, int, int *, int *); > ^ > /Users/shulga/projects/mariadb/server-10.2/plugin/auth_pam/mapper/pam_user_map.c:95:46: error: passing > 'gid_t *' (aka 'unsigned int *') to parameter of type 'int *' converts between pointers to integer types > with different sign [-Werror,-Wpointer-sign] > (void) getgrouplist(user, user_group_id, loc_groups, &ng); > ^~~~~~~~~~ > /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/includ > e/unistd.h:650:43: note: > passing argument to parameter here > int getgrouplist(const char *, int, int *, int *); > ^ > 2 errors generated. > > In case MariaDB serer is build with -DCMAKE_BUILD_TYPE=Debug it results in > uild error. > > The reason of compiler warnings is that declaration of the Posix C API function > getgrouplist() on MacOS differs from declration of getgrouplist() proposed > by Posix. > > To suppress this compiler warning cmake configure was adapted to detect what > kind of getgrouplist() function is declared on the building platform and > set the macros HAVE_POSIX_GETGROUPLIST in case the building platform suppors > Posix compatible interface of the getgrouplist() function. Depending on > whether this macros is set the compatible type of arguments is used to pass > parameter values to the function. > > --- > config.h.cmake | 1 + > configure.cmake | 22 ++++++++++++++++++++++ > plugin/auth_pam/mapper/pam_user_map.c | 13 ++++++++++++- > 3 files changed, 35 insertions(+), 1 deletion(-) > > diff --git a/config.h.cmake b/config.h.cmake > index 0e19dd44694..07e4647f0bf 100644 > --- a/config.h.cmake > +++ b/config.h.cmake > @@ -188,6 +188,7 @@ > #cmakedefine HAVE_POSIX_FALLOCATE 1 > #cmakedefine HAVE_LINUX_FALLOC_H 1 > #cmakedefine HAVE_FALLOC_PUNCH_HOLE_AND_KEEP_SIZE 1 > +#cmakedefine HAVE_POSIX_GETGROUPLIST 1 > #cmakedefine HAVE_PREAD 1 > #cmakedefine HAVE_PAUSE_INSTRUCTION 1 > #cmakedefine HAVE_FAKE_PAUSE_INSTRUCTION 1 > diff --git a/configure.cmake b/configure.cmake > index 58e1111bfc7..c6db3ecb912 100644 > --- a/configure.cmake > +++ b/configure.cmake > @@ -1031,3 +1031,25 @@ IF(NOT MSVC) > HAVE_FALLOC_PUNCH_HOLE_AND_KEEP_SIZE > ) > ENDIF() > + > +# Check whether getgrouplist uses git_t for second and third arguments. > +SET(CMAKE_REQUIRED_FLAGS -Werror) > +CHECK_C_SOURCE_COMPILES( > +" > +#ifdef HAVE_GRP_H > +#include <grp.h> > +#endif > +#ifdef HAVE_UNISTD_H > +#include <unistd.h> > +#endif > +int main() { > + char *arg_1; > + gid_t arg_2, arg_3; > + int arg_4; > + (void)getgrouplist(arg_1,arg_2,&arg_3,arg_4); > + return 0; > +} > +" > +HAVE_POSIX_GETGROUPLIST > +) > +SET(CMAKE_REQUIRED_FLAGS) > diff --git a/plugin/auth_pam/mapper/pam_user_map.c b/plugin/auth_pam/mapper/pam_user_map.c > index 9d7ed53f8b1..b0d2affac5c 100644 > --- a/plugin/auth_pam/mapper/pam_user_map.c > +++ b/plugin/auth_pam/mapper/pam_user_map.c > @@ -31,6 +31,8 @@ These comments are written to the syslog as 'authpriv.debug' > and usually end up in /var/log/secure file. > */ > > +#include <my_config.h> > + > #include <stdlib.h> > #include <stdio.h> > #include <ctype.h> > @@ -72,8 +74,13 @@ static const char debug_keyword[]= "debug"; > > static int populate_user_groups(const char *user, gid_t **groups) > { > +#ifdef HAVE_POSIX_GETGROUPLIST > gid_t user_group_id; > gid_t *loc_groups= *groups; > +#else > + int user_group_id; > + int *loc_groups= (int*)*groups; > +#endif > int ng; > > { > @@ -88,12 +95,16 @@ static int populate_user_groups(const char *user, gid_t **groups) > { > /* The rare case when the user is present in more than */ > /* GROUP_BUFFER_SIZE groups. */ > +#ifdef HAVE_POSIX_GETGROUPLIST > loc_groups= (gid_t *) malloc(ng * sizeof (gid_t)); > +#else > + loc_groups= (int *) malloc(ng * sizeof (gid_t)); > +#endif > if (!loc_groups) > return 0; > > (void) getgrouplist(user, user_group_id, loc_groups, &ng); > - *groups= loc_groups; > + *groups= (gid_t*)loc_groups; > } > > return ng; > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 5e71a3e0df7: MDEV-23925: Fix warnings generated during compilation of mysys_ssl/openssl.c on MacOS
by Sergei Golubchik 19 Oct '20

19 Oct '20

Hi, Dmitry! ok to push (again, comments below) On Oct 19, Dmitry Shulga wrote: > revision-id: 5e71a3e0df7 (mariadb-10.2.31-489-g5e71a3e0df7) > parent(s): 8984d779103 > author: Dmitry Shulga <dmitry.shulga(a)mariadb.com> > committer: Dmitry Shulga <dmitry.shulga(a)mariadb.com> > timestamp: 2020-10-09 20:55:14 +0700 > message: > > MDEV-23925: Fix warnings generated during compilation of mysys_ssl/openssl.c on MacOS > > Compiler warnings listed below are generated during server build on MacOS: > In file included from server-10.2-MDEV-23564/mysys_ssl/openssl.c:33: > In file included from /usr/local/include/openssl/evp.h:16: > In file included from /usr/local/include/openssl/bio.h:20: > /usr/local/include/openssl/crypto.h:206:10: warning: 'CRYPTO_cleanup_all_ex_data' macro redefined [-Wmacro-redefined] > ^ > /mariadb/server-10.2-MDEV-23564/include/ssl_compat.h:46:9: note: previous definition is here > ^ > In file included from /mariadb/server-10.2-MDEV-23564/mysys_ssl/openssl.c:33: > /usr/local/include/openssl/evp.h:501:10: warning: 'EVP_MD_CTX_init' macro redefined [-Wmacro-redefined] > ^ > /mariadb/server-10.2-MDEV-23564/include/ssl_compat.h:33:9: note: previous definition is here > ^ > In file included from /mariadb/server-10.2-MDEV-23564/mysys_ssl/openssl.c:33: > /usr/local/include/openssl/evp.h:627:11: warning: 'EVP_CIPHER_CTX_init' macro redefined [-Wmacro-redefined] > ^ > /mariadb/server-10.2-MDEV-23564/include/ssl_compat.h:35:9: note: previous definition is here > ^ > In file included from /mariadb/server-10.2-MDEV-23564/mysys_ssl/openssl.c:33: > /usr/local/include/openssl/evp.h:866:11: warning: 'EVP_cleanup' macro redefined [-Wmacro-redefined] > ^ > /mariadb/server-10.2-MDEV-23564/include/ssl_compat.h:44:9: note: previous definition is here > ^ > 4 warnings generated. generally these long commit comments are not any more clear than if you'd only listed one compilation failure. > > In case MariaDB serer is build with -DCMAKE_BUILD_TYPE=Debug it results in > build error. > > The reason of compiler warnings is that header file <ssl_compat.h> > included before the openssl system header files. File ssl_compat.h > contains some macros with the name as SSL API functions declared > in the openssl system header files. It resulted in duplicate > symbols thats produces compiler warnings. > > To fix the issue the header file ssl_compat.h should be included > after a line where openssl system header is included. > > --- > mysys_ssl/openssl.c | 4 ++-- > 1 file changed, 2 insertions(+), 2 deletions(-) > > diff --git a/mysys_ssl/openssl.c b/mysys_ssl/openssl.c > index f3dc8f4277c..a11db113b14 100644 > --- a/mysys_ssl/openssl.c > +++ b/mysys_ssl/openssl.c > @@ -15,7 +15,6 @@ > Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA */ > > #include <my_global.h> > -#include <ssl_compat.h> > > /* > The check is only done for OpenSSL 1.1.x. > @@ -25,13 +24,14 @@ > */ > > #ifndef HAVE_OPENSSL11 > +#include <ssl_compat.h> > int check_openssl_compatibility() > { > return 0; > } > #else > #include <openssl/evp.h> > - > +#include <ssl_compat.h> > static uint testing, alloc_size, alloc_count; > > static void *coc_malloc(size_t size, const char *f __attribute__((unused)), > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] b92597c220f: MDEV-23564: CMAKE failing due to deprecated Apple GSS method
by Sergei Golubchik 19 Oct '20

19 Oct '20

Hi, Dmitry! ok to push. but see comments below On Oct 19, Dmitry Shulga wrote: > revision-id: b92597c220f (mariadb-10.2.31-479-gb92597c220f) > parent(s): 577c61e8be0 > author: Dmitry Shulga <dmitry.shulga(a)mariadb.com> > committer: Dmitry Shulga <dmitry.shulga(a)mariadb.com> > timestamp: 2020-10-08 00:48:47 +0700 > message: > > MDEV-23564: CMAKE failing due to deprecated Apple GSS method > > Some GSS-API functions like gss_import_name(), gss_release_buffer() > used in plugin/auth_gssapi and libmariadb/plugins/auth are marked > as deprecated in MacOS starting from version 10.14+. It results in > extra warnings output on server building. > > To eliminate extra warnings the flag '-Wno-deprecated-declarations' > has been added to compiler invocation string for those source > files that invoke deprecated GSS-API functions. > > diff --git a/libmariadb b/libmariadb > index a746c3af449..fd0dc672309 160000 > --- a/libmariadb > +++ b/libmariadb > @@ -1 +1 @@ > -Subproject commit a746c3af449a8754e78ad7971e59e79af7957cdb > +Subproject commit fd0dc6723095f066648a7c3b2a48523b3002e392 be careful when pushing this commit. you'd need something like 1. rebase your libmariadb fix on top of 3.1, push into 3.1 2. rebase your server fix on top of 10.2 2. update submodule reference in the server repo 3. amend your commit fix and push > diff --git a/plugin/auth_gssapi/CMakeLists.txt b/plugin/auth_gssapi/CMakeLists.txt > index bca4f5af3a1..7a3c32a00b1 100644 > --- a/plugin/auth_gssapi/CMakeLists.txt > +++ b/plugin/auth_gssapi/CMakeLists.txt > @@ -18,6 +18,13 @@ ELSE() > SET(GSSAPI_SERVER gssapi_server.cc) > SET(GSSAPI_ERRMSG gssapi_errmsg.cc) > > + IF(APPLE) > + SET_SOURCE_FILES_PROPERTIES( > + ${GSSAPI_CLIENT} ${GSSAPI_SERVER} ${GSSAPI_ERRMSG} > + PROPERTY COMPILE_FLAGS "-Wno-deprecated-declarations") > +# SET_SOURCE_FILES_PROPERTIES(${GSSAPI_SERVER} PROPERTY COMPILE_FLAGS "-Wno-deprecated-declarations") > +# SET_SOURCE_FILES_PROPERTIES(${GSSAPI_ERRMSG} PROPERTY COMPILE_FLAGS "-Wno-deprecated-declarations") you don't need these commented lines anymore, do you? Is it possible to use a compiler on APPLE that doesn't understand -Wno-deprecated-declarations? If not - your fix it ok. If it is - better use MY_CHECK_AND_SET_COMPILER_FLAG > + ENDIF() > SET(CMAKE_REQUIRED_INCLUDES ${GSSAPI_INCS}) > SET(CMAKE_REQUIRED_LIBRARIES ${GSSAPI_LIBS}) > INCLUDE(CheckCXXSymbolExists) Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 2d73406dad8: MDEV-16481: set global system_versioning_asof=sf() crashes in specific case
by Sergei Golubchik 19 Oct '20

19 Oct '20

Hi, Nikita! ok to push (better late than never :) On Oct 19, Nikita Malyavin wrote: > revision-id: 2d73406dad8 (mariadb-10.3.16-39-g2d73406dad8) > parent(s): 609f358a9bd > author: Nikita Malyavin <nikitamalyavin(a)gmail.com> > committer: Sergei Golubchik <serg(a)mariadb.com> > timestamp: 2020-10-19 17:37:04 +0200 > message: > > MDEV-16481: set global system_versioning_asof=sf() crashes in specific case > > * sys_vars.h: add `MYSQL_TIME` field to `set_var::save_result` > * sys_vars.ic: get rid of calling `var->value->get_date()` from `Sys_var_vers_asof::update()` > * versioning.sysvars: add test; remove double warning > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] e3f45b2f9ea: MDEV-10267 Add ngram fulltext parser plugin
by Sergei Golubchik 19 Oct '20

19 Oct '20

Hi, Rinat! On Oct 19, Rinat Ibragimov wrote: > revision-id: e3f45b2f9ea (mariadb-10.2.31-368-ge3f45b2f9ea) > parent(s): 44c48cefdbe > author: Rinat Ibragimov <ri(a)tempesta-tech.com> > committer: Rinat Ibragimov <ri(a)tempesta-tech.com> > timestamp: 2020-08-25 01:59:06 +0300 > message: > > MDEV-10267 Add ngram fulltext parser plugin > > Ngram is a fulltext parser plugin that splits words into overlapping > segments of fixed lengths. For example, if 3-grams are in use, string > "abcdef" is split into "abc", "bcd", "cde", "def", thus allowing an > efficient substring search for CJK texts where splitting by words in not > feasible. Only word characters are going into n-grams. Spaces and > punctuation characters are treated as separators. > > diff --git a/plugin/fulltext/CMakeLists.txt b/plugin/fulltext/CMakeLists.txt > index 3c37666cb3e..9704733a9aa 100644 > --- a/plugin/fulltext/CMakeLists.txt > +++ b/plugin/fulltext/CMakeLists.txt > @@ -18,3 +18,7 @@ set(CMAKE_CXX_FLAGS_RELWITHDEBINFO "${CMAKE_CXX_FLAGS_RELWITHDEBINFO} -DNDEBUG") > > MYSQL_ADD_PLUGIN(ftexample plugin_example.c > MODULE_ONLY MODULE_OUTPUT_NAME "mypluglib" COMPONENT Test) > + > +# n-gram parser > +MYSQL_ADD_PLUGIN(ftngram ngram_parser/plugin_ngram.cc > + MODULE_ONLY MODULE_OUTPUT_NAME "fulltext_ngram") why to put the plugin under plugin/fulltext/ngram_parser/ ? it's only one file, just put it in plugin/fulltext/ > diff --git a/plugin/fulltext/mysql-test/ngram/disabled.def b/plugin/fulltext/mysql-test/ngram/disabled.def > new file mode 100644 > index 00000000000..0ca9dd33ef7 > --- /dev/null > +++ b/plugin/fulltext/mysql-test/ngram/disabled.def > @@ -0,0 +1 @@ > +ngram_gb18030 : GB18030 character set is not supported yet. I don't think there's any need to add tests for unsupported charsets. The plugin is trivial, no need to test it for every possible charset that we have and don't have. > diff --git a/plugin/fulltext/mysql-test/ngram/ngram_basic.inc b/plugin/fulltext/mysql-test/ngram/ngram_basic.inc > --- /dev/null > +++ b/plugin/fulltext/mysql-test/ngram/ngram_basic.inc > @@ -0,0 +1,25 @@ > +--echo > +--eval CREATE TABLE t (lang TEXT, str TEXT, FULLTEXT(str) WITH PARSER ngram) ENGINE=$engine CHARSET $charset > + > +# Some character are not possible to represent in some encodings. Ignore them. > +--disable_warnings > +--disable_query_log > +# http://www.gerolf.org/doc/polyglot/polyglot.htm. > +INSERT IGNORE INTO t VALUES > + ('Chinese', '他们为什么不说中文？'), > + ('English', 'Why can''t they just speak English?'), > + ('Japanese', 'なぜ、みんな日本語を話してくれないのか？'), > + ('Korean', '세계의 모든 사람들이 한국어를 이해한다면 얼마나 좋을까?'); > +--enable_query_log > +--enable_warnings > + > +SELECT * FROM t ORDER BY lang; > + > +SELECT * FROM t WHERE MATCH(str) AGAINST('之一' IN BOOLEAN MODE) ORDER BY lang; > +SELECT * FROM t WHERE MATCH(str) AGAINST('数据' IN BOOLEAN MODE) ORDER BY lang; > +SELECT * FROM t WHERE MATCH(str) AGAINST('zzz' IN BOOLEAN MODE) ORDER BY lang; > +SELECT * FROM t WHERE MATCH(str) AGAINST('gli' IN BOOLEAN MODE) ORDER BY lang; > +SELECT * FROM t WHERE MATCH(str) AGAINST('ない' IN BOOLEAN MODE) ORDER BY lang; > +SELECT * FROM t WHERE MATCH(str) AGAINST('람들' IN BOOLEAN MODE) ORDER BY lang; > + > +DROP TABLE t; > diff --git a/plugin/fulltext/mysql-test/ngram/ngram_basic.test b/plugin/fulltext/mysql-test/ngram/ngram_basic.test > new file mode 100644 > index 00000000000..eaa8829c3ee > --- /dev/null > +++ b/plugin/fulltext/mysql-test/ngram/ngram_basic.test > @@ -0,0 +1,25 @@ > +# Ngram fulltext parser basic functionality. > + > +SET NAMES utf8mb4; > +SELECT @@ngram_token_size; > + > +let $engine_index= 3; > +while ($engine_index) { > + if ($engine_index == 3) { let $engine= InnoDB; } > + if ($engine_index == 2) { let $engine= MyISAM; } > + if ($engine_index == 1) { let $engine= Aria; } > + > + let $charset_index= 6; > + while ($charset_index) { > + if ($charset_index == 6) { let $charset= utf8mb4; } > + if ($charset_index == 5) { let $charset= big5; } > + if ($charset_index == 4) { let $charset= gbk; } > + if ($charset_index == 3) { let $charset= gb2312; } > + if ($charset_index == 2) { let $charset= sjis; } > + if ($charset_index == 1) { let $charset= ujis; } > + > + source ngram_basic.inc; > + dec $charset_index; > + } eh. unroll the loop please, write let $engine= InnoDB; let $charset= utf8mb4; source ngram_basic.inc; let $charset= big5; source ngram_basic.inc; ... Or, better: CREATE TABLE t (lang TEXT, str TEXT, FULLTEXT(str) WITH PARSER ngram) ENGINE=InnoDB CHARSET utf8mb4; source ngram_basic.inc; CREATE TABLE t (lang TEXT, str TEXT, FULLTEXT(str) WITH PARSER ngram) ENGINE=InnoDB CHARSET big5; source ngram_basic.inc; ... > + dec $engine_index; > +} > diff --git a/plugin/fulltext/mysql-test/ngram/ngram_missing_plugin.result b/plugin/fulltext/mysql-test/ngram/ngram_missing_plugin.result > new file mode 100644 > index 00000000000..aa17b1d6053 > --- /dev/null > +++ b/plugin/fulltext/mysql-test/ngram/ngram_missing_plugin.result > @@ -0,0 +1,23 @@ there are tests for the missing fulltest parser plugin. they don't depend on the individial plugin (that is, btw, *not installed*). No need to repeat them for every plugin, remove this test. > +SET NAMES utf8mb4; > +CREATE TABLE t (str TEXT, FULLTEXT(str) WITH PARSER ngram) > +ENGINE=InnoDB; > +INSERT INTO t VALUES ('Hello World'), ('abcdef ghijklm'); > +# Restart with ngram plugin disabled. > +SET NAMES utf8mb4; > +SELECT * FROM t WHERE MATCH(str) AGAINST('bcd') ORDER BY str; > +ERROR HY000: Plugin 'ngram' is not loaded > +SELECT * FROM t ORDER BY str; > +ERROR HY000: Plugin 'ngram' is not loaded > +INSERT INTO t VALUES ('something'); > +ERROR HY000: Plugin 'ngram' is not loaded > +# Restart with ngram plugin enabled again. > +SET NAMES utf8mb4; > +SELECT * FROM t WHERE MATCH(str) AGAINST('bcd') ORDER BY str; > +str > +abcdef ghijklm > +SELECT * FROM t ORDER BY str; > +str > +abcdef ghijklm > +Hello World > +INSERT INTO t VALUES ('something'); > +DROP TABLE t; > diff --git a/plugin/fulltext/mysql-test/ngram/ngram_token_size.result b/plugin/fulltext/mysql-test/ngram/ngram_token_size.result > new file mode 100644 > index 00000000000..7043e147d5b > --- /dev/null > +++ b/plugin/fulltext/mysql-test/ngram/ngram_token_size.result > @@ -0,0 +1,109 @@ > +# Check 'ngram_token_size' variable. > +# > +# It's a global variable. > +SELECT @@global.ngram_token_size; > +@@global.ngram_token_size > +2 > +SHOW GLOBAL VARIABLES LIKE 'ngram_token_size'; > +Variable_name Value > +ngram_token_size 2 > +SELECT * FROM information_schema.global_variables WHERE variable_name='ngram_token_size'; > +VARIABLE_NAME VARIABLE_VALUE > +NGRAM_TOKEN_SIZE 2 > +# > +# Not a session variable. > +SELECT @@session.ngram_token_size; > +ERROR HY000: Variable 'ngram_token_size' is a GLOBAL variable > +SHOW SESSION VARIABLES LIKE 'ngram_token_size'; > +Variable_name Value > +ngram_token_size 2 > +SELECT * FROM information_schema.session_variables WHERE variable_name='ngram_token_size'; > +VARIABLE_NAME VARIABLE_VALUE > +NGRAM_TOKEN_SIZE 2 > +# > +# 0 is outside allowed range [1, 10], so it's changed to 1. > +SET GLOBAL ngram_token_size= 0; > +Warnings: > +Warning 1292 Truncated incorrect ngram_token_size value: '0' > +SELECT * FROM information_schema.global_variables WHERE variable_name='ngram_token_size'; > +VARIABLE_NAME VARIABLE_VALUE > +NGRAM_TOKEN_SIZE 1 > +# > +# 10 is an acceptable value. > +SET GLOBAL ngram_token_size= 10; > +SELECT * FROM information_schema.global_variables WHERE variable_name='ngram_token_size'; > +VARIABLE_NAME VARIABLE_VALUE > +NGRAM_TOKEN_SIZE 10 > +# > +# 1 is an acceptable value. > +SET GLOBAL ngram_token_size= 1; > +SELECT * FROM information_schema.global_variables WHERE variable_name='ngram_token_size'; > +VARIABLE_NAME VARIABLE_VALUE > +NGRAM_TOKEN_SIZE 1 > +# > +# 11 is outside allowed range [1, 10], so it's changed to 10. > +SET GLOBAL ngram_token_size= 11; > +Warnings: > +Warning 1292 Truncated incorrect ngram_token_size value: '11' > +SELECT * FROM information_schema.global_variables WHERE variable_name='ngram_token_size'; > +VARIABLE_NAME VARIABLE_VALUE > +NGRAM_TOKEN_SIZE 10 ^^^^ all tests above aren't needed, it's enough to do query_vertical select * from information_schema.system_variables where variable_name='ngram_token_size'; all the behavior above is implemented outside of the plugin, no need to test it. > +# > +# Try 3-grams. > +SET GLOBAL ngram_token_size= 3; > +CREATE TABLE t (str TEXT, FULLTEXT INDEX (str) WITH PARSER ngram) Engine=InnoDB; > +INSERT INTO t VALUES > +('Lorem ipsum dolor sit amet, consectetur adipiscing elit.'), > +('Maecenas non pellentesque odio.'), > +('Sed fermentum egestas sapien vel consequat.'); > +# Only one row contains substring 'api'. > +SELECT str FROM t WHERE MATCH(str) AGAINST('api'); > +str > +Sed fermentum egestas sapien vel consequat. > +# 'lent' is parsed into ('len', 'ent') list. Two rows contain at least one of the tokens. > +SELECT str FROM t WHERE MATCH(str) AGAINST('lent*'); > +str > +Maecenas non pellentesque odio. > +Sed fermentum egestas sapien vel consequat. > +# 'zzz' is not present anywhere, so result is empty. > +SELECT str FROM t WHERE MATCH(str) AGAINST('zzzz'); > +str > +# 'piz' cannot be found anywhere, but 'api' is in one of the rows. > +SELECT str FROM t WHERE MATCH(str) AGAINST('apiz'); > +str > +Sed fermentum egestas sapien vel consequat. > +# > +# Try switching to 2-grams while keeping the index. > +SET GLOBAL ngram_token_size= 2; > +# 'api' is now parsed into ('ap', 'pi'). Since index only contains 3-grams, nothing is found. > +SELECT str FROM t WHERE MATCH(str) AGAINST('api'); > +str > +# > +# Add some data while ngram size is set to 2. > +INSERT INTO t VALUES('1234 56789'); > +# > +# Now index contains 3-grams from the rows added before and a number of > +# 2-grams from the just added row. > +# Querying for 'api' still returns nothing, but line with '678' is found. > +SELECT str from t WHERE MATCH(str) AGAINST('api'); > +str > +SELECT str FROM t WHERE MATCH(str) AGAINST('678'); > +str > +1234 56789 > +# > +# Switching back to 3-grams make intitially added rows searchable. > +SET GLOBAL ngram_token_size= 3; > +SELECT str FROM t WHERE MATCH(str) AGAINST('api'); > +str > +Sed fermentum egestas sapien vel consequat. > +# > +# But row with '678' in it cannot be found, as it was added with 2-grams. > +SELECT str FROM t WHERE MATCH(str) AGAINST('678'); > +str > +# > +# Try matching in boolean mode. > +SELECT str FROM t WHERE MATCH(str) AGAINST('+ent -api' IN BOOLEAN MODE); > +str > +Maecenas non pellentesque odio. > +# > +DROP TABLE t; > diff --git a/plugin/fulltext/mysql-test/ngram/ngram_ucs2.result b/plugin/fulltext/mysql-test/ngram/ngram_ucs2.result > new file mode 100644 > index 00000000000..9d82743bd24 > --- /dev/null > +++ b/plugin/fulltext/mysql-test/ngram/ngram_ucs2.result > @@ -0,0 +1,8 @@ > +CREATE TABLE t (str TEXT, FULLTEXT INDEX (str) WITH PARSER ngram) Engine=InnoDB CHARSET=UCS2; > +ERROR HY000: Column 'str' cannot be part of FULLTEXT index > +CREATE TABLE t (str TEXT, FULLTEXT INDEX (str) WITH PARSER ngram) Engine=InnoDB CHARSET=UTF16; > +ERROR HY000: Column 'str' cannot be part of FULLTEXT index > +CREATE TABLE t (str TEXT, FULLTEXT INDEX (str) WITH PARSER ngram) Engine=InnoDB CHARSET=UTF16LE; > +ERROR HY000: Column 'str' cannot be part of FULLTEXT index > +CREATE TABLE t (str TEXT, FULLTEXT INDEX (str) WITH PARSER ngram) Engine=InnoDB CHARSET=UTF32; > +ERROR HY000: Column 'str' cannot be part of FULLTEXT index same as above. This is fulltext index limitation, it does not depend on the parser plugin. The plugin doesn't need to test for it. > diff --git a/plugin/fulltext/mysql-test/ngram/suite.opt b/plugin/fulltext/mysql-test/ngram/suite.opt > new file mode 100644 > index 00000000000..ab37cea996e > --- /dev/null > +++ b/plugin/fulltext/mysql-test/ngram/suite.opt > @@ -0,0 +1,4 @@ > +--plugin-load-add=$FULLTEXT_NGRAM_SO > +--loose-innodb no need to enable innodb explicitly, `source have_innodb.inc` does that. > +--loose-enable-innodb_ft_index_table > +--loose-innodb_ft_index_cache why did you enable these two plugins if you don't use them in your tests? > diff --git a/plugin/fulltext/ngram_parser/plugin_ngram.cc b/plugin/fulltext/ngram_parser/plugin_ngram.cc > new file mode 100644 > index 00000000000..80498c3299b > --- /dev/null > +++ b/plugin/fulltext/ngram_parser/plugin_ngram.cc > @@ -0,0 +1,256 @@ > +/* > + Copyright (c) 2014, 2017, Oracle and/or its affiliates. All rights reserved. > + Copyright (c) 2020, MariaDB. > + > + This program is free software; you can redistribute it and/or modify > + it under the terms of the GNU General Public License, version 2.0, > + as published by the Free Software Foundation. > + > + This program is also distributed with certain software (including > + but not limited to OpenSSL) that is licensed under separate terms, > + as designated in a particular file or component or in included license > + documentation. The authors of MySQL hereby grant you an additional > + permission to link the program and your derivative works with the > + separately licensed software that they have included with MySQL. > + > + This program is distributed in the hope that it will be useful, > + but WITHOUT ANY WARRANTY; without even the implied warranty of > + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > + GNU General Public License, version 2.0, for more details. > + > + You should have received a copy of the GNU General Public License > + along with this program; if not, write to the Free Software > + Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA */ > + > +#include <stddef.h> > +#include <limits.h> > +#include <my_attribute.h> > +#include <my_global.h> > +#include "../storage/innobase/include/fts0tokenize.h" No, the plugin should be self-contained and not include anything from other plugins. At least if a plugin is as trivial as that one. > + > +#define U_LOW_LINE 0x5f // Code point for underscore '_'. > +#define is_alnum(ct) ((ct) & (_MY_U | _MY_L | _MY_NMR)) > + > +static int ngram_token_size; > + > +static MYSQL_SYSVAR_INT(token_size, ngram_token_size, > + 0, > + "Ngram full text plugin parser token size in characters", > + NULL, NULL, 2, 1, 10, 0); > + > +static int my_ci_charlen(CHARSET_INFO *cs, const uchar *str, const uchar *end) > +{ > + return my_charlen(cs, reinterpret_cast<const char *>(str), > + reinterpret_cast<const char *>(end)); > +} > + > +static int my_ci_ctype(CHARSET_INFO *cs, int *char_type, const uchar *str, > + const uchar *end) > +{ > + return cs->cset->ctype(cs, char_type, str, end); > +} > + > +static bool is_underscore(CHARSET_INFO *cs, const uchar *str, const uchar *end) > +{ > + my_wc_t wc; > + cs->cset->mb_wc(cs, &wc, str, end); > + return wc == U_LOW_LINE; > +} > + > +// Splits a string into ngrams and emits them. > +static int split_into_ngrams(MYSQL_FTPARSER_PARAM *param, const uchar *doc, > + int len, MYSQL_FTPARSER_BOOLEAN_INFO *bool_info) > +{ > + const CHARSET_INFO *cs= param->cs; > + const uchar *start= doc; > + const uchar *end= doc + len; > + const uchar *next= start; > + int n_chars= 0; > + int ret= 0; > + bool is_first= true; > + > + while (next < end) { > + int char_type; > + int char_len= my_ci_ctype(cs, &char_type, next, end); > + > + // Broken data? > + if (next + char_len > end || char_len == 0) > + break; > + > + // Avoid creating partial n-grams by stopping at word boundaries. > + // Underscore is treated as a word character too, since identifiers in > + // programming languages often contain them. > + if (!is_alnum(char_type) && !is_underscore(cs, next, end)) { > + start= next + char_len; > + next= start; > + n_chars= 0; > + continue; I don't think it's a good idea. On the opposite, I would rather create ngrams over word boundaries, so that, say, I'd split "n-gram plugin" to "n-g", "-gr", "gra", "ram", "am ", "m p", " pl", "plu", "lug", "ugi", "gin". May be better would be to split it into "n-gr", "gra", "ram", "am p", "m pl", "plu", "lug", "ugi", "gin". (that is, always three letters in an n-gram, not three characters). This needs testing to find out what approach works better. > + } > + > + next += char_len; > + n_chars++; > + > + if (n_chars == ngram_token_size) { > + bool_info->position= static_cast<uint>(start - doc); nope, we don't have MYSQL_FTPARSER_FULL_BOOLEAN_INFO::position and this plugin is definitely not the reason to break the API and add it. > + ret= param->mysql_add_word(param, reinterpret_cast<const char *>(start), > + static_cast<int>(next - start), bool_info); > + if (ret != 0) > + return ret; > + > + start += my_ci_charlen(cs, start, end); why do you scan the string again here, you've already counted character by character above? > + n_chars= ngram_token_size - 1; > + is_first= false; > + } > + } > + > + // If nothing was emitted yet, emit the remainder. what reminder, what is it for? > + switch (param->mode) { > + case MYSQL_FTPARSER_FULL_BOOLEAN_INFO: > + case MYSQL_FTPARSER_WITH_STOPWORDS: > + if (n_chars > 0 && is_first) { > + assert(next > start); > + assert(n_chars < ngram_token_size); > + > + bool_info->position= static_cast<uint>(start - doc); > + ret= param->mysql_add_word(param, reinterpret_cast<const char *>(start), > + static_cast<int>(next - start), bool_info); > + } > + break; > + > + default: > + break; > + } > + > + return ret; > +} > + > +static int number_of_chars(const CHARSET_INFO *cs, const uchar *buf, int len) > +{ > + const uchar *ptr= buf; > + const uchar *end= buf + len; > + int size= 0; > + > + while (ptr < end) { > + ptr += my_ci_charlen(cs, ptr, end); > + size += 1; > + } > + > + return size; > +} > + > +// Convert term into phrase and handle wildcard. > +static int ngram_term_convert(MYSQL_FTPARSER_PARAM *param, const uchar *token, > + int len, MYSQL_FTPARSER_BOOLEAN_INFO *bool_info) > +{ > + MYSQL_FTPARSER_BOOLEAN_INFO token_info= > + { FT_TOKEN_WORD, 0, 0, 0, 0, 0, ' ', 0 }; > + const CHARSET_INFO *cs= param->cs; > + int ret= 0; > + > + assert(bool_info->type == FT_TOKEN_WORD); > + assert(bool_info->quot == NULL); > + > + /* Convert rules: > + * 1. if term with wildcard and term length is less than ngram_token_size, > + * we keep it as normal term search. > + * 2. otherwise, term is converted to phrase and wildcard is ignored. > + * e.g. 'abc' and 'abc*' are both equivalent to '"ab bc"'. > + */ > + > + if (bool_info->trunc && number_of_chars(cs, token, len) < ngram_token_size) { > + ret= param->mysql_add_word(param, reinterpret_cast<const char *>(token), > + len, bool_info); > + } else { > + bool_info->type= FT_TOKEN_LEFT_PAREN; > + bool_info->quot= reinterpret_cast<char*>(1); > + > + ret= param->mysql_add_word(param, NULL, 0, bool_info); > + if (ret != 0) > + return ret; > + > + ret= split_into_ngrams(param, token, len, &token_info); > + if (ret != 0) > + return ret; > + > + bool_info->type= FT_TOKEN_RIGHT_PAREN; > + ret= param->mysql_add_word(param, NULL, 0, bool_info); > + > + assert(bool_info->quot == NULL); > + bool_info->type= FT_TOKEN_WORD; > + } > + > + return ret; > +} > + > +static int ngram_parser_parse(MYSQL_FTPARSER_PARAM *param) > +{ > + MYSQL_FTPARSER_BOOLEAN_INFO bool_info= > + { FT_TOKEN_WORD, 0, 0, 0, 0, 0, ' ', 0 }; > + const CHARSET_INFO *cs= param->cs; > + uchar *start= reinterpret_cast<uchar *>(const_cast<char *>(param->doc)); > + uchar *end= start + param->length; > + FT_WORD word= {NULL, 0, 0}; > + int ret= 0; > + > + switch (param->mode) { > + case MYSQL_FTPARSER_SIMPLE_MODE: > + case MYSQL_FTPARSER_WITH_STOPWORDS: > + ret= split_into_ngrams(param, start, static_cast<int>(end - start), > + &bool_info); > + break; > + > + case MYSQL_FTPARSER_FULL_BOOLEAN_INFO: > + // Reusing InnoDB's boolean mode parser to handle all special characters. > + while (fts_get_word(cs, &start, end, &word, &bool_info)) { nope. InnoDB might not even be compiled it. Use param->mysql_parse() > + if (bool_info.type == FT_TOKEN_WORD) { > + if (bool_info.quot != NULL) { > + ret= split_into_ngrams(param, word.pos, word.len, &bool_info); > + } else { > + ret= ngram_term_convert(param, word.pos, word.len, &bool_info); > + } > + } else { > + ret= param->mysql_add_word(param, reinterpret_cast<char*>(word.pos), > + word.len, &bool_info); > + } > + > + if (ret != 0) > + return ret; > + } > + > + break; > + } > + > + return ret; > +} > + > +static struct st_mysql_ftparser ngram_parser_descriptor= > +{ > + MYSQL_FTPARSER_INTERFACE_VERSION, > + ngram_parser_parse, > + NULL, > + NULL, > +}; > + > +static struct st_mysql_sys_var *ngram_system_variables[]= > +{ > + MYSQL_SYSVAR(token_size), > + NULL > +}; > + > +maria_declare_plugin(ngram_parser) > +{ > + MYSQL_FTPARSER_PLUGIN, // Type. > + &ngram_parser_descriptor, // Descriptor. > + "ngram", // Name. > + "Oracle Corp", // Author. > + "Ngram Full-Text Parser", // Description. > + PLUGIN_LICENSE_GPL, // License. > + NULL, // Initialization function. > + NULL, // Deinitialization function. > + 0x0100, // Numeric version. > + NULL, // Status variables. > + ngram_system_variables, // System variables. > + "1.0", // String version representation. > + MariaDB_PLUGIN_MATURITY_EXPERIMENTAL, // Maturity. > +} > +maria_declare_plugin_end; > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 12d5c4f2813: Fixed problems with S3 after "DROP TABLE FORCE" changes
by Sergei Golubchik 19 Oct '20

19 Oct '20

Hi, Monty! On Oct 16, Michael Widenius wrote: > revision-id: 12d5c4f2813 (mariadb-10.5.4-190-g12d5c4f2813) > parent(s): afa5f735f30 > author: Michael Widenius <michael.widenius(a)gmail.com> > committer: Michael Widenius <michael.widenius(a)gmail.com> > timestamp: 2020-09-13 17:48:15 +0300 > message: > > Fixed problems with S3 after "DROP TABLE FORCE" changes > > MDEV-23691 S3 storage engine: delayed slave can drop the table please put the MDEV on the first line of the commit comment > > This commit also fixes all failing replication S3 tests. > > A slave is delayed if it is trying to execute replicated queries on a > table that is already converted to S3 by the master. What do you mean, a slave is delayed? > > Fixes for replication events on S3 tables for delayed slaves: > - INSERT and INSERT ... SELECT and CREATE TABLE are ignored but written > to the binary log. UPDATE & DELETE will be fixed in a future commit. > > Other things: > - On slaves with --s3-slave-ignore-updates set, allow S3 tables to be > opened in read-write mode. This was done to be able to > ignore-but-replicate queries like insert. Without this change any > open of an S3 table failed with 'Table is read only' which is too > early to be able to replicate the original query. > - Errors are now printed if handler::extra() call fails in > wait_while_tables_are_used() > - Error message for row changes are changed from HA_ERR_WRONG_COMMAND > to HA_ERR_TABLE_READONLY > - Disable some maria_extra() calls for S3 tables. This could cause > S3 tables to fail in some cases. > - Added missing thr_lock_delete() to ma_open() in case of failure > > diff --git a/sql/log.cc b/sql/log.cc > index 2a887a68606..9c656387e90 100644 > --- a/sql/log.cc > +++ b/sql/log.cc > @@ -2151,6 +2151,12 @@ static int binlog_commit(handlerton *hton, THD *thd, bool all) > > DBUG_RETURN(0); > } > + /* > + This is true if we are doing an alter table that is replicated as > + CREATE TABLE ... SELECT > + */ > + if (thd->variables.option_bits & OPTION_BIN_COMMIT_OFF) > + DBUG_RETURN(0); why binlog_commit is even called here? > > DBUG_PRINT("debug", > ("all: %d, in_transaction: %s, all.modified_non_trans_table: %s, stmt.modified_non_trans_table: %s", > diff --git a/sql/sql_class.cc b/sql/sql_class.cc > index f4cbda5490c..7c103756566 100644 > --- a/sql/sql_class.cc > +++ b/sql/sql_class.cc > @@ -7476,6 +7476,34 @@ int THD::binlog_query(THD::enum_binlog_query_type qtype, char const *query_arg, > DBUG_RETURN(0); > } > > + > +/** > + Binlog current query as a statement if binary log is open > + > + @retval false ok > + @retval true error > +*/ > + > + > +bool THD::binlog_current_query() > +{ > + if (!mysql_bin_log.is_open()) > + return 0; > + > + /* > + The following is needed if the query was "ignored" by a shared distributed > + engine, in which case decide_logging_format() will set the binlog filter > + */ > + reset_binlog_local_stmt_filter(); > + clear_binlog_local_stmt_filter(); > + return binlog_query(THD::STMT_QUERY_TYPE, query(), query_length(), > + /* is_trans */ FALSE, > + /* direct */ FALSE, > + /* suppress_use */ FALSE, > + /* Error */ 0) > 0; > +} you created a generally looking function with a rather special behavior. Please rename it to make sure it's not a general purpose helper. For example, call it binlog_current_query_unfiltered() or force_binlog_current_query(). And the function comment should say Force-binlog current query as a statement ignoring the binlog filter setting > + > + > void > THD::wait_for_wakeup_ready() > { > diff --git a/sql/sql_insert.cc b/sql/sql_insert.cc > index 4ad4c478937..5f59e064a35 100644 > --- a/sql/sql_insert.cc > +++ b/sql/sql_insert.cc > @@ -724,6 +724,7 @@ bool mysql_insert(THD *thd, TABLE_LIST *table_list, > Item *unused_conds= 0; > DBUG_ENTER("mysql_insert"); > > + bzero((char*) &info,sizeof(info)); why did you move it up? > create_explain_query(thd->lex, thd->mem_root); > /* > Upgrade lock type if the requested lock is incompatible with > @@ -764,16 +765,27 @@ bool mysql_insert(THD *thd, TABLE_LIST *table_list, > DBUG_RETURN(TRUE); > value_count= values->elements; > > - if (mysql_prepare_insert(thd, table_list, table, fields, values, > - update_fields, update_values, duplic, > - &unused_conds, FALSE)) > + if ((res= mysql_prepare_insert(thd, table_list, table, fields, values, > + update_fields, update_values, duplic, > + &unused_conds, FALSE))) > + { > + retval= thd->is_error(); > + if (res < 0) > + { > + /* > + Insert should be ignored but we have to log the query in statement > + format in the binary log > + */ > + res= thd->binlog_current_query(); abort label doesn't look at `res` it returns `retval`. So I don't think you need `retval= thd->is_error()` at all, and instead you need if (res < 0) retval= thd->binlog_current_query(); > + } > goto abort; > + } > + /* mysql_prepare_insert sets table_list->table if it was not set */ > + table= table_list->table; > > /* Prepares LEX::returing_list if it is not empty */ > if (returning) > result->prepare(returning->item_list, NULL); > - /* mysql_prepare_insert sets table_list->table if it was not set */ > - table= table_list->table; why did you move this up? > > context= &thd->lex->first_select_lex()->context; > /* > @@ -1575,21 +1587,29 @@ bool mysql_prepare_insert(THD *thd, TABLE_LIST *table_list, > DBUG_ASSERT (!select_insert || !values); > > if (mysql_handle_derived(thd->lex, DT_INIT)) > DBUG_RETURN(1); > if (table_list->handle_derived(thd->lex, DT_MERGE_FOR_INSERT)) > DBUG_RETURN(1); > if (thd->lex->handle_list_of_derived(table_list, DT_PREPARE)) > DBUG_RETURN(1); > > if (duplic == DUP_UPDATE) > { > /* it should be allocated before Item::fix_fields() */ > if (table_list->set_insert_values(thd->mem_root)) > DBUG_RETURN(1); > + } > + > + if (!table) > + table= table_list->table; As far as I can see, table in mysql_prepare_insert is *always* either table_list->table or NULL. At least in 10.5. So I'd rather remove the table argument completely and let mysql_prepare_insert always use table_list->table. > + > + if (table->file->check_if_updates_are_ignored("INSERT")) > + { > + DBUG_RETURN(-1); > } > > if (mysql_prepare_insert_check_table(thd, table_list, fields, select_insert)) > DBUG_RETURN(1); > > /* Prepare the fields in the statement. */ > if (values) > diff --git a/sql/sql_table.cc b/sql/sql_table.cc > index 15d190c3139..2843496f783 100644 > --- a/sql/sql_table.cc > +++ b/sql/sql_table.cc > @@ -5206,7 +5215,19 @@ int create_table_impl(THD *thd, const LEX_CSTRING &orig_db, > goto err; > } > else if (options.if_not_exists()) > + { > + /* > + We never come here as part of normal create table as table existance > + is checked in open_and_lock_tables(). We may come here as part of > + ALTER TABLE > + */ How so? There is no ALTER IF NOT EXISTS. And this is the old code, so it seems to be for CREATE TABLE IF NOT EXISTS. > + > + /* Log CREATE IF NOT EXISTS on slave for distributed engines */ > + if (thd->slave_thread && (exists_hton && exists_hton->flags & > + HTON_IGNORE_UPDATES)) > + thd->log_current_statement= 1; > goto warn; Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] MDEV-21265: IN predicate conversion to IN subquery should be allowed for a broader set of datatype comparison
by Sergey Petrunia 16 Oct '20

16 Oct '20

Hi Varun, > Commit 3e29c5a1620deef2fa9b25a0d42b4c07fe59e96f > > MDEV-21265: IN predicate conversion to IN subquery should be allowed for a broader set of datatype comparison > > Allow materialization strategy when the collation of the columns of the > left side of the IN subquery are not same as the one on the right side > of the IN subquery but are compatible with each other. > This allows conversion from utf8mb3 to utf8mb4 as the former is a subset of the later. Please fix the comment to say what the patch does: When collations on the inner and outer sides are the same, and the character set of the inner side is a proper subset of the charset on the outer. > > diff --git a/mysql-test/main/subselect4.result b/mysql-test/main/subselect4.result > index 020a70642913..f3ae2398d45e 100644 > --- a/mysql-test/main/subselect4.result > +++ b/mysql-test/main/subselect4.result > @@ -2718,3 +2718,172 @@ Warning 1931 Query execution was interrupted. The query examined at least 3020 r > SET join_cache_level= @save_join_cache_level; > DROP TABLE t1,t2,t3,t4; > # End of 10.2 tests > +# > +# MDEV-21265: IN predicate conversion to IN subquery should be allowed for a broader set of datatype comparison > +# > +CREATE TABLE t1 ( col1 varchar (32) collate utf8_general_ci); > +CREATE TABLE t2 ( col1 varchar (32) collate utf8mb4_general_ci, key(col1)); > +INSERT INTO t1 select seq from seq_1_to_100; > +INSERT INTO t2 values (_utf8mb4 X'F09F9883'), (1),('?'); ... > EXPLAIN FORMAT=JSON SELECT HEX(col1), col1 IN (SELECT col1 FROM t1) FROM t2; Please insert the '?' into the table t1, not t2. The idea of the check is to check for the following: - We get smiley symbol from table t2. - We make the lookup key from it. This results in an error and the '?' as output. - Then, a lookup is made in t1. And we are trying to provoke a wrong result by having the '?' in t1. OK to push into the custom-build tree when the above is addressed. BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] 0d05c9f8699: Add mytop to Client component for rpm package
by Sergei Golubchik 13 Oct '20

13 Oct '20

Hi, Anel! About the diff to 1.9, I tend to agree that our mytop is a superset. I did have some comments when looking at the diff, couldn't help it :) Here they are: > diff --git a/scripts/mytop.sh b/../mytop-1.9.1-4 > index 321b79537fc..056b7a9a806 100644 > --- b/../mytop-1.9.1-4 > +++ a/scripts/mytop.sh > @@ -1,13 +1,13 @@ > -#!/usr/bin/perl -w > +#!/usr/bin/env perl I thought we're getting rid of "env perl" ? > # > -# $Id: mytop,v 1.91 2012/01/18 16:49:12 mgrennan Exp $ > +# $Id: mytop,v 1.99-maria6 2019/10/22 14:53:51 jweisbuch Exp $ > > =pod > > =head1 NAME > > -mytop - display MySQL server performance info like `top' > +mytop - display MariaDB server performance info like `top' may be "MariaDB/MySQL" here and below? > > =cut > > @@ -242,11 +253,15 @@ my $dsn; > > ## Socket takes precedence. > > -$dsn ="DBI:mysql:database=$config{db};mysql_read_default_group=mytop;"; > +if (eval {DBI->install_driver("MariaDB")}) { > + $dsn = "DBI:MariaDB:database=$config{db};mariadb_read_default_group=mytop;"; > +} else { > + $dsn = "DBI:mysql:database=$config{db};mysql_read_default_group=mytop;"; > +} > > if ($config{socket} and -S $config{socket}) > { > - $dsn .= "mysql_socket=$config{socket}"; > + $dsn .= "mariadb_socket=$config{socket}"; if you support both drivers, you need to use the correct option here > } > else > { > @@ -1027,10 +918,15 @@ sub GetData() > } > } > > - open L, "</proc/loadavg"; > - my $l = <L>; > - close L; > - chomp $l; > + my $l; > + if (-e "/proc/loadavg") > + { > + ## To avoid warnings if the OS is not Linux > + open (my $fh, "<", "/proc/loadavg"); > + ## Only the first 3 values are interresting rr? > + $l = join(" ", (split /\s+/, <$fh>)[0..2]); > + close $fh; > + } > > $last_time = $now_time; > > @@ -1960,11 +1694,15 @@ sub PrintHelp() > S - change slow query hightlighting > t - switch to thread view (default) > u - show only a specific user > + V - show variables > : - enter a command (not yet implemented) > ! - Skip an error that has stopped replications (at your own risk) > + L - show full queries (do not strip to terminal width) > + w - adjust the User and DB columns width > + a - toggle the progress column > > -${GREEN}http://jeremy.zawodny.com/mysql/mytop/${RESET} > -${GREEN}http://www.mysqlfanboy.com/mytop-3/${RESET} > +Base version from ${GREEN}http://www.mysqlfanboy.com/mytop-3${RESET} if it's mytop-3, perhaps it should have the version 3.x and not 1.99-maria6 ? > +This version comes as part of the ${GREEN}MariaDB${RESET} distribution. > ]; > > print $help; Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

3 2

Re: [Maria-developers] d19f380d62c: MDEV-21445 Strange/inconsistent behavior of IN condition when mixing numbers and strings
by Sergei Golubchik 12 Oct '20

12 Oct '20

Hi, Alexander! On Oct 12, Alexander Barkov wrote: > revision-id: d19f380d62c (mariadb-10.3.21-213-gd19f380d62c) > parent(s): 90c8d773ed0 > author: Alexander Barkov <bar(a)mariadb.com> > committer: Alexander Barkov <bar(a)mariadb.com> > timestamp: 2020-08-18 13:19:21 +0400 > message: > > MDEV-21445 Strange/inconsistent behavior of IN condition when mixing numbers and strings > > Comparison is now performed as DECIMAL for pairs (int,string) and (string,int). Are there tests with strings like '1e100', outside of bigint range ? > Before the patch the comparison was performed in DOUBLE format, > so precision loss happened on huge values, e.g.: > > '97716021308405775' was converted to > 97716021308405770 (notice, the last digit was lost) > > diff --git a/sql/field.h b/sql/field.h > index be3a648617b..c9f5a34a3ae 100644 > --- a/sql/field.h > +++ b/sql/field.h > @@ -311,7 +311,7 @@ class Value_source > const char *end) > { > Converter_str2my_decimal_with_warn(NULL, Warn_filter_all(), > - E_DEC_FATAL_ERROR & ~E_DEC_BAD_NUM, > + 0, why did you remove warnings (here and elsewhere)? would be good to have it in the commit comment too, not just in email. > cs, cptr, end - cptr, decimal_value); > return decimal_value; > } > diff --git a/sql/sql_type.cc b/sql/sql_type.cc > index 9d3a47adfa5..7a847c90b8f 100644 > --- a/sql/sql_type.cc > +++ b/sql/sql_type.cc > @@ -764,6 +764,11 @@ Type_handler_hybrid_field_type::aggregate_for_comparison(const Type_handler *h) > m_type_handler= &type_handler_datetime; > } > } > + else if ((a == INT_RESULT && b == STRING_RESULT) || > + (b == INT_RESULT && a == INT_RESULT)) looks like a typo, the second line should be b == INT_RESULT && a == STRING_RESULT please add some tests to cover it. > + { > + m_type_handler= &type_handler_newdecimal; > + } > else if ((a == INT_RESULT || a == DECIMAL_RESULT) && > (b == INT_RESULT || b == DECIMAL_RESULT)) > { Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0